Skip to main content
Conversational AI Engine: Create advanced voice apps by combining Agora’s real-time audio with powerful conversational AI.

Receive event notifications

Real-time conversational AI applications require responsive user interfaces that react to agent events. This page demonstrates how to implement event handling to create dynamic, interactive experiences with conversational agents.

Understand the tech​

Agora provides a flexible, scalable, and standardized conversational AI engine toolkit. The toolkit supports iOS, Android, and Web platforms, and encapsulates scenario-based APIs. You can use these APIs to integrate the capabilities of the Agora Chat SDK and Video SDK to enable the following features:

The component provides a set of callback methods that allow you to listen for various agent-related events and system information:

  • onAgentStateChanged : Listen for agent state changes through Signaling presence events. Possible states include: silent, listening, thinking, and speaking. Use this callback to update the agent UI or track the conversation flow.

  • onAgentInterrupted : Listen for agent interruption events through Signaling messages. Use this to handle scenarios where the agent is interrupted mid-response.

  • onAgentMetrics : Listen for agent performance metrics through Signaling messages. Metrics include LLM reasoning delay, TTS synthesis delay, and more. Use this callback to monitor system performance.

  • onAgentError : Listen for agent error events through Signaling messages. These may include errors in agent modules such as LLM or TTS. Use this callback for error monitoring, logging, and implementing graceful degradation strategies.

Prerequisites​

Before you begin, ensure the following:

  • You have implemented the Conversational AI Engine REST quickstart.
  • Your app integrates Video SDK v4.5.1 or later and includes the video quickstart implementation.
  • You have enabled Signaling in the Agora Console and completed Signaling quickstart for basic messaging.
  • You maintain active and authenticated RTC and Signaling instances that persist beyond the component's lifecycle. The toolkit does not manage the initialization, lifecycle, or authentication of RTC or Signaling.

Implementation​

This section describes how to implement Conversational AI engine events using the toolkit.

  1. Integrate the toolkit

    Copy the convoaiApi folder to your project and import the toolkit before calling the toolkit API. Refer to Folder structure to understand the role of each file.

  2. Create a toolkit instance

    Create a configuration object with the Video SDK and Signaling engine instances. Use the configuration to create a toolkit instance.

    // Create a configuration object for the Video SDK and Chat SDK instances val config = ConversationalAIAPIConfig(     rtcEngine = rtcEngineInstance,     rtmClient = rtmClientInstance,     enableLog = true ) // Create the toolkit instance val api = ConversationalAIAPIImpl(config)

  3. Register events

    Call the addHandler method to register and implement agent-related event callbacks.

    // Register event callbacks api.addHandler(object : IConversationalAIAPIEventHandler {     // Listen for agent state changes     override fun onAgentStateChanged(agentUserId: String, event: StateChangeEvent) {         when (event.state) {             AgentState.SILENT -> {                 updateAgentStatus("Waiting...")             }             AgentState.LISTENING -> {                 updateAgentStatus("Listening...")             }             AgentState.THINKING -> {                 updateAgentStatus("Thinking...")             }             AgentState.SPEAKING -> {                 updateAgentStatus("Speaking...")             }             AgentState.UNKNOWN -> {                 Log.w("AgentState", "Unknown agent state: $event")             }         }     }     // Listen for agent interruption events     override fun onAgentInterrupted(agentUserId: String, event: InterruptEvent) {         Log.d("AgentInterrupt", "Agent $agentUserId interrupted at turn ${event.turnId}")         showInterruptNotification()     }     // Monitor agent performance metrics     override fun onAgentMetrics(agentUserId: String, metric: Metric) {         when (metric.type) {             ModuleType.LLM -> {                 Log.d("Metrics", "LLM latency: ${metric.value} ms")             }             ModuleType.TTS -> {                 Log.d("Metrics", "TTS latency: ${metric.value} ms")             }             else -> {                 Log.d("Metrics", "${metric.type}: ${metric.name} = ${metric.value}")             }         }     }     // Handle agent errors     override fun onAgentError(agentUserId: String, error: ModuleError) {         Log.e("AgentError", "Error in ${error.type}: ${error.message} (code: ${error.code})")         when (error.type) {             ModuleType.LLM -> {                 showErrorMessage("AI processing failed. Please try again later.")             }             ModuleType.TTS -> {                 showErrorMessage("Speech synthesis failed.")             }             else -> {                 showErrorMessage("System error: ${error.message}")             }         }     }     // Handle debug logs     override fun onDebugLog(log: String) {         if (BuildConfig.DEBUG) {             Log.d("ConvoAI", log)         }         // Optional: Send logs to a remote server for diagnostics     } })

  4. Subscribe to the channel

    Subtitles are delivered through Signaling channel messages. To receive subtitle data, call subscribeMessage before starting the agent session.

    api.subscribeMessage("channelName") { error ->     if (error != null) {         // Handle error     } }

  5. Add a Conversational AI agent to the channel

    To start a Conversational AI agent, configure the following parameters in your POST request:

    ParameterDescriptionRequired
    advanced_features.enable_rtm: trueStarts the Signaling serviceYes
    parameters.data_channel: "rtm"Enables Signaling as the data transmission channelYes
    parameters.enable_metrics: trueEnables agent performance data collectionOptional
    parameters.enable_error_message: trueEnables reporting of agent error eventsOptional

    After a successful response, the agent joins the specified Video SDK channel and is ready to interact with the user.

  6. Unsubscribe from the channel

    After an agent session ends, unsubscribe from channel messages to release subtitle-related resources:

    api.unsubscribeMessage("channelName") { error ->
         if (error != null) {
             // Handle the error
         }
     }

  7. Release resources

    At the end of each call, use the destroy method to clean up the cache.

    api.destroy()

Reference​

This section contains content that completes the information on this page, or points you to documentation that explains other aspects to this product.

Sample project​

Refer to the following open-source sample code for your reference.

Folder structure​

To integrate the client components, only the files and folders listed below are required. You do not need to copy any other files.

  • IConversationalAIAPI.kt: Defines the API interface, data structures, and enumerations.
  • ConversationalAIAPIImpl.kt: Contains the main implementation logic of the ConversationalAI API.
  • ConversationalAIUtils.kt: Provides utility functions and manages event callbacks.
    • v3/: Contains the subtitle rendering module.
      • TranscriptionController.kt: Controls subtitle rendering and synchronization.
      • MessageParser.kt: Parses transcription and message data.

API reference​