Client-side events

Real-time Conversational AI applications require responsive user interfaces that react to agent events. This page explains how to implement client-side event handling with the Conversational AI Engine toolkit. Use this page when you need in-session events in a mobile or web client. If you need backend monitoring, alerting, or post-session analysis, use Webhooks instead.

For an overview that compares client toolkit and webhook delivery, see Get runtime events.

Understand the tech

Agora provides a flexible, scalable, and standardized Conversational AI Engine toolkit. The toolkit supports Android, iOS, and Web platforms, and encapsulates scenario-based APIs. You can use these APIs to integrate Agora Signaling SDK and Video SDK capabilities to enable the following features:

Interrupt agents
Display live transcripts
Monitor agent status, errors, and performance
Set optimal audio parameters for iOS and Android

The toolkit exposes callback methods that let you listen for various agent-related events and system information:

onAgentStateChanged: Listen for agent state changes such as silent, listening, thinking, and speaking.
onAgentListeningChanged: Track when the agent starts or stops listening.
onAgentThinkingChanged: Track when the agent starts or stops processing.
onAgentSpeakingChanged: Track when the agent starts or stops playing speech.
onAgentInterrupted: Handle interruption events triggered during a conversation.
onAgentMetrics: Observe performance metrics such as LLM and TTS latency.
onAgentError: Handle module-level failures such as LLM or TTS errors.

Prerequisites

Implemented the Conversational AI Engine quickstart.
Your app integrates Agora Video SDK v4.5.1 or later and includes the Video SDK quickstart.
Enabled Signaling in the Agora Console and completed the Signaling quickstart for basic messaging.
Maintain active and authenticated RTC and Signaling instances that persist beyond the component lifecycle. The toolkit does not manage RTC or Signaling initialization, lifecycle, or authentication.

Implementation

Integrate the toolkit

Copy the convoaiApi folder to your project and import the toolkit before calling the toolkit API. See Folder structure to understand the role of each file.

Create a toolkit instance

val config = ConversationalAIAPIConfig(
    rtcEngine = rtcEngineInstance,
    rtmClient = rtmClientInstance,
    enableLog = true
)

val api = ConversationalAIAPIImpl(config)

Register events

api.addHandler(object : IConversationalAIAPIEventHandler {
    override fun onAgentStateChanged(agentUserId: String, event: StateChangeEvent) {
        when (event.state) {
            AgentState.SILENT -> updateAgentStatus("Waiting...")
            AgentState.LISTENING -> updateAgentStatus("Listening...")
            AgentState.THINKING -> updateAgentStatus("Thinking...")
            AgentState.SPEAKING -> updateAgentStatus("Speaking...")
            AgentState.UNKNOWN -> Log.w("AgentState", "Unknown agent state: $event")
        }
    }

    override fun onAgentInterrupted(agentUserId: String, event: InterruptEvent) {
        Log.d("AgentInterrupt", "Agent $agentUserId interrupted at turn ${event.turnId}")
        showInterruptNotification()
    }

    override fun onAgentMetrics(agentUserId: String, metric: Metric) {
        when (metric.type) {
            ModuleType.LLM -> Log.d("Metrics", "LLM latency: ${metric.value} ms")
            ModuleType.TTS -> Log.d("Metrics", "TTS latency: ${metric.value} ms")
            else -> Log.d("Metrics", "${metric.type}: ${metric.name} = ${metric.value}")
        }
    }

    override fun onAgentError(agentUserId: String, error: ModuleError) {
        Log.e("AgentError", "Error in ${error.type}: ${error.message} (code: ${error.code})")
    }
})

Subscribe to the channel

api.subscribeMessage("channelName") { error ->
    if (error != null) {
        // Handle error
    }
}

Add a conversational AI agent to the channel

To start a conversational AI agent, configure:

Parameter	Description	Required
`advanced_features.enable_rtm: true`	Starts the Signaling service	Yes
`parameters.data_channel: "rtm"`	Enables Signaling as the data transmission channel	Yes
`parameters.enable_metrics: true`	Enables agent performance data collection	Optional
`parameters.enable_error_message: true`	Enables reporting of agent error events	Optional

Unsubscribe and release resources

api.unsubscribeMessage("channelName") { error ->
    if (error != null) {
        // Handle the error
    }
}

api.destroy()

Integrate the toolkit

Copy the ConversationalAIAPI folder to your project and import the toolkit before calling the toolkit API. See Folder structure to understand the role of each file.

Create a toolkit instance

let config = ConversationalAIAPIConfig(
    rtcEngine: rtcEngine,
    rtmEngine: rtmEngine,
    enableLog: true
)

convoAIAPI = ConversationalAIAPIImpl(config: config)

Register events

class ConversationViewController: UIViewController, ConversationalAIAPIEventHandler {
    func onAgentStateChanged(agentUserId: String, event: StateChangeEvent) {
        DispatchQueue.main.async {
            self.updateAgentStatus(event.state)
        }
    }

    func onAgentInterrupted(agentUserId: String, event: InterruptEvent) {
        print("Agent \(agentUserId) interrupted at turn \(event.turnId)")
    }

    func onAgentMetrics(agentUserId: String, metrics: Metric) {
        switch metrics.type {
        case .llm:
            print("LLM latency: \(metrics.value)ms")
        case .tts:
            print("TTS latency: \(metrics.value)ms")
        case .unknown:
            print("Unknown metric: \(metrics.name) = \(metrics.value)")
        }
    }

    func onAgentError(agentUserId: String, error: ModuleError) {
        print("Error in \(error.type): \(error.message) (code: \(error.code))")
    }
}

convoAIAPI.addHandler(handler: self)

Subscribe to the channel

convoAIAPI.subscribeMessage(channelName: channelName) { error in
    if let error = error {
        print("Subscription failed: \(error.message)")
    } else {
        print("Subscription successful")
    }
}

Add a conversational AI agent to the channel

To start a conversational AI agent, configure:

Parameter	Description	Required
`advanced_features.enable_rtm: true`	Starts the Signaling service	Yes
`parameters.data_channel: "rtm"`	Enables Signaling as the data transmission channel	Yes
`parameters.enable_metrics: true`	Enables collection of agent performance data	Optional
`parameters.enable_error_message: true`	Enables reporting of agent error events	Optional

Unsubscribe and release resources

convoAIAPI.unsubscribeMessage(channelName: channelName) { error in
    if let error = error {
        print("Unsubscription failed: \(error.message)")
    } else {
        print("Unsubscribed successfully")
    }
}

convoAIAPI.destroy()

Integrate the toolkit

Copy the conversational-ai-api file to your project and import the toolkit before calling the toolkit API. See Folder structure to understand the role of each file.

Create a toolkit instance

ConversationalAIAPI.init({
    rtcEngine,
    rtmEngine,
})

const conversationalAIAPI = ConversationalAIAPI.getInstance()

Register events

conversationalAIAPI.on(EConversationalAIAPIEvents.AGENT_STATE_CHANGED, onAgentStateChanged)
conversationalAIAPI.on(EConversationalAIAPIEvents.AGENT_METRICS, onAgentMetricsChanged)
conversationalAIAPI.on(EConversationalAIAPIEvents.AGENT_ERROR, onAgentError)

Subscribe to the channel

conversationalAIAPI.subscribeMessage(channel_name)

Add a conversational AI agent to the channel

To start a conversational AI agent, configure:

Parameter	Description	Required
`advanced_features.enable_rtm: true`	Starts the Signaling service	Yes
`parameters.data_channel: "rtm"`	Enables Signaling as the data transmission channel	Yes
`parameters.enable_metrics: true`	Enables agent performance data collection	Optional
`parameters.enable_error_message: true`	Enables reporting of agent error events	Optional

Unsubscribe and release resources

conversationalAIAPI.unsubscribeMessage(channel_name)
conversationalAIAPI.destroy()

Reference

Sample project

Conversational-AI-Demo

Folder structure

IConversationalAIAPI.kt: API interface, data structures, and enumerations
ConversationalAIAPIImpl.kt: main implementation logic
ConversationalAIUtils.kt: utility functions and event callback management
v3/
- TranscriptionController.kt: transcript rendering and synchronization
- MessageParser.kt: transcription and message parsing

ConversationalAIAPI.swift: API interface, data structures, and enumerations
ConversationalAIAPIImpl.swift: main implementation logic
Transcription/
- TranscriptionController.swift: transcript rendering and control

index.ts: main API class
type.ts: API interfaces, data structures, and enumerations
utils/
- index.ts: general utility functions
- events.ts: event management class
- sub-render.ts: transcript rendering module

Understand the tech

Prerequisites

Implementation

Reference

Sample project

Folder structure

API reference

Contact Us

Contact Us