Android toolkit API

Installation

Add the toolkit to your project using Maven or by copying the source code.

implementation 'io.agora.agents:agora-agent-client-toolkit:2.9.0'

The Android toolkit API for Conversational AI Engine provides the following classes and methods.

IConversationalAIAPI class

chat

Send a message to the agent.

Support for sending different types of messages through the ChatMessage sealed class hierarchy:

• ImageMessage: Used to send picture messages.

fun chat(agentUserId: String, message: ChatMessage, completion: (error: ConversationalAIAPIError?) -> Unit)

addHandler

Registers an event handler to receive agent session events.

fun addHandler(handler: IConversationalAIAPIEventHandler)

removeHandler

Removes a registered event handler.

fun removeHandler(handler: IConversationalAIAPIEventHandler)

subscribeMessage

Subscribes to a channel to receive agent conversation events.

fun subscribeMessage(channelName: String, completion: (error: ConversationalAIAPIError?) -> Unit)

unsubscribeMessage

Unsubscribes from the channel and stops receiving events.

fun unsubscribeMessage(channelName: String, completion: (error: ConversationalAIAPIError?) -> Unit)

interrupt

Interrupts the agent’s speech.

fun interrupt(agentUserId: String, completion: (error: ConversationalAIAPIError?) -> Unit)

manualSOS

Sends a manual Start of Speech (SoS) signal to the agent over RTM, explicitly declaring that the current user has started speaking.

fun manualSOS(agentUserId: String, completion: (requestId: String, error: ConversationalAIAPIError?) -> Unit)

Keep the following in mind when calling this method:

• The component generates requestId internally. After the request is sent, the component returns the requestId for this request in completion. Your app can store this value to correlate it with the server-side callback.
• completion only indicates whether the RTM publish succeeded — it does not indicate that the server processed the request successfully.
• Whether the server successfully processed the request is determined by the subsequent onUserManualSosEvent or onUserManualEosEvent callback.
• The client does not check locally whether manual mode is enabled. Mode validation is handled by the server.
• Common failure reasons include: the current mode configuration does not allow manual signaling, there is no active user speech to mark, or a manual SoS or EoS that has already taken effect is resubmitted within the same turn.

manualEOS

Sends a manual End of Speech (EoS) signal to the agent over RTM, explicitly declaring that the current user has finished speaking.

fun manualEOS(agentUserId: String, completion: (requestId: String, error: ConversationalAIAPIError?) -> Unit)

Keep the following in mind when calling this method:

• The component generates requestId internally. After the request is sent, the component returns the requestId for this request in completion. Your app can store this value to correlate it with the server-side callback.
• completion only indicates whether the RTM publish succeeded — it does not indicate that the server processed the request successfully.
• Whether the server successfully processed the request is determined by the subsequent onUserManualSosEvent or onUserManualEosEvent callback.
• The client does not check locally whether manual mode is enabled. Mode validation is handled by the server.
• Common failure reasons include: the current mode configuration does not allow manual signaling, there is no active user speech to mark, or a manual SoS or EoS that has already taken effect is resubmitted within the same turn.

loadAudioSettings

Sets audio parameters to optimize the agent’s conversation performance.

fun loadAudioSettings(scenario: Int = Constants.AUDIO_SCENARIO_AI_CLIENT)

destroy

Destroys the API instance and releases resources.

Call this method when the instance is no longer needed. After calling, the instance becomes unusable.

fun destroy()

IConversationalAIAPIEventHandler class

onMessageError

Message processing error callback.

This callback is triggered when an error occurs during message processing. For example, if a chat message fails to be sent, this callback is triggered and the error message is returned.

fun onMessageError(agentUserId: String, error: MessageError)

onMessageReceiptUpdated

Message receipt update callback.

fun onMessageReceiptUpdated(agentUserId: String, receipt: MessageReceipt)

onUserManualSosEvent

Manual SoS result callback.

Receives the server's processing result for a manual SoS request. Both success and failure results are returned through this callback — check event.payload.success to determine the outcome.

This callback has a default empty implementation, so existing integrations do not need to implement it.

fun onUserManualSosEvent(agentUserId: String, event: UserManualSosEvent)

onUserManualEosEvent

Manual EoS result callback.

Receives the server's processing result for a manual EoS request. Both success and failure results are returned through this callback — check event.payload.success to determine the outcome.

This callback has a default empty implementation, so existing integrations do not need to implement it.

fun onUserManualEosEvent(agentUserId: String, event: UserManualEosEvent)

onAgentManualEosEvent

Server-triggered automatic EoS notification callback.

Triggered when a user's single speaking turn exceeds the server-side duration threshold. In this case, the server ends the current user's turn on its own and notifies the client through this callback. This event is not a result of calling manualEOS().

This callback has a default empty implementation, so existing integrations do not need to implement it.

fun onAgentManualEosEvent(agentUserId: String, event: AgentManualEosEvent)

onAgentStateChanged

Triggered when the agent’s state changes.

fun onAgentStateChanged(agentUserId: String, event: StateChangeEvent)

onAgentListeningChanged

Triggered when the agent's listening state changes.

This callback is a fine-grained complement to onAgentStateChanged, used to monitor whether the agent is actively listening to user input.

fun onAgentListeningChanged(agentUserId: String, isListening: Boolean)

onAgentThinkingChanged

Triggered when the agent's thinking state changes.

This callback is a fine-grained complement to onAgentStateChanged, used to monitor whether the agent is actively processing a request.

fun onAgentThinkingChanged(agentUserId: String, isThinking: Boolean)

onAgentSpeakingChanged

Triggered when the agent's speaking state changes.

This callback is a fine-grained complement to onAgentStateChanged, used to monitor whether the agent is actively playing back speech.

fun onAgentSpeakingChanged(agentUserId: String, isSpeaking: Boolean)

onAgentInterrupted

Triggered when an interrupt event occurs.

fun onAgentInterrupted(agentUserId: String, event: InterruptEvent)

onAgentMetrics

Triggered when performance metrics become available.

fun onAgentMetrics(agentUserId: String, metric: Metric)

onAgentError

Triggered when an agent error occurs.

fun onAgentError(agentUserId: String, error: ModuleError)

onTranscriptUpdated

Triggered when the transcription content is updated.

fun onTranscriptUpdated(agentUserId: String, transcript: Transcript)

onDebugLog

Triggered for internal debug logs.

fun onDebugLog(log: String)

Structures

StateChangeEvent

Represents an agent state change event.
Tracks session flow and updates status indications in the UI, including the event timestamp.

data class StateChangeEvent(
    val state: AgentState,
    val turnId: Long,
    val timestamp: Long,
)

InterruptEvent

Indicates an interrupt event.
Triggered when the user or system interrupts the agent or when the system detects a high-priority message. Used for logging and handling.

data class InterruptEvent(
    val turnId: Long,
    val timestamp: Long
)

Metric

Used to record and transmit system performance data. This data can be used for performance monitoring, system optimization, and user experience improvement.

data class Metric(
    val type: ModuleType,
    val name: String,
    val value: Double,
    val timestamp: Long
)

ModuleError

Processes and reports agent-related error information.

data class ModuleError(
    val type: ModuleType,
    val code: Int,
    val message: String,
    val timestamp: Long,
    val turnId: Long? = null
)

ChatMessage

Sealed base class for all message types sent to agents.

This sealed class hierarchy provides a type-safe way to handle messages of different content types.

sealed class ChatMessage

ImageMessage

Used to send image content to the agent.

Supports specifying image files via HTTP/HTTPS links.

data class ImageMessage(
    val uuid: String,
    val imageUrl: String,
) : ChatMessage()

MessageReceipt

MessageReceipt represents message receipt information and supports processing multiple media types through MediaInfo.

data class MessageReceipt(
    val type: ModuleType,
    val chatMessageType: ChatMessageType,
    val turnId: Long,
    val message: String
)

ChatMessageType

Used to distinguish different types of messages in the session system.

enum class ChatMessageType(val value: String) {
    Text("text"),
    Image("picture"),
    UNKNOWN("unknown");

    companion object {
        fun fromValue(value: String): ChatMessageType {
            return ChatMessageType.entries.find { it.value == value } ?: UNKNOWN
        }
    }
}

MessageError

Used to process and report message error information.

data class MessageError(
    val chatMessageType: ChatMessageType,
    val code: Int,
    val message: String,
    val timestamp: Long
)

UserManualEventPayload

Common payload for manual SoS/EoS events.

data class UserManualEventPayload(
    val success: Boolean,
    val requestId: String,
    val turnId: Long?,
    val errorMessage: String?
)

UserManualSosEvent

Represents a manual SoS event.

data class UserManualSosEvent(
    val eventId: String,
    val timestamp: Long,
    val payload: UserManualEventPayload
)

UserManualEosEvent

Represents a manual EoS event.

data class UserManualEosEvent(
    val eventId: String,
    val timestamp: Long,
    val payload: UserManualEventPayload
)

AgentManualEosPayload

Payload for a server-triggered automatic EoS event.

data class AgentManualEosPayload(
    val reason: String,
    val maxDurationMs: Long,
    val turnId: Long
)

AgentManualEosEvent

Represents a server-triggered automatic EoS event.

data class AgentManualEosEvent(
    val eventId: String,
    val timestamp: Long,
    val payload: AgentManualEosPayload
)

Transcript

Represents a full transcribed message used to render the UI.

data class Transcript(
    val turnId: Long,
    val userId: String = "",
    val text: String,
    var status: TranscriptStatus,
    var type: TranscriptType
)

ConversationalAIAPIConfig

Holds configuration parameters for initializing the Conversational AI API.

data class ConversationalAIAPIConfig(
    val rtcEngine: RtcEngine,
    val rtmClient: RtmClient,
    val renderMode: TranscriptRenderMode = TranscriptRenderMode.Word,
    val enableLog: Boolean = true,
    val enableRenderModeFallback: Boolean = true
)

ConversationalAIAPIError

Represents an error type in the Conversational AI API.

sealed class ConversationalAIAPIError : Exception() {
    data class RtmError(val code: Int, val msg: String) : ConversationalAIAPIError()
    data class RtcError(val code: Int, val msg: String) : ConversationalAIAPIError()
    data class UnknownError(val msg: String) : ConversationalAIAPIError()

    val errorCode: Int
        get() = when (this) {
            is RtmError -> this.code
            is RtcError -> this.code
            is UnknownError -> -100
        }

    val errorMessage: String
        get() = when (this) {
            is RtmError -> this.msg
            is RtcError -> this.msg
            is UnknownError -> this.msg
        }
}

Enum classes

Priority

Controls the priority with which the agent handles incoming messages during an interaction.

AgentState

Represents the current state of the agent.

ModuleType

Performance module type enumeration.

MessageType

Used to distinguish different types of messages in the system.

TranscriptRenderMode

Transcript rendering mode.

TranscriptType

Transcript source type.

TranscriptStatus

Indicates the current status of the transcription.