Web toolkit API

Installation

Add the toolkit to your project using a package manager or by copying the source code.

The Web toolkit API provides the following classes and methods.

API overview

ConversationalAIAPI class

Class for managing conversational AI engine interactions through Agora's RTC and RTM services.

chat

Send chat messages to a conversational agent.

public async chat(agentUserId: string, message: IChatMessageText | IChatMessageImage)

Sample code

// Send an image message
const imageMessage: IChatMessageImage = {
  messageType: EChatMessageType.IMAGE,
  imageData: urlImageData
};
await api.chat("user123", imageMessage);

Return Value
If the method call succeeds, a Promise is returned, indicating the result of the message sending.

getInstance

Gets a singleton instance of ConversationalAIAPI. This method ensures that only one instance exists throughout the entire application lifecycle.

public static getInstance()

Return Value: If the method call succeeds, a ConversationalAIAPI instance is returned. If not initialized, a NotFoundError exception is thrown.

init

Initializes the ConversationalAIAPI singleton instance.

public static async init(cfg: IConversationalAIAPIConfig): Promise<ConversationalAIAPI>

This method sets the RTC and RTM engines, rendering mode, and logging options. You must call this method before calling other methods of ConversationalAIAPI. Awaiting this method returns the initialized instance directly, so a separate call to getInstance is not required immediately after initialization.

Return Value: If the method call succeeds, a Promise resolving to the initialized ConversationalAIAPI instance.

subscribeMessage

Subscribes to the messaging channel to get real-time updates.

public subscribeMessage(channel: string)

This method binds the necessary RTC and RTM events and starts CovSubRenderController to process received messages.

unsubscribe

Unsubscribes from the message channel and cleans up resources.

public unsubscribe()

interrupt

Sends an interrupt message to the specified agent user.

public async interrupt(agentUserId: string)

manualSOS

Sends a manual Start of Speech (SoS) signal to the agent over RTM, explicitly declaring that the current user has started speaking.

public async manualSOS(agentUserId: string): Promise<string>

Keep the following in mind when calling this method:

• The component generates requestId internally and resolves it in the returned Promise. Your app can store this value to correlate it with the server-side event.
• A resolved Promise only indicates that the RTM publish succeeded — it does not indicate that the server processed the request successfully.
• Whether the server successfully processed the request is determined by the subsequent USER_MANUAL_SOS_RESULT or USER_MANUAL_EOS_RESULT event. See EConversationalAIAPIEvents.
• The client does not check locally whether manual mode is enabled. Mode validation is handled by the server.
• Common failure reasons include: the current mode configuration does not allow manual signaling, there is no active user speech to mark, or a manual SoS or EoS that has already taken effect is resubmitted within the same turn.

Return Value
If the method call succeeds, a Promise<string> is returned, resolving to the requestId for this request. This result only indicates that the RTM publish succeeded.

manualEOS

Sends a manual End of Speech (EoS) signal to the agent over RTM, explicitly declaring that the current user has finished speaking.

public async manualEOS(agentUserId: string): Promise<string>

Keep the following in mind when calling this method:

• The component generates requestId internally and resolves it in the returned Promise. Your app can store this value to correlate it with the server-side event.
• A resolved Promise only indicates that the RTM publish succeeded — it does not indicate that the server processed the request successfully.
• Whether the server successfully processed the request is determined by the subsequent USER_MANUAL_SOS_RESULT or USER_MANUAL_EOS_RESULT event. See EConversationalAIAPIEvents.
• The client does not check locally whether manual mode is enabled. Mode validation is handled by the server.
• Common failure reasons include: the current mode configuration does not allow manual signaling, there is no active user speech to mark, or a manual SoS or EoS that has already taken effect is resubmitted within the same turn.

Return Value
If the method call succeeds, a Promise<string> is returned, resolving to the requestId for this request. This result only indicates that the RTM publish succeeded.

destroy

Destroys the ConversationalAIAPI instance and cleans up resources.

public destroy(): void

IConversationalAIAPIEventHandlers interface

Event handler interface for the Conversational AI API module.

A set of event handlers that respond to various events emitted by the conversational AI system, including agent state changes, interruptions, performance metrics, errors, and transcription updates.

Types and Interfaces

TMessageReceipt

Message receipt type definition.

export type TMessageReceipt = {
  moduleType: EModuleType
  messageType: EChatMessageType
  message: string
  turnId: number
}

TUserManualEventPayload

Common payload for manual SoS/EoS events.

export type TUserManualEventPayload = {
  success: boolean
  requestId: string
  turnId?: number
  errorMessage?: string
}

TUserManualSosEvent

Represents a manual SoS event.

export type TUserManualSosEvent = {
  eventId: string
  timestamp: number
  payload: TUserManualEventPayload
}

TUserManualEosEvent

Represents a manual EoS event.

export type TUserManualEosEvent = {
  eventId: string
  timestamp: number
  payload: TUserManualEventPayload
}

TAgentManualEosPayload

Payload for a server-triggered automatic EoS event.

export type TAgentManualEosPayload = {
  reason: string
  maxDurationMs: number
  turnId: number
}

TAgentManualEosEvent

Represents a server-triggered automatic EoS event.

export type TAgentManualEosEvent = {
  eventId: string
  timestamp: number
  payload: TAgentManualEosPayload
}

IChatMessageBase

The IChatMessageBase interface that contains the properties of the underlying message type.

export interface IChatMessageBase {
  messageType: EChatMessageType
}

IChatMessageImage

Represents an image-based chat message, which can contain URL image data.

export interface IChatMessageImage extends IChatMessageBase {
  messageType: EChatMessageType.IMAGE
  uuid: string
  url?: string
}

IConversationalAIAPIConfig

Parameters used to configure the interface.

export interface IConversationalAIAPIConfig {
 rtcEngine: IAgoraRTCClient;
 rtmEngine: RTMClient;
 renderMode?: ETranscriptHelperMode;
 enableLog?: boolean;
 enableRenderModeFallback?: boolean;
}

TStateChangeEvent

Event type used to describe a change in the state of a voice agent.

export type TStateChangeEvent = {
 state: EAgentState
 turnID: number
 timestamp: number
 reason: string
}

TAgentMetric

Used to store metric data during agent runtime.

export type TAgentMetric = {
 type: EModuleType
 name: string
 value: number
 timestamp: number
}

TModuleError

Used to represent error messages from different AI modules, including error type, error code, error message, and timestamp.

export type TModuleError = {
 type: EModuleType
 code: number
 message: string
 timestamp: number
}

ISubtitleHelperItem

Transcript auxiliary item interface.

The data structure for a single transcript item in the transcription system. This includes information such as user ID, stream ID, turn ID, timestamp, text content, status, and metadata.

export interface ISubtitleHelperItem<T> {
 uid: string;
 stream_id: number;
 turn_id: number;
 _time: number;
 text: string;
 status: ETurnStatus;
 metadata: T | null;
}

Enum classes

EConversationalAIAPIEvents

Event types that can be triggered by the Conversational AI API.

EChatMessagePriority

Used to set the chat message processing priority.

EChatMessageType

Chat message types supported in conversational AI.

EAgentState

Agent state enumeration.

export enum EAgentState {
 IDLE = "idle",
 LISTENING = "listening", 
 THINKING = "thinking",
 SPEAKING = "speaking",
 SILENT = "silent"
}

EModuleType

Enumeration of module types for AI capabilities.

ETranscriptHelperMode

The mode type for transcript processing.

ETurnStatus

TURN connection status enumeration.