# Web toolkit API (/en/api-reference/api-ref/conversational-ai/client-toolkit/web)

> For AI agents: see the complete documentation index at [llms.txt](/llms.txt).

The [Web toolkit API](https://github.com/AgoraIO-Community/Conversational-AI-Demo/tree/main/Web/Scenes/VoiceAgent/src/conversational-ai-api) provides the following classes and methods.

## API overview [#api-overview]

| API                                     | Description                                                          |
| --------------------------------------- | -------------------------------------------------------------------- |
| [`chat`](#chat)                         | Send chat messages to a conversational agent.                        |
| [`getInstance`](#getinstance)           | Gets a singleton instance of `ConversationalAIAPI`.                  |
| [`init`](#init)                         | Initializes the `ConversationalAIAPI` singleton instance.            |
| [`subscribeMessage`](#subscribemessage) | Subscribes to the messaging channel to get real-time updates.        |
| [`unsubscribe`](#unsubscribe)           | Unsubscribes from the message channel and cleans up resources.       |
| [`interrupt`](#interrupt)               | Sends an interrupt message to the specified agent user.              |
| [`destroy`](#destroy)                   | Destroys the `ConversationalAIAPI` instance and cleans up resources. |

## ConversationalAIAPI class [#conversationalaiapi-class]

Class for managing conversational AI engine interactions through Agora's RTC and RTM services.

### `chat` [#chat]

Send chat messages to a conversational agent.

<CalloutContainer type="info">
  <CalloutDescription>
    Since v1.7
  </CalloutDescription>
</CalloutContainer>

```javascript
public async chat(agentUserId: string, message: IChatMessageText | IChatMessageImage)
```

#### Sample code [#sample-code]

```js
// Send an image message
const imageMessage: IChatMessageImage = {
  messageType: EChatMessageType.IMAGE,
  imageData: urlImageData
};
await api.chat("user123", imageMessage);
```

| Parameter     | Type                                      | Description                                                                                                               |
| ------------- | ----------------------------------------- | ------------------------------------------------------------------------------------------------------------------------- |
| `agentUserId` | `string`                                  | Unique identifier of the agent.                                                                                           |
| `message`     | `IChatMessageText` \| `IChatMessageImage` | Chat message to be sent. Should be of type `IChatMessageImage`, see details in [`IChatMessageImage`](#ichatmessageimage). |

**Return Value**\
If the method call succeeds, a Promise is returned, indicating the result of the message sending.

### `getInstance` [#getinstance]

Gets a singleton instance of `ConversationalAIAPI`. This method ensures that only one instance exists throughout the entire application lifecycle.

```javascript
public static getInstance()
```

<CalloutContainer type="info">
  <CalloutDescription>
    * You must call [`init`](#init) before using this method.
    * If not initialized, an error is thrown.
  </CalloutDescription>
</CalloutContainer>

**Return Value:**
If the method call succeeds, a `ConversationalAIAPI` instance is returned. If not initialized, a `NotFoundError` exception is thrown.

### `init` [#init]

Initializes the `ConversationalAIAPI` singleton instance.

```javascript
public static init(cfg: IConversationalAIAPIConfig)
```

This method sets the RTC and RTM engines, rendering mode, and logging options. You must call this method before calling other methods of `ConversationalAIAPI`.

<CalloutContainer type="info">
  <CalloutDescription>
    * Only one instance can be initialized at a time.
    * If already initialized, an error is thrown.
  </CalloutDescription>
</CalloutContainer>

| Parameter | Type                         | Description                                                                                                       |
| --------- | ---------------------------- | ----------------------------------------------------------------------------------------------------------------- |
| `cfg`     | `IConversationalAIAPIConfig` | Configuration object used to initialize the API. See [`IConversationalAIAPIConfig`](#iconversationalaiapiconfig). |

**Return Value:**
If the method call succeeds, a `ConversationalAIAPI` instance is returned.

### `subscribeMessage` [#subscribemessage]

Subscribes to the messaging channel to get real-time updates.

```javascript
public subscribeMessage(channel: string)
```

This method binds the necessary RTC and RTM events and starts `CovSubRenderController` to process received messages.

<CalloutContainer type="info">
  <CalloutDescription>
    * You must call [`init`](#init) before using this method.
    * If not initialized, an error is thrown.
  </CalloutDescription>
</CalloutContainer>

| Parameter | Type     | Description                               |
| --------- | -------- | ----------------------------------------- |
| `channel` | `string` | The channel to subscribe to for messages. |

### `unsubscribe` [#unsubscribe]

Unsubscribes from the message channel and cleans up resources.

```javascript
public unsubscribe()
```

<CalloutContainer type="info">
  <CalloutDescription>
    This method unbinds RTC and RTM events, clears channels, and cleans up `CovSubRenderController`. You must call `subscribeMessage` before calling this method, otherwise an error is thrown.
  </CalloutDescription>
</CalloutContainer>

### `interrupt` [#interrupt]

Sends an interrupt message to the specified agent user.

```javascript
public async interrupt(agentUserId: string)
```

<CalloutContainer type="info">
  <CalloutDescription>
    * You must call [`init`](#init) before using this method.
    * If not initialized or sending fails, an error is thrown.
  </CalloutDescription>
</CalloutContainer>

| Parameter     | Type     | Description                                      |
| ------------- | -------- | ------------------------------------------------ |
| `agentUserId` | `string` | The user ID of the agent user to be interrupted. |

### `destroy` [#destroy]

Destroys the `ConversationalAIAPI` instance and cleans up resources.

```javascript
public destroy(): void
```

<CalloutContainer type="info">
  <CalloutDescription>
    * You must call `unsubscribe` before calling this method.
    * If not initialized, an error is thrown.
  </CalloutDescription>
</CalloutContainer>

## IConversationalAIAPIEventHandlers interface [#iconversationalaiapieventhandlers-interface]

Event handler interface for the Conversational AI API module.

A set of event handlers that respond to various events emitted by the conversational AI system, including agent state changes, interruptions, performance metrics, errors, and transcription updates.

| Parameter       | Type                                                                        | Description                                                                                                                    |
| --------------- | --------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------ |
| `agentUserId`   | `string`                                                                    | Unique identifier for the AI agent.                                                                                            |
| `event`         | `TStateChangeEvent \| { turnID: number; timestamp: number }`                | Event data. The type depends on the event type. See [`TStateChangeEvent`](#tstatechangeevent).                                 |
| `metrics`       | `TAgentMetric`                                                              | Performance indicator data of the agent. See [`TAgentMetric`](#tagentmetric).                                                  |
| `error`         | `TModuleError`                                                              | Error message when an error occurs in the agent. See [`TModuleError`](#tmoduleerror).                                          |
| `transcription` | `ISubtitleHelperItem<Partial<IUserTranscription \| IAgentTranscription>>[]` | An array of transcripts of the conversation between the user and the agent. See [`ISubtitleHelperItem`](#isubtitlehelperitem). |
| `message`       | `string`                                                                    | Debug log message string.                                                                                                      |

## Types and Interfaces [#types-and-interfaces]

### `TMessageReceipt` [#tmessagereceipt]

Message receipt type definition.

<CalloutContainer type="info">
  <CalloutDescription>
    Since v1.7
  </CalloutDescription>
</CalloutContainer>

```js
export type TMessageReceipt = {
  moduleType: EModuleType
  messageType: EChatMessageType
  message: string
  turnId: number
}
```

| Parameter     | Type               | Description                                                                |
| ------------- | ------------------ | -------------------------------------------------------------------------- |
| `moduleType`  | `EModuleType`      | The type of module sending the message. See [`EModuleType`](#emoduletype). |
| `messageType` | `EChatMessageType` | The type of message. See [`EChatMessageType`](#echatmessagetype).          |
| `message`     | `string`           | The content of the message.                                                |
| `turnId`      | `number`           | Unique identifier for the conversation turn.                               |

### `IChatMessageBase` [#ichatmessagebase]

The `IChatMessageBase` interface that contains the properties of the underlying message type.

<CalloutContainer type="info">
  <CalloutDescription>
    Since v1.7
  </CalloutDescription>
</CalloutContainer>

```js
export interface IChatMessageBase {
  messageType: EChatMessageType
}
```

| Parameter     | Type               | Description                                                               |
| ------------- | ------------------ | ------------------------------------------------------------------------- |
| `messageType` | `EChatMessageType` | Type of message. See [`EChatMessageType`](#echatmessagetype) for details. |

### `IChatMessageImage` [#ichatmessageimage]

Represents an image-based chat message, which can contain URL image data.

<CalloutContainer type="info">
  <CalloutDescription>
    Since v1.7
  </CalloutDescription>
</CalloutContainer>

```js
export interface IChatMessageImage extends IChatMessageBase {
  messageType: EChatMessageType.IMAGE
  uuid: string
  url?: string
}
```

| Parameter     | Type                     | Description                                                                 |
| ------------- | ------------------------ | --------------------------------------------------------------------------- |
| `messageType` | `EChatMessageType.IMAGE` | Message type, must be `IMAGE`. See [`EChatMessageType`](#echatmessagetype). |
| `uuid`        | `string`                 | Unique identifier for the image message.                                    |
| `url`         | `string`                 | Optional. URL pointing to the image resource.                               |

### `IConversationalAIAPIConfig` [#iconversationalaiapiconfig]

Parameters used to configure the interface.

```javascript
export interface IConversationalAIAPIConfig {
 rtcEngine: IAgoraRTCClient;
 rtmEngine: RTMClient;
 renderMode?: ESubtitleHelperMode;
 enableLog?: boolean;
}
```

| Parameter    | Type                  | Description                                                                                                                        |
| ------------ | --------------------- | ---------------------------------------------------------------------------------------------------------------------------------- |
| `rtcEngine`  | `IAgoraRTCClient`     | Agora RTC engine instance. See [`IAgoraRTCClient`](https://api-ref.agora.io/en/video-sdk/web/4.x/interfaces/iagorartcclient.html). |
| `rtmEngine`  | `RTMClient`           | Agora RTM engine instance. See the [Signaling Web API reference](https://api-ref.agora.io/en/signaling-sdk/web/2.x/index.html).    |
| `renderMode` | `ESubtitleHelperMode` | Rendering mode. See [`ESubtitleHelperMode`](#esubtitlehelpermode).                                                                 |
| `enableLog`  | `boolean`             | Whether to enable logging: `true` enables logging, `false` disables logging.                                                       |

### `TStateChangeEvent` [#tstatechangeevent]

Event type used to describe a change in the state of a voice agent.

```javascript
export type TStateChangeEvent = {
 state: EAgentState
 turnID: number
 timestamp: number
 reason: string
}
```

<CalloutContainer type="info">
  <CalloutDescription>
    The state change event is triggered when the state of the voice agent changes. The timestamp uses the UNIX timestamp in milliseconds.
  </CalloutDescription>
</CalloutContainer>

| Parameter   | Type                          | Description                                                              |
| ----------- | ----------------------------- | ------------------------------------------------------------------------ |
| `state`     | [`EAgentState`](#eagentstate) | The current state of the voice agent. See [`EAgentState`](#eagentstate). |
| `turnID`    | `number`                      | Unique identifier for the current session turn.                          |
| `timestamp` | `number`                      | Timestamp in milliseconds when the state change occurred.                |
| `reason`    | `string`                      | Description of the reason for the status change.                         |

### `TAgentMetric` [#tagentmetric]

Used to store metric data during agent runtime.

```javascript
export type TAgentMetric = {
 type: EModuleType
 name: string
 value: number
 timestamp: number
}
```

| Parameter   | Type                          | Description                                                       |
| ----------- | ----------------------------- | ----------------------------------------------------------------- |
| `type`      | [`EModuleType`](#emoduletype) | Indicator module type. See [`EModuleType`](#emoduletype).         |
| `name`      | `string`                      | The name of the indicator.                                        |
| `value`     | `number`                      | The indicator value.                                              |
| `timestamp` | `number`                      | Timestamp in milliseconds since Unix epoch (January 1, 1970 UTC). |

### `TModuleError` [#tmoduleerror]

Used to represent error messages from different AI modules, including error type, error code, error message, and timestamp.

```javascript
export type TModuleError = {
 type: EModuleType
 code: number
 message: string
 timestamp: number
}
```

| Parameter   | Type                          | Description                                                                  |
| ----------- | ----------------------------- | ---------------------------------------------------------------------------- |
| `type`      | [`EModuleType`](#emoduletype) | The module type where the error occurred. See [`EModuleType`](#emoduletype). |
| `code`      | `number`                      | Module-specific error code.                                                  |
| `message`   | `string`                      | Readable error description that makes it easier to understand and handle.    |
| `timestamp` | `number`                      | Timestamp in milliseconds since Unix epoch (January 1, 1970 UTC).            |

### `ISubtitleHelperItem` [#isubtitlehelperitem]

Transcript auxiliary item interface.

The data structure for a single transcript item in the transcription system. This includes information such as user ID, stream ID, turn ID, timestamp, text content, status, and metadata.

```javascript
export interface ISubtitleHelperItem<T> {
 uid: string;
 stream_id: number;
 turn_id: number;
 _time: number;
 text: string;
 status: ETurnStatus;
 metadata: T | null;
}
```

| Parameter   | Type                          | Description                                                           |
| ----------- | ----------------------------- | --------------------------------------------------------------------- |
| `uid`       | `string`                      | Unique user identifier.                                               |
| `stream_id` | `number`                      | Stream identifier.                                                    |
| `turn_id`   | `number`                      | The turn identifier within the session.                               |
| `_time`     | `number`                      | Timestamp in milliseconds since Unix epoch (January 1, 1970 UTC).     |
| `text`      | `string`                      | Transcript text content.                                              |
| `status`    | [`ETurnStatus`](#eturnstatus) | The status of the transcript item. See [`ETurnStatus`](#eturnstatus). |
| `metadata`  | `T \| null`                   | Additional metadata information.                                      |

## Enum classes [#enum-classes]

### EConversationalAIAPIEvents [#econversationalaiapievents]

Event types that can be triggered by the Conversational AI API.

<CalloutContainer type="info">
  <CalloutDescription>
    Since v1.6.0
  </CalloutDescription>
</CalloutContainer>

| Value                     | Description                                                      |
| ------------------------- | ---------------------------------------------------------------- |
| `AGENT_STATE_CHANGED`     | `"agent-state-changed"`: Agent state change event.               |
| `AGENT_LISTENING_CHANGED` | `"agent-listening-changed"`: Agent listening state change event. |
| `AGENT_THINKING_CHANGED`  | `"agent-thinking-changed"`: Agent thinking state change event.   |
| `AGENT_SPEAKING_CHANGED`  | `"agent-speaking-changed"`: Agent speaking state change event.   |
| `AGENT_INTERRUPTED`       | `"agent-interrupted"`: Agent interrupted event.                  |
| `AGENT_METRICS`           | `"agent-metrics"`: Agent metrics event.                          |
| `AGENT_ERROR`             | `"agent-error"`: Agent error event.                              |
| `TRANSCRIPT_UPDATED`      | `"transcript-updated"`: Transcript updated event.                |
| `DEBUG_LOG`               | `"debug-log"`: Debug log event.                                  |
| `MESSAGE_RECEIPT_UPDATED` | `"message-receipt-updated"`: Message receipt updated event.      |
| `MESSAGE_ERROR`           | `"message-error"`: Message error event.                          |

### EChatMessagePriority [#echatmessagepriority]

<CalloutContainer type="info">
  <CalloutDescription>
    Since v1.7
  </CalloutDescription>
</CalloutContainer>

Used to set the chat message processing priority.

| `Value`       | `description`                                                                          |
| ------------- | -------------------------------------------------------------------------------------- |
| `INTERRUPTED` | (`'interrupted'`): Interrupts current processing and immediately handles this message. |
| `APPEND`      | (`'append'`): Adds the message to the processing queue to be handled in order.         |
| `IGNORE`      | (`'ignore'`): Discards the message without processing.                                 |

### EChatMessageType [#echatmessagetype]

Chat message types supported in conversational AI.

<CalloutContainer type="info">
  <CalloutDescription>
    Since v1.7
  </CalloutDescription>
</CalloutContainer>

| `Value`   | `Description`                        |
| --------- | ------------------------------------ |
| `TEXT`    | (`'text'`): Text message.            |
| `IMAGE`   | (`'image'`): Image message.          |
| `UNKNOWN` | (`'unknown'`): Unknown message type. |

### EAgentState [#eagentstate]

Agent state enumeration.

```javascript
export enum EAgentState {
 IDLE = "idle",
 LISTENING = "listening", 
 THINKING = "thinking",
 SPEAKING = "speaking",
 SILENT = "silent"
}
```

| Value       | Description                                                  |
| ----------- | ------------------------------------------------------------ |
| `IDLE`      | ("idle"): The agent is idle and ready for new interactions.  |
| `LISTENING` | ("listening"): The agent is receiving user input.            |
| `THINKING`  | ("thinking"): The agent is processing the input it receives. |
| `SPEAKING`  | ("speaking"): The agent is outputting a response.            |
| `SILENT`    | ("silent"): The agent intentionally does not respond.        |

### EModuleType [#emoduletype]

Enumeration of module types for AI capabilities.

| Value     | Description                                |
| --------- | ------------------------------------------ |
| `LLM`     | ("llm"): Large Language Model.             |
| `MLLM`    | ("mllm"): Multimodal Large Language Model. |
| `TTS`     | ("tts"): Text-to-speech module.            |
| `CONTEXT` | ("context"): Context management module.    |
| `UNKNOWN` | ("unknown"): Unknown module type.          |

### ESubtitleHelperMode [#esubtitlehelpermode]

The mode type for transcript processing.

| Value     | Description                         |
| --------- | ----------------------------------- |
| `TEXT`    | Processes transcript in text mode.  |
| `WORD`    | Processes transcript in word mode.  |
| `UNKNOWN` | Unknown transcript processing mode. |

### ETurnStatus [#eturnstatus]

TURN connection status enumeration.

| Value         | Description                                 |
| ------------- | ------------------------------------------- |
| `IN_PROGRESS` | (0): TURN connection in progress.           |
| `END`         | (1): TURN connection has ended.             |
| `INTERRUPTED` | (2): TURN connection has been disconnected. |