Start a conversational AI agent
Updated
Creates and starts a Conversational AI agent instance.
https://api.agora.io/api/conversational-ai-agent/v2/projects/{appid}/joinUse this endpoint to create and start a Conversational AI agent instance.
Path Parameters
The App ID of the project.
Request Body
application/json
The unique identifier of the agent. The same identifier cannot be used repeatedly.
Deprecated. Use credential_mode: "managed" within the asr, llm, or tts block instead. See Use managed mode.
A comma-separated string of one or more presets. Each preset provides a predefined configuration for ASR, LLM, and TTS. You can specify a preset for any or all of ASR, LLM, and TTS. When a preset is specified, you do not need to provide the endpoint URL, API key, or model for the preset providers. Use the asr, llm, and tts fields to configure additional settings.
The unique ID of a published agent in Conversational AI Studio. When provided, the saved agent configuration is used as the base configuration. Any fields specified in properties override the corresponding agent settings. When you specify a pipeline_id, the asr, tts, and llm fields in properties are optional.
The unique ID of a published agent in STUDIO. When provided, the saved agent configuration is used as the base configuration. Any fields specified in properties override the corresponding agent settings. When you specify a pipeline_id, the asr, tts, and llm fields in properties are optional.
Configuration details of the agent.
The name of the channel to join.
The authentication token used by the agent to join the channel.
The user ID of the agent in the channel. All UIDs within an RTC channel must be unique. Ensure no other user or service bot is using this UID. A value of 0 means that a unique random UID is generated and assigned. Set the token accordingly.
A list of user IDs that the agent subscribes to in the channel. Only subscribed users can interact with the agent.
Note
Currently, only one user ID is supported.
Whether to enable String uid:
-
true: Both agent and subscriber user IDs use strings. -
false: Both agent and subscriber user IDs must be integers.
falseThe maximum idle time in seconds after all users specified in remote_rtc_uids are detected to have left the channel. When the idle time exceeds this value, the agent automatically stops and exits the channel. The valid range is 0 to 259200 (72 hours).
- When set to
0, the agent does not exit due to channel idle timeout. - Regardless of the
idle_timeoutvalue, the maximum running time for a single session is 72 hours. The agent automatically exits after 72 hours.
Agent lifecycle best practice
For precise and reliable control over the agent's lifecycle, use the leave API to terminate the agent as soon as its task is complete.
30[0, 259200]Regional access restriction configuration. Use this to limit which Agora servers the Conversational AI Engine can access based on geographic regions.
The allowed region for server access.
GLOBAL | NORTH_AMERICA | EUROPE | ASIA | INDIA | JAPANThe excluded region. Only available when area is set to GLOBAL.
NORTH_AMERICA | EUROPE | ASIA | INDIA | JAPANAdvanced features configuration.
Use mllm.enable instead.
Enable Multimodal Large Language Model for voice-to-voice processing. Enabling MLLM automatically disables ASR, LLM, and TTS since the MLLM handles end-to-end voice processing directly. See turn_detection.type for turn detection options available with MLLM.
falseWhether to enable the Signaling (RTM) service. When enabled, the agent can combine the capabilities provided by Signaling to implement advanced functions, such as delivering custom information.
Note
Before enabling the Signaling service, make sure the token includes both
RTC and RTM privileges. When an agent joins an RTM channel, it reuses
the token specified in the token field. For more information, see
"How can I generate a token with both RTC and Signaling
privileges?".
falseEnable Selective Attention Locking (SAL). When enabled, configure the sal field to set up speaker recognition or locking modes. See the sal parameter for configuration details.
falseEnable tool invocation. When enabled, the agent can invoke tools provided by the MCP server to implement advanced functionality.
falseAutomatic Speech Recognition (ASR) configuration.
The credential mode for this provider:
-
managed: Use Agora-managed credentials. You do not need to provide an API key. See Use managed mode. -
byok: Bring your own key. Provide your own provider credentials, including the API key.
byokmanaged | byokThe BCP-47 language tag identifying the primary language used for agent interaction. If params contains a vendor-specific language code, it takes precedence over this setting.
en-USASR provider:
ares: Adaptive Recognition Engine for Speechmicrosoft: Microsoft Azuredeepgram: Deepgramopenai: OpenAIspeechmatics: Speechmaticsassemblyai: AssemblyAIamazon: Amazon Transcribegoogle: Googlesarvam: Sarvamxai: xAI
aresares | microsoft | deepgram | openai | google | amazon | assemblyai | sarvam | xaiThe configuration parameters for the selected ASR vendor. For vendor-specific parameters, see the corresponding ASR provider page listed in asr.vendor.
Text-to-speech (TTS) module configuration.
The credential mode for this provider:
-
managed: Use Agora-managed credentials. You do not need to provide an API key. See Use managed mode. -
byok: Bring your own key. Provide your own provider credentials, including the API key.
byokmanaged | byokTTS provider.
amazon: Amazon Pollycartesia: Cartesiadeepgram: Deepgramelevenlabs: ElevenLabsfishaudio: Fish Audiogeneric_http: Generic TTS, integrated using the OpenAI TTS protocol. Requirestts.url,tts.headers, andtts.params.google: Googlegradium: Gradiumhumeai: Hume AImicrosoft: Microsoft Azureminimax: MiniMaxmistral: Mistralmurf: Murfopenai: OpenAIrime: Rimesarvam: Sarvamxai: xAI
amazon | cartesia | deepgram | elevenlabs | fishaudio | generic_http | google | gradium | humeai | microsoft | minimax | mistral | openai | rime | sarvam | xaiThe OpenAI TTS protocol compatible service endpoint. Required only when tts.vendor is generic_http.
Custom request headers to include in requests to the TTS service, such as authentication credentials. Required only when tts.vendor is generic_http.
Example:
{
"Authorization": "Bearer your_tts_api_key"
}
The configuration parameters for the selected TTS vendor. For vendor-specific parameters, see the corresponding TTS provider page listed in tts.vendor. Required when tts.vendor is generic_http.
Controls whether the TTS module skips bracketed content when reading LLM response text. This prevents the agent from vocalizing structural prompt information like tone indicators, action descriptions, and system prompts, creating a more natural and immersive listening experience. Enable this feature by specifying one or more values:
1: Skip content in Chinese parentheses（）2: Skip content in Chinese square brackets【】3: Skip content in parentheses( )4: Skip content in square brackets[ ]5: Skip content in curly braces{ }
Note
-
Nested brackets: When input text contains nested brackets and multiple bracket types are configured to be skipped, the system processes only the outermost brackets. The system matches from the beginning of the text and skips the first outermost bracket pair that meets the skip rule, including all nested content.
-
Agent memory: The agent's short-term memory always contains the complete, unfiltered LLM text, regardless of live captioning settings.
-
Real-time transcript: When enabled, transcript excludes filtered content during TTS playback but restores the complete text after each sentence finishes.
Large language model (LLM) configuration.
The credential mode for this provider:
-
managed: Use Agora-managed credentials. You do not need to provide an API key. See Use managed mode. -
byok: Bring your own key. Provide your own provider credentials, including the API key.
byokmanaged | byokThe LLM callback address.
The LLM verification API key. The default value is an empty string. Ensure that you enable the API key in a production environment.
A set of predefined information used as input to the LLM, including prompt words and examples.
Additional LLM configuration parameters, such as the model used, and the maximum token limit. For vendor-specific parameters, see the corresponding LLM provider page listed in llm.vendor.
The number of conversation history messages cached in the LLM. History includes user and agent dialog messages, tool call information, and timestamps. Agent and user messages are recorded separately.
32LLM input modalities:
["text"]: Text only["text", "image"]: Text plus image. Recommended configuration, requires the selected LLM to support visual input
["text"]LLM output modalities:
["text"]: The output text is converted to speech by the TTS module and then published to the RTC channel.["audio"]: Voice only. Voice is published directly to the RTC channel.["text", "audio"]: Text plus voice. Write your own logic to process the output of LLM as needed.
["text"]Agent greeting broadcast configuration.
Determines when the agent sends greeting messages to users joining the channel.
single_every: Broadcasts a greeting every time a user joins the channel.single_first: Broadcasts a greeting only once to the first user who joins the channel.
single_everysingle_every | single_firstThe delay in milliseconds before the agent plays the greeting message after a user joins the channel.
true: Follows the globalinterruptionconfiguration.false: Uninterruptible. The greeting plays in its entirety. If the user speaks multiple times while the greeting plays, the system merges the speech segments after the greeting ends and sends them to the LLM for a single response.
trueThe timeout in milliseconds for downloading the greeting audio file. Only takes effect when greeting_audio_url is configured. If the download times out, the system falls back to TTS synthesis using greeting_message.
1000[200, 10000]The sample rate in Hz of the greeting audio file when the format is PCM. Only takes effect when greeting_audio_url is configured and the audio format is PCM. Not required for mp3 or wav files.
16000The URL of a pre-recorded audio file to play as the agent's greeting. When configured, the system downloads and plays this recording instead of synthesizing the greeting via TTS.
Supported formats: mp3, wav, pcm.
- Only
http://orhttps://URLs are supported. An empty string, non-URL string, or a URL exceeding2048bytes causes the request to return a400error. - For
pcmfiles, setgreeting_configs.audio_pcm_sample_rateto specify the sample rate. - The system resamples the imported audio to match
tts.params.sample_ratebefore playback.
You must also configure greeting_message as a fallback when using greeting_audio_url; otherwise the request returns a 400 error. If the recording fails to download, decode, or is in an unsupported format, the system automatically falls back to TTS synthesis using greeting_message.
Agent greeting text. This field serves the following purposes:
- Standalone greeting: If provided without
greeting_audio_url, the agent greets the first user who joins the channel using TTS synthesis. - Audio greeting fallback: When
greeting_audio_urlis configured, this field is required as a fallback. If the audio file fails to download, decode, or is in an unsupported format, the agent falls back to TTS synthesis using this text. - Interrupted greeting context: When
greeting_configs.interruptableistrueand the audio greeting is interrupted, the system uses this text to estimate the portion already played and injects it into short-term memory to maintain conversation context.
Prompt for agent activation failure. If provided, it is returned through TTS when the custom LLM call fails.
LLM provider, supports the following settings:
openai: OpenAIazure: Azure OpenAIxai: xAI Grokcustom: Custom LLM. When you set this option, the agent includes the following fields, in addition toroleandcontentwhen making requests to the custom LLM:turn_id: A unique identifier for each conversation turn. It starts from0and increments with each turn. One user-agent interaction corresponds to oneturn_id.timestamp: The request timestamp, in milliseconds.
openai | azure | xai | customThe request style for chat completion:
openai: For OpenAI and OpenAI-compatible APIsgemini: For Google Gemini and Google Vertex API formatanthropic: For Anthropic Claude API formatdify: For Dify API format
For vendor-specific parameters, see the corresponding LLM provider page listed in llm.vendor.
openaiopenai | gemini | anthropic | difyTemplate parameter configuration used to insert variables into the agent's system_messages, greeting_message, failure_message, and parameters.silence_config.content text. Uses key-value pairs, where the key is the variable name and the value is the variable's value. Template variables, combined with prompt customization and SIP outbound calling functionality, enable dynamic content injection, automating processes such as automatic hang-up, voicemail recognition, automatic message leaving, and call transfer.
To insert defined variables in the prompt text, use the syntax {{variable_name}}. The system automatically replaces each variable with the corresponding value defined in template_variables.
Note
Variable values cannot reference other variables. For example, if you
define "farewell": "Looking forward to seeing you again, {{name}}",
the {{name}} variable will not be resolved.
MCP (Model Context Protocol) server configuration. By configuring MCP servers, agents can call tools provided by external services to implement advanced functionality.
A unique identifier for the MCP server. Maximum 48 characters. Accepts only English letters and numbers.
The endpoint address of the MCP server. The agent uses this to communicate with the MCP server.
Transport protocol type.
streamable_http: Streaming HTTP protocol
HTTP header information to include when requesting the MCP server, such as authentication information.
A list of tools that the agent is allowed to invoke. The agent can only use tools on this list.
Behavior:
- Empty or omitted: All tools are enabled.
- Empty array
[]: No tools are enabled. ["*"]: All tools are enabled.- Specific tools
["aa", "bb", "cc"]: Onlyaa,bb, andccare enabled. - Mix with wildcard
["aa", "bb", "*"]: All tools are enabled (wildcard takes precedence).
The MCP server request timeout in milliseconds. After timeout, the agent stops waiting for the MCP server's response and continues executing subsequent logic.
Custom headers to include in requests to the LLM. Use this field to pass business-specific information such as custom fields or tenant identifiers.
Note
-
These headers are merged with the headers generated by the Conversational AI Engine. If a key conflict occurs, the engine-generated header takes precedence. For example, authentication-related headers.
-
Header keys are merged using exact string matching and are case-sensitive. Agora recommends using standard capitalization to avoid ambiguity from duplicate keys with different casing.
Multimodal Large Language Model (MLLM) configuration for real-time audio and text processing. mllm is an exclusive alternative to the standard asr + llm + tts pipeline."
Enable Multimodal Large Language Model for voice-to-voice processing. Enabling MLLM automatically disables ASR, LLM, and TTS since the MLLM handles end-to-end voice processing directly. Replaces the deprecated advanced_features.enable_mllm.
The MLLM WebSocket URL for real-time communication.
The API key used for MLLM authentication.
Array of conversation items used for short-term memory management. Uses the same structure as item.content from the OpenAI Realtime API.
Additional MLLM configuration parameters.
- Modalities override: The
modalitiessetting in params is overridden byinput_modalitiesandoutput_modalities. - Turn detection override: The
turn_detectionsetting inparamsis overridden bymllm.turn_detection.
For vendor-specific parameters, see the corresponding MLLM provider page listed in mllm.vendor.
MLLM input modalities:
["audio"]: Audio only["audio", "text"]: Audio plus text
["audio"]MLLM output modalities:
["text", "audio"]: Text plus audio
["text","audio"]Agent greeting message. If provided, the first user in the channel is automatically greeted with this message upon joining.
MLLM provider. Currently supports:
openai: OpenAI Realtime APIgemini: Google Gemini Livevertexai: Google Gemini Live (Vertex AI)xai: xAI Grok
openai | gemini | vertexai | xaiTurn detection configuration for the MLLM module.
Note
When mllm.turn_detection is defined, the top-level turn_detection
object has no effect.
agora_vad: Agora VAD-based detection.server_vad: Vendor-side VAD-based detection. Supported by OpenAI Realtime API and Gemini Live.semantic_vad: Semantic-based detection. Supported by OpenAI Realtime API only.
agora_vad | server_vad | semantic_vadConfiguration for Agora VAD-based turn detection. Applicable when mode is agora_vad.
Minimum duration of speech in milliseconds required to trigger an interruption.
Duration of audio in milliseconds to include before the detected speech start.
Duration of silence in milliseconds required to determine end of speech.
VAD sensitivity threshold. A higher value reduces false positives.
Configuration for vendor-side VAD-based turn detection. Applicable when mode is server_vad. Parameters are passed through to the vendor.
Duration of audio in milliseconds to include before the detected speech start.
Duration of silence in milliseconds required to determine end of speech.
VAD sensitivity threshold. Applicable to OpenAI Realtime API only.
Idle timeout in milliseconds. Applicable to OpenAI Realtime API only.
Sensitivity for start of speech detection. Applicable to Gemini Live only.
START_SENSITIVITY_HIGH | START_SENSITIVITY_LOWSensitivity for end of speech detection. Applicable to Gemini Live only.
END_SENSITIVITY_HIGH | END_SENSITIVITY_LOWConfiguration for semantic-based turn detection. Applicable when mode is semantic_vad. Supported by OpenAI Realtime API only.
Controls how eagerly the model ends its turn.
auto | low | medium | highAvatar configuration.
Whether to enable the avatar function for the agent. To enable, set to true and configure the vendor and params fields.
falseAvatar vendor. Supports the following values:
akool: Akoolliveavatar: LiveAvataranam: Anamgeneric: Generic
akool | liveavatar | anam | genericThe configuration parameters for the selected avatar vendor. For vendor-specific parameters, see the corresponding avatar provider page listed in avatar.vendor.
Conversation turn detection settings. Controls the logic for voice activity detection and conversation turn determination. The previous version of turn_detection is deprecated. Refer to Deprecated parameters for details. Agora recommends switching to the latest parameters.
Note
This object has no effect when mllm.enable is true. Use
mllm.turn_detection instead.
Note
Starting with v2.6, turn_detection only handles Start of Speech (SoS) and
End of Speech (EoS) detection. Interruption handling strategies, including
keyword-based interruption and disabling interruption, have moved to the
top-level interruption field.
This configuration supports multiple combinations of detection modes:
-
Start of Speech (SoS): Supports three modes: VAD, Keyword, and Disable.
-
End of Speech (EoS): Supports VAD and Semantic modes.
Conversation turn detection mode:
default: Uses standard conversation turn detection configuration.
defaultDeprecated turn detection mechanism. agora_vad uses Agora VAD, server_vad lets the model detect speech start and end based on audio volume, and semantic_vad uses semantic estimation with VAD.
Compatible with both cascade (ASR/LLM/TTS) and MLLM modes.
agora_vadagora_vad | server_vad | semantic_vadDeprecated interruption behavior when human voice interrupts the agent while it is speaking or thinking. Sets the agent's behavior when human voice interrupts the agent while it is interacting (speaking or thinking).
Note
Only the interrupt mode is supported when you integrate an mllm.
interruptinterrupt | append | ignore | keywords | adaptiveThe amount of time in milliseconds that the user's voice must exceed the VAD threshold before an interruption is triggered.
160Keywords that trigger an interruption when turn_detection.interrupt_mode is set to keyword.
Specifies the list of keywords that trigger an interruption when the turn_detection.interrupt_mode is set to "keyword".
Note
- Keyword recognition capabilities, such as support for multiple languages or dialects, depend on the ASR provider you choose.
- You can configure up to 128 keywords.
The extra forward padding time in milliseconds before processing starts.
800The duration of audio silence in milliseconds after which the agent assumes the user has stopped speaking.
640Voice activity detection sensitivity. The valid range is (0.0, 1.0).
Identification sensitivity determines the level of sound in the audio signal that is considered voice activity.
0.5(0, 1)The eagerness of the model to respond. Only available in semantic_vad mode when using OpenAI Realtime API.
autoauto | low | highDetailed configuration for conversation turn detection.
Voice activity detection sensitivity. Determines the sound level in the audio signal that is considered voice activity. Lower values make it easier for the agent to detect speech, and higher values ignore weak sounds.
0.5(0, 1)Start of Speech (SoS) detection configuration. Determines when a user begins speaking.
Start of speech detection mode:
vad: Based on VAD (Voice Activity Detection). Uses audio signal detection.manual: Manual SoS mode. Disables automatic start-of-speech detection. The client explicitly signals the start of user speech via RTM.keywords: Deprecated. Useinterruption.mode = "keywords"instead.disabled: Deprecated. Useinterruption.enable = falsewithinterruption.disabled_config.strategyto configure the handling strategy.
Note
When mode is manual, you must also set advanced_features.enable_rtm = true and parameters.data_channel = "rtm". This mode does not support manual_config; no additional configuration is required.
vad | manual | keywords | disabledStart of speech detection configuration parameters. The structure and supported fields vary depending on the detection mode. Configuration examples:
vad_config
"vad_config": {
"interrupt_duration_ms": 160,
"speaking_interrupt_duration_ms": 160,
"prefix_padding_ms": 800
}
keywords_config
"keywords_config": {
"interrupt_duration_ms": 160,
"prefix_padding_ms": 800,
"triggered_keywords": ["Are you there", "hello"]
}
disabled_config
"disabled_config": {
"strategy": "append"
}
Note
-
The configuration type must match
mode. For example, whenmodeisvad, you must providevad_config. -
You cannot provide multiple mode configurations simultaneously.
The amount of time in milliseconds that the user's voice must exceed the VAD threshold before an interruption is triggered.
160Interruption duration in milliseconds while the agent is speaking.
160The extra forward padding time in milliseconds before the processing system starts to process the speech input. This padding helps capture the beginning of speech.
800Specifies the list of keywords that trigger an interruption. When the agent detects any of these keywords in the user's speech, it immediately stops its current interaction and processes the new input.
Voice processing strategy when the agent is interacting (speaking or thinking):
append: Append mode. Human voice does not interrupt the agent. The agent processes the human voice input after the current interaction ends.ignore: Ignore mode. The agent ignores human voice input. If the agent receives human voice while speaking or thinking, the agent discards the input without storing it in context.
append | ignoreEnd of Speech (EoS) detection configuration. Determines when a user ends their speech.
End of speech detection mode. Possible values:
vad: Based on VAD (Voice Activity Detection). Detects silence duration.semantic: Based on semantic triggering. Uses semantic understanding to determine when conversation ends.manual: Manual EoS mode. Disables automatic end-of-speech detection. The client explicitly signals the end of user speech via RTM.
Note
- When
modeissemantic, EoS detection supports English and Chinese only. For unsupported languages, the engine falls back to VAD. - When
modeismanual, you must also setadvanced_features.enable_rtm = trueandparameters.data_channel = "rtm". This mode does not supportmanual_config; no additional configuration is required.
vad | semantic | manualEnd of speech detection configuration parameters. The structure and supported fields vary depending on the detection mode. Configuration examples:
vad_config
"vad_config": {
"silence_duration_ms": 640
}
semantic_config
"semantic_config": {
"silence_duration_ms": 320,
"max_wait_ms": 3000,
"pause_state_enabled": true
}
Note
-
The configuration type must match
mode. For example, whenmodeisvad, you must providevad_config. -
You cannot provide multiple mode configurations simultaneously.
Default: 640 in vad_config, 320 in semantic_config
Silence duration threshold in milliseconds. The minimum silence duration at the end of a speech segment, to ensure that a brief pause does not prematurely end the speech segment.
-1 means forever.
Maximum wait time in milliseconds. The maximum time to wait for semantic determination. After timeout, the conversation end is determined based on the current state.
3000Whether to detect user intent to pause the conversation:
true: The agent uses semantic understanding to determine if the user intends to pause the conversation. For example, when the user's input ends with phrases such as "hold on" or "just a moment", the agent waits for further input rather than treating the utterance as complete and sending it to the LLM.false: The agent does not detect intent to pause the conversation.
trueInterruption control configuration. Provides unified management of the agent's behavior when interrupted by the user.
Whether to enable agent interruption:
true: Enable interruption.false: Disable interruption. When disabled, the agent cannot be interrupted mid-response.
trueThe interruption trigger mode:
start_of_speech: Trigger interruption when the user starts speaking.keywords: Trigger interruption when the user speaks a specified keyword. Configure the trigger keywords inkeywords_config.
start_of_speechstart_of_speech | keywordsConfiguration for keyword-based interruption triggering. Applicable only when mode is keywords.
The list of keywords that trigger an interruption. A maximum of 128 keywords is supported.
Configuration for agent behavior when interruption is disabled. Applicable only when interruption.enable is false.
The processing strategy when interruption is disabled:
append: User speech does not interrupt the agent. The agent processes the user's input after the current interaction ends.ignore: The agent ignores user speech. If the agent receives user speech while speaking or thinking, it discards the input without storing it in context.
append | ignoreSelective Attention Locking (SAL) configuration. (Beta)
Selective attention lock mode. Supports the following options:
-
locking: Speaker Lock Mode. The agent locks onto the speaker, blocking 95% of ambient human voices and noise. You can enable this mode in two ways: -
Seamless mode: When a user speaks loudly and clearly at the beginning of a conversation, the intelligent agent automatically recognizes the user as the speaker.
-
Personalized mode: When creating an agent, a speaker's voiceprint URL is pre-registered through the
sample_urlsfield. The agent then locates the speaker based on the pre-registered voiceprint. -
recognition: Voiceprint recognition mode. You can pre-register only one voiceprint URL using thesample_urlsfield. The agent identifies different speakers and suppresses other background voices and environmental noise. The target speaker is identified through thevpidsfield in themetadatafield and sent to the LLM. Setllm.vendorto "custom" and refer to Custom LLM for instructions on how to make the LLM process speaker information.
lockinglocking | recognitionThe registered voiceprint URL as a key-value pair, where the key is the voiceprint name and the value is the download URL for the speaker's voiceprint. Only one voiceprint URL is supported. Example:
{
"speaker1": "https://example.com/speaker1.pcm"
}
Note
-
Do not set the incoming voiceprint name to "unknown"; this is a reserved keyword used to identify unknown speakers.
-
For a registered voiceprint, ensure that:
-
Size: The voiceprint file does not exceed 2 MB.
-
Duration: Contains 10 to 15 seconds of audio, with at least 8 seconds of effective audio excluding silent segments.
-
Format: 16kHz sampling rate, 16-bit depth, mono PCM audio file. The file name extension must be ".pcm".
Custom labels in key-value pair format, where the key is the label name and the value is the label value. Enables agents to carry custom business information.
These labels are bound to the agent and returned in the payload field of all message notification callbacks from the conversational AI engine. Use them to implement custom business logic, such as tagging activity IDs, customer groups, and business scenarios.
RTC media encryption configuration.
The encryption key for RTC media content. The key has no length limit. Agora recommends using a 32-byte key. If no encryption key is set or if the key is empty, built-in encryption is not used.
The salt value used for encryption. This is a Base64-encoded string that is 32 bytes long after decoding. This parameter only takes effect when encryption_mode is set to 7 (AES_128_GCM2) or 8 (AES_256_GCM2). Ensure that the salt parameter is not empty for these encryption modes.
The built-in encryption mode.
1:AES_128_XTS- 128-bit AES encryption, XTS mode.2:AES_128_ECB- 128-bit AES encryption, ECB mode.3:AES_256_XTS- 256-bit AES encryption, XTS mode.4:SM4_128_ECB- 128-bit SM4 encryption, ECB mode.5:AES_128_GCM- 128-bit AES encryption, GCM mode.6:AES_256_GCM- 256-bit AES encryption, GCM mode.7:AES_128_GCM2- 128-bit AES encryption, GCM mode. Requires settingencryption_salt.8:AES_256_GCM2- 256-bit AES encryption, GCM mode. Requires settingencryption_salt.
Agora recommends using either 7 (AES_128_GCM2) or 8 (AES_256_GCM2) mode. Both modes support cryptographic salts to enhance security.
1 | 2 | 3 | 4 | 5 | 6 | 7 | 8Filler word configuration. Plays filler words while waiting for LLM responses to reduce user anxiety and improve conversation flow.
Filler word playback follows these rules:
- Playback order: When multiple filler words or LLM responses are waiting to be played, they are played in the order they arrive.
- Interruption control: Inherits the interruption mode setting from the
interruptionfield.
Whether to enable filler words:
true: Enable filler words.false: Disable filler words.
falseFiller word trigger configuration. Defines when to trigger filler word playback.
Filler word trigger mode:
fixed_time: Fixed time trigger. Triggers filler word playback when LLM response wait time exceeds the threshold.
Filler word trigger configuration parameters. The parameter name and structure vary depending on the trigger mode. Configuration example:
"fixed_time_config": {
"response_wait_ms": 1500
}
Note
-
The configuration type must match
mode. For example, whenmodeisfixed_time, you must providefixed_time_config. -
You cannot provide multiple mode configurations simultaneously.
LLM response wait threshold in milliseconds. Triggers filler word playback when the LLM waits this duration without generating a response, such as when waiting for RAG retrieval or tool call results.
1500Filler word content configuration. Defines the source and selection rules for filler words.
Filler word content mode:
static: Static filler words. Uses a predefined list of filler words.
Filler word content configuration parameters. The parameter name and structure vary depending on the content mode.
Note
-
The configuration type must match
mode. For example, whenmodeisstatic, you must providestatic_config. -
You cannot provide multiple mode configurations simultaneously.
Static filler word configuration example:
"static_config": {
"phrases": [
"Please wait.",
"Okay.",
"Uh-huh."
],
"selection_rule": "shuffle"
}
List of filler word phrases.
Limits:
- Maximum 100 filler words.
- Each filler word must not exceed 50 English words.
Filler word selection rule:
shuffle: Random shuffle. Already-used filler words are not repeated until all filler words have been used once. After all filler words are played, they are reshuffled randomly and a new round begins.round_robin: Round-robin. Selects and plays filler words sequentially from the list. After all filler words are played once, a new cycle begins.
shuffle | round_robinAgent configuration parameters.
Settings related to agent silence behavior.
Note
silence_config does not apply when you integrate a mllm.
Specifies the maximum duration (in milliseconds) that the agent can remain silent. After the agent is successfully created and the user joins the channel, any time during which the agent is not listening, thinking, or speaking is considered silent time. When the silent time reaches the specified value, the agent broadcasts a silent reminder message. This feature is useful for prompting users when they become inactive.
0: Disables the silent reminder feature.(0, 60000]: Enables the silent reminder. You must also setcontent; otherwise, the configuration is invalid.
0Specifies how the agent behaves when the silent timeout is reached. Valid values:
speak: Uses the TTS module to announce the silent prompt (content).think: Appends the silent prompt (content) to the context and passes it to the LLM.
speakSpecifies the silent prompt message. The message use depends on the value of action parameter.
Graceful hang-up settings for the agent.
Enable graceful leave:
true: Enabled. When enabled, calling the POST method to stop the agent ensures that the agent is in anIDLEstate before leaving the channel.false: Disabled.
falseGraceful exit timeout (in seconds). Represents the maximum time to wait for the agent to enter an IDLE state before exiting the channel. After this time, the agent will exit the channel immediately, even if it is not in an idle state. This field is only effective when graceful_enabled is true.
30Agent data transmission channel:
rtm: Use RTM transmission. This configuration takes effect only whenadvanced_features.enable_rtmistrue.datastream: Use RTC data stream transport.
datastreamWhether to receive agent performance data:
true: Receive agent performance data.false: Do not receive agent performance data.
This setting only takes effect when advanced_features.enable_rtm is true. See Listen to agent events to learn how to use client components to receive agent performance data.
falseWhether to receive agent error events:
true: Receive agent error events.false: Do not receive agent error events.
This setting only takes effect when advanced_features.enable_rtm is true. See Listen to agent events to learn how to use client components to receive agent error events.
falseThe audio scenario for the RTC channel.
default: Maps toaiserver.chorus: Real-time chorus scenario, where users have good network conditions and require ultra-low latency.aiserver: Optimized for interactions between the user and the conversational AI agent in terms of latency and network resilience.
defaultdefault | chorus | aiserverWhether to disable data retention for the current session.
false: (Default) Data retention is enabled.true: Data retention is disabled. When disabled, session interaction text and audio are not retained, which means troubleshooting, performance review, and agent optimization based on session history will not be available.
Note
To support service operation, agent optimization, and troubleshooting, session interaction text and audio are temporarily retained for the minimum necessary period by default. If your business has strict security or compliance requirements, enable this setting to prevent session data from being stored.
falseResponse
-
If the returned status code is
200, the request was successful. The response body contains the result of the request. -
If the returned status code is not
200, the request failed. The response body includes thedetailandreasonfor failure. Refer to status codes to understand the possible reasons for failure. -
The following turn detection configuration is deprecated. To create more natural conversations and reduce unintended interruptions, Agora recommends using the latest version of
turn_detectionabove.
Response Body
application/json
application/json
Response schema
200The request was successful. The response body contains the result of the request.
Unique id of the agent instance
Timestamp of when the agent was created
Current status.
-
IDLE(0): Agent is idle. -
STARTING(1): The agent is being started. -
RUNNING(2): The agent is running. -
STOPPING(3): The agent is stopping. -
STOPPED(4): The agent has exited. -
FAILED(6): The agent failed to execute.
IDLE | STARTING | RUNNING | STOPPING | STOPPED | FAILEDRequest examples
curl --request post \
--url https://api.agora.io/api/conversational-ai-agent/v2/projects/:appid/join \
--header 'Authorization: Basic <your_base64_encoded_credentials>' \
--data '
{
"name": "unique_name",
"properties": {
"channel": "channel_name",
"token": "token",
"agent_rtc_uid": "1001",
"remote_rtc_uids": [
"1002"
],
"idle_timeout": 120,
"llm": {
"url": "https://api.openai.com/v1/chat/completions",
"api_key": "<your_llm_key>",
"system_messages": [
{
"role": "system",
"content": "You are a helpful chatbot."
}
],
"max_history": 32,
"greeting_message": "Hello, how can I assist you today?",
"failure_message": "Please hold on a second.",
"params": {
"model": "gpt-4o-mini"
}
},
"tts": {
"vendor": "microsoft",
"params": {
"key": "<your_tts_api_key>",
"region": "eastus",
"voice_name": "en-US-AndrewMultilingualNeural"
}
},
"asr": {
"language": "en-US"
}
}
}'
Response example
{ "agent_id": "1NT29X10YHxxxxxWJOXLYHNYB", "create_ts": 1737111452, "status": "RUNNING"}