Start a conversational AI agent
Start a conversational AI agent
https://api.agora.io/api/conversational-ai-agent/v2/projects/{appid}/joinUse this endpoint to create and start a Conversational AI agent instance.
Request
Path parameters
The App ID of the project.
Request body
BODYrequired
- name stringrequired
The unique identifier of the agent. The same identifier cannot be used repeatedly.
- properties objectrequired
Configuration details of the agent.
Show propertiesHide properties
- channel stringrequired
The name of the channel to join.
- token stringrequired
The authentication token used by the agent to join the channel.
- agent_rtc_uid stringrequired
The user ID of the agent in the channel. A value of
0means that a random UID is generated and assigned. Set thetokenaccordingly. - remote_rtc_uids array[string]required
A list of user IDs that the agent subscribes to in the channel. Only subscribed users can interact with the agent. Use
"*"to subscribe to all users in the channel.infoThe
"*"selector includes all UIDs present in the channel, which may include other AI agents. If you're running multiple agents in the same channel, review the best practices under theidle_timeoutparameter to avoid unintended behavior and unnecessary usage costs. - enable_string_uid booleannullable
Default:
falseWhether to enable String uid:
true: Both agent and subscriber user IDs use strings.false: Both agent and subscriber user IDs must be integers.
- idle_timeout integernullable
Default:
30Sets the timeout after all the users specified in
remote_rtc_uidsare detected to have left the channel. When the timeout value is exceeded, the agent automatically stops and exits the channel. A value of0means that the agent does not exit until it is stopped manually.Multi-agent use casesIf multiple AI agents are active in the same channel and are configured to subscribe to all users using
remote_rtc_uids: ["*"], they detect each other's presence. As a result, theidle_timeoutcondition, when all other users have left, might never be triggered. This can cause agents to run indefinitely and lead to significant unintended usage.Agent lifecycle best practiceFor precise and reliable control over the agent's lifecycle, use the
leaveAPI to terminate the agent as soon as its task is complete. - advanced_features objectnullable
Advanced features configuration.
Show propertiesHide properties
- enable_aivad booleannullable
Default:
falseWhether to enable the intelligent interruption handling function (AIVAD). This feature is currently available only for English.
- enable_mllm booleannullable
Default:
falseEnable Multimodal Large Language Model. Enabling MLLM automatically disables ASR, LLM, and TTS. When you set this parameter to true,
enable_aivadis also disabled. - enable_rtm booleannullable
Default:
falseWhether to enable the Signaling (RTM) service. When enabled, the agent can combine the capabilities provided by Signaling to implement advanced functions, such as delivering custom information.
infoBefore enabling the Signaling service, make sure the token includes both RTC and RTM privileges. When an agent joins an RTM channel, it reuses the token specified in the
tokenfield. For more information, see "How can I generate a token with both RTC and Signaling privileges?".
- asr objectnullable
Automatic Speech Recognition (ASR) configuration.
Show propertiesHide properties
- language stringnullable
Default:
en-USThe BCP-47 language tag identifying the primary language used for agent interaction. If
paramscontains a vendor-specific language code, it takes precedence over this setting. - vendor stringnullable
Default:
aresPossible values:
ares,microsoft,deepgramASR provider:
ares: Adaptive Recognition Engine for Speechmicrosoft: Microsoft Azuredeepgram: Deepgram
- params objectrequired
The configuration parameters for the ASR vendor. See ASR Overview for details.
- tts objectrequired
Text-to-speech (TTS) module configuration.
Show propertiesHide properties
- vendor stringrequired
Possible values:
microsoft,elevenlabs,cartesia,openaiTTS provider.
microsoft: Microsoft Azureelevenlabs: ElevenLabscartesia: Cartesiaopenai: OpenAI
- params objectrequired
The configuration parameters for the TTS vendor. See TTS Overview for details.
- skip_patterns array[integer]nullable
Controls whether the TTS module skips bracketed content when reading LLM response text. This prevents the agent from vocalizing structural prompt information like tone indicators, action descriptions, and system prompts, creating a more natural and immersive listening experience. Enable this feature by specifying one or more values:
1: Skip content in Chinese parentheses（）2: Skip content in Chinese square brackets【】3: Skip content in parentheses( )4: Skip content in square brackets[ ]5: Skip content in curly braces{ }
info- Nested brackets: When input text contains nested brackets and multiple bracket types are configured to be skipped, the system processes only the outermost brackets. The system matches from the beginning of the text and skips the first outermost bracket pair that meets the skip rule, including all nested content.
- Agent memory: The agent's short-term memory always contains the complete, unfiltered LLM text, regardless of live captioning settings.
- Real-time subtitles: When enabled, subtitles exclude filtered content during TTS playback but restore the complete text after each sentence finishes.
- llm objectrequired
Large language model (LLM) configuration.
Show propertiesHide properties
- url stringrequired
The LLM callback address.
- api_key stringnullable
The LLM verification API key. The default value is an empty string. Ensure that you enable the API key in a production environment.
- system_messages array[object]nullable
A set of predefined information used as input to the LLM, including prompt words and examples.
- params objectnullable
Additional LLM configuration parameters, such as the
modelused, and the maximum token limit. For details about each supported LLM, refer to Supported LLMs. - max_history integernullable
Default:
32The number of conversation history messages cached in the custom LLM. The minimum value is
1. History includes user and agent dialog messages, tool call information, and timestamps. Agent and user messages are recorded separately. - input_modalities array[string]nullable
Default:
["text"]LLM input modalities:
["text"]: Text only["text", "image"]: Text plus image; requires the selected LLM to support visual input
- output_modalities array[string]nullable
Default:
["text"]LLM output modalities:
["text"]: The output text is converted to speech by the TTS module and then published to the RTC channel.["audio"]: Voice only. Voice is published directly to the RTC channel.["text", "audio"]: Text plus voice. Write your own logic to process the output of LLM as needed.
- greeting_message stringnullable
Agent greeting. If provided, the first user in the channel is automatically greeted with the message upon joining.
- failure_message stringnullable
Prompt for agent activation failure. If provided, it is returned through TTS when the custom LLM call fails.
- vendor stringnullable
LLM provider, supports the following settings:
custom: Custom LLM. When you set this option, the agent includes the following fields, in addition toroleandcontentwhen making requests to the custom LLM:turn_id: A unique identifier for each conversation turn. It starts from0and increments with each turn. One user-agent interaction corresponds to oneturn_id.timestamp: The request timestamp, in milliseconds.
- style stringnullable
Default:
openaiPossible values:
openai,gemini,anthropic,difyThe request style for chat completion:
openai: For OpenAI and OpenAI-compatible APIsgemini: For Google Gemini and Google Vertex API formatanthropic: For Anthropic Claude API formatdify: For Dify API format
For details, refer to Supported LLMs.
- mllm objectnullable
Multimodal Large Language Model (MLLM) configuration for real-time audio and text processing.
Show propertiesHide properties
- url stringrequired
The MLLM WebSocket URL for real-time communication.
- api_key stringrequired
The API key used for MLLM authentication.
- messages array[object]nullable
Array of conversation items used for short-term memory management. Uses the same structure as
item.contentfrom the OpenAI Realtime API. - params objectnullable
Additional MLLM configuration parameters.
- Modalities override: The
modalitiessetting in params is overridden byinput_modalitiesandoutput_modalities. - Turn detection override: The
turn_detectionsetting in params is overridden by theturn_detectionsection outside ofmllm.
See MLLM Overview for details.
- Modalities override: The
- max_history integernullable
Default:
32The number of conversation history messages cached in the MLLM. The minimum value is
1. Cannot exceed the model's context window. - input_modalities array[string]nullable
Default:
["audio"]MLLM input modalities:
["audio"]: Audio only["audio", "text"]: Audio plus text
- output_modalities array[string]nullable
Default:
["text", "audio"]MLLM output modalities:
["text", "audio"]: Text plus audio
- greeting_message stringnullable
Agent greeting message. If provided, the first user in the channel is automatically greeted with this message upon joining.
- vendor stringnullable
Possible values:
openaiMLLM provider. Currently supports:
openai: OpenAI Realtime API
- style stringnullable
Default:
openaiPossible values:
openaiThe request style for MLLM completion:
openai: For OpenAI Realtime API format
- vad objectdeprecatednullable
Voice Activity Detection (VAD) configuration.
Deprecation NoticeThe
vadconfiguration section is deprecated and will not be supported in future versions. When the same parameter exists in bothvadandturn_detectionsections,turn_detectionparameters have higher priority. Agora recommends migrating toturn_detectionconfiguration for voice activity detection settings.Show propertiesHide properties
- interrupt_duration_ms numbernullable
Default:
160The amount of time in milliseconds that the user's voice must exceed the VAD threshold before an interruption is triggered.
- prefix_padding_ms integernullable
Default:
800The extra forward padding time in milliseconds before the processing system starts to process the speech input. This padding helps capture the beginning of the speech.
- silence_duration_ms integernullable
Default:
640The duration of audio silence in milliseconds. If no voice activity is detected during this period, the agent assumes that the user has stopped speaking.
- threshold numbernullable
Default:
0.5Identification sensitivity determines the level of sound in the audio signal that is considered voice activity. The value range is
(0.0, 1.0). Lower values make it easier for the agent to detect speech, and higher values ignore weak sounds.
- turn_detection objectnullable
Conversation turn detection settings.
Show propertiesHide properties
- type stringnullable
Default:
agora_vadPossible values:
agora_vad,server_vad,semantic_vadTurn detection mechanism.
agora_vad: Agora VAD.server_vad: The model detects the start and end of speech based on audio volume and responds at the end of user speech. Only available whenmllmis enabled and OpenAI is selected.semantic_vad: Uses a turn detection model in conjunction with VAD to semantically estimate whether the user has finished speaking, then dynamically sets a timeout based on this probability for more natural conversations. Only available whenmllmis enabled and OpenAI is selected.
- interrupt_mode stringnullable
Default:
interruptSets the agent's behavior when human voice interrupts the agent while it is interacting (speaking or thinking). Choose from the following values:
interrupt: The agent immediately stops the current interaction and processes the human voice input.append: The agent completes the current interaction, then processes the human voice input.ignore: The agent discards the human voice input without processing or storing it in the context.
infoOnly the
interruptmode is supported when you integrate amllm. - interrupt_duration_ms numbernullable
Default:
160The amount of time in milliseconds that the user's voice must exceed the VAD threshold before an interruption is triggered.
- prefix_padding_ms integernullable
Default:
800The extra forward padding time in milliseconds before the processing system starts to process the speech input. This padding helps capture the beginning of the speech.
- silence_duration_ms integernullable
Default:
640The duration of audio silence in milliseconds. If no voice activity is detected during this period, the agent assumes that the user has stopped speaking.
- threshold numbernullable
Default:
0.5Identification sensitivity determines the level of sound in the audio signal that is considered voice activity. The value range is
(0.0, 1.0). Lower values make it easier for the agent to detect speech, and higher values ignore weak sounds. - create_response booleannullable
Default:
trueWhether to automatically generate a response when a VAD stop event occurs. Only available in
server_vadandsemantic_vadmodes when using OpenAI Realtime API. - interrupt_response booleannullable
Default:
trueWhether to automatically interrupt any ongoing response when a VAD start event occurs. Only available in
server_vadandsemantic_vadmodes when using OpenAI Realtime API. - eagerness stringnullable
Default:
autoPossible values:
auto,low,highThe eagerness of the model to respond:
auto: Equivalent to mediumlow: Wait longer for the user to continue speakinghigh: Respond more quickly
Only available in
semantic_vadmode when using OpenAI Realtime API.
- parameters objectnullable
Agent configuration parameters.
Show propertiesHide properties
- silence_config objectnullable
Settings related to agent silence behavior.
infosilence_configdoes not apply when you integrate amllm.Show propertiesHide properties
- timeout_ms integernullable
Default:
0Possible values:
0 to 60000Specifies the maximum duration (in milliseconds) that the agent can remain silent. After the agent is successfully created and the user joins the channel, any time during which the agent is not listening, thinking, or speaking is considered silent time. When the silent time reaches the specified value, the agent broadcasts a silent reminder message. This feature is useful for prompting users when they become inactive.
0: Disables the silent reminder feature.(0, 60000]: Enables the silent reminder. You must also setcontent; otherwise, the configuration is invalid.
- action stringnullable
Default:
speakSpecifies how the agent behaves when the silent timeout is reached. Valid values:
speak: Uses the TTS module to announce the silent prompt (content).think: Appends the silent prompt (content) to the context and passes it to the LLM.
- content stringnullable
Specifies the silent prompt message. The message use depends on the value of
actionparameter.
- data_channel stringnullable
Default:
datastreamAgent data transmission channel:
rtm: Use RTM transmission. This configuration takes effect only whenadvanced_features.enable_rtmistrue.datastream: Use RTC data stream transport.
- enable_metrics booleannullable
Default:
falseWhether to receive agent performance data:
true: Receive agent performance data.false: Do not receive agent performance data.
This setting only takes effect when
advanced_features.enable_rtmistrue. See Listen to agent events to learn how to use client components to receive agent performance data. - enable_error_message booleannullable
Default:
falseWhether to receive agent error events:
true: Receive agent error events.false: (Default) Do not receive agent error events.
This setting only takes effect when
advanced_features.enable_rtmistrue. See Listen to agent events to learn how to use client components to receive agent error events.
Response
-
If the returned status code is
200, the request was successful. The response body contains the result of the request.OK
- agent_id string
Unique id of the agent instance
- create_ts integer
Timestamp of when the agent was created
- status string
Possible values:
IDLE,STARTING,RUNNING,STOPPING,STOPPED,RECOVERING,FAILEDCurrent status.
IDLE(0): Agent is idle.STARTING(1): The agent is being started.RUNNING(2): The agent is running.STOPPING(3): The agent is stopping.STOPPED(4): The agent has exited.RECOVERING(5): The agent is recovering.FAILED(6): The agent failed to execute.
-
If the returned status code is not
200, the request failed. The response body includes thedetailandreasonfor failure. Refer to status codes to understand the possible reasons for failure.