Start a conversational AI agent

Updated 2026/09/11 14:11:21

Creates and starts a Conversational AI agent instance.

POST

https://api.agora.io/api/conversational-ai-agent/v2/projects/{appid}/join

Use this endpoint to create and start a Conversational AI agent instance.

Path Parameters

appidstring

Required

The App ID of the project.

Request Body

application/json

namestring

Required

The unique identifier of the agent. The same identifier cannot be used repeatedly.

pipeline_idstring

The unique ID of a published agent in Conversational AI Studio. When provided, the saved agent configuration is used as the base configuration. Any fields specified in properties override the corresponding agent settings. When you specify a pipeline_id, the asr, tts, and llm fields in properties are optional.

propertiesobject

Required

Configuration details of the agent.

channelstring

Required

The name of the channel to join.

tokenstring

Required

The authentication token used by the agent to join the channel.

agent_rtc_uidstring

Required

The user ID of the agent in the channel. All UIDs within an RTC channel must be unique. Ensure no other user or service bot is using this UID. A value of 0 means that a unique random UID is generated and assigned. Set the token accordingly.

remote_rtc_uidsarray<string>

Required

A list of user IDs that the agent subscribes to in the channel. Only subscribed users can interact with the agent.

Note

Currently, only one user ID is supported.

enable_string_uidboolean

Default:

false

Whether to enable String uid:

true: Both agent and subscriber user IDs use strings.
false: Both agent and subscriber user IDs must be integers.

idle_timeoutinteger

Default:

Range:

0 <= value <= 259200

The maximum idle time in seconds after all users specified in remote_rtc_uids are detected to have left the channel. When the idle time exceeds this value, the agent automatically stops and exits the channel. The valid range is 0 to 259200 (72 hours).

When set to 0, the agent does not exit due to channel idle timeout.
Regardless of the idle_timeout value, the maximum running time for a single session is 72 hours. The agent automatically exits after 72 hours.

Agent lifecycle best practice

For precise and reliable control over the agent's lifecycle, use the leave API to terminate the agent as soon as its task is complete.

geofenceobject

Regional access restriction configuration. Use this to limit which Agora servers the Conversational AI Engine can access based on geographic regions.

advanced_featuresobject

Advanced features configuration.

asrobject

Automatic Speech Recognition (ASR) configuration.

ttsobject

Required

Text-to-speech (TTS) module configuration.

credential_modestring

Default:

"byok"

Allowed values:

managed, byok

The credential mode for this provider:

managed: Use Agora-managed credentials. You do not need to provide an API key. See Use managed mode.
byok: Bring your own key. Provide your own provider credentials, including the API key.

vendorstring

Required

Allowed values:

amazon, cartesia, deepgram, elevenlabs, fishaudio, generic_http, google, gradium, humeai, microsoft, minimax, mistral, murf, openai, rime, sarvam, typecast, xai

TTS provider.

amazon: Amazon Polly
cartesia : Cartesia
deepgram: Deepgram
elevenlabs: ElevenLabs
fishaudio: Fish Audio
generic_http: Generic TTS, integrated using the OpenAI TTS protocol. Requires tts.url, tts.headers, and tts.params.
google: Google
gradium: Gradium
humeai: Hume AI
microsoft: Microsoft Azure
minimax: MiniMax
mistral: Mistral
murf: Murf
openai: OpenAI
rime: Rime
sarvam: Sarvam
typecast: Typecast
xai: xAI

urlstring

The OpenAI TTS protocol compatible service endpoint. Required only when tts.vendor is generic_http.

headersobject

Custom request headers to include in requests to the TTS service, such as authentication credentials. Required only when tts.vendor is generic_http. Example:

{
"Authorization": "Bearer your_tts_api_key"
}

paramsobject

Required

The configuration parameters for the selected TTS vendor. For vendor-specific parameters, see the corresponding TTS provider page listed in tts.vendor. Required when tts.vendor is generic_http.

skip_patternsarray<integer>

Controls whether the TTS module skips bracketed content when reading LLM response text. This prevents the agent from vocalizing structural prompt information like tone indicators, action descriptions, and system prompts, creating a more natural and immersive listening experience. Enable this feature by specifying one or more values:

1: Skip content in Chinese parentheses （）
2: Skip content in Chinese square brackets 【】
3: Skip content in parentheses ( )
4: Skip content in square brackets [ ]
5: Skip content in curly braces { }

Note

Nested brackets: When input text contains nested brackets and multiple bracket types are configured to be skipped, the system processes only the outermost brackets. The system matches from the beginning of the text and skips the first outermost bracket pair that meets the skip rule, including all nested content.
Agent memory: The agent's short-term memory always contains the complete, unfiltered LLM text, regardless of live captioning settings.
Real-time transcript: When enabled, transcript excludes filtered content during TTS playback but restores the complete text after each sentence finishes.

llmobject

Required

Large language model (LLM) configuration.

credential_modestring

Default:

"byok"

Allowed values:

managed, byok

The credential mode for this provider:

managed: Use Agora-managed credentials. You do not need to provide an API key. See Use managed mode.
byok: Bring your own key. Provide your own provider credentials, including the API key.

urlstring

Required

The LLM callback address.

api_keystring

Default:

The LLM verification API key. The default value is an empty string. Ensure that you enable the API key in a production environment.

system_messagesarray<object>

A set of predefined information used as input to the LLM, including prompt words and examples.

paramsobject

Additional LLM configuration parameters, such as the model used, and the maximum token limit. For vendor-specific parameters, see the corresponding LLM provider page listed in llm.vendor.

max_historyinteger

Default:

The number of conversation history messages cached in the LLM. History includes user and agent dialog messages, tool call information, and timestamps. Agent and user messages are recorded separately.

input_modalitiesarray<string>

Default:

[  "text"]

LLM input modalities:

["text"]: Text only
["text", "image"]: Text plus image. Recommended configuration, requires the selected LLM to support visual input

output_modalitiesarray<string>

Default:

[  "text"]

LLM output modalities:

["text"]: The output text is converted to speech by the TTS module and then published to the RTC channel.
["audio"]: Voice only. Voice is published directly to the RTC channel.
["text", "audio"]: Text plus voice. Write your own logic to process the output of LLM as needed.

greeting_configsobject

Agent greeting broadcast configuration.

greeting_audio_urlstring

The URL of a pre-recorded audio file to play as the agent's greeting. When configured, the system downloads and plays this recording instead of synthesizing the greeting via TTS.

Supported formats: mp3, wav, pcm.

Only http:// or https:// URLs are supported. An empty string, non-URL string, or a URL exceeding 2048 bytes causes the request to return a 400 error.
For pcm files, set greeting_configs.audio_pcm_sample_rate to specify the sample rate.
The system resamples the imported audio to match tts.params.sample_rate before playback.

You must also configure greeting_message as a fallback when using greeting_audio_url; otherwise the request returns a 400 error. If the recording fails to download, decode, or is in an unsupported format, the system automatically falls back to TTS synthesis using greeting_message.

greeting_messagestring

Agent greeting text. This field serves the following purposes:

Standalone greeting: If provided without greeting_audio_url, the agent greets the first user who joins the channel using TTS synthesis.
Audio greeting fallback: When greeting_audio_url is configured, this field is required as a fallback. If the audio file fails to download, decode, or is in an unsupported format, the agent falls back to TTS synthesis using this text.
Interrupted greeting context: When greeting_configs.interruptable is true and the audio greeting is interrupted, the system uses this text to estimate the portion already played and injects it into short-term memory to maintain conversation context.

failure_messagestring

Prompt for agent activation failure. If provided, it is returned through TTS when the custom LLM call fails.

vendorstring

Allowed values:

openai, azure, xai, custom

LLM provider, supports the following settings:

openai: OpenAI
azure: Azure OpenAI
xai: xAI Grok
custom: Custom LLM. When you set this option, the agent includes the following fields, in addition to role and content when making requests to the custom LLM:
turn_id: A unique identifier for each conversation turn. It starts from 0 and increments with each turn. One user-agent interaction corresponds to one turn_id.
timestamp: The request timestamp, in milliseconds.

stylestring

Default:

"openai"

Allowed values:

openai, gemini, anthropic, dify

The request style for chat completion:

openai: For OpenAI and OpenAI-compatible APIs
gemini: For Google Gemini and Google Vertex API format
anthropic: For Anthropic Claude API format
dify: For Dify API format

For vendor-specific parameters, see the corresponding LLM provider page listed in llm.vendor.

template_variablesobject

Template parameter configuration used to insert variables into the agent's system_messages, greeting_message, failure_message, and parameters.silence_config.content text. Uses key-value pairs, where the key is the variable name and the value is the variable's value. Template variables, combined with prompt customization and SIP outbound calling functionality, enable dynamic content injection, automating processes such as automatic hang-up, voicemail recognition, automatic message leaving, and call transfer. To insert defined variables in the prompt text, use the syntax {{variable_name}}. The system automatically replaces each variable with the corresponding value defined in template_variables.

Note

Variable values cannot reference other variables. For example, if you define "farewell": "Looking forward to seeing you again, {{name}}", the {{name}} variable will not be resolved.

mcp_serversarray<object>

MCP (Model Context Protocol) server configuration. By configuring MCP servers, agents can call tools provided by external services to implement advanced functionality. See Use MCP tools.

toolsarray<object>

Items:

items <= 32

A list of custom tools. When configured, the LLM can select a tool based on the conversation and generate its parameters. The Conversational AI Engine then validates the parameters, calls the corresponding HTTPS endpoint, and returns the result to the LLM to continue generating a response.

Custom tools, MCP tools, built-in tools, and model-native tools share the same tool name namespace. Name comparisons aren't case-sensitive and use ASCII rules. After MCP discovery and filtering, a maximum of 32 tools total can be exposed to the LLM. If names conflict or the limit is exceeded, agent startup fails. The engine doesn't automatically overwrite, rename, or truncate tools.

Note

Set advanced_features.enable_tools to true before using tools.
Only supported for synchronous calls to standard text LLMs. Custom LLMs, MLLMs, Dify, asynchronous tools, and stored tool references aren't supported.

typestring

Required

Allowed values:

function

The tool type. Only function is supported.

functionobject

Required

namestring

Required

Match:

^[a-zA-Z][a-zA-Z0-9]{0,63}$

The tool name visible to the model. Must start with a letter and contain only letters and numbers, 1 to 64 characters long.

Custom tools, MCP tools, built-in tools, and model-native tools can't share a name. Name comparisons aren't case-sensitive and use ASCII rules. Names can't use the mcp prefix or the value ragsearch_tool.

descriptionstring

Required

Length:

1 <= length <= 1024

A description of the tool, used to help the LLM decide when to call it and what it returns. Don't include sensitive information such as secrets.

parametersobject

Required

The tool's parameters, defined as JSON Schema and passed to the LLM using OpenAI's non-strict function-calling mode. See OpenAI Function Calling.

Usage rules:

The root type must be object.
Common JSON Schema types are supported: object, array, string, integer, number, boolean, and null.
Standard JSON Schema keywords relevant to tool parameters are supported, such as type, description, enum, properties, required, additionalProperties, and items.
Objects can use properties, required, and additionalProperties. additionalProperties can be omitted; when omitted, extra properties are allowed per JSON Schema semantics. To disallow undeclared extra parameters, set it to false explicitly.
Array parameters should specify an element schema with items. Support for tuple schemas depends on the selected LLM.
Don't add optional parameters to required. Don't rely on a specific model's support for advanced keywords such as $ref, $defs, format, pattern, range constraints, or composition schemas. Confirm your LLM's compatibility before using them.
The serialized schema size is capped at 64 KiB.

Example:

{

  "type": "object",
  "properties": {
    "order_id": {
      "type": "string",
      "description": "Order ID"
    }
  },
  "required": ["order_id"],
  "additionalProperties": false
}

executionobject

The tool's execution mode.

serverobject

Required

The HTTP request configuration for the custom tool.

Templates use single-pass substitution; they aren't expanded recursively. The following variables are supported:

{{args.<name>}}: A validated LLM tool parameter. Can be used in the URL path or query parameters and in the request body, but not in headers.
{{template_variables.<name>}}: A template variable configured when creating the agent. Can be used in the URL path or query parameters, header values, and the request body.
{{tool_call_id}}: The ID correlating this tool call. Can be used in the URL, header values, or the request body.

Template variable rules:

Headers and request body: Each value must be either a fixed value or one complete placeholder, such as {{args.order_id}}. Concatenating a placeholder with other text, such as order-{{args.order_id}}, isn't supported, and nested fields, array indexes, and recursive references aren't supported either.
URL path and query parameters: Placeholders can be used within a fixed URL, such as /orders/{{args.order_id}}.
If a placeholder references a variable that doesn't exist or is null, the tool call returns UnresolvedTemplate.
Variable values in the URL and headers must be a string, number, or boolean.

Authentication information may only be passed as a complete header value. Don't put authentication information in the URL or request body. All header values are treated as sensitive and aren't passed to the LLM or written to regular logs, events, subtitles, or TTS output.

methodstring

Required

Allowed values:

GET, POST

The HTTP request method:

GET: For querying data. Doesn't support a request body; configuring body causes request validation to fail.
POST: For submitting data. Supports URL query parameters, request headers, and a request body together.

urlstring

Required

Match:

^https://

Length:

1 <= length <= 2048

The absolute HTTPS URL for the custom tool request.

The URL path and query parameters can embed single-level placeholders:

{{args.<name>}}
{{template_variables.<name>}}
{{tool_call_id}}

For example: https://api.example.com/orders/{{args.order_id}}?tenant={{template_variables.tenant_id}}.

Usage limits:

Placeholder names must be complete; nested fields, array indexes, and recursive expansion aren't supported.
Only the URL path and query parameters can use placeholders; the scheme, userinfo, host, and port can't be templated.
Usernames, passwords, or other authentication information can't be embedded in the URL.
Placeholder values in the URL must be a string, number, or boolean.
Each rendered path variable is capped at 1024 UTF-8 bytes, each query variable at 4096 UTF-8 bytes, and the final URL at 8192 UTF-8 bytes.

headersobject

Properties:

properties <= 32

Custom headers to send with the tool request, up to 32 entries.

Header values support:

A static string, number, or boolean.
A complete {{template_variables.<name>}} placeholder.
A complete {{tool_call_id}} placeholder.

Usage limits:

Header names must be 1 to 128 characters and must conform to the HTTP token specification.
Header names can't be templated, and header values don't support {{args.<name>}}.
A placeholder must be the entire header value; it can't be concatenated with other text, and nested fields, array indexes, or recursive expansion aren't supported.
Each rendered header value is capped at 4096 UTF-8 bytes, and all headers combined are capped at 16 KiB.
If a custom header name conflicts with a system-generated header, the system-generated value takes precedence.

All header values are treated as sensitive: they aren't passed to the LLM and aren't written to regular logs, events, subtitles, or TTS output.

bodyunknown

The HTTP request body for the custom tool. Only used with POST; configuring this field for GET causes request validation to fail.

The request body supports object, array, string, number, boolean, and null, in any level of JSON structure. Leaf values in the request body can be:

A JSON constant.
A complete {{args.<name>}} placeholder.
A complete {{template_variables.<name>}} placeholder.
A complete {{tool_call_id}} placeholder.

A placeholder must be the entire leaf value; it can't be concatenated with other text, and nested fields, array indexes, or recursive expansion aren't supported. After substitution, a placeholder keeps the variable's original JSON type; tool parameters are automatically escaped when serialized into the JSON request body.

timeout_msinteger

Default:

10000

Range:

1000 <= value <= 100000

The HTTP request timeout in milliseconds, in the range [1000, 100000]. Defaults to 10000.

headersobject

Custom headers to include in requests to the LLM. Use this field to pass business-specific information such as custom fields or tenant identifiers.

Note

These headers are merged with the headers generated by the Conversational AI Engine. If a key conflict occurs, the engine-generated header takes precedence. For example, authentication-related headers.
Header keys are merged using exact string matching and are case-sensitive. Agora recommends using standard capitalization to avoid ambiguity from duplicate keys with different casing.

mllmobject

Multimodal Large Language Model (MLLM) configuration for real-time audio and text processing. mllm is an exclusive alternative to the standard asr + llm + tts pipeline."

enableboolean

Enable Multimodal Large Language Model for voice-to-voice processing. Enabling MLLM automatically disables ASR, LLM, and TTS since the MLLM handles end-to-end voice processing directly. Replaces the deprecated advanced_features.enable_mllm.

urlstring

The MLLM WebSocket URL for real-time communication.

api_keystring

Required

The API key used for MLLM authentication.

messagesarray<object>

Array of conversation items used for short-term memory management. Uses the same structure as item.content from the OpenAI Realtime API.

paramsobject

Additional MLLM configuration parameters.

Modalities override: The modalities setting in params is overridden by input_modalities and output_modalities.
Turn detection override: The turn_detection setting in params is overridden by mllm.turn_detection.

For vendor-specific parameters, see the corresponding MLLM provider page listed in mllm.vendor.

input_modalitiesarray<string>

Default:

[  "audio"]

MLLM input modalities:

["audio"]: Audio only
["audio", "text"]: Audio plus text

output_modalitiesarray<string>

Default:

[  "text",  "audio"]

MLLM output modalities:

["text", "audio"]: Text plus audio

greeting_messagestring

Agent greeting message. If provided, the first user in the channel is automatically greeted with this message upon joining.

vendorstring

Allowed values:

openai, azure, gemini, vertexai, xai, openai_gpt_live

MLLM provider. Currently supports:

openai: OpenAI Realtime API
azure: Azure OpenAI Realtime API
gemini: Google Gemini Live
vertexai: Google Gemini Live (Vertex AI)
xai: xAI Grok
openai_gpt_live: OpenAI GPT-Live

mcp_serversarray<object>

MCP (Model Context Protocol) server configuration for the MLLM module. By configuring MCP servers, agents can call tools provided by external services to implement advanced functionality. See Use MCP tools.

turn_detectionobject

Turn detection configuration for the MLLM module.

Note

When mllm.turn_detection is defined, the top-level turn_detection object has no effect.

modestring

Allowed values:

agora_vad, server_vad, semantic_vad

agora_vad: Agora VAD-based detection.
server_vad: Vendor-side VAD-based detection. Supported by OpenAI Realtime API and Gemini Live.
semantic_vad: Semantic-based detection. Supported by OpenAI Realtime API only.

agora_vad_configobject

Configuration for Agora VAD-based turn detection. Applicable when mode is agora_vad.

server_vad_configobject

Configuration for vendor-side VAD-based turn detection. Applicable when mode is server_vad. Parameters are passed through to the vendor.

semantic_vad_configobject

Configuration for semantic-based turn detection. Applicable when mode is semantic_vad. Supported by OpenAI Realtime API only.

avatarobject

Avatar configuration.

turn_detectionobject

Conversation turn detection settings. Controls the logic for voice activity detection and conversation turn determination. The previous version of turn_detection is deprecated. Refer to Deprecated parameters for details. Agora recommends switching to the latest parameters.

Note

This object has no effect when mllm.enable is true. Use mllm.turn_detection instead.

Note

Starting with v2.6, turn_detection only handles Start of Speech (SoS) and End of Speech (EoS) detection. Interruption handling strategies, including keyword-based interruption and disabling interruption, have moved to the top-level interruption field.

This configuration supports multiple combinations of detection modes:

Start of Speech (SoS): Supports three modes: VAD, Keyword, and Disable.
End of Speech (EoS): Supports VAD and Semantic modes.

modestring

Default:

"default"

Conversation turn detection mode:

default: Uses standard conversation turn detection configuration.

typestring

Deprecated

Default:

"agora_vad"

Allowed values:

agora_vad, server_vad, semantic_vad

Deprecated turn detection mechanism. agora_vad uses Agora VAD, server_vad lets the model detect speech start and end based on audio volume, and semantic_vad uses semantic estimation with VAD. Compatible with both cascade (ASR/LLM/TTS) and MLLM modes.

interrupt_modestring

Deprecated

Default:

"interrupt"

Allowed values:

interrupt, append, ignore, keywords, adaptive

Deprecated interruption behavior when human voice interrupts the agent while it is speaking or thinking. Sets the agent's behavior when human voice interrupts the agent while it is interacting (speaking or thinking).

Note

Only the interrupt mode is supported when you integrate an mllm.

interrupt_duration_msnumber

Deprecated

Default:

160

The amount of time in milliseconds that the user's voice must exceed the VAD threshold before an interruption is triggered.

interrupt_keywordsarray<string>

Deprecated

Keywords that trigger an interruption when turn_detection.interrupt_mode is set to keyword. Specifies the list of keywords that trigger an interruption when the turn_detection.interrupt_mode is set to "keyword".

Note

Keyword recognition capabilities, such as support for multiple languages or dialects, depend on the ASR provider you choose.
You can configure up to 128 keywords.

prefix_padding_msinteger

Deprecated

Default:

800

The extra forward padding time in milliseconds before processing starts.

silence_duration_msinteger

Deprecated

Default:

640

The duration of audio silence in milliseconds after which the agent assumes the user has stopped speaking.

thresholdnumber

Deprecated

Default:

0.5

Range:

0 < value < 1

Voice activity detection sensitivity. The valid range is (0.0, 1.0). Identification sensitivity determines the level of sound in the audio signal that is considered voice activity.

configobject

Detailed configuration for conversation turn detection.

speech_thresholdnumber

Default:

0.5

Range:

0 < value < 1

Voice activity detection sensitivity. Determines the sound level in the audio signal that is considered voice activity. Lower values make it easier for the agent to detect speech, and higher values ignore weak sounds.

start_of_speechobject

Start of Speech (SoS) detection configuration. Determines when a user begins speaking.

modestring

Required

Allowed values:

vad, manual, keywords, disabled

Start of speech detection mode:

vad: Based on VAD (Voice Activity Detection). Uses audio signal detection.
manual: Manual SoS mode. Disables automatic start-of-speech detection. The client explicitly signals the start of user speech via RTM.
keywords: Deprecated. Use interruption.mode = "keywords" instead.
disabled: Deprecated. Use interruption.enable = false with interruption.disabled_config.strategy to configure the handling strategy.

Note

When mode is manual, you must also set advanced_features.enable_rtm = true and parameters.data_channel = "rtm". This mode does not support manual_config; no additional configuration is required.

{mode}_configobject

Start of speech detection configuration parameters. The structure and supported fields vary depending on the detection mode. Configuration examples:

vad_config

"vad_config": {
"interrupt_duration_ms": 160,
"speaking_interrupt_duration_ms": 160,
"prefix_padding_ms": 800
}

keywords_config

"keywords_config": {
"interrupt_duration_ms": 160,
"prefix_padding_ms": 800,
"triggered_keywords": ["Are you there", "hello"]
}

disabled_config

"disabled_config": {
"strategy": "append"
}

Note

The configuration type must match mode. For example, when mode is vad, you must provide vad_config.
You cannot provide multiple mode configurations simultaneously.

end_of_speechobject

End of Speech (EoS) detection configuration. Determines when a user ends their speech.

modestring

Allowed values:

vad, semantic, manual

End of speech detection mode. Possible values:

vad: Based on VAD (Voice Activity Detection). Detects silence duration.
semantic: Based on semantic triggering. Uses semantic understanding to determine when conversation ends.
manual: Manual EoS mode. Disables automatic end-of-speech detection. The client explicitly signals the end of user speech via RTM.

Note

When mode is semantic, EoS detection supports English and Chinese only. For unsupported languages, the engine falls back to VAD.
When mode is manual, you must also set advanced_features.enable_rtm = true and parameters.data_channel = "rtm". This mode does not support manual_config; no additional configuration is required.

{mode}_configobject

End of speech detection configuration parameters. The structure and supported fields vary depending on the detection mode. Configuration examples:

vad_config

"vad_config": {
"silence_duration_ms": 640
}

semantic_config

"semantic_config": {
"silence_duration_ms": 320,
"max_wait_ms": 3000,
"pause_state_enabled": true
}

Note

The configuration type must match mode. For example, when mode is vad, you must provide vad_config.
You cannot provide multiple mode configurations simultaneously.

interruptionobject

Interruption control configuration. Provides unified management of the agent's behavior when interrupted by the user.

Note

This object has no effect when mllm.enable is true. Use mllm.turn_detection instead.

salobject

Selective Attention Locking (SAL) configuration. (Beta)

sal_modestring

Default:

"locking"

Allowed values:

locking, recognition

Selective attention lock mode. Supports the following options:

locking: Speaker Lock Mode. The agent locks onto the speaker, blocking 95% of ambient human voices and noise. You can enable this mode in two ways:
Seamless mode: When a user speaks loudly and clearly at the beginning of a conversation, the intelligent agent automatically recognizes the user as the speaker.
Personalized mode: When creating an agent, a speaker's voiceprint URL is pre-registered through the sample_urls field. The agent then locates the speaker based on the pre-registered voiceprint.
recognition: Voiceprint recognition mode. You can pre-register only one voiceprint URL using the sample_urls field. The agent identifies different speakers and suppresses other background voices and environmental noise. The target speaker is identified through the vpids field in the metadata field and sent to the LLM. Set llm.vendor to "custom" and refer to Custom LLM for instructions on how to make the LLM process speaker information.

sample_urlsobject

The registered voiceprint URL as a key-value pair, where the key is the voiceprint name and the value is the download URL for the speaker's voiceprint. Only one voiceprint URL is supported. Example:

{
"speaker1": "https://example.com/speaker1.pcm"
}

Note

Do not set the incoming voiceprint name to "unknown"; this is a reserved keyword used to identify unknown speakers.
For a registered voiceprint, ensure that:
Size: The voiceprint file does not exceed 2 MB.
Duration: Contains 10 to 15 seconds of audio, with at least 8 seconds of effective audio excluding silent segments.
Format: 16kHz sampling rate, 16-bit depth, mono PCM audio file. The file name extension must be ".pcm".

labelsobject

Custom labels in key-value pair format, where the key is the label name and the value is the label value. Enables agents to carry custom business information.

These labels are bound to the agent and returned in the payload field of all message notification callbacks from the conversational AI engine. Use them to implement custom business logic, such as tagging activity IDs, customer groups, and business scenarios.

rtcobject

RTC media encryption configuration.

filler_wordsobject

Filler word configuration. Plays filler words while waiting for LLM responses to reduce user anxiety and improve conversation flow.

Filler word playback follows these rules:

Playback order: When multiple filler words or LLM responses are waiting to be played, they are played in the order they arrive.
Interruption control: Inherits the interruption mode setting from the interruption field.

enableboolean

Default:

false

Whether to enable filler words:

true: Enable filler words.
false: Disable filler words.

triggerobject

Filler word trigger configuration. Defines when to trigger filler word playback.

modestring

Filler word trigger mode:

fixed_time: Fixed time trigger. Triggers filler word playback when LLM response wait time exceeds the threshold.

{mode}_configobject

Filler word trigger configuration parameters. The parameter name and structure vary depending on the trigger mode. Configuration example:

"fixed_time_config": {
"response_wait_ms": 1500
}

Note

The configuration type must match mode. For example, when mode is fixed_time, you must provide fixed_time_config.
You cannot provide multiple mode configurations simultaneously.

contentobject

Filler word content configuration. Defines the source and selection rules for filler words.

modestring

Filler word content mode:

static (default): Static filler words. Uses a predefined list of filler words.
generated: Generated filler words. Agora-hosted generation produces one filler word phrase based on the current context. If the generated result isn't available in time, the static filler words in static_config are used as a fallback.

static_configobject

Static filler word configuration. Required whenever filler words are enabled. When mode is generated, this configuration is used as the static fallback for when generation isn't available.

Note

Static filler word configuration example:


"static_config": {

"phrases": [

"Please wait.",

"I'm looking into it."

],

"selection_rule": "shuffle"

}

generated_configobject

Generated filler word configuration. Only takes effect when mode is generated.

When mode is generated, Agora generates a context-relevant filler word phrase in parallel with the main LLM request.
If the generated result isn't available by response_wait_ms, the static filler words in static_config are used as a fallback.

parametersobject

Agent configuration parameters.

silence_configobject

Settings related to agent silence behavior.

Note

silence_config does not apply when you integrate a mllm.

farewell_configobject

Graceful hang-up settings for the agent.

data_channelstring

Default:

"datastream"

Agent data transmission channel:

rtm: Use RTM transmission. This configuration takes effect only when advanced_features.enable_rtm is true.
datastream: Use RTC data stream transport.

enable_metricsboolean

Default:

false

Whether to receive agent performance data:

true: Receive agent performance data.
false: Do not receive agent performance data.

This setting only takes effect when advanced_features.enable_rtm is true. See Listen to agent events to learn how to use client components to receive agent performance data.

enable_error_messageboolean

Default:

false

Whether to receive agent error events:

true: Receive agent error events.
false: Do not receive agent error events.

This setting only takes effect when advanced_features.enable_rtm is true. See Listen to agent events to learn how to use client components to receive agent error events.

audio_scenariostring

Default:

"default"

Allowed values:

default, chorus, aiserver

The audio scenario for the RTC channel.

default: Maps to aiserver.
chorus: Real-time chorus scenario, where users have good network conditions and require ultra-low latency.
aiserver: Optimized for interactions between the user and the conversational AI agent in terms of latency and network resilience.

opt_outboolean

Default:

false

Whether to disable data retention for the current session.

false: (Default) Data retention is enabled.
true: Data retention is disabled. When disabled, session interaction text and audio are not retained, which means troubleshooting, performance review, and agent optimization based on session history will not be available.

Note

To support service operation, agent optimization, and troubleshooting, session interaction text and audio are temporarily retained for the minimum necessary period by default. If your business has strict security or compliance requirements, enable this setting to prevent session data from being stored.

presetstring

Deprecated

Deprecated. Use credential_mode: "managed" within the asr, llm, or tts block instead. See Use managed mode.

A comma-separated string of one or more presets. Each preset provides a predefined configuration for ASR, LLM, and TTS. You can specify a preset for any or all of ASR, LLM, and TTS. When a preset is specified, you do not need to provide the endpoint URL, API key, or model for the preset providers. Use the asr, llm, and tts fields to configure additional settings.

Response

If the returned status code is 200, the request was successful. The response body contains the result of the request.
If the returned status code is not 200, the request failed. The response body includes the detail and reason for failure. Refer to status codes to understand the possible reasons for failure.
The following turn detection configuration is deprecated. To create more natural conversations and reduce unintended interruptions, Agora recommends using the latest version of turn_detection above.

Response Body

The request was successful. The response body contains the result of the request.

agent_idstring

Unique id of the agent instance

create_tsinteger

Timestamp of when the agent was created

statusstring

Allowed values:

IDLE, STARTING, RUNNING, STOPPING, STOPPED, FAILED

Current status.

IDLE (0): Agent is idle.
STARTING (1): The agent is being started.
RUNNING (2): The agent is running.
STOPPING (3): The agent is stopping.
STOPPED (4): The agent has exited.
FAILED (6): The agent failed to execute.

Authorization

This endpoint requires authentication.

tokenAuthbasicAuth

Request examples

curl --request post \--url https://api.agora.io/api/conversational-ai-agent/v2/projects/:appid/join \--header 'Authorization: Basic <your_base64_encoded_credentials>' \--data '{    "name": "unique_name",    "properties": {        "channel": "channel_name",        "token": "token",        "agent_rtc_uid": "1001",        "remote_rtc_uids": [            "1002"        ],        "idle_timeout": 120,        "llm": {            "url": "https://api.openai.com/v1/chat/completions",            "api_key": "<your_llm_key>",            "system_messages": [                {                    "role": "system",                    "content": "You are a helpful chatbot."                }            ],            "max_history": 32,            "greeting_message": "Hello, how can I assist you today?",            "failure_message": "Please hold on a second.",            "params": {                "model": "gpt-4o-mini"            }        },        "tts": {            "vendor": "microsoft",            "params": {                "key": "<your_tts_api_key>",                "region": "eastus",                "voice_name": "en-US-AndrewMultilingualNeural"            }        },        "asr": {            "language": "en-US"        }    }}'

Response example

{  "agent_id": "1NT29X10YHxxxxxWJOXLYHNYB",  "create_ts": 1737111452,  "status": "RUNNING"}

Start a conversational AI agent

Path Parameters

Request Body

Response

Response Body

Authorization

Request examples

Response example

Contact Us

Contact Us