# Reference overview (/en/api-reference)
# Recipes (/en/api-reference/recipes) # SDKs (/en/api-reference/sdks) # Contribute an ASR Extension (/en/ai/create_asr_extension) This page has moved to [Contribute an ASR extension](./reference/ten-agent/create-asr-extension). # Contribute a TTS Extension (/en/ai/create_tts_extension) This page has moved to [Contribute a TTS extension](./reference/ten-agent/create-tts-extension). # Voice Agent overview (/en/ai) A Voice Agent is a realtime system that listens to a user, reasons over the conversation, and responds with speech. On Agora, that experience is built from a shared foundation: realtime transport, an agent runtime, AI models, and the endpoint where the conversation is delivered. Agora supports two Voice Agent paths: **in apps** and **on dedicated devices**. Choose **In apps** if your agent lives in a web, mobile, desktop, or backend-assisted app experience. Choose **On dedicated devices** if your endpoint is embedded or hardware-first, such as a toy, wearable, kiosk, or companion. Use the quickstart that matches your endpoint to jump directly into the right implementation path. ## Capabilities [#capabilities] No matter where you ship it, a Voice Agent usually has the same core layers: * **Realtime transport**: moves audio, events, and conversation state between the user and the agent with low latency. * **Agent runtime**: manages the session lifecycle, turn taking, interruptions, memory, and tool orchestration. * **Models**: provide ASR, LLM, TTS, and optional multimodal capabilities. * **Endpoint**: presents the experience to the user, either through an app or a dedicated device. The first three layers stay conceptually similar across the docs. The main split in the sidebar happens at the **endpoint** layer, because app teams and device teams need different implementation workflows. ## How this section is organized [#how-this-section-is-organized] The AI sidebar is structured in two paths: * **In apps**: for web, mobile, desktop, and backend-assisted app experiences. * **On dedicated devices**: for embedded or hardware-first endpoints such as toys, wearables, kiosks, and companions. Both paths share the same Voice Agent idea, but they differ in what teams need to build next. App teams usually care first about session orchestration, UI, model behavior, and runtime events. Device teams usually care first about bring-up, networking, firmware, controls, and hardware integration. # Release notes (/en/ai/release-notes) This document tracks notable changes and improvements to the Conversational AI Engine. ## Releases [#releases] ### v2.9 [#v29] Released on July 1, 2026. #### New features [#new-features] Included in this release: * **Manual turn control** You can now explicitly control when a user's speech starts and ends by sending manual Start of Speech (SoS) and End of Speech (EoS) requests over Signaling (RTM), instead of relying on automatic detection. This supports scenarios with strict turn-boundary requirements, such as AI interviews, interactive quizzes, or push-to-talk. Set [`start_of_speech.mode`](/en/api-reference/api-ref/conversational-ai/join#properties-turn-detection-config-start-of-speech-mode) and [`end_of_speech.mode`](/en/api-reference/api-ref/conversational-ai/join#properties-turn-detection-config-end-of-speech-mode) to `manual` when starting the agent. For details, see [Manually control start and end of speech](/en/ai/best-practices/manual-turn-control). * **Agent state notifications over RTM Message** You can now track agent state changes in real time using Signaling (RTM) Message events. The agent pushes `state.listening`, `state.thinking`, and `state.speaking` events as individual RTM messages, making it easier to record state change timestamps, process events sequentially, or consume state events alongside other RTM Message events. For details, see [Get agent state](/en/ai/best-practices/get-agent-state). * **Pre-recorded greeting audio** You can now play a pre-recorded audio file as the agent's greeting instead of synthesizing it with TTS. Set [`properties.llm.greeting_audio_url`](/en/api-reference/api-ref/conversational-ai/join#properties-llm-greeting-audio-url) to the URL of an `mp3`, `wav`, or `pcm` file when starting the agent. If the audio fails to download, decode, or is in an unsupported format, the agent automatically falls back to TTS synthesis using `greeting_message`. * **Managed mode** You can now use supported ASR, LLM, and TTS providers with Agora-managed credentials, without supplying your own API keys. Set [`credential_mode`](/en/api-reference/api-ref/conversational-ai/join#properties-asr-credential-mode) to `"managed"` within the `asr`, `llm`, or `tts` block when starting an agent. Managed mode replaces the deprecated `preset` parameter. For details, see [Use managed mode](/en/ai/build/custom-model-integration/managed-mode). #### Improvements [#improvements] This release includes the following enhancements. * **Speech captured during an uninterruptible greeting** When [`properties.llm.greeting_configs.interruptable`](/en/api-reference/api-ref/conversational-ai/join#properties-llm-greeting-configs-interruptable) is set to `false`, the agent now silently collects the user's ASR results while the greeting plays. After the greeting ends, the system concatenates the collected speech segments into a single user message and sends it to the LLM for a unified response, instead of discarding speech that occurs during playback. #### API changes [#api-changes] This release introduces the following changes to the RESTful API. * Changes to [**Start a conversational AI agent**](/en/api-reference/api-ref/conversational-ai/join) * **New parameters** * [`properties.llm.greeting_audio_url`](/en/api-reference/api-ref/conversational-ai/join#properties-llm-greeting-audio-url): The URL of a pre-recorded audio file (`mp3`, `wav`, or `pcm`) to play as the agent's greeting instead of synthesizing it with TTS. * [`properties.llm.greeting_configs.audio_download_timeout_ms`](/en/api-reference/api-ref/conversational-ai/join#properties-llm-greeting-configs-audio-download-timeout-ms): The timeout in milliseconds for downloading the greeting audio file. Defaults to `1000`. * [`properties.llm.greeting_configs.audio_pcm_sample_rate`](/en/api-reference/api-ref/conversational-ai/join#properties-llm-greeting-configs-audio-pcm-sample-rate): The sample rate in Hz of the greeting audio file when the format is PCM. Defaults to `16000`. * [`properties.asr.credential_mode`](/en/api-reference/api-ref/conversational-ai/join#properties-asr-credential-mode), [`properties.llm.credential_mode`](/en/api-reference/api-ref/conversational-ai/join#properties-llm-credential-mode), and [`properties.tts.credential_mode`](/en/api-reference/api-ref/conversational-ai/join#properties-tts-credential-mode): Set to `"managed"` to use Agora-managed credentials for the provider without supplying your own API key. * **Updated parameters** * [`properties.turn_detection.config.start_of_speech.mode`](/en/api-reference/api-ref/conversational-ai/join#properties-turn-detection-config-start-of-speech-mode): Now accepts `manual`, which disables automatic start-of-speech detection so the client can explicitly signal the start of user speech over RTM. * [`properties.turn_detection.config.end_of_speech.mode`](/en/api-reference/api-ref/conversational-ai/join#properties-turn-detection-config-end-of-speech-mode): Now accepts `manual`, which disables automatic end-of-speech detection so the client can explicitly signal the end of user speech over RTM. * [`properties.llm.greeting_message`](/en/api-reference/api-ref/conversational-ai/join#properties-llm-greeting-message): In addition to serving as a standalone greeting, this field is now required as a fallback when `greeting_audio_url` is configured, and is used to maintain conversation context when an audio greeting is interrupted. * **Deprecated parameters** * [`preset`](/en/api-reference/api-ref/conversational-ai/join#preset): Deprecated in favor of setting `credential_mode` to `"managed"` within the `asr`, `llm`, or `tts` block. See [Use managed mode](/en/ai/build/custom-model-integration/managed-mode). * Changes to [**Notification event types**](/en/ai/reference/event-types) * **Updated events** * [`110 agent error`](/en/ai/reference/event-types#110-agent-error): Now reports greeting audio URL errors when the greeting audio file fails to download, times out, is in an unsupported format, or fails to decode. The agent automatically falls back to TTS synthesis using `greeting_message`. When `parameters.enable_error_message` is `true`, these errors are also delivered to the client via the `onMessageError` RTM callback. ### v2.8 [#v28] Released on June 11, 2026. #### New features [#new-features-1] Included in this release: * **Session data retention control** By default, session interaction text and audio are temporarily retained for the minimum necessary period to support service operation, agent optimization, and troubleshooting. To disable data retention for a session, set [`properties.parameters.opt_out`](/en/api-reference/api-ref/conversational-ai/join#properties-parameters-opt-out) to `true` when starting an agent. * **RTC token expiry warning** A new [`104 agent expire`](reference/event-types#104-agent-expire) event is triggered when the agent's RTC token is about to expire. Upon receiving this event, call the [Update agent](/en/api-reference/api-ref/conversational-ai/update) API with a new token to refresh it. The agent does not exit the channel or interrupt the session when this event is triggered. * **New ASR provider** * [xAI](models/asr/xai) * **New LLM provider** * [xAI Grok](models/llm/xai) * **New TTS provider** * [xAI](models/tts/xai) #### Improvements [#improvements-1] This release includes the following enhancements. Important These changes may affect your existing integration. Review each change and update your integration as needed. * **Idle timeout behavior change** Setting `properties.idle_timeout` to `0` no longer means the agent runs indefinitely. From this release, `0` only disables the channel idle timeout. Regardless of the `idle_timeout` value, the maximum running time for a single session is 72 hours, after which the agent automatically exits. If your integration previously relied on `idle_timeout = 0` to keep an agent running until manually stopped, update your task scheduling and lifecycle management logic to account for the 72-hour limit. #### API changes [#api-changes-1] This release introduces the following changes to the RESTful API. * Changes to [**Start a conversational AI agent**](/en/api-reference/api-ref/conversational-ai/join) * **New parameters** * [`properties.parameters.opt_out`](/en/api-reference/api-ref/conversational-ai/join#properties-parameters-opt-out): Set to `true` to disable data retention for the current session. * **Updated parameters** * [`properties.idle_timeout`](/en/api-reference/api-ref/conversational-ai/join#properties-idle-timeout): Valid range is now `0` to `259200` seconds (72 hours). Regardless of the value set, the maximum running time for a single session is 72 hours. * Changes to [**Retrieve agent history**](/en/api-reference/api-ref/conversational-ai/history) * **New response fields** * [`contents[].speech_start_ms`](/en/api-reference/api-ref/conversational-ai/history#contents-speech-start-ms): Unix timestamp in milliseconds indicating when the user started speaking or the agent started TTS playback. Only returned when `llm.vendor` is `custom`. * [`contents[].speech_end_ms`](/en/api-reference/api-ref/conversational-ai/history#contents-speech-end-ms): Unix timestamp in milliseconds indicating when the user stopped speaking, or when TTS playback completed or was interrupted. Only returned when `llm.vendor` is `custom`. * [`contents[].speech_algorithmic_delay`](/en/api-reference/api-ref/conversational-ai/history#contents-speech-algorithmic-delay): The total delay in milliseconds introduced by audio processing algorithms after audio is captured from the user's microphone. Only returned when `llm.vendor` is `custom`, `contents[].role` is `user`, and actual voice input is present. * Changes to [**Notification event types**](reference/event-types) * **New events** * [`104 agent expire`](reference/event-types#104-agent-expire): Triggered when the agent's RTC token is about to expire. Call the Update agent API with a new token upon receiving this event. * **Updated events** * [`102 agent left`](reference/event-types#102-agent-left): Added `Task lifetime limit exceeded` as a new exit reason when the 72-hour session limit is reached. * [`103 agent history`](reference/event-types#103-agent-history): Added `contents[].speech_start_ms`, `contents[].speech_end_ms`, and `contents[].speech_algorithmic_delay`. Only returned when `llm.vendor` is `custom`. * Changes to [**Retrieve a list of agents**](/en/api-reference/api-ref/conversational-ai/list) * **Updated parameters** * `state`: Now accepts a comma-separated list of values to filter by multiple agent states in a single request. For example, `state=0,1,2`. ### v2.7 [#v27] Released on May 20, 2026. #### New features [#new-features-2] Included in this release: * **New MLLM provider** * [xAI Grok](models/mllm/xai) * **New Avatar providers** * [Generic avatar (Beta)](models/avatar/generic) * **Greeting interruption control** The `properties.llm.greeting_configs` object in the [Start a conversational AI agent](/en/api-reference/api-ref/conversational-ai/join) API now supports an `interruptable` parameter, which controls whether user speech can interrupt a greeting during playback. #### Improvements [#improvements-2] This release includes the following enhancements: Important These changes may affect your existing integration. Review each change and update your integration as needed. * **Response status codes updated** This release adjusts the HTTP response status codes and `reason` values returned by the RESTful API. New status codes `401`, `429`, and `500` are added. New `reason` values `ServiceNotEnabled`, `AccountSuspended`, and `ResourceAllocationFailed` are introduced. The existing `InvalidRequest` reason is replaced by `InvalidRequestBody`, `MissingRequiredField`, and `InvalidFieldValue`. For details, see [Status codes and error messages](/en/api-reference/api-ref/conversational-ai/status-codes). If your integration relies on `reason` values for error handling, retry logic, or alerting, update your logic to handle the new values accordingly. * **Query conversation turns API now returns paginated results** The [Query conversation turn information](/en/api-reference/api-ref/conversational-ai/turns) API no longer returns all turns in a single response. Results are now paginated, defaulting to page 1 with up to 50 turns per page. This affects sessions with more than 50 conversation turns. Update your integration to handle the new `total_turn_count` and `pagination` fields in the response, and retrieve subsequent pages as needed. This release also adds the [112 turns finished](reference/event-types#112-turns-finished) notification event, which delivers conversation turn data in batches after a session ends as an alternative way to retrieve turn data. * **`on_listening_action` default behavior change** The default value of `on_listening_action` in the [Send a custom instruction](/en/api-reference/api-ref/conversational-ai/think) API has changed from `inject` to `interrupt`. When set to `interrupt`, the API immediately interrupts the current flow and initiates a new round of dialogue. If you rely on the previous default behavior, explicitly set `on_listening_action` to `inject` in your requests. #### API changes [#api-changes-2] This release introduces the following changes to the RESTful API. * Changes to [**Start a conversational AI agent**](/en/api-reference/api-ref/conversational-ai/join) * **New parameters** * [`properties.llm.greeting_configs.interruptable`](/en/api-reference/api-ref/conversational-ai/join#properties-llm-greeting-configs-interruptable): Controls whether user speech can interrupt a greeting during playback. * Changes to [**Query conversation turn information**](/en/api-reference/api-ref/conversational-ai/turns) * **New query parameters** Adds `page_index` and `page_size` fields to [query parameters](/en/api-reference/api-ref/conversational-ai/turns#query-parameters). * **New response parameters** The [response](/en/api-reference/api-ref/conversational-ai/turns#response) now includes additional top-level fields: * `agent_id`: The unique identifier of the agent. * `name`: The name of the agent. * `channel`: The name of the RTC channel the agent joined. * `total_turn_count`: The total number of dialogue turns in the session. * `pagination`: Pagination details for the response, including `page_index`, `total_pages`, and `is_last_page`. * Changes to [**Send a custom instruction**](/en/api-reference/api-ref/conversational-ai/think) * **Behavior changes** * [`on_listening_action`](/en/api-reference/api-ref/conversational-ai/think#on-listening-action): The default value has changed from `inject` to `interrupt`. When set to `interrupt`, the API immediately interrupts the current flow and initiates a new round of dialogue. * Changes to [**Notification event types**](reference/event-types) * **New events** * `112 turns finished`: Triggered after a session ends, delivering conversation turn data in batches for post-session analysis and processing. ### v2.6 [#v26] Released on April 22, 2026. #### New features [#new-features-3] Included in this release: * **New TTS provider** * [Deepgram](models/tts/deepgram) * **New endpoint: Send a custom instruction** Adds a new endpoint to inject a custom text instruction into the agent's current conversation pipeline. The instruction is processed as user input, enabling scenarios such as implicit instruction injection, client-side event triggering, and voice and text collaboration. For details, see [Send a custom instruction](/en/api-reference/api-ref/conversational-ai/think). * **MLLM turn detection refactoring** Introduces a new [`mllm.turn_detection`](/en/api-reference/api-ref/conversational-ai/join#properties-mllm-turn-detection) object for configuring turn detection when using MLLM. When defined, this overrides the top-level `turn_detection` object. Supported modes vary by vendor: * OpenAI Realtime API: `agora_vad`, `server_vad`, `semantic_vad` * Google Gemini Live: `agora_vad`, `server_vad` * **Interruption control** Adds a new top-level [`interruption`](/en/api-reference/api-ref/conversational-ai/join#properties-interruption) object for unified management of agent interruption behavior. Supports `start_of_speech` and `keywords` trigger modes, and configurable handling strategies when interruption is disabled. #### Improvements [#improvements-3] This release includes the following enhancements: * **New agent state callbacks** Adds three callbacks to the [Android](/en/api-reference/api-ref/conversational-ai/client-toolkit/android#onagentlisteningchanged), [iOS](/en/api-reference/api-ref/conversational-ai/client-toolkit/ios#onagentlisteningchanged), and [web](/en/api-reference/api-ref/conversational-ai/client-toolkit/web#econversationalaiapievents) Toolkit event handler interface: * `onAgentListeningChanged`: Listen for changes in the agent's listening state to monitor when the agent starts or stops listening to user input. * `onAgentThinkingChanged`: Listen for changes in the agent's thinking state to monitor when the agent starts or stops processing a request. * `onAgentSpeakingChanged`: Listen for changes in the agent's speaking state to monitor when the agent starts or stops playing back speech. * **Added `name` field to all [NCS event](reference/event-types) payloads** Returns the agent name provided when starting the agent. #### API changes [#api-changes-3] This release introduces the following changes to the RESTful API. * **New endpoint** * [Send a custom instruction](/en/api-reference/api-ref/conversational-ai/think): `POST /v2/projects/{appid}/agents/{agentId}/think` * Changes to [**Start a conversational AI agent**](/en/api-reference/api-ref/conversational-ai/join) * **New parameters** * [`interruption`](/en/api-reference/api-ref/conversational-ai/join#properties-interruption): Unified interruption control configuration. Supports `start_of_speech` and `keywords` trigger modes, and configurable handling strategies when interruption is disabled. * [`greeting_configs.delay_ms`](/en/api-reference/api-ref/conversational-ai/join#properties-llm-greeting-configs-delay-ms): Specifies the delay in milliseconds before the agent plays the greeting message after the user joins the channel. * [`llm.headers`](/en/api-reference/api-ref/conversational-ai/join#properties-llm-headers): Lets you pass custom headers in LLM requests, such as business-specific fields or tenant identifiers. * [`mllm.enable`](/en/api-reference/api-ref/conversational-ai/join#properties-mllm-enable): Enables the MLLM module. Replaces the deprecated `advanced_features.enable_mllm`. * [`mllm.turn_detection`](/en/api-reference/api-ref/conversational-ai/join#properties-mllm-turn-detection): Turn detection configuration for the MLLM module. When defined, overrides the top-level `turn_detection` object. * [`mllm.turn_detection.server_vad_config.idle_timeout_ms`](/en/api-reference/api-ref/conversational-ai/join#properties-mllm-turn-detection-server-vad-config-idle-timeout-ms): Idle timeout in milliseconds for server VAD mode. Applicable to OpenAI Realtime API only. * [`mllm.vendor`](/en/api-reference/api-ref/conversational-ai/join#properties-mllm-vendor) value `gemini`: Enables Google Gemini Live integration using the Gemini Developer API. * **Behavior changes** * `turn_detection` now handles SoS and EoS detection only. Interruption handling strategies have moved to the new top-level `interruption` field. * `turn_detection.config.start_of_speech.mode` values `keywords` and `disabled` are deprecated. Use the new `interruption` field instead. * When `turn_detection.end_of_speech.mode` is set to `semantic`, EOS detection supports English and Chinese only. For unsupported languages, the engine automatically falls back to VAD. * **Deprecated fields** * `advanced_features.enable_mllm`: Use `mllm.enable` instead. * `turn_detection.config.start_of_speech.mode` value `keywords`: Use `interruption.mode = "keywords"` instead. * `turn_detection.config.start_of_speech.mode` value `disabled`: Use `interruption.enable = false` with `interruption.disabled_config.strategy` instead. * **Removed fields** * `mllm.style` * `mllm.create_response` and `mllm.interrupt_response` for OpenAI Realtime API. * Changes to [**Notification event types**](reference/event-types) * **New fields** * Added `name` to all [NCS event payloads](reference/event-types). Returns the agent name provided when starting the agent. ### v2.5 [#v25] Released on March 31, 2026. #### New features [#new-features-4] Included in this release: * **Preset model configurations** Adds a [`preset`](/en/api-reference/api-ref/conversational-ai/join#preset) parameter to the Join API. Pass a comma-separated list of preset names to apply predefined configurations for ASR, LLM, and TTS providers. When you use a preset, you do not need to provide the endpoint URL, API key, or model for the preset provider. Use the `asr`, `llm`, and `tts` fields to configure additional settings. When you use a preset, the corresponding ASR, LLM, or TTS service is provided through Agora-managed accounts. For details, see the [pricing](reference/pricing) page. Available presets: * ASR: `deepgram_nova_2`, `deepgram_nova_3` * LLM: `openai_gpt_4o_mini`, `openai_gpt_4_1_mini`, `openai_gpt_5_nano`, `openai_gpt_5_mini` * TTS: `minimax_speech_2_6_turbo`, `minimax_speech_2_8_turbo`, `openai_tts_1` * **Pause state detection** When enabled, the agent uses semantic understanding to detect when a user intends to pause the conversation. For example, if the user says "hold on" or "just a moment", the agent waits for further input instead of forwarding the utterance to the LLM. Controlled by the new [`pause_state_enabled`](/en/api-reference/api-ref/conversational-ai/join#properties-turn-detection-config-end-of-speech--mode-config-pause-state-enabled) parameter. * **Conversation turn insights** You can now query per-turn start, end, and latency metrics for completed agent sessions. See [Query conversation turn information](/en/api-reference/api-ref/conversational-ai/turns). * **Real-time TTS parameter updates with custom LLM** In scenarios with a custom LLM integrated, this release adds support for real-time TTS parameter updates during conversations, enabling a more natural conversational experience. This feature applies to the following use cases: * The custom LLM detects a user request for the agent to change its voice or speaking style. * The custom LLM identifies a user's positive emotion and adjusts TTS volume, pitch, speech rate, and other parameters in real time to better match the user's mood. For implementation details, refer to [Real-time TTS parameter updates](build/custom-model-integration/custom-llm#update-tts-parameters-in-real-time). * **New TTS provider** * [Murf (Beta)](models/tts/murf) * **New Avatar provider** * [Anam (Beta)](models/avatar/anam) #### Improvements [#improvements-4] This release includes the following enhancements: * **Migration from HeyGen API to LiveAvatar (Beta) API** Updates the HeyGen avatar model configuration to use the [LiveAvatar (Beta)](models/avatar/heygen) API. #### API changes [#api-changes-4] This release introduces the following changes to the RESTful API. * **New endpoint: Query conversation turns** Adds a new GET endpoint to query conversation turn information for an agent session. After a conversation ends, use this endpoint to retrieve the start event, end event, and latency metrics for each turn within the session. For details, see [Query conversation turn information](/en/api-reference/api-ref/conversational-ai/turns). * Changes to [**Start a conversational AI agent**](/en/api-reference/api-ref/conversational-ai/join) * **New parameters added** * [`preset`](/en/api-reference/api-ref/conversational-ai/join#preset) * [`pause_state_enabled`](/en/api-reference/api-ref/conversational-ai/join#properties-turn-detection-config-end-of-speech--mode-config-pause-state-enabled) * **Deprecated fields deleted** The following deprecated fields are deleted: * `properties.silence_timeout` * `properties.advanced_features.enable_aivad` * `properties.llm.silence_message` ### v2.4 [#v24] Released on February 2, 2026. #### New features [#new-features-5] Included in this release: * **MCP integration** Adds a [`llm.mcp_servers`](/en/api-reference/api-ref/conversational-ai/join#properties-llm-mcp-servers) field to the [Start a conversational AI agent](/en/api-reference/api-ref/conversational-ai/join) API to connect the LLM to an MCP (Model Context Protocol) server. Set [`advanced_features.enable_tools`](/en/api-reference/api-ref/conversational-ai/join#properties-advanced-features-enable-tools) to `true` to enable tool calls. This allows the agent to call tools provided by external services to extend functionality. * **Filler phrases** Adds a [`properties.filler_words`](/en/api-reference/api-ref/conversational-ai/join#properties-filler-words) field to insert pre-set or LLM-generated filler phrases during conversations to fill periods of silence while waiting for LLM output. This feature smooths dialogue flow, reduces user anxiety, and makes agent speech sound more natural. #### Improvements [#improvements-5] This release includes the following enhancements: * **Turn detection configuration optimization** The `turn_detection` parameter structure has been updated to provide more flexible conversation turn detection. The new structure uses a `mode` + `config` pattern with separate Start of Speech (SoS) and End of Speech (EoS) detection configurations. **Basic configuration** * `mode`: Conversation turn detection mode (currently supports `default`) * `config.speech_threshold`: Voice activity detection sensitivity **Start of Speech (SoS) detection** Detects when the user begins speaking. When SoS is detected while the agent is speaking, an interruption occurs. The [`config.start_of_speech`](/en/api-reference/api-ref/conversational-ai/join#properties-turn-detection-config-start-of-speech) parameter supports three detection modes: * **VAD mode** (`vad`): Triggered by voice activity detection. Supports configuration of interruption threshold, prefix padding, and ignore word list. * **Keyword mode (Beta)** (`keywords`): Based on keyword triggering. The agent begins conversation after detecting a specified keyword. * **Disabled mode** (`disabled`): Disables interruption. Supports append or ignore strategies. **End of Speech (EoS) detection** Detects when the user finishes speaking. When EoS is detected, the agent generates a response. The [`config.end_of_speech`](/en/api-reference/api-ref/conversational-ai/join#properties-turn-detection-config-end-of-speech) parameter supports two detection modes: * **VAD mode** (`vad`): Determines conversation end based on silence duration. * **Semantic mode** (`semantic`): Determines conversation end based on semantic understanding. Supports configurable maximum wait time. #### API changes [#api-changes-5] This release introduces the following changes to the RESTful API. * Changes to [**Start a conversational AI agent**](/en/api-reference/api-ref/conversational-ai/join) * **[`turn_detection`](/en/api-reference/api-ref/conversational-ai/join#properties-turn-detection) has a revamped structure** All previous fields under `turn_detection` have been deprecated and replaced with a new data structure. * Deprecated * `turn_detection.interrupt_mode` * `turn_detection.interrupt_keywords` * `turn_detection.interrupt_duration_ms` * `turn_detection.prefix_padding_ms` * `turn_detection.silence_duration_ms` * `turn_detection.threshold` * **AIVAD activation** The `advanced_features.enable_aivad` field is now deprecated; to configure AIVAD activation use [`turn_detection.config.end_of_speech.mode = "semantic"`](/en/api-reference/api-ref/conversational-ai/join#properties-turn-detection-config-end-of-speech-mode). * **New parameters added** * [`advanced_features.enable_tools`](/en/api-reference/api-ref/conversational-ai/join#properties-advanced-features-enable-tools) * [`llm.mcp_servers`](/en/api-reference/api-ref/conversational-ai/join#properties-llm-mcp-servers) * [`properties.filler_words`](/en/api-reference/api-ref/conversational-ai/join#properties-filler-words) * [`turn_detection.mode`](/en/api-reference/api-ref/conversational-ai/join#properties-turn-detection-mode) * [`turn_detection.config`](/en/api-reference/api-ref/conversational-ai/join#properties-turn-detection-config) ### v2.3 [#v23] Released on January 7, 2026. #### New features [#new-features-6] Included in this release: * **Control LLM response interruption in custom LLM scenarios** Adds a `metadata.interruptable` field in the first chunk of the `chat.completion.custom_metadata` object to control whether user speech can interrupt the agent's TTS output when using custom LLMs with streaming response (SSE). Use this to prevent interruptions when delivering critical information such as regulations, policies, or pricing. For more information, see [Configure LLM response interruption](build/custom-model-integration/custom-llm#configure-llm-response-interruption). * **RTC media content encryption** Adds an `rtc` parameter to the [Start a conversational AI agent](/en/api-reference/api-ref/conversational-ai/join) API for configuring RTC encryption. * **New ASR provider** * [Sarvam (Beta)](models/asr/sarvam) #### Improvements [#improvements-6] This release includes the following enhancements: * **ElevenLabs TTS compatibility optimization** Supports SSML parsing and forwarding to ensure ElevenLabs TTS correctly renders intended speech effects such as pauses, emphasis, and pronunciation. * **Latency optimization best practices** Adds a guide to [Optimize conversation latency](best-practices/optimize-latency). * **Cloud Recording best practices** Adds a guide to [Record conversations with Cloud Recording](best-practices/record-agent-conversation). #### API changes [#api-changes-6] This release introduces the following changes to the RESTful API. * **Stop agent API is now asynchronous** The [Stop a conversational AI agent](/en/api-reference/api-ref/conversational-ai/leave) API now responds immediately after request parameters are validated. The request is processed asynchronously after the API returns. * Changes to [**Start a conversational AI agent**](/en/api-reference/api-ref/conversational-ai/join) * **New parameters added** * `properties.rtc` ### v2.2 [#v22] Released on December 15, 2025. #### New features [#new-features-7] Included in this release: * **Geofencing** Use `geofence` configuration when you [Start a conversational AI agent](/en/api-reference/api-ref/conversational-ai/join) to limit which Agora servers the Conversational AI Engine can access based on geographic regions. See [Restrict agent zones](best-practices/regional-restrictions) for details. * **Agent greeting mode** Adds a field `llm.greeting_configs.mode` to the [Start a conversational AI agent](/en/api-reference/api-ref/conversational-ai/join) request to set the agent's greeting broadcast mode. The following modes are supported: * `single_every` (Default): The agent broadcasts a greeting every time a user joins a channel where there are no other users. * `single_first`: The agent broadcasts a greeting only when the first user joins a channel. * **New TTS providers** * [Sarvam (Beta)](models/tts/sarvam) #### Improvements [#improvements-7] This release includes the following enhancements: * **TTS parameter update** Adds `base_url` field to the TTS parameters for [OpenAI](models/tts/openai) and [ElevenLabs](models/tts/elevenlabs). This field specifies the endpoint URL for the TTS service. #### API changes [#api-changes-7] This release introduces the following changes to the RESTful API. * Changes to [**Start a conversational AI agent**](/en/api-reference/api-ref/conversational-ai/join) * **New parameters added** * `properties.geofence` * `llm.greeting_configs.mode` ### v2.1 [#v21] Released on December 5, 2025. #### New features [#new-features-8] Included in this release: * **Template variables** This version adds the `llm.template_variables` field to the [Start a conversational AI agent](/en/api-reference/api-ref/conversational-ai/join) API, used to insert variables into the agent's `system_messages`, `greeting_message`, `failure_message`, and `parameters.silence_config.content` text. By configuring these variables, the Conversational AI engine automatically replaces them with the corresponding values defined in `llm.template_variables`. Template variables, combined with prompt customization and SIP outbound calling functionality, enable you to dynamically inject content to automate processes such as automatic hang-up, voicemail recognition, automatic message leaving, and call transfer. * **Custom labels** This version adds a `labels` field to the [Start a conversational AI agent](/en/api-reference/api-ref/conversational-ai/join) API, enabling agents to carry custom labels. These labels are bound to the agent and returned in the `payload` field of all message notification callbacks from the Conversational AI engine, allowing you to implement custom business logic, such as tagging activity IDs, customer groups, and business scenarios. In SIP outbound call scenarios, you can pass a custom label in the `properties.labels` field when calling the outbound call interface to mark the call. #### API changes [#api-changes-8] This release introduces the following changes to the RESTful API. * Changes to [**Start a conversational AI agent**](/en/api-reference/api-ref/conversational-ai/join) * **New parameters added** * `properties.labels` * `llm.template_variables` ### v2.0 [#v20] Released on November 15, 2025. #### New features [#new-features-9] Included in this release: * **Telephony (Beta)** This version adds an outbound calling feature that enables the conversational AI agent to initiate an outbound call to a specified number through a POST API. After the call is answered, the agent can engage in real-time dialogue with the callee. This feature supports a wide range of outbound AI call scenarios. A set of phone number management APIs is also added for handling numbers connected to Conversational AI Engine. Refer to the API changes section for details. Telephony pricing The telephony feature is currently in **Beta** and is provided free of charge. Pricing terms may change upon official release. * **Selective attention locking (Beta)** This version adds the selective attention lock feature. Register voiceprints to enable the agent to identify specific speakers and suppress background voices and environmental noise, ensuring clearer, more focused conversations. * **Graceful exit** This version adds a new `farewell_config` field to [Start a conversational AI agent](/en/api-reference/api-ref/conversational-ai/join) API to configure the graceful exit feature. When enabled, calling the [Stop a Conversational Agent](/en/api-reference/api-ref/conversational-ai/leave) API causes the agent to enter an `IDLE` state before leaving the channel. * **Keyword interruption mode** This release adds a new option to the `turn_detection.interrupt_mode` field in the [Start a conversational AI agent](/en/api-reference/api-ref/conversational-ai/join) API. Set this field to `"keyword"` to enable keyword interruption mode. When this mode is enabled, the agent stops its current behavior after detecting any of the keywords specified in the `turn_detection.interrupt_keywords` field. * **Adaptive interruption mode** This release adds a new option to the `turn_detection.interrupt_mode` field in the [Start a conversational AI agent](/en/api-reference/api-ref/conversational-ai/join) API. Set this field to `"adaptive"` to enable adaptive interruption mode. When this mode is enabled, the agent dynamically increases the voice continuity threshold while speaking to reduce accidental interruptions. * **New ASR, LLM, MLLM, and TTS providers** * ASR * [OpenAI (Beta)](models/asr/openai) * [Speechmatics](models/asr/speechmatics) * [Google (Beta)](models/asr/google) * Amazon Transcribe (Beta) * [AssemblyAI (Beta)](models/asr/assembly-ai) * LLM * [Groq](models/llm/groq) * [Amazon Bedrock](models/llm/amazon) * MLLM * [Google Gemini Live](models/mllm/gemini) * TTS * [Rime (Beta)](models/tts/rime) * [Fish Audio (Beta)](models/tts/fish-audio) * [Google (Beta)](models/tts/google) * [Amazon Polly (Beta)](models/tts/amazon) * **New webhook notification events** This release adds three new webhook notification event types to support metrics reporting and call-state monitoring: * `111`: [agent metrics](reference/event-types#111-agent-metrics) Notifies real-time performance metrics for each dialogue turn, including ASR, LLM, and TTS latency measurements. * `201`: [inbound call state](reference/event-types#201-inbound-call-state) Reports state changes for incoming calls, such as when a call starts, is answered, transferred, or hung up. * `202`: [outbound call state](reference/event-types#202-outbound-call-state) Reports state changes for outbound calls initiated by the agent, including call start, dialing, ringing, answer, and hang-up events. #### Improvements [#improvements-8] This release includes the following enhancements: * **Support for avatars with MLLMs** Added support for using avatars with MLLMs. #### API changes [#api-changes-9] This release introduces the following changes to the RESTful API. * Changes to [**Start a conversational AI agent**](/en/api-reference/api-ref/conversational-ai/join) * **New parameters added** * `properties.parameters.farewell_config` * `properties.advanced_features.enable_sal` * `properties.sal` * `properties.sal.sal_mode` * `properties.sal.sample_urls` * `properties.turn_detection.interrupt_mode` (supports `adaptive` and `keyword` values) * `properties.turn_detection.interrupt_keywords` * `properties.turn_detection.interrupt_duration_ms` (migrated from `vad.interrupt_duration_ms`) * `properties.turn_detection.prefix_padding_ms` (migrated from `vad.prefix_padding_ms`) * `properties.turn_detection.silence_duration_ms` (migrated from `vad.silence_duration_ms`) * `properties.turn_detection.threshold` (migrated from `vad.threshold`) * **Deprecated** * The `vad` interface is deprecated. All configuration items have been moved to the `turn_detection` field. * **New APIs** * Telephony (Beta) * Initiate an outbound call * Retrieve call records * Hang-up a call * Retrieve call status * Phone number management (Beta) * Retrieve list of numbers * Import number * Retrieve number information * Update number configuration * Delete number #### Toolkit API [#toolkit-api] This release renames all APIs and parameters containing the word `transcription` in the client-side subtitle API to use `transcript`, as shown below: #### Android [#android] * `onTranscriptionUpdated` renamed to `onTranscriptUpdated` * `TranscriptionRenderMode` renamed to `TranscriptRenderMode` * `TranscriptionType` renamed to `TranscriptType` * `TranscriptionStatus` renamed to `TranscriptStatus` * `Transcription` renamed to `Transcript` #### iOS [#ios] * `onTranscriptionUpdated` renamed to `onTranscriptUpdated` * `TranscriptionRenderMode` renamed to `TranscriptRenderMode` * `TranscriptionType` renamed to `TranscriptType` * `TranscriptionStatus` renamed to `TranscriptStatus` * `Transcription` renamed to `Transcript` #### Web [#web] * `TRANSCRIPTION_UPDATED` renamed to `TRANSCRIPT_UPDATED` ### v1.7 [#v17] Released on July 31, 2025. #### New features [#new-features-10] * **AI avatars** Create visual avatar representations for your conversational agents using third-party avatar providers. AI avatars provide a visual presence during voice interactions, making conversations feel more natural and engaging. Enable AI avatars by setting `avatar.enable` to `true` and configuring the `avatar.vendor` and `avatar.params` fields when calling [Start a conversational AI agent](/en/api-reference/api-ref/conversational-ai/join) to create your agent. Info AI avatars require video streaming and incur additional charges. See [video calling pricing](/en/realtime-media/video/reference/pricing) for details. * **Selective attention locking (Beta)** This version introduces the selective attention locking feature, which uses voiceprint recognition technology to identify and filter out the speaker while suppressing background noise. This enhances the efficiency of conversational AI, particularly improving speech recognition accuracy. To experience this feature, contact [technical support](mailto\:support@agora.io). * **Send picture messages (Beta)** The toolkit now includes an API for [sending picture messages](build/send-multimodal-messages). You can send image URLs to the main model, which automatically references the image in future interactions to generate more relevant responses. A new callback is available to receive image message receipt details after successful transmission. Info * The picture messaging feature is currently in Beta and free for a limited time. * Image processing depends on the capabilities of the integrated LLM. Ensure the LLM you connect to the Conversational AI Engine supports image input. #### API changes [#api-changes-10] This release introduces the following modifications to the RESTful API. * [Start a conversational AI agent](/en/api-reference/api-ref/conversational-ai/join) * **New parameters added** * `avatar.enable` * `avatar.vendor` * `avatar.params` ##### Toolkit API [#toolkit-api-1] * Android: * `chat` * `ImageMessage` * `onMessageReceiptUpdated` * `MessageReceipt` * iOS: * `chat` * `ChatMessage` * `ChatMessageType` * `ImageMessage` * `onMessageReceiptUpdated` * `MessageReceipt` * Web: * `chat` * `TMessageReceipt` * `EChatMessagePriority` * `EChatMessageType` * `IChatMessageBase` * `IChatMessageImage` ### v1.6 [#v16] Released on July 15, 2025. #### New features [#new-features-11] * **Support for OpenAI realtime API** Integrate Multimodal Large Language Models (MLLMs) with Conversational AI Engine to enable end-to-end real-time audio and text interactions. See [OpenAI Realtime API](models/mllm/openai) for integration details. * **Support for more TTS vendors** Conversational AI Engine now supports the following additional TTS vendors: * [Cartesia](models/tts/cartesia) * [OpenAI](models/tts/openai) * **Custom ASR provider support** To improve flexibility in configuring conversational agents, this release allows you to select a custom automatic speech recognition (ASR) provider. The [Start a conversational AI agent](/en/api-reference/api-ref/conversational-ai/join) API now includes the following new parameters: * `asr.vendor`: Specify the ASR provider * `asr.params`: Configure ASR parameters The following ASR providers are supported: * **ARES** (default) * **Microsoft Azure** * **Deepgram** **Billing update:** In earlier versions, the service fee included the cost of the Ares ASR provider. Starting in v1.6, the pricing is restructured as follows: * If you use **ARES ASR**, the total price remains unchanged: ***Total cost = Conversational AI Engine Audio Basic Task + ARES ASR Task*** * If you use **a different ASR provider**, you are charged **only** the new **Conversational AI Engine Audio Basic Task** fee. For further details, see [Pricing](reference/pricing). * **Multi-platform toolkit** Agora now offers a toolkit to help you quickly build conversational agent apps. The toolkit is available for [**iOS**](/en/api-reference/api-ref/conversational-ai/client-toolkit/ios), [**Android**](/en/api-reference/api-ref/conversational-ai/client-toolkit/android), and [**Web**](/en/api-reference/api-ref/conversational-ai/client-toolkit/web), and includes APIs for common scenarios. Call these APIs to combine the capabilities of the Agora Voice SDK and Signaling SDK to achieve the following functions: * [**Display live transcript**](build/transcripts) Display real-time text output of user-agent conversations. The transcript component is now more robust, with better error handling, session management, and extensibility. * [**Interrupt the agent**](build/shape-the-conversation/interrupt-agent) Stop the agent from speaking or thinking mid-conversation. * [**Client-side events**](build/handle-runtime-events/event-notifications) Track changes in conversation state, performance metrics, and error events. * [**Optimize audio settings**](best-practices/audio-setup) Quickly apply best-practice audio configurations to improve agent responsiveness and clarity. #### API changes [#api-changes-11] ##### REST API [#rest-api] This release introduces several important modifications to the RESTful API. * [Start a conversational AI agent](/en/api-reference/api-ref/conversational-ai/join) * **New parameters added** * `asr.vendor` * `asr.params` * `advanced_features.enable_mllm` * `properties.mllm` * `turn_detection.type` * `turn_detection.interrupt_duration_ms` * `turn_detection.prefix_padding_ms` * `turn_detection.silence_duration_ms` * `turn_detection.threshold` * `turn_detection.create_response` * `turn_detection.interrupt_response` * `turn_detection.eagerness` * `parameters.enable_metrics` * `parameters.data_channel` * `parameters.enable_error_message` ##### Toolkit APIs [#toolkit-apis] * [Android SDK](/en/api-reference/api-ref/conversational-ai/client-toolkit/android) * [iOS SDK](/en/api-reference/api-ref/conversational-ai/client-toolkit/ios) * [Web SDK](/en/api-reference/api-ref/conversational-ai/client-toolkit/web) ### v1.5 [#v15] Released on June 9, 2025. #### New features [#new-features-12] * **Voice interruption mode** This release adds the `turn_detection.interrupt_mode` parameter to the [Start a conversational AI agent](/en/api-reference/api-ref/conversational-ai/join) API, allowing you to control how the agent handles human voice interruptions. The following modes are supported: * **`interrupt`**: (Default) The human voice immediately interrupts the agent. The agent terminates the current interaction and processes the new human voice input. * **`append`**: The human voice does not interrupt the agent. The agent processes the newly received human voice request after the current interaction ends. * **`ignore`**: The agent ignores human voice requests received during speaking or thinking. These requests are discarded and not stored in the context. * **TTS filtering** This release adds the `tts.skip_patterns` parameter to the [Start a conversational AI agent](/en/api-reference/api-ref/conversational-ai/join) API. This parameter controls whether the TTS module skips bracketed content when reading LLM response text. This prevents the agent from vocalizing structural prompt information like tone indicators, action descriptions, and system prompts, creating a more natural and immersive listening experience. #### API changes [#api-changes-12] This release introduces several important modifications to the RESTful API. * [Start a conversational AI agent](/en/api-reference/api-ref/conversational-ai/join) * **New parameters added** * `turn_detection.interrupt_mode` * `parameters.silence_config` * `tts.skip_patterns` ### v1.4 [#v14] Released on May 29, 2025. #### New features [#new-features-13] * **Metadata support for LLM requests** This release adds the `llm.vendor` parameter to the [Start a conversational AI agent](/en/api-reference/api-ref/conversational-ai/join) API. When set to `"custom"`, the agent includes additional metadata when calling the LLM, such as `turn_id` and `timestamp`. * **Support for Anthropic** Conversational AI Engine now supports `anthropic` as a request style for chat completion. Refer to the `llm.style` parameter in [Start a conversational AI agent](/en/api-reference/api-ref/conversational-ai/join). #### Improvements [#improvements-9] This release includes the following enhancements: * **Advanced LLM configuration**: The [Update agent configuration](/en/api-reference/api-ref/conversational-ai/update) API now supports: * `llm.system_messages` for updating system prompts * `llm.params` for modifying configuration parameters used when calling the large language model #### API changes [#api-changes-13] This release introduces several important modifications to the RESTful API. * [Start a conversational AI agent](/en/api-reference/api-ref/conversational-ai/join) * **New parameters added** * `llm.vendor` * **Removed parameters:** * `agent_rtm_uid` * [Update agent configuration](/en/api-reference/api-ref/conversational-ai/update) * **New parameters added** * `llm.system_messages` * `llm.params` ### v1.3 [#v13] Released on April 16, 2025. #### New features [#new-features-14] * **Agent conversation history**: This version adds two methods to retrieve an agent's history. The history includes messages exchanged between the user and the agent and timestamps of agent creation and exit. * Call the RESTful API `history` endpoint to [Retrieve agent history](/en/api-reference/api-ref/conversational-ai/history). * Subscribe to the [agent history event](reference/event-types#103-agent-history) through the [Agora message notification service](build/handle-runtime-events/webhooks). When the agent stops, Agora automatically sends the agent's history to your business server through a Webhook callback. #### Improvements [#improvements-10] * **Customize the priority of broadcast information**: This version upgrades the [Broadcast a message using TTS](/en/api-reference/api-ref/conversational-ai/speak) interface and adds two new configuration parameters related to broadcast interruption logic: * `priority`: Sets the priority of the message broadcast. Supports setting the following priorities: * `INTERRUPT` High priority * `APPEND`: Medium priority * `IGNORE`: Low priority * `interruptable`: Configure whether to allow human voice to interrupt the agent's broadcast. #### API changes [#api-changes-14] * Adds the [Retrieve agent history](/en/api-reference/api-ref/conversational-ai/history) method. * Adds `priority` and `interruptable` parameters to the [Broadcast a message using TTS](/en/api-reference/api-ref/conversational-ai/speak) method. ### v1.2 [#v12] Released on April 10, 2025. #### New features [#new-features-15] * **Broadcast a message using TTS**: A new message broadcast interface enables a specified agent to deliver a custom message. When interacting with an agent, calling this interface interrupts the agent's speech and thinking process, allowing the TTS module to immediately broadcast the custom message. * **Interrupt the agent**: The interrupt agent endpoint lets you stop the specified agent's speech and thinking process. #### API changes [#api-changes-15] This version adds the following APIs: * [Broadcast a message using TTS](/en/api-reference/api-ref/conversational-ai/speak) * [Interrupt the agent](/en/api-reference/api-ref/conversational-ai/interrupt) ### v1.1 [#v11] Released on March 27, 2025. #### New features [#new-features-16] The [Start a conversational agent](/en/api-reference/api-ref/conversational-ai/join) API adds the `enable_rtm` and `agent_rtm_uid` parameters to enable Signaling integration with the conversational AI agent. When this feature is enabled, the agent can leverage the Signaling SDK to obtain a user's custom context information such as speaking status, selected text, signature, and score, and pass this data to the agent to generate more relevant content. For details, see [Transmit custom information](build/shape-the-conversation/custom-information). #### Improvements [#improvements-11] To help you quickly integrate a custom large language model (LLM), this version adds documentation for [Custom LLMs](build/custom-model-integration/custom-llm). Refer to the sample code in the documentation to integrate your custom model into the Conversational AI Engine and enable advanced capabilities such as Retrieval-Augmented Generation (RAG), multi-modal processing, and tool invocation. #### API changes [#api-changes-16] The `POST` method to [Start a conversational AI agent](/en/api-reference/api-ref/conversational-ai/join) now includes the `enable_rtm` and `agent_rtm_uid` parameters. ### v1.0 (Public Beta) [#v10-public-beta] This version, released on March 4, 2025, adds pricing information for the Agora Conversational AI Engine. For more information, see [Pricing](reference/pricing). #### Integration guide [#integration-guide] To achieve the best conversation experience, use Agora Conversational AI Engine with the following Agora SDKs: * Agora RTC Native SDK, v4.5.1 or later. * Agora RTC Web SDK, version 4.23.2 or later. #### New features [#new-features-17] * **Live transcripts**: Supports real-time text output of conversations between users and the AI agent for transcript display in your app's UI. Agora provides an open-source transcript processing module. Integrate the module and call its API to implement live transcript. For details, see [Display live transcripts](build/transcripts). * **Message Notification Service**: Introduces a new Conversational AI Engine message notification service. Configure it in the Agora console and subscribe to agent creation, stop, and error events. When a subscribed event occurs, Agora sends the details to your specified callback address. See [Receive event notifications](build/handle-runtime-events/webhooks). * **Keywords**: Enhances recognition accuracy of Conversational AI Engine for proprietary words by adding keywords. This feature is currently in Beta stage. For details, [contact technical support](https://agoraio.zendesk.com/hc/en-us). ### v1.0 (Private Beta) [#v10-private-beta] Released on February 18, 2025. The first beta release of the Conversational AI Engine brings natural, smooth, low-latency, and highly reliable real-time voice conversations with AI agents to Agora channels. It enables you to efficiently build intelligent and immersive interactive experiences. See [Product overview](/en/ai) for details. #### Core features [#core-features] * **Real-time voice conversation** Supports natural and smooth real-time voice conversations with AI. It delivers a low-latency, ultra-responsive interactive experience as if the user is communicating with a real person. * **Intelligent noise suppression** Intelligently identifies and suppresses background noise, ensuring clear sound transmission even in noisy environments to provide users with a high-quality audio experience. * **Background human voice suppression** Suppresses background voices and noise while accurately preserving the primary speaker's voice. This ensures a clear and focused interactive experience in multi-speaker environments. * **Intelligent interruption handling** Allows users to interrupt AI at any time to ensure quick and natural responses. This feature enables smooth transitions and avoids mechanical interactions. * **Intelligent transmission** An AI-optimized transmission algorithm ensures stable voice data delivery even in weak network conditions where packet loss reaches 80%. This guarantees conversation continuity and reliability across diverse network environments. * **Flexible arrangement** Supports multiple Large Language Model (LLM) and Text-to-Speech (TTS) providers, enabling flexible orchestration to meet diverse business needs and deliver highly customizable AI dialogue solutions. * **Multi-platform support** Compatible with iOS, Android, Web, and various embedded hardware platforms, providing a seamless and consistent cross-platform experience. #### Integration guide [#integration-guide-1] * For the best conversational experience, Agora recommends using Conversational AI Engine with specific Agora Video/Voice SDK versions. For details, [contact technical support](mailto\:support@agora.io). * The number of Peak Concurrent Users (PCU) allowed to call the server API under a single App ID is limited to 20. If you need to increase this limit, [contact technical support](mailto\:support@agora.io). # About Agora (/en/introduction/about-agora) ## What Agora is [#what-agora-is] Agora is a realtime platform for products where people, devices, services, and AI systems need to stay connected inside the same live session. Instead of treating voice, video, messaging, AI orchestration, and media processing as separate stacks, Agora gives you a shared realtime foundation you can build on. You can start with one immediate use case, such as voice calling or an AI agent, and then extend the same session architecture into messaging, cloud processing, analytics, and production operations. Think in live sessions, not isolated features Agora is easiest to understand when you treat it as a shared realtime layer for interaction, coordination, AI participation, and media workflows. ## What problems it solves [#what-problems-it-solves] Teams usually come to Agora when they need to solve more than one live-product problem at the same time: * Keep voice and video interactions usable under real-world network conditions. * Coordinate room state, presence, events, and messaging alongside live media. * Add voice AI into a live session instead of bolting AI onto a disconnected backend flow. * Record, transcribe, translate, transform, or distribute media after the session starts. * Operate production systems with the credentials, security, monitoring, and governance live products require. In practice, this means Agora is most useful when your product is not just sending a message or making a single API call, but managing an ongoing realtime experience. ## Product families [#product-families] Agora's docs are easier to navigate when you group the platform into four product families instead of reading it as a list of unrelated SKUs. Build voice agents, assistants, tutors, support flows, and multimodal AI experiences that participate in a live realtime session. Build calling, meetings, classrooms, livestream interaction, smart-device media flows, and other low-latency communication products. Keep users, devices, and room state synchronized with chat, presence, metadata, signaling, and collaboration surfaces. Extend live sessions with recording, speech-to-text, translation, push and pull workflows, transcoding, and distribution tooling. ## How these products fit together [#how-these-products-fit-together] The most useful mental model is to think in layers of responsibility: 1. **Projects and credentials** define access, billing scope, and service boundaries. 2. **Realtime sessions** connect users, devices, and agents inside a shared live context. 3. **Voice, video, and messaging** carry interaction, room state, and events through that context. 4. **AI and server-side control** add orchestration, automation, callbacks, and business logic. 5. **Cloud media services** capture, transform, translate, distribute, and observe the live session. You do not need every layer on day one. The value of the platform is that these layers can be added over time without forcing you to replace the underlying session model. ## Common solution patterns [#common-solution-patterns] Most implementations start from a small number of repeatable patterns: Start with live voice or video when the main requirement is low-latency communication between participants. Add room messaging, presence, and coordination when participants also need shared state, chat, or event-driven flows. Add a voice agent when an AI system needs to join the session as a live participant instead of a separate backend service. Extend a live session with archive, compliance, captions, transcripts, translation, or post-session processing. Build smart-device or embedded experiences where microphones, speakers, sensors, and agents operate through the same realtime path. These patterns are helpful because they let you choose a narrow first milestone while keeping a clear path to the rest of the platform. ## Where to start [#where-to-start] Choose the next page based on the first problem you need to solve: * Start with [Start with AI](/en/introduction/start-with-ai) if you want an AI-assisted path using Skills, MCP, and agent-friendly docs. * Go to [Conversational AI](/en/introduction/conversational-ai) if your main product is a voice agent, assistant, tutor, or spoken AI experience. * Go to [Real-Time Voice & Video](/en/introduction/realtime-audio-video) if your first milestone is calling, meetings, classrooms, livestream interaction, or device media transport. * Go to [Messaging & Presence](/en/introduction/messaging-presence) if chat, presence, signaling, or shared room state is your main coordination problem. * Go to [Cloud Media Services](/en/introduction/cloud-media-services) if you need recording, transcription, translation, media routing, or stream transformation around a live session. * Read [Core Concepts](/en/introduction/core-concepts) if you want the platform primitives before choosing a build path. When you move from prototype to production, continue with [Projects](/en/introduction/projects), [Console setup](/en/introduction/console-setup), [Billing](/en/introduction/billing), and [Security and privacy](/en/introduction/security-privacy). # Account (/en/introduction/account) This page shows you how to sign up for an Agora account, create a new project, and get the App ID and App Certificate to generate a temporary token. ## Get started with Agora [#get-started-with-agora] To join an Agora session, you need an Agora App ID. This section shows you how to set up an Agora account, create an Agora project and get the required information from [Agora Console](https://console.agora.io/v2). ### Sign up for an Agora account [#sign-up-for-an-agora-account] To use Agora products and services, create an Agora account with your email, phone number, or a third-party account. Once you sign up successfully, your account is automatically logged in. Follow the on-screen instructions to create your first project and test out real-time communications. For later visits, log in to [Agora Console](https://console.agora.io/v2) with your phone number, email address, or linked third-party account. ### Create an Agora project [#create-an-agora-project] To create an Agora project, do the following: 1. In [Agora Console](https://console.agora.io/v2), open the [Projects](https://console.agora.io/v2/project-management) page. 2. Click **Create New**. 3. Follow the on-screen instructions to enter a project name and use case, and check **Secured mode: APP ID + Token (Recommended)** as the authentication mechanism. ![configure\_project](https://assets-docs.agora.io/images/signaling/create_new_project.png) 4. Click **Submit**. You see the new project on the **Projects** page. ### Get the App ID [#get-the-app-id] Agora automatically assigns a unique identifier to each project, called an App ID. To copy this App ID, find your project on the [Projects](https://console.agora.io/v2/project-management) page in Agora Console, and click the copy icon in the **App ID** column. ![configure\_project](https://assets-docs.agora.io/images/signaling/app-id.png) ## Security and authentication [#security-and-authentication] Use the following features from your Agora account to implement security and authentication features in your apps. ### Get the App Certificate [#get-the-app-certificate] When generating an authentication token on your app server, you need an App Certificate, in addition to the App ID. To get an App Certificate, do the following: 1. On the [Projects](https://console.agora.io/v2/project-management) page, click the pencil icon to edit the project you want to use. ![Console project management page](https://assets-docs.agora.io/images/common/console-project-management-page.png) 2. Click the copy icon under **Primary Certificate**. ![Console primary certificate](https://assets-docs.agora.io/images/common/console-primary-certificate.png) For full App Certificate management details, see [Projects](/en/introduction/projects#manage-app-certificates). ### Generate temporary tokens [#generate-temporary-tokens] To ensure communication security, best practice is to use tokens to authenticate the users who log in from your app. To generate a temporary RTC token for use in your project: 1. On the [Projects](https://console.agora.io/v2/project-management) page, click the pencil icon next to your project. 2. On the **Security** panel, click **Generate Temp Token**, enter a channel name in the pop-up box and click **Generate**. 3. Copy the generated RTC token. To generate a token for other Agora products: 1. In your browser, navigate to the [Agora token builder](https://agora-token-generator-demo.vercel.app/). 2. Choose the Agora product your user wants to log in to. Fill in **App ID** and **App Certificate** with the details of your project in Agora Console. 3. Customize the token for each user. The required fields are visible in the Agora token builder. 4. In token-based authentication, each token is specific to a user ID in your app. Generate a token for each user, and make sure the user ID in your SDK login or join call matches the user ID used to create the token. 5. Click **Generate Token**. The token appears in Token Builder. 6. Copy the token and use it in your app. For more information on managing other aspects of your Agora account, see [Console setup](/en/introduction/console-setup). # Agora CLI (/en/introduction/agora-cli) Install the Agora CLI when you want the shortest path to a working Agora starter. Use it to log in, initialize an official starter, write environment values, switch projects, and diagnose setup problems from the terminal. If you want to get a demo running before you fine-tune prompts or architecture, start here. After the starter works, add [Agora MCP](/en/introduction/agora-mcp) or [Agora Skills](/en/introduction/agora-skills) if you want your assistant to stay grounded in current docs and Agora-specific workflows. ## Install the Agora CLI [#install-the-agora-cli] macOS and Linux Windows (PowerShell) ```bash curl -fsSL https://agoraio.github.io/cli/install.sh | sh -s -- --add-to-path agora --help ``` ```powershell irm https://agoraio.github.io/cli/install.ps1 | iex agora --help ``` ## Quick start [#quick-start] Follow this path if you want to get an official starter running as quickly as possible. TypeScript Python Go ```bash agora login agora init my-nextjs-demo --template nextjs cd my-nextjs-demo pnpm install pnpm dev ``` ```bash agora login agora init my-python-demo --template python cd my-python-demo bun install bun run dev ``` ```bash agora login agora init my-go-demo --template go cd my-go-demo make setup make dev ``` ### What happens in this flow [#what-happens-in-this-flow] 1. `agora login` authenticates your machine. 2. `agora init` clones an official starter, binds it to your Agora project, and writes the environment you need locally. 3. The runtime-specific install and dev commands start the application. If the starter runs, move on to application-specific work. If not, troubleshoot in the same order you experienced the setup below. ## If something fails [#if-something-fails] ### The `agora` command is not available [#the-agora-command-is-not-available] If `agora --help` does not work after installation, rerun the installer first. On macOS and Linux, make sure you used the install command with `--add-to-path`. If PowerShell blocks the Windows install script, download `install.ps1` and run: ```text powershell -ExecutionPolicy Bypass -File .\install.ps1 ``` ### Login or starter setup fails [#login-or-starter-setup-fails] Run the setup path again in order: ```bash agora login agora init my-demo --template nextjs ``` This covers the two most common first-run failures: the machine is not authenticated, or the starter was not initialized correctly. ### The starter runs incorrectly [#the-starter-runs-incorrectly] Use the CLI to check readiness, credentials, and feature enablement: ```bash agora project doctor ``` ### Project settings or credentials look wrong [#project-settings-or-credentials-look-wrong] Inspect the current environment values that the CLI exported for your active project: ```bash agora project env --shell ``` This is the right check when the local app starts but the agent does not join, transcripts do not appear, or the active project looks wrong. ## Build from scratch instead [#build-from-scratch-instead] If you are not using an official starter, create or switch projects first, then export credentials into the local environment. TypeScript Python Go ```bash agora login agora project create conv-ai-tutorial --feature rtc --feature convoai agora project use conv-ai-tutorial agora project env --shell ``` ```bash agora login agora project create conv-ai-tutorial --feature rtc --feature convoai agora project use conv-ai-tutorial agora project env --shell ``` ```bash agora login agora project create conv-ai-tutorial --feature rtc --feature convoai agora project use conv-ai-tutorial agora project env --shell ``` You can also switch the active project later or create a new one explicitly: ```bash agora project create my-project --feature rtc --feature convoai agora project use my-project ``` The CLI gives your assistant reproducible local steps it can execute and verify, especially during starter bootstrapping, project binding, environment generation, and first-line diagnosis. # Agora MCP (/en/introduction/agora-mcp) Install Agora MCP when you want your assistant to check the latest Agora docs before it writes or changes code. Use it to confirm current SDK methods, parameters, quickstarts, and product or platform differences while you build or debug. This is most useful when documentation freshness is the risky part of the task. If you also want help choosing the right Agora workflow or starter, add [Agora Skills](/en/introduction/agora-skills). ## Install the MCP server [#install-the-mcp-server] `Agora MCP` is included when you install [Agora Skills](/en/introduction/agora-skills), but you can also install the MCP server on its own: ```text https://mcp.agora.io ``` Cursor Claude Code Codex Gemini CLI Manual installation Add the following JSON to your MCP configuration: ```json { "mcpServers": { "agora-docs": { "url": "https://mcp.agora.io" } } } ``` ```bash claude mcp add --transport http agora-docs https://mcp.agora.io ``` ```bash codex mcp add --url https://mcp.agora.io agora-docs ``` ```bash gemini mcp add --transport http agora-docs https://mcp.agora.io ``` Add `https://mcp.agora.io` to your MCP client and use `http` or `Streamable HTTP` transport if the client asks for a transport type. ## Start with a prompt [#start-with-a-prompt] Install the server in your coding tool, then ask your assistant to search Agora docs before it writes code. Include the Agora product, the target platform, and the task type in the same prompt. ```text Use Agora MCP to find the current web quickstart for Conversational AI and explain the minimum setup for a browser voice demo. ``` ## Make docs lookup part of the workflow [#make-docs-lookup-part-of-the-workflow] If your repository includes `AGENTS.md` or similar instructions, tell your assistant to prefer Agora MCP for product-specific lookups. That turns docs checks into a normal part of the workflow instead of an optional extra step. If your MCP client is not available, these fallback resources still help: * `https://docs.agora.io/llms.txt` * `https://docs-md.agora.io/...` for Markdown page versions # Agora Skills (/en/introduction/agora-skills) Install Agora Skills when you want your assistant to choose the right Agora workflow before it starts coding. Skills helps you pick the right product path, starter, and setup sequence, so your assistant can follow official Agora patterns instead of guessing. Use Skills when docs lookup alone is not enough and the harder part is deciding what to build first or which official path to follow. If you also want fresh documentation while your assistant works, pair Skills with [Agora MCP](/en/introduction/agora-mcp). ## Install Agora Skills [#install-agora-skills] Skills CLI Manual installation ```bash npx skills add AgoraIO/skills ``` Skills activate automatically when the agent detects a relevant Agora task. Clone the repository and point your coding assistant to the Agora skill files: ```bash git clone https://github.com/AgoraIO/skills.git ~/agora-skills ``` Use `skills/agora/` as the directory entry point, or load `SKILL.md` directly if your tool expects a single root file. If you install Skills manually, place the files where your assistant already looks for local instructions: Claude Code Cursor Windsurf GitHub Copilot Other tools Run the following commands inside Claude Code: * **User-level** (available across all your projects) ```bash /plugin marketplace add AgoraIO/skills /plugin install agora ``` * **Project-level** (shared with your team via version control) ```bash /plugin marketplace add AgoraIO/skills /plugin install agora --scope project ``` Copy or symlink `skills/agora/` into `.cursor/rules/`. Add `skills/agora/` to Cascade context. Reference the files with `@workspace` or add them to Copilot instructions. Point the tool at `skills/agora/` or the top-level `SKILL.md`. ## Start with a prompt [#start-with-a-prompt] Install Skills, then ask your assistant to choose the correct Agora path before it starts scaffolding or editing code. ```text Use Agora Skills and Agora MCP to help me build a web-based conversational AI demo. Check the official docs first, choose the right starter, and use the Agora CLI for setup. ``` From there, let your assistant choose the starter and setup path before it edits code. # Cloud Media Services (/en/introduction/cloud-media-services) Cloud Media Services is the capability layer for products that need to do more with live media than simply transport it between participants. This category covers the cloud-side workflows that capture, transform, route, translate, and extend live sessions. Agora's product map for this layer includes: * **Recording** in cloud or local-server form * **Media Push / Pull** and RTMP Gateway workflows * **Speech-to-Text / Translation** * **Transcoding / Compositing** * **Extensions / Analytics** and adjacent cloud-side operational services ## What this category is good for [#what-this-category-is-good-for] This category is the right entry point when you need: * recording for archive, replay, compliance, or review * pushing or pulling media between Agora and external streaming systems * speech transcription or live translation as a cloud-side service * stream composition, transcoding, and output shaping * analytics, extensions, and post-session media workflows The shared requirement is that the media session must continue to create value after, around, or beyond the primary live interaction. ## Capability map [#capability-map] Capture live sessions into replayable and reviewable assets through cloud-managed or self-managed recording paths. Move media between Agora and external live-streaming or online media systems through push, pull, and gateway workflows. Turn live speech into text and translated text for accessibility, AI preprocessing, records, and multilingual products. Reshape streams into layouts, output formats, and distribution-friendly media workflows for playback and broadcast. Extend product behavior and add production observation, diagnosis, and ecosystem integrations around the media path. ## Start here [#start-here] ## Key characteristics [#key-characteristics] Handle what happens to live media before, during, and after the user-facing session. Connect live interaction with storage, broadcast delivery, analytics, moderation, and external streaming workflows. Work with recordings, transformed streams, transcripts, translations, and playback-oriented outputs as one service layer. Add capabilities that teams usually need once a live experience moves beyond prototype stage. Reuse the same cloud media services whether the upstream source is RTC, AI interaction, or host-audience live streaming. ## What to read next [#what-to-read-next] * [Media processing and distribution](/en/realtime-media/media-processing-and-distribution) * [Cloud Recording](/en/realtime-media/cloud-recording) * [RTMP Gateway](/en/realtime-media/rtmp-gateway) * [Transcoding](/en/realtime-media/transcoding) * [Realtime Transcription and Translation](/en/realtime-media/speech-to-text) # Community Resources (/en/introduction/community-resources) Agora Console is the unified portal for configuring, purchasing, and managing Agora products and services. ## Dashboard overview [#dashboard-overview] After logging in to Console, the overview dashboard provides quick access to: * projects * billing * usage * members and roles * documentation * code samples * support * Agora Analytics * Extensions Marketplace ## Additional resources [#additional-resources] * [Support](/en/introduction/support) * [Release Notes](/en/ai/release-notes) * [Usage Analytics](/en/introduction/usage-analytics) * [AI best practices](/en/ai/best-practices) * [AI product map](/en/ai/domain-overview) ## Status and operations [#status-and-operations] Use the Agora Status Page and the Support Center to monitor service health, track incidents, and follow maintenance events. # Console Setup (/en/introduction/console-setup) [Agora Console](https://console.agora.io/v2) is the unified portal for you to configure, purchase, and manage Agora products and services. This page shows you how to use the new **beta** version of Agora Console to manage all aspects of your Agora account. ## Dashboard overview [#dashboard-overview] After logging in to Agora Console, you see the overview dashboard. The dashboard displays a summary of important statistics and provides quick access to projects, billing, documentation, and code samples. ![Console dashboard](https://assets-docs.agora.io/images/common/console-dashboard.png) Agora Console provides access to the following: * Create and manage projects Create a project, and get the App ID and App Certificate to generate a temporary token. See [Projects](/en/introduction/projects). * Enable and configure features Configure CHAT, Signaling, WHITE, CREC, MPUSH, MPULL, Notifications, FC, Cloud proxy, and Co-Host Authentication. * Check usage Check the usage amount generated by each project or all of them. See [Usage Analytics](/en/introduction/usage-analytics). * Manage members and roles Add members to your account, and add them to different teams with specified permissions. See [Members & Roles](/en/introduction/members-roles). * Manage billing Add money to your account, view your transactions, and make withdrawal requests. See [Billing center](#billing-center). * Submit a ticket If you have any questions about Agora products, [submit a ticket](#submit-a-ticket). * Generate Customer ID and Customer Secret To use Agora RESTful APIs, you need a set of Customer ID and Customer Secret. * Agora Analytics Track and analyze the usage and quality data of real-time communication products through highly visualized reports in order to spot issues and find root causes. For more information, see the [Agora Analytics overview](/en/realtime-media/agora-analytics/product-overview). * Agora Extensions Marketplace Gain access to extensions that rapidly add fun features to your app on top of Agora SDKs. For more information, see the [Extensions Marketplace overview](/en/realtime-media/marketplace). ## Manage profile [#manage-profile] This section shows how to view and edit the profile of your Agora account. ### Open the profile page [#open-the-profile-page] To open the **Profile** page, follow these steps: 1. Log in to [Agora Console](https://console.agora.io/v2). 2. Click your account name in the top right corner, and select your account name from the dropdown menu. On the **Profile** page, you can view and edit your phone number, email address, password, and login methods. ![Console profile page](https://assets-docs.agora.io/images/common/console-profile-page.png) ### Change phone number, email address, or password [#change-phone-number-email-address-or-password] 1. Find **Phone Number**, **Email**, or **Password** fields on the profile page, then click **Update**. 2. Follow the on-screen instructions to complete. ### Link a third-party account [#link-a-third-party-account] 1. On the profile page, find **GitHub**, **WeChat**, or **Google**, then click **Connect**. 2. Follow the on-screen instructions to complete. ### Reset your password [#reset-your-password] If you forget your password, follow these steps to reset it: 1. Go to the [login page](https://sso.agora.io/login/) and click **Forgot password**. 2. Fill in your email address and click **Send email**. Agora sends an email to you. 3. Follow the instructions in the email to reset your password. ## Create and manage projects [#create-and-manage-projects] This section shows how to create a project, view project information, and manage App Certificates in Agora Console. ### Restrictions [#restrictions] * If your account has multiple members, only those assigned to the teams of **Admin**, **Engineer**, or an authorized custom team have access to the **Projects** page. * Each Agora account can create up to 20 projects. If you need to create more projects, contact Agora by [submitting a ticket](https://agora-ticket.agora.io/). ### Create an Agora project [#create-an-agora-project] To create a project, do the following: 1. Open the [Projects](https://console.agora.io/v2/project-management) page. 2. Click **Create New**. 3. Follow the on-screen instructions to enter a project name and use case, and check **Secured mode: APP ID + Token (Recommended)** as the authentication mechanism. 4. Click **Submit**. You can now see the project on the **Projects** page. ### View project information [#view-project-information] For a project listed on the **Projects** page, you can do the following: * View basic information, such as the last updated and created dates, name, and security status. * Copy the [App ID](/en/introduction/glossary#app-id) of the project. * Click the pencil icon to open the project details page, where you can configure **Project Name**, copy **App Certificate** and **App ID**, and generate a **Temporary Token**. ### Manage app certificates [#manage-app-certificates] An App Certificate is a string generated by Agora Console to enable token authentication. For different security requirements, Agora provides the following app certificate options: * Primary Certificate: This certificate is used to generate tokens for temporary use or production environments. * Secondary Certificate: This certificate is used to generate tokens for production environments only and does not apply to RESTful APIs. ### Enable the Primary Certificate [#enable-the-primary-certificate] For high-security use-cases, enable the Primary Certificate as follows: * If you choose **APP ID + Token** as the authentication mechanism when creating a project, the Primary Certificate is enabled by default. On the project details page, you can click the copy icon under **Primary Certificate** to view and copy it. In this case, you need to use the token generated by the **Primary Certificate** for authentication. * If you choose **APP ID** as the authentication mechanism when creating a project, you need to enable the Primary Certificate manually. On the project detail page, click **Enable** under **Primary Certificate**. ![Enable Primary Certificate](https://assets-docs.agora.io/images/common/console-enable-primary-certificate.png) Once the Primary Certificate is enabled, you can click the copy icon to view and copy it. In this case, you can either complete authentication with the App ID only or use tokens generated by the **Primary Certificate** for authentication. ### Enable the Secondary Certificate [#enable-the-secondary-certificate] If you need to change the Primary Certificate after enabling it, you can enable the Secondary Certificate. On the project detail page, click **Add a Certificate**. ![Add a certificate](https://assets-docs.agora.io/images/common/console-add-certificate.png) The **Secondary Certificate** is now enabled. ![Enable econdary Certificate](https://assets-docs.agora.io/images/common/console-enable-secondary-certificate.png) Once the Secondary Certificate is enabled, you can click the copy icon to view and copy it. In this case, both the Primary and Secondary Certificate can be used to generate tokens for authentication. ### Switch to a new Primary Certificate [#switch-to-a-new-primary-certificate] If you suspect that your primary certificate has been compromised, swap the primary certificate with the secondary certificate and delete the original primary certificate to avoid risks. Replacing the primary certificate may impact your online business. Best practice is to replace the primary certificate only when necessary. If you replace the primary certificate but do not delete the secondary certificate (the original primary certificate), the tokens previously generated using this certificate can still be used. To switch to a new Certificate, follow these steps: 1. [Enable the Secondary Certificate](#enable-the-secondary-certificate). 2. On the project detail page, click the arrows icon between the Certificates to swap them. 3. Turn off the switch next to the **Secondary Certificate** to disable it. ![Enable Secondary Certificate](https://assets-docs.agora.io/images/common/console-enable-secondary-certificate.png) 4. Follow the on-screen instructions to complete verification. If verified successfully, the status of the Secondary Certificate is updated to **Disabled**, and the **Delete** icon appears. ![Delete Certificate](https://assets-docs.agora.io/images/common/console-delete-certificate.png) 5. Click the **Delete** icon. 6. Follow the on-screen instructions to complete verification. If verified successfully, the **Delete** button disappears, and the current Secondary Certificate (the original Primary Certificate) is deleted. Once a certificate is deleted, it cannot be restored. Before deleting the secondary certificate, make sure that most users have switched to the new certificate to avoid the following consequences: * All tokens generated using the deleted certificate become invalid, and users can no longer use these tokens to join channels. * Users who have joined the channel cannot specify new tokens through `renewToken`. ## Check usage [#check-usage] This section explains how to check your usage of Agora products in Agora Console, such as Interactive Live Streaming, Broadcast Streaming, Media Push, On-Premise Recording, Cloud Recording, and Interactive Whiteboard. ### Restrictions [#restrictions-1] * If your account has multiple members, only those assigned to the team of **Admin**, **Product/Operation**, or an authorized custom team have access to the **Usage** page. * You can only view the usage data for the past 12 months. * Data on the **Usage** page can differ from the actual amount. For the most accurate usage data, see the bills sent to your account. ### Procedure [#procedure] Follow these steps to check your usage: 1. Enter the [Usage](https://console.agora.io/v2/usage) page in Agora Console. 2. (Optional) Add filters for the usage data, as follows: * Select a time frame and data granularity. * Select **All Projects** or a specific project from the dropdown menu. * Select a product to view its usage data. * Check **View Breakdown**. ![Console Usage](https://assets-docs.agora.io/images/common/console-usage-details.png) ## Manage members and teams [#manage-members-and-teams] Once you add members to your Agora account, you are assigned to the team of **Admin** and can add these members to different teams with specified permissions. ### Add a member [#add-a-member] Follow these steps to add a member to your account: 1. Log in to [Agora Console](https://console.agora.io/v2), click your account name in the top-right corner, and click **Settings** in the dropdown menu. 2. In the left navigation panel, click **Teams and members**, then select the **Members** tab. 3. Click the **Add New Member** button, fill in the email address of the new member, and choose a team from the dropdown menu. Then click **OK**. ![Add new member](https://assets-docs.agora.io/images/common/console-add-new-member.png) 4. Agora sends a confirmation email to this address. The new member should follow the instructions in the email to finish joining the project. ### Manage members [#manage-members] On the [Teams and members](https://console.agora.io/v2/teams-members) page, **Members** tab, you can do the following: * Click the pencil icon to change the team that a member belongs to. * Delete a member. Information Only the main account can delete a member account. Member accounts assigned to the Admin team cannot delete a member account. ### Manage teams and permissions [#manage-teams-and-permissions] On the **Teams and members** page, **Teams** tab, you can view the permissions assigned to teams. Agora predefines five teams for you. The permissions of each predefined team are as follows: * **Admin** has full access to all projects. The administrator can view usage data and finance information, manage members and teams, manage projects, and view Agora Analytics reports. * **Finance** can view finance information of all projects. * **Product/Operation** can view usage data of all projects. * **CS/Maintenance** can view Agora Analytics reports of all projects. * **Engineer** can manage projects and view Agora Analytics reports of all projects. ### Add a custom team [#add-a-custom-team] Follow these steps to add a custom team: 1. On the **Teams and Members** page, **Teams** tab, click **Add New Team**. ![Add new team](https://assets-docs.agora.io/images/common/console-add-team.png) 2. Fill in the team name, and select the permissions of this team in the **Usage**, **Finance**, **Teams and Members**, **Project**, **Agora Analytics**, and **Data** columns. 3. Click **OK**. ## Integrate with Okta [#integrate-with-okta] If you are using [Okta](https://www.okta.com/) as the identity provider for your app, you can configure Agora Console so that your team members can log into Agora using their Okta accounts. You will need: * An Okta developer account with admin privileges * An Agora account assigned to the **Admin** team * A Premium or Enterprise Agora Support package ### Configuration steps [#configuration-steps] 1. Apply for Okta integration in Agora Console. Log in to your Agora account. On the [SSO Management](https://console.agora.io/settings/sso-management) page, click **Request**. ![okta-application](https://assets-docs.agora.io/images/common/okta-application.png) Once you apply, a request is sent to Agora to enable integration for you. ![okta-applied](https://assets-docs.agora.io/images/common/okta-applied.png) When your application is processed by Agora, **SAML Configuration** on the same page is automatically enabled and a set of **SSO URL** and **Audience URI** is generated. 2. Create a SAML integration for Agora in Okta Console. 1. Log in to your Okta developer account as a user with administrative privileges and click **Admin** in the upper right corner. ![okta-admin](https://assets-docs.agora.io/images/common/okta-admin.png) 2. Go to **Applications** > **Applications** and click **Create App Integration**. ![okta-create-app-integration](https://assets-docs.agora.io/images/common/okta-create-app-integration.png) 3. In the **Sign-in method** section, select **SAML 2.0** and click **Next**. 4. On the **General Settings** tab, enter a name for this integration, for example, `Agora`, and click **Next**. 5. On the **Configure SAML** tab, fill in the following SAML configurations generated in Agora Console: * In the **Single sign on URL** field, enter `https://sso2.agora.io/api/v0/saml/idp/callback`. * In the **Audience URI (SP Entity ID)** field, enter `https://sso2.agora.io/{companyId}/saml/SSO` where `{companyId}` is your company ID. * In the **Attribute Statements (optional)** section, add the following attribute: | Name | Value | | :------ | :----------- | | `email` | `user.email` | ![okta-saml-config](https://assets-docs.agora.io/images/common/okta-saml-config.png) 6. On the **Feedback** tab, select **I'm an Okta customer adding an internal app** and tick **This is an internal app that we have created**, then click **Finish**. ![okta-feedback](https://assets-docs.agora.io/images/common/okta-feedback.png) 3. View IdP information in Okta Console 1. In Okta Console, select the **Sign On** tab for Agora integration and click **View Setup Instructions**. You see the following information: * **Identity Provider Single Sign-On URL**: At the end of this procedure, your team members use this URL to sign in to Agora Console * **Identity Provider Issuer** * **X.509 Certificate** ![okta-saml-setup](https://assets-docs.agora.io/images/common/okta-saml-setup.png) 4. Configure SAML settings in Agora Console In the **SAML Configuration** section on the [SSO Management](https://console.agora.io/settings/sso-management) page in Agora Console, enter the following information from Okta Console: * **Identity Provider Single Sign-On URL** * **Identity Provider Issuer** * **X.509 Certificate**, including the `BEGIN CERTIFICATE` and `END CERTIFICATE lines`. Click **Save**. 5. Manage access You can manage access for your team members manually using SAML, which is enabled when you apply for Okta integration in Agora Console, or automatically, if you additionally enable SCIM. Manual management means that adding, managing, and removing a team member's access must be done in both, Okta and Agora Console. Automatic management means that SCIM automatically adds, manages, and removes members' access to Agora Console when you make changes in Okta. **Note**: Once enabled, SAML is applicable for both, existing and newly created Agora accounts for team members. For new accounts, however, log in as the admin in Agora Console and create an account *before* it is configured in Okta. SCIM is applicable only for new accounts. * Manual management with SAML 1. In Agora Console, add team members and choose their teams. For details, see [Manage members and teams](#manage-members-and-teams). 2. In Okta Console, go to **Directory** > **People**. Ensure that the email address of each team member is the same as that in Agora Console. 3. Go to **Applications** > **Applications** > **Agora integration**. 4. On the **Assignments** tab, click **Assign** and select **Assign to People**. 5. Enter the team members that need to sign in to Agora Console and click **Assign** for each. Click **Done**. * Automated management with SCIM 1. In Agora Console, enable **SCIM API Basic Auth** on the **SSO Management** page. A set of username and password is generated, along with the SCIM connector base URL. ![scim-enabled](https://assets-docs.agora.io/images/common/scim-enabled.png) 2. In Okta Console, select the **General** tab under **Agora integration** and click **Edit**. 3. In the **Provisioning** section, select **SCIM** and click **Save**. 4. Select the **Provisioning** tab under **Agora integration**, then under **Settings** > **Integration** click **Edit**. Make the following changes: * In **SCIM connector base URL**, enter the corresponding URL from Agora Console. * In **Unique identifier field for users**, enter `email`. * Under **Supported provisioning actions**, tick all the checkboxes. * In **Authentication Mode**, select **Basic Auth**. * Under the **Basic Auth** section, enter the username and password from the SSO management page in Agora Console. ![scim-configured](https://assets-docs.agora.io/images/common/scim-configured.png) After enabling and configuring Okta integration, the [Big Bang feature](https://help.okta.com/oie/en-us/content/topics/reference/glossary.htm) is enabled by default. This means that login to Agora Console with email and password is no longer available and your team members can log in only through Okta. ## Billing center [#billing-center] This section shows how to check your account balance, add money to your account, view transactions, and make a withdrawal in Agora Console. ### Restrictions [#restrictions-2] If your account has multiple members, only those assigned to the teams of **Admin**, **Finance**, or an authorized custom team have access to the **Billing** page. ### Check account balance [#check-account-balance] After logging in to [Agora Console](https://console.agora.io/v2), you can see your account balance in the **Billing** panel on the **Overview** page. ### Add money to account [#add-money-to-account] You can add money to your account either with a credit card or via bank transfer. ### Credit card [#credit-card] To add money to your account with a credit card, follow these steps: 1. In [Billing](https://console.agora.io/v2/billing), select the **Credit Card** tab. 2. (Optional) If you have not added a credit card to your account, follow these steps: 1. Click **Add New Card**. ![Add new card](https://assets-docs.agora.io/images/common/console-add-new-card.png) 2. Enter the card number, card holder name, expiration date, and CVC. 3. Choose whether or not to set this card as the default credit card. 4. Click **Continue**. 3. Select a credit card, enter the **Payment Amount**, and click **Pay**. If the payment is successful, you see a "Payment Succeeded" message. ### Bank transfer [#bank-transfer] To add money to your account via bank transfer, follow these steps: 1. In [Billing](https://console.agora.io/v2/billing), select the **Bank Transfer** tab. ![Bank transfer](https://assets-docs.agora.io/images/common/console-bank-transfer.png) 2. Follow the on-screen instructions to complete. ### View invoices and transactions [#view-invoices-and-transactions] In [Billing](https://console.agora.io/v2/billing), scroll down to see **Your Invoices** and **Your Transactions**, respectively. You can also click **View all** > **Export CSV** in the top right corner to export transactions and invoices as a CSV file. ## Submit a ticket [#submit-a-ticket] To ask Agora support a question, follow these steps: 1. Log in to [Agora Console](https://console.agora.io/v2). 2. Click **Get Support** > **Create Support Ticket** ![Create a support ticket](https://assets-docs.agora.io/images/common/console-create-support-ticket.png) 3. Type in your question or keywords to see if the question has been answered. If you cannot find your answer, select a category and submit a ticket to Agora’s customer support. ![](https://web-cdn.agora.io/docs-files/1658716858508) You can track the status of your ticket under **Tickets**. ## Delete your Agora account [#delete-your-agora-account] This section describes how to delete your Agora account created with an email address or phone number. Before deleting your Agora account, ensure that the following requirements are met: * You are the creator of the account to be deleted. * The account to be deleted is an Agora account created in [Agora Console](https://console.agora.io/v2), not a reseller account created in [Agora Reseller Console](https://reseller.agora.io/). * There are no active projects under your account. If there are any, go to the [Projects](https://console.agora.io/v2/project-management) page to disable all active projects. * Your account balance is zero. If your account balance is positive, request a withdrawal. If your account balance is negative, add money to your account. * Your bill for the previous month has been issued. * No usage is generated in the current month. * All of your packages have either expired or been used up, including: * Video SDK and Cloud Recording packages. * Support packages. * Extensions Marketplace packages. * There are no members under your account. If there are any, go to the [Teams and Members](https://console.agora.io/v2/teams-members) page to delete all members. ### Procedure [#procedure-1] Follow these steps to delete your Agora account: 1. Log in to [Agora Console](https://dashboard.agora.io/), click your account name in the top right corner and select **Settings** in the dropdown. 2. In the left navigation panel, click **Account**. ![Delete an account](https://assets-docs.agora.io/images/common/console-delete-account.png) 3. Ensure that you meet all the prerequisites, and click **Delete Account** at the bottom of the page. 4. Read the pop-up prompt carefully and fill in the confirmation field. Click **Delete Account**. If any prerequisite is not met, an error message appears after you click **Delete Account**. Check the requirements above and try again. # Conversational AI (/en/introduction/conversational-ai) Conversational AI is the capability layer for products where the primary interface is a live conversation with an AI system. In this category, voice is not just an input channel. It is the product experience itself. Agora groups several related surfaces under this layer: * **Conversational AI Engine** for production voice-agent workflows * **AI developer toolkits** for client and server integration * **OpenAI Realtime integration** for teams that want to combine Agora transport with OpenAI Realtime-style model interaction ## What this category is good for [#what-this-category-is-good-for] This category is the right entry point when you need: * voice-first AI assistants and copilots * customer service and sales agents * tutors, role-play systems, and spoken learning experiences * AI-powered smart devices and hardware endpoints * multimodal products where an agent participates in a live session The common requirement is natural turn-taking, low response latency, interruption handling, and a live connection between the user and the AI system. ## Products and paths in this category [#products-and-paths-in-this-category] The main product path for building realtime voice agents with managed speech, reasoning, interruption handling, and agent lifecycle control. Client and server integration surfaces for web, mobile, and backend teams that need to build custom AI experiences on top of Agora. A guided path for teams that want to connect Agora realtime transport with OpenAI Realtime-style voice interaction flows. ## Start here [#start-here] ## Key characteristics [#key-characteristics] Keep spoken interaction responsive enough for natural back-and-forth conversation rather than turn-based chatbot behavior. Combine realtime audio transport with ASR, LLM, TTS, interruption handling, and agent lifecycle control in one interaction model. Start from Agora-managed paths or integrate custom LLM, ASR, and TTS combinations depending on control and quality requirements. Build across web, mobile, backend, and device surfaces with a shared mental model for sessions, events, and agent behavior. Add custom data, event subscriptions, server-side orchestration, and downstream analytics without redesigning the whole interaction loop. Reuse Agora's voice path and session model so AI can participate as part of a live system, not only as a detached API call. ## What to read next [#what-to-read-next] * [Conversational AI](/en/ai) * [AI quickstart](/en/ai/get-started/quickstart) * [OpenAI Realtime](/en/ai/reference/openai-realtime-integration) * [API Reference](/en/api-reference/conversational-ai) # Core concepts (/en/introduction/core-concepts) RTC (Real-Time Communication) refers to real-time communication technology, that allows almost instant exchange of audio, video, and other data between the sender and the receiver. Agora SDKs provide real-time audio and video interaction services, with multi-platform and multi-device support. This includes high-definition video calls, voice-only calls, interactive live streaming, as well as one-on-one and multi-group chats. This article introduces the key processes and concepts you need to know to use Agora SDKs. ## Using the Agora Console [#using-the-agora-console] To use Agora SDKs, create an audio and video project in the Agora Console first. See [Agora account management](/en/introduction/account) for details. ![Create project in Agora Console](https://assets-docs.agora.io/images/common/create-project.svg) #### Agora Console [#agora-console] [Agora Console](https://console.agora.io/v2) is the main dashboard where you manage your Agora projects and services. Agora Console provides an intuitive interface for developers to query and manage their Agora account. After registering an [Agora Account](https://console.agora.io/v2), you use the Agora Console to perform the following tasks: * Manage the account * Create and configure Agora projects and services * Get an App ID * Manage members and roles * Check call quality and usage * Check bills and make payments * Access product resources Agora also provides RESTful APIs that you use to implement features such as creating a project and fetching usage numbers programmatically. ## General concepts [#general-concepts] Agora uses the following basic concepts: ### App ID [#app-id] The App ID is a random string generated within [Agora Console](https://console.agora.io/v2) when you create a new project. You can create multiple projects in your account; each project has a different App ID. This App ID enables your app users to communicate securely with each other. When you initialize Agora Engine in your app, you pass the App ID as an argument. The App ID is also used to create the authentication tokens that ensure secure communication in a channel. You [retrieve your App ID](https://console.agora.io/v2/project-management) using Agora Console. Agora uses this App ID to identify each app, provide billing and other statistical data services. For applications requiring high security in a production environment, you must choose a **App ID + Token** mechanism for user authentication when creating a new project. Without an authentication token, your environment is open to anyone with access to your App ID. ### App certificate [#app-certificate] An App certificate is a string generated by Agora Console to enable token authentication. It is required for generating a Video SDK or Signaling authentication token. To use your App certificate for setting up a token server, see [Deploy a token server](/en/realtime-media/video/build/authenticate-users/deploy-token-server). ### Token [#token] A token is a dynamic key that is used by the Agora authentication server to check user permissions. You use Agora Console to generate a temporary token for testing purposes during the development process. In a production environment, you implement a token server in your security infrastructure to control access to your channels. After obtaining the App ID, App Certificate, and Token in the Agora Console, you can start implementing basic audio and video communication in your app. ### Channel [#channel] Agora uses the *channel name* to identify a channel. Users who specify the same *channel name* join a common channel and interact with each other. A channel is created when the first user joins. It ceases to exist when the last user leaves. You create a channel by calling the methods for transmitting real-time data. Agora uses different channels to transmit different types of data. The Video SDK channel transmits audio or video data, while the Signaling channel transmits messaging or signaling data. The Video SDK and Signaling channels are independent of each other. Additional components provided by Agora, such as On-Premise Recording and Cloud Recording, join the Video SDK channel and provide real-time recording, transmission acceleration, media playback, and content moderation. ### Channel profile [#channel-profile] The SDK applies different optimization methods according to the selected channel profile. Agora supports the following channel profiles: | Channel profile | Description | | ------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `COMMUNICATION` | This profile is suitable for one-on-one or group calls, where all users in the channel talk freely. | | `LIVE_BROADCASTING` | In a live streaming channel, users have two client roles: *host* and *audience*. The *host* sends and receives audio or video, while the *audience* only receives audio or video with the sending function disabled. | ### Stream [#stream] A stream is a sequence of digitally-encoded coherent signals that contains audio or video data. Users in a channel [publish](#publish) local streams and [subscribe](#subscribe) to remote streams from other users. ### Publish [#publish] Publishing is the act of sending a user's audio or video data to the channel. Usually, the published stream is created by the audio data sampled from a microphone or the video data captured by a camera. You can also publish media streams from other sources, such as an online music file or the user's screen. After successfully publishing a stream, the SDK continues sending media data to other users in the channel. By publishing the local stream and subscribing to remote streams, users communicate with each other in real-time. ### Subscribe [#subscribe] Subscribing is the act of receiving media streams published by remote users to the channel. A user receives audio and video data from other users by subscribing to one or more of their streams. You either directly play the subscribed streams or process incoming data for other purposes such as recording or capturing screenshots. ### User ID [#user-id] A User ID (UID) identifies a user in a channel. Each user in a channel should have a unique user ID. If you do not specify a user ID when the user joins a channel, a UID is automatically generated and assigned to the user. ### User role [#user-role] A user role is used to define whether users in the channel have permission to publish streams. There are two user roles: * Host: A user who can publish streams in a channel. * Audience: A user who cannot publish streams in a channel. Users with this role can only subscribe to remote audio and video streams. ### Connection (RtcConnection) [#connection-rtcconnection] The connection between the SDK and the channel. When you need to publish or receive multiple streams in multiple channels, a connection is used to specify the target channel. ### Agora SDRTN® [#agora-sdrtn] Agora's core engagement services are powered by its Software-Defined Real-Time Network (SDRTN®) that is accessible and available anytime, anywhere around the world. The software-defined network isn't confined by device, phone numbers, or a telecommunication provider's coverage area like traditional networks. Agora SDRTN® has data centers globally that cover over 200+ countries and regions. The network delivers sub-second latency and high availability of real-time video and audio anywhere on the globe. With Agora SDRTN®, Agora can deliver live user engagement experiences in the form of real-time communication (RTC) with the following advantages: * Unmatched quality of service * High availability and accessibility * True scalability * Low Cost # Firewall requirements (/en/introduction/firewall) To allow you to use Agora products in environments with restricted network access, Agora provides the following solutions: the firewall whitelist and the Agora cloud proxy. The following table lists the support of Agora products for the two solutions: | Agora Products | Firewall Whitelist | Agora Cloud Proxy | | ------------------------------------------ | :----------------: | :---------------: | | Video SDK (Native, third-party frameworks) | ✘ | ✔ | | Video SDK (Web) | ✔ | ✔ | | Signaling SDK (Native) | ✔ | ✘ | | Signaling SDK (Web) | ✔ | ✔ | | On-Premise Recording SDK | ✘ | ✔ | | Interactive Gaming SDK | ✘ | ✘ | * When using the firewall whitelist, add the domains and ports to the firewall whitelist, and do not set restrictions on IP addresses. * When using Agora cloud proxy, refer to [Connect to Agora through a restricted network](/en/realtime-media/video/build/manage-connection-and-quality/cloud-proxy) ### Video SDK (Web) [#video-sdk-web] Add the following destination domains and the corresponding ports to your firewall whitelist. #### Domains [#domains] ```text .agora.io .edge.agora.io .sd-rtn.com .edge.sd-rtn.com .ap.sd-rtn.com .statscollector.sd-rtn.com .webrtc-cloud-proxy.sd-rtn.com .rtnsvc.com .edge.rtnsvc.com .rtesvc.com .edge.rtesvc.com ``` To improve connectivity over public networks, we recommend adding the domains listed above to your firewall allowlist. #### Ports [#ports] | Destination ports | Port type | Operation | | -------------------------------------------------------------------------------------------------------- | --------- | --------- | | 80; 443; 3433; 3478; 4700 - 5000; 5668; 5669; 6080; 6443; 8667; 9667; 30011 - 30013 (for RTMP converter) | TCP | Allow | | 3478; 4700 - 5000 (2.9.0 or later); 10000 - 65535 (before 2.9.0) | UDP | Allow | ### Signaling SDK (Web) [#signaling-sdk-web] #### Message channel [#message-channel] For a message channel, you need to add the following content to the firewall whitelist: * **Domains**: ```text .edge.agora.io .edge.sd-rtn.com web-1.ap.sd-rtn.com web-2.ap.sd-rtn.com web-3.ap.sd-rtn.com web-4.ap.sd-rtn.com ap-web-1.agora.io ap-web-2.agora.io ap-web-3.agora.io ap-web-4.agora.io webcollector-rtm.agora.io logservice-rtm.agora.io rtm.statscollector.sd-rtn.com rtm.logservice.sd-rtn.com ``` * **Ports**: | **Destination port** | **Protocol** | **Operate** | | ---------------------- | ------------ | ----------- | | 443; 9591; 9593; 27387 | TCP | Allow | Info If you are using Signaling 1.x, also add port 9601. #### Stream channel [#stream-channel] For a stream channel, you need to add the following to your firewall whitelist: * **Domains**: ```text .agora.io .edge.agora.io .sd-rtn.com .edge.sd-rtn.com ``` * **Ports**: | **Destination port** | **Protocol** | **Operate** | | --------------------------------------------------------- | ------------ | ----------- | | 80; 3433; 4700 - 5000; 5668; 5669; 6080; 6443; 8667; 9667 | TCP | Allow | | 3478; 4700 - 5000 | UDP | Allow | ### Signaling SDK (Native) [#signaling-sdk-native] #### Message channel [#message-channel-1] For a message channel, you need to add the following content to the firewall whitelist: * **Domains**: ```text .agora.io ``` * **Ports**: | **Destination port** | **Protocol** | **Operate** | | ----------------------------------------------------- | ------------ | ----------- | | 443; 7384; 8443; 9130; 9131; 9136; 9137; 9140; 9141 | TCP | Allow | | 1080; 3000; 8000; 8130; 8443; 9120; 9121; 9700; 25000 | UDP | Allow | #### Stream channel [#stream-channel-1] For a stream channel, you need to add the following to your firewall whitelist: * **Ports**: | **Destination port** | **Protocol** | **Operate** | | -------------------- | ------------ | ----------- | | 4001 - 4150 | UDP | Allow | Info The target ports listed in this section may be adjusted according to actual conditions. If you encounter any issues, contact [rtm@agora.io](mailto\:rtm@agora.io). # Glossary (/en/introduction/glossary) ## A [#a] ### Agora Analytics [#agora-analytics] Agora Analytics is a site for developers to track and analyze the usage and quality of calls. ### Agora Cloud Backup [#agora-cloud-backup] Agora Cloud Backup is a backup cloud storage service used in cloud recording. If the recording service cannot upload the recorded files to the specified third-party cloud storage, then the service automatically and temporarily stores them in the backup cloud. ### Agora Console [#agora-console] Agora Console is a site for developers to manage Agora projects and services. ### App ID [#app-id] An app ID is a randomly generated string provided by Agora and is the unique identifier of an app. ### App certificate [#app-certificate] An app certificate is a randomly generated string provided by Agora for enabling token authentication. It is one of the required arguments for generating a token. ### Audience [#audience] Audience are users who do not have streaming permissions in a channel. An audience user can subscribe to remote audio and video streams, but cannot publish audio and video streams. For more information, see [user role](#user-role). ### Audience (becoming) [#audience-becoming] Becoming an audience describes a use-case within an Interactive Live Streaming channel (the channel profile is Live-Broadcast) when a host switches the user role and becomes an audience. ### Audio mixing [#audio-mixing] Audio mixing means combining multiple audio streams into one. ### Audio profile [#audio-profile] An audio profile includes the sample rate, encoding scheme, number of channels, and bitrate for encoded audio data. ### Audio route [#audio-route] The audio route is the pathway audio data takes through audio hardware components during playback. ## C [#c] ### Callee [#callee] A callee is a Signaling user who receives a [call invitation](#call-invitation). ### Caller [#caller] A caller is a Signaling user who sends a [call invitation](#call-invitation). ### Call invitation [#call-invitation] Call invitation is a communication protocol based on the peer-to-peer messaging functionality of the Agora Signaling SDK. Call invitation supports starting, ending, accepting, and refusing calls. ### Channel [#channel] In Agora's platform, a channel is a way of grouping users together and is identified by a unique channel name. Users who connect to the same channel can communicate with each other. A channel is created when the first user joins and ceases to exist when the last user leaves. ### Channel attribute [#channel-attribute] Channel attributes are tags added to Signaling channels, including the property name, property value, the ID of the last Signaling user who updated the attribute, and the time of the last update. ### Channel message [#channel-message] A channel message is a message that a Signaling user sends to all Signaling users in a channel. ### Channel profile [#channel-profile] The channel profile is a configuration that Agora uses to apply optimized algorithms for different real-time use-cases. ### Cloud proxy [#cloud-proxy] Cloud proxy is a proxy service that enables users to connect to Agora services through a firewall by using fixed IP addresses. ### CREC [#crec] CREC is a component provided by Agora for recording and saving voice and video calls and interactive streaming on a third-party cloud storage through RESTful APIs. ### Co-hosting [#co-hosting] Co-hosting describes a use-case with more than one host. ### Composite recording mode [#composite-recording-mode] Composite recording mode generates a single mixed audio and video file for all UIDs in a channel. ### Custom rendering [#custom-rendering] Custom rendering is the process where developers collect raw data from the SDK and process it according to specific needs. ### Custom source [#custom-source] Custom source is the process where an app captures raw data by itself. ## D [#d] ### Delay [#delay] In real-time audio and video communication, delay refers to the time elapsed from when the data is sent to when it is received. ### Dual-stream mode [#dual-stream-mode] In the dual-stream mode, the Video SDK simultaneously transmits a higher-resolution video stream along with an additional low-resolution, low bitrate video stream. ## F [#f] ### Freeze [#freeze] Freeze refers to choppy audio or video playback caused by a poor network connection or limited device performance during real-time audio and video communication. ## H [#h] ### High-quality video stream [#high-quality-video-stream] In dual-stream mode, the SDK transmits two video streams of differing quality at the same time. See [dual stream mode](#dual-stream-mode) for details. ### Host [#host] The host refers to a user who has streaming permissions in a channel. A host can publish audio and video. A host may also subscribe to audio and video published by other hosts. ### Host (becoming) [#host-becoming] Becoming a host describes a use-case within an Interactive Live Streaming channel (the channel profile is Live-Broadcast) when an audience switches the user role and becomes a host. ## I [#i] ### Individual recording mode [#individual-recording-mode] Individual recording mode records audio and video of each UID as separate files. ### Inject online media stream [#inject-online-media-stream] Inject online media stream refers to injecting an online media stream in an Interactive Live Streaming channel to share the stream with all users in the channel. The Agora Video SDK provides a method for developers to inject an online mixed audio and video stream or an audio only stream to a channel. ### Interactive Live Streaming [#interactive-live-streaming] Enabled by either Agora’s Video SDK or Voice SDK, Interactive Live Streaming gives you full control over the streaming experience from a standard one-to-many stream to a highly-interactive live event. ## J [#j] ### Jitter [#jitter] In real-time audio and video communication, jitter is the variation in the delay of data packets transmitted continuously on the network. ## L [#l] ### Last mile [#last-mile] The last mile refers to the network between the Agora edge server and the end user's device. ### Loopback test [#loopback-test] A loopback test sends a signal from a communication device and is then returned (looped back) to it. It is often used to determine whether a device is working properly. ### Low-quality video stream [#low-quality-video-stream] In dual-stream mode, the SDK transmits two video streams of differing quality at the same time. The low-quality video stream has a lower resolution and bitrate than the high-quality video stream. See [dual stream mode](#dual-stream-mode) for details. ## M [#m] ### MediaPlayer kit [#mediaplayer-kit] The mediaplayer kit is a plug-in of the Video SDK to play local and online media resources and publish the media streams to other users in an Interactive Live Streaming channel. ### Media stream [#media-stream] A media stream is an object that contains media data. ### MPUSH [#mpush] MPUSH enables you to upload audio and video streams from Agora channels and upload them to a Content Delivery Network (CDN) to reach a larger audience. ### Mirror [#mirror] Mirroring is an effect that a video image renders. ## O [#o] ### Offline [#offline] Offline describes the status of a Signaling user who has successfully logged out of Signaling. ### Offline message [#offline-message] An offline message is a peer-to-peer message that an online Signaling user sends to an offline Signaling user. ### Online [#online] Online describes the status of a user who has successfully logged in to the Agora Signaling system or stays disconnected from the Agora Signaling system for more than 30 seconds. ### OPREC [#oprec] OPREC is a component provided by Agora for recording and saving voice and video calls and interactive streaming on a Linux server. ## P [#p] ### Packet loss [#packet-loss] Packet loss refers to the data packets transmitted on the network failing to arrive at their intended destination. ### Peer-to-peer message [#peer-to-peer-message] A peer-to-peer message is a message that an online Signaling user sends to an online or offline user. ### Publish [#publish] Publishing is the action of sending the user's audio and/or video data to the channel. ## R [#r] ### Raw data [#raw-data] Raw data, including raw audio data and raw video data, is the unprocessed data which developers can collect during real-time communication. ### Render the first video frame [#render-the-first-video-frame] Rendering the first video frame is the action of rendering the first video frame on the local device. ## S [#s] ### Agora SDRTN® [#agora-sdrtn] Software-Defined Real-Time Network (SDRTN®) is a real-time transmission network built by Agora and is the only network infrastructure specifically designed for real-time communications in the world. ### Signaling SDK [#signaling-sdk] You use the Signaling SDK to implement real-time messaging use-cases that require low latency and high concurrency for a global audience. ### Slice [#slice] Slicing means cutting recorded audio or video into separate files according to specific rules. During an Agora CREC, the recording service cuts the streams and generates multiple slice files (TS or WebM files) and M3U8 files that serve as a playlist of the slice files. ### Sound localization [#sound-localization] Sound localization means determining the distance to and direction of a sound through hearing the difference of volume, time, and timbre between users' ears. ### Stream fallback [#stream-fallback] In use-cases where multiple users engage in real-time audio and video communication, user experience can be impaired if the network condition is too poor to guarantee both audio and video at the same time. ### Stream mixing [#stream-mixing] Stream mixing means combining multiple media streams into one. It may include the mixing of video streams (video mixing) and audio streams (audio mixing). ### Subscribe [#subscribe] In the Agora Video SDK, subscribing is the action of receiving media streams published to the channel. In the Agora Signaling SDK, subscribing is the action of monitoring the online status of one or multiple Signaling users. ## T [#t] ### TCP [#tcp] TCP (Transmission Control Protocol) is a connection-oriented and reliable transport layer communication protocol. ### Token [#token] A token, also known as a dynamic key, is used for authentication when an app user joins a channel or logs onto the Agora Signaling. ### Transcoding [#transcoding] Transcoding is the process of decoding audio and video data and then re-encoding them into the target conversion output or format. ## U [#u] ### UDP [#udp] UDP (User Datagram Protocol) is a connectionless-oriented and unreliable transport layer communication protocol. ### User attribute [#user-attribute] User attributes are tags added to Signaling users, including property names and property values. ### User ID (uid) [#user-id-uid] In the Agora Video SDK, a user ID identifies a user in the channel. The user ID is a 32-bit signed integer, with a value range from -231 to 231-1, that you can specify yourself. If you specify `0` for the user ID when joining a channel, the SDK generates a random number and returns the value in the join channel success callback. In the Agora Signaling SDK, a user ID identifies a user in Signaling. The user ID in the Agora Video SDK and the Agora Signaling SDK are independent of each other. ### User role [#user-role] The type of user role determines whether the user in the channel has streaming permissions. ## V [#v] ### Video layout [#video-layout] Video layout arranges the display of users when multiple users are mixed into one stream, such as in Media Push or a composite recording. ### Video mixing [#video-mixing] Video mixing means combining multiple video streams into one. ### Video profile [#video-profile] The video profile refers to a set of video attributes, such as resolution, bitrate, and frame rate. ### Video SDK [#video-sdk] An SDK developed by Agora to enable developers to add real-time audio interaction to their projects. ### Voice SDK [#voice-sdk] Agora provides the Voice SDK to enable real-time audio communication. ## W [#w] ### Web page recording mode [#web-page-recording-mode] In web page recording mode, the content and audio of a specified web page are recorded in a single file. # Agora Documentation (/en/introduction) ## Build an Agent [#build-an-agent] ## Build Real-Time Applications [#build-real-time-applications] ## Start with AI Tools [#start-with-ai-tools] ## Realtime use cases [#realtime-use-cases] ## SDKs and APIs [#sdks-and-apis]
## Get Help [#get-help] # Members & Roles (/en/introduction/members-roles) ## Manage members and teams [#manage-members-and-teams] Once you add members to your Agora account, you are assigned to the **Admin** team and can add these members to different teams with specified permissions. ### Add a member [#add-a-member] 1. Log in to Agora Console, click your account name in the top-right corner, and click **Settings**. 2. In the left navigation panel, click **Teams and members**, then select the **Members** tab. 3. Click **Add New Member**, fill in the email address of the new member, and choose a team from the dropdown. 4. Click **OK**. 5. Agora sends a confirmation email to this address. The new member must follow the instructions in the email to finish joining the project. ### Manage members [#manage-members] On the **Teams and members** page, **Members** tab, you can do the following: * Change the team that a member belongs to. * Delete a member. Only the main account can delete a member account. Member accounts assigned to the **Admin** team cannot delete a member account. ### Manage teams and permissions [#manage-teams-and-permissions] On the **Teams and members** page, **Teams** tab, you can view the permissions assigned to teams. Agora predefines the following teams: * **Admin**: full access to all projects; can view usage data and finance information, manage members and teams, manage projects, and view Analytics reports. * **Finance**: can view finance information for all projects. * **Product/Operation**: can view usage data for all projects. * **CS/Maintenance**: can view Analytics reports for all projects. * **Engineer**: can manage projects and view Analytics reports for all projects. ### Add a custom team [#add-a-custom-team] 1. On the **Teams and Members** page, **Teams** tab, click **Add New Team**. 2. Fill in the team name. 3. Select the permissions of this team in the **Usage**, **Finance**, **Teams and Members**, **Project**, **Analytics**, and **Data** columns. 4. Click **OK**. # Messaging & Presence (/en/introduction/messaging-presence) ![Signaling overview](https://assets-docs.agora.io/images/signaling/signaling-overview.png) Messaging & Presence is the capability layer for products that need more than media. This category covers human messaging, participant presence, room state, and shared collaboration surfaces that keep live systems coordinated. Agora's product map for this layer includes: * **IM / Chat** * **RTM / Signaling** * **Interactive Whiteboard** ## What this category is good for [#what-this-category-is-good-for] This category is the right entry point when you need: * in-room chat and social messaging * user presence and online state * metadata synchronization and coordination signals * collaborative session control and shared state * whiteboard-backed visual collaboration inside a live product The shared requirement is that participants must stay aligned with each other and with the state of the session in real time. ## Capability map [#capability-map] IM-oriented communication for user conversation, room interaction, and product workflows that depend on message exchange. RTM-oriented channels, topics, presence, metadata, and coordination events that support realtime systems. Shared visual state for drawing, annotation, presentation, and collaborative interaction around content. ## Start here [#start-here] ## Key characteristics [#key-characteristics] Support both user-facing communication and the metadata flows that keep live products synchronized. Track who is online, what state they are in, and how session behavior should react as users and devices change status. Extend from text and events into shared whiteboard or content collaboration when the product needs a richer interaction layer. Combine easily with RTC sessions and AI agents when messages and events need to travel alongside audio and video. Choose between IM-style user communication and RTM-style signaling depending on whether the main problem is conversation, coordination, or both. Keep users, agents, and session tools operating against the same live room state instead of fragmenting the product experience. ## What to read next [#what-to-read-next] * [Realtime Messaging RTM](/en/realtime-media/rtm) * [Instant Messaging IM](/en/realtime-media/im) * [Interactive Whiteboard](/en/realtime-media/whiteboard) # Projects (/en/introduction/projects) ## Why this page matters [#why-this-page-matters] In Agora, a project is the operational boundary for a product environment. It holds the App ID, certificate configuration, enabled services, and billing context that the rest of your integration depends on. If your project setup is wrong, the rest of the docs may still be correct but your implementation will fail later through missing credentials, disabled services, or the wrong security mode. ## What you do here [#what-you-do-here] Use the **Projects** page in Agora Console to: * create a new project for a product or environment * choose the authentication mode for that project * copy the App ID used by clients and services * manage App Certificates for token-based access * check whether the project is configured the way your integration expects ## Recommended setup [#recommended-setup] For most teams: * create separate projects for development, testing, and production * use **App ID + Token** instead of App ID only * keep certificate management on the server side * confirm the correct project is being used before debugging client or backend code ## Create a project [#create-a-project] 1. Open the **Projects** page in Agora Console. 2. Click **Create New**. 3. Enter a project name and use case. 4. Select **App ID + Token (Recommended)** as the authentication mechanism. 5. Click **Submit**. ## What to check in project details [#what-to-check-in-project-details] On the project details page, confirm that you can: * copy the App ID used by your application * view whether token-based security is enabled * access the App Certificate when your backend needs token generation * verify that the project is the one your current environment should use ## Service access [#service-access] To access most Agora products you need: * An App ID. * An App Certificate if your backend generates tokens. * A project with the required service enabled. For production, use token-based authentication instead of App ID only. ## What to verify before launch [#what-to-verify-before-launch] Before moving beyond a prototype, confirm: * the required service is enabled for the correct project * the authentication mode matches your intended production design * the billing unit makes sense for your usage pattern * the right people on your team can access billing and financial records ## Manage App Certificates [#manage-app-certificates] Agora provides two certificate roles: * **Primary Certificate** for normal token generation * **Secondary Certificate** for production rotation and controlled cutovers If you suspect a certificate is compromised: 1. Enable the Secondary Certificate. 2. Swap the certificates. 3. Disable the old secondary certificate. 4. Delete the original primary certificate after clients and services have moved over. Deleting a certificate invalidates all tokens generated with that certificate. ## Restrictions [#restrictions] * If your account has multiple members, only those assigned to the **Admin**, **Engineer**, or an authorized custom team can access the **Projects** page. * Each Agora account can create up to 20 projects. If you need to create more projects, submit a support ticket. ## Where to go next [#where-to-go-next] * Open [Console setup](/en/introduction/console-setup) to understand the first-time console workflow around credentials and service activation. * Read [Billing](/en/introduction/billing) to understand free allowances, settlement, and package options before launch. * Read [Security and privacy](/en/introduction/security-privacy) before production rollout. # Real-Time Voice & Video (/en/introduction/realtime-audio-video) ![Video Calling overview](https://assets-docs.agora.io/images/video-calling/video-calling-overview.png) Real-Time Audio & Video is the capability layer for products where live media is the primary user experience. This category covers two-way voice and video interaction, host-audience live experiences, large-scale broadcast delivery, and device-oriented realtime connectivity. Agora's product map for this layer includes: * **Voice Calling** * **Video Calling** * **Interactive Live Streaming** * **Fusion CDN broadcast delivery** * **IoT / RTSA edge-device connectivity** ## What this category is good for [#what-this-category-is-good-for] This category is the right entry point when you need: * one-to-one or group voice and video calling * meetings, classrooms, and telehealth sessions * creator-led or host-audience live interaction * broadcast-style delivery to larger audiences * smart-device, camera, or edge-terminal media connectivity The shared requirement is stable, low-latency media transport with clear session behavior across networks, devices, and participant roles. ## Capability map [#capability-map] Low-latency voice interaction for calling, social audio, customer communication, and lightweight live sessions. Realtime face-to-face interaction for meetings, classrooms, telehealth, and collaboration products. Host-audience interaction for creator events, live communities, and interactive broadcast-style products. Fusion CDN-based distribution for larger viewer populations and playback-oriented live delivery. RTSA-based device connectivity for smart cameras, displays, embedded endpoints, and IoT-oriented realtime media. ## Start here [#start-here] ## Key characteristics [#key-characteristics] Use the same realtime transport model across calling, live interaction, broadcast expansion, and device connectivity. Support peer-to-peer, group, host-audience, and device-oriented topologies without switching to a different conceptual model. Start from interactive sessions, then extend into larger audience delivery or downstream cloud media services. Reach web, mobile, desktop, and edge-device products with consistent session and media semantics. Combine media transport with messaging, AI, transcription, recording, and analytics as product requirements grow. Keep the live experience usable under real-world packet loss, geography, and device constraints. ## What to read next [#what-to-read-next] * [RTC](/en/realtime-media/rtc) * [Fusion CDN](/en/realtime-media/fusion-cdn) * [RTSA](/en/realtime-media/rtsa) # Security & Privacy (/en/introduction/security-privacy) ## Information security policy [#information-security-policy] This section provides information security policy for Agora services. Agora services provide built-in encryption and customized encryption. You can use either of them to implement encryption. The following diagram describes the encrypted data transmission process: ![Encrypted data transmission process](https://assets-docs.agora.io/images/common/encrypted-data-transmission-process.svg) #### Purpose [#purpose] Agora is committed to safeguarding the confidentiality, integrity, and availability of all users' physical and electronic information assets. * Confidentiality against unauthorized access and eavesdropping * Integrity against tampering and forgery * Availability of data transmission through the Agora SDRTN® #### Scope [#scope] This article describes how Agora protects customer data with security controls. ### Data classification [#data-classification] All customer data, in all formats or media types, is classified according to the following categories and protected accordingly. | Category | Description | | --------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Customer Account Data | Information related to the customers' Agora accounts, including customer ID, customer IP, network type, operating system, email address, telephone, product interest, programming platform, UTM information, identity, company name, company URL, billing information, trade information, purchased package information, project status, ticket, member information and console access records. | | End User Data | Information collected by customers related to end users' personal devices and network, including microphone and camera information, CPU status, memory status, battery status, system version, phone model, phone signal level, received signal strength indicator (RSSI), network type, user attributes and channel attributes. | | Call Content | Audio and video data of the user during a call. | | Log | Media server logs generated by the Agora servers when accessing the Agora SDRTN®. Media server logs do not contain text messages or personal information. | ### Data security [#data-security] The communication between the user and the Agora server is protected by transmission protocols, such as the Agora private transmission protocol, Transport Layer Security (TLS) and WebSocket Secure (WSS). You can also use Advanced Encryption Standard (AES) or a customized encryption algorithm for the encryption of audio and video data. During data transmission, the Agora SDRTN® does not transmit any encryption key information. Call content information can only be decrypted on the terminal device (such as the client app and the customer's on-premise recording server) through the client authorization key. ### Data availability [#data-availability] * Large and distributed data centers: Agora has multiple data centers providing services globally, and any attack on one data center cannot affect others. * Rapid recovery: When a data center is subjected to malicious attacks that are difficult to prevent, such as a distributed denial-of-service (DDoS) attack, Agora will automatically isolate the data center and avoid affecting users' services. * DDoS attack prevention: Agora has deployed anti-DDoS firewalls in each core cloud data center. Agora has more than two hundred distributed data centers around the world, which guarantees sufficient capabilities and resources to control the risk of DDoS attacks. ### Data storage [#data-storage] Agora provides customers with the Agora On-Premise Recording SDK and Agora Cloud Recording, enabling customers to record part or all of the call contents. When using the recording services, all recorded video or audio files are stored on the storage server provided by the customer. ### Access authorization [#access-authorization] End users can access the Agora SDRTN® using a dynamic key. For details, see [Secure authentication with tokens](/en/realtime-media/video/build/authenticate-users/authentication-workflow). ### Electronic access controls [#electronic-access-controls] Agora strictly controls the data access on all internal systems. All users have independent internal accounts and authorization systems, and must pass two-step verification. All access records are recorded. All servers involving user data are strictly audited and protected. Agora employees will only access the production server when necessary, and only by obtaining temporary authorization. Operation records are kept for the whole process. ### Physical access controls [#physical-access-controls] All operating servers are hosted in computer rooms that meet ISO 27001 or above information security management certification standards, and all computer rooms protect information security in accordance with the escrow agreement or the Data Protection Association (DPA). Both third-party hosting providers and Agora employees must be approved before they can access servers and equipment. ### Roles and responsibilities [#roles-and-responsibilities] Agora sets the responsibilities for its own staff and customers. See the following sections for more information. ### Roles and responsibilities within Agora [#roles-and-responsibilities-within-agora] Security roles and responsibilities within Agora are categorized as follows: | Role | Responsibilities | | ---------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Information Security Sub-Committee | The Information Security Sub-committee (ISSC) is responsible for the development and implementation of policies and procedures. The ISSC monitors company adherence, and conducts regular technical and non-technical evaluations of Agora security policies. The ISSC also designates which employees are authorized employees. | | Authorized Employee | An authorized employee has access to the production machines required for the support and maintenance of the Agora service as determined by the ISSC. | | All Other Staff | All other staff are required to maintain confidentiality as required by their terms of employment and are required to immediately report any security breach. | ### Shared responsibilities [#shared-responsibilities] When evaluating the Agora SDRTN®, it is important for customers to understand and distinguish the following security measures: * The security measures that Agora implements and operates. * The security measures that customers implement and operate to ensure the security of call content information and applications when using the Agora SDRTN® services. Customers retain control of the security measures they choose to protect their personal information, platform, applications, systems, and networks. Agora customers should be responsible for all information they collect from users and themselves, such as application logins, identities, passwords, payment information, names, and addresses. Agora recommends adding obvious prompts before accessing the users' personal information. For example, add an Enable/Disable button with which the user can agree or disagree to share personal information. ### Security training [#security-training] Agora conducts information protection, security, and compliance training for all new employees. Agora also ensures that all employees receive information confidentiality training at least once a year. All authorized employees (employees who have access to the production machines) will receive additional training. ### Non-compliance [#non-compliance] Employees must comply with confidentiality agreements and internal security systems. If a non-compliant situation occurs, Agora will take the corresponding measures depending on the severity of the situation, including but not limited to having conversations with the at-fault employees, strengthening training and education, dissolving labor agreements, and pursuing other legal liabilities. ### Reporting potential security risks [#reporting-potential-security-risks] Please report any potential risks of Agora services that you may notice to [security@agora.io](mailto\:security@agora.io). Please use our PGP public key (Key ID: 2F4553BE) to encrypt your report, as potential security risks are usually sensitive. We assure you that our security experts will handle the issue immediately once they receive your report. To facilitate troubleshooting and verification, please include the following information in your email: * Your contact information. * The version of the impacted SDK or solution. * Description of the potential risks. * Miscellaneous technical details, such as your system configurations, methods or steps for reproducing the issue. To better protect our system and customers, Agora also invites security researchers to report any bugs or vulnerabilities they discover to us. Click [Bug Bounty](#agora-bug-bounty-program) to learn more. ### Security FAQ [#security-faq] To ensure the security of transmitted data, Agora services provide encryption for audio and video data. Customers can use AES-128/AES-256 or other algorithms preset by Agora, or use customized encryption algorithms. For details, see [Channel Encryption](/en/realtime-media/video/build/secure-and-protect-channels/media-stream-encryption). The encryption key is completely generated and distributed by the customer. Agora recommends that customers use separate keys for each channel to achieve the highest level of data security. For network protocol encryption, Agora SDKs use AES-128 mode to encrypt network payloads. For channel encryption, Agora SDKs support AES-128/AES-256 mode to encrypt audio and video. For those who have higher requirements for content security, Agora recommends using our customized encryption. Note that the On-Premise Recording SDK and Cloud Recording do not support customized encryption. For details, please send an email to [support@agora.io](mailto\:support@agora.io). Yes. Agora regularly scans core network nodes to check and clear possible security holes. Agora also configures anti-DDoS firewalls in each core cloud data center to protect them from attacks. Agora has more than two hundred distributed data centers around the world, which provides sufficient capabilities and resources to control the risk of DDoS attacks. When using Agora services, the audio and video data are transmitted through the Agora servers. Agora servers cache the audio and video data for 10 seconds during the transmission and release all audio and video data immediately after the call. ## Whitepaper [#whitepaper] People engage longer when they see, hear, and interact with each other. The future of meaningful human connections is made possible now with Agora’s Real-Time Engagement Platform. People rely on Agora’s Real-Time Engagement Platform to exchange millions of calls and messages, with vivid voice and video embedded in any application, on any device, anywhere. Agora, Inc. is proud to offer a professional Real-Time Engagement Platform as a Service (RTE PaaS) with Compliance, Safety, Security, and Trust. Our commitment to compliance, data and information security, and privacy protection is part of the core values of our company. From our Software-Defined Real-Time Network (SDRTN®) architecture to our day-to-day business operations, Agora continually invests in innovations and business processes that build trust with our customers, investors, and developer community. Agora works to high standards to follow the best security practices and comply with strict privacy regulations and standards as we respect the privacy of all our customers. The information contained in this document is intended to provide transparency in relation to Agora’s security stance and processes. If you think you may have found a security vulnerability within any of Agora’s services, please contact our security team directly at [security@agora.io](mailto\:security@agora.io). Agora strives to incorporate security into all our products and services and integrates the best security practices into everyday business operations. To meet these primary goals and improve the overall information security posture in an efficient and effective manner, Agora has built its security framework against the ISO/IEC 27001 Information Security Management Standard. As threats to information security continue to evolve, having dedicated security resources is essential. Agora’s Executive Security Committee meets regularly to address security concerns and coordinate company-wide security initiatives. The Agora Strategic Security Program Roadmap has been developed and approved by the committee to guide the implementation of our security programs. Our dedicated security team, led by the Chief Information Security Officer, has the responsibility for building and enforcing information security programs. Agora is always looking to better protect its systems and customers. Therefore, we continuously monitor and improve our information security programs by implementing the following: * Agora conducts an internal audit of its information security management system at least once a year to ensure effectiveness; * Agora follows industry best practice software development lifecycle management processes and conducts internal security reviews and testing before deployment into production systems; * Agora engages third-party security experts to carry out regular penetration tests; * Through our Bug Bounty Program, Agora works with security researchers to keep our customers' data more secure by identifying and reporting vulnerabilities in our products and services. Agora adheres to regional and international information security standards as well as industry requirements and is committed to using international best practices. We engage with independent third parties to verify the compliance of Agora. Certified by various reputable agencies across the globe, we are recognized by industry and security organizations for excellence. **ISO/IEC 27001 Information Security Management Standard** Agora is certified to ISO/IEC 27001:2022 by DNV GL, demonstrating our information security maturity level. Our security team implements the Information Security Management System in partnership with Ernst & Young. Security is a top priority at Agora, and this achievement demonstrates our commitment and continuous efforts to improve the efficiency of information security controls. Download the certificate: [ISO/IEC 27001:2022](https://docs.agora.io/en/assets/files/Agora_ISO_27001-d408e0245f77e7e09e0eef8527103483.pdf) **ISO/IEC 27018 Information Technology – Security techniques – Code of practice for protection of personally identifiable information (PII) in public clouds acting as PII processors** Agora is certified to ISO/IEC 27018 by DNV GL. This standard is a Code of Practice for protecting personal data in the cloud environment. Agora continuously strives to protect our customers' sensitive data. Download the certificate: [ISO/IEC 27018:2019](https://docs.agora.io/en/assets/files/Agora_ISO_27018-0f20342310eefc1424b81ff98caf707d.pdf) **SOC 2 Report** Agora is confident in our security practices, and we continue to engage independent third parties to perform strict SOC 2 audits on our internal processes, security controls, and the design of Agora products. We meet the audit requirements set by the American Institute of Certified Public Accountants (AICPA) standards for security, availability, and confidentiality, and have obtained a SOC 2 report. **PCI DSS** Agora prioritizes the security and privacy of its customers. Since we are neither a merchant nor a service provider as defined by the PCI Security Standards Council, we do not need to conduct an annual PCI assessment. However, Agora ensures that all transactions and customer data are handled with the highest level of security, in alignment with industry best practices. Should you have any questions or require further information regarding our security measures, please feel free to contact us at [security@agora.io](mailto\:security@agora.io). **General Data Protection Regulation - GDPR** Agora is aligned with GDPR and we are committed to providing GDPR-compliant products and services to our customers in the EU region or with our customers who conduct business within the EU. **Health Insurance Portability and Accountability Act - HIPAA** Agora is aware of the sensitivity of transmitting and processing health information and we have invested in both the creation and ongoing maintenance of a HIPAA compliance program. **California Consumer Privacy Act - CCPA** The CCPA is the first comprehensive privacy law in the United States that aims to provide a variety of privacy rights to California consumers. As a service provider, Agora is aligned with CCPA through the implementation of our security programs. **Children’s Online Privacy Protection Act - COPPA** The COPPA regulates the privacy protection requirements for children under the age of thirteen. Agora has engaged privacy experts in meeting the requirements of COPPA. Agora strongly believes in the principles of Secure by Design and Defense in Depth. Therefore, Agora adopts industry-recognized security standards and best security practices at every layer, from infrastructure to application, to perfect our products and environment and to secure the organization and our customers. Securing access to your environment starts with identity and access controls. Agora provides you with a solution to ensure that only authorized people can access your services and resources. The Agora Console is a role-based access control tool that you use to restrict access based on the "need-to-know" principle. The console is an interactive interface where you can easily create accounts, remove members, and assign roles and permissions. This tool can help you enforce your security policies. Furthermore, Agora provides static key, dynamic key, and hybrid authentication methods to secure the communication channels in different use cases. At Agora, you choose how your content is secured. We offer you various options for your content in transit and provide you with full control of your own encryption keys. These features include: * Agora Software Development Kits (SDKs) provide built-in encryption algorithms, including AES-128 and AES-256, to protect all data transmitted between the end users and Agora services. * We also provide you with the choice to use your own encryption algorithm to protect users’ media streams during real-time engagement. The encryption key is completely under your control. * The communication between the end users and Agora network (that is, Agora SDRTN®) is protected by encrypted transmission protocols such as the Agora Private Transmission Protocol, Transport Layer Security (TLS) and WebSocket Secure (WSS). Agora offers our customers the ability to record real-time communication with Agora On-Premise Recording SDK and Agora Cloud Recording SDK. The recording files can be stored on users’ local devices or in a designated cloud storage service chosen by our customers. The local recordings or cloud recordings can be further encrypted through any encryption form of your choosing. Agora does not store any streaming data or user data except for caching for transmission purposes. The cached streaming data of users will be immediately released after the service. Data centers hosting Agora services are maintained by certified and industry-leading cloud service providers, offering state-of-the-art physical protection for the servers and infrastructure that comprise the Agora environment. The production environment, where all our customers' data and functional servers reside, is completely separated from our internal organization network, including the development and testing environments. This guarantees that all our customers' data will stay in the production environment and never be used for development or testing purposes. Access to the Agora production and non-production network is minimized to the greatest extent. Agora also implements network segmentation in the production network based on various factors, such as the type of business, the criticality of data, and potential risks, to secure sensitive customer data. **DDoS prevention** Agora regularly scans our core network nodes in the production environment to check and clear potential security vulnerabilities. Anti-DDoS firewalls are configured in each core cloud data center for protection. With more than two hundred distributed data centers around the world, Agora can provide sufficient capabilities and resources to minimize the impact of DDoS attacks and ensure high availability of real-time video and audio anywhere around the globe. **Monitoring, logging and analysis** Agora continuously monitors and analyzes log events to gain a comprehensive view of the security state of our production environment. The logging covers both successful and unsuccessful security events, with an emphasis on the event data of critical infrastructure. To provide customers with better visibility and security insights, Agora Analytics is made available to consumers as a tracking and analysis tool. This tool enables customers to efficiently locate quality issues and identify root causes for a better end user experience. The tool contains the Real-Time Alarm function, which enables you to monitor call quality and informs you in real time when the user’s communication experience is below expectation. With Real-Time Alarm, you can undertake the following actions in real time: * Monitor users who are having a poor communication experience. * Identify abnormalities, analyze quality factors, and locate the source of an abnormal issue. **Network geo-fencing** Agora has embedded Network Geo-Fencing in Real-Time Voice, Video and Messaging SDKs to ensure your data is protected against rising concerns about network security and privacy breaches. Agora Network Geo-Fencing establishes a virtual boundary within Agora SDRTN®, and you have the choice to restrict your network traffic to one or more designated regions. **Network redundancy** Agora has more than 200 data center POPs (Points of Presence) across the world, covering the United States, Europe, China, Japan, India, the Asia-Pacific region, and other areas. The POPs in the SDRTN® network adopt the full mesh topology with superior routing capabilities. This ensures that the network services are not interrupted due to a single point of failure. The POPs build fault tolerance and disaster recovery capabilities for Agora services across regions. POPs also measure the performance of every possible path through the global network to find the “optimized” paths to ensure high data packet delivery success rate within the smallest time window. Agora provides expert guidance to our customers on how to leverage our security features and embed best practices into every layer of your application. Agora is continuously monitoring, auditing, and improving the design and operating effectiveness of our security controls. These activities are regularly performed by both third-party credentialed assessors and Agora’s internal risk and compliance team. Audit results are shared with senior management and all findings are tracked to resolution in a timely manner. In addition to third-party security compliance audits, Agora engages Trustwave SpiderLabs to conduct network penetration tests at least annually. Results of the penetration testing are shared with senior management and are triaged, prioritized, and remediated in a timely manner. Agora customers may receive executive summaries of these activities by requesting them from their account managers. Data security and user privacy are the top priorities of Agora. Agora is committed to building a professional RTE PaaS with Compliance, Safety, Security, and Trust. It is a critical responsibility for Agora to help ensure the confidentiality, integrity, and availability of systems and data, and Agora continues to work hard to maintain that trust. If you have any questions or concerns, please contact our security team or account managers. ## ISO certificates [#iso-certificates] Security and compliance are basic requirements for real-time interaction. Agora complies with the compliance requirements of different countries and industries to create safe and reliable cloud services. The ISO certification of Agora has passed the supervision and accreditation of three different certification organizations: IFA CNAS (China National Accreditation Service for Conformity Assessment), UKAS (United Kingdom Accreditation Service), and DNV (Det Norske Veritas). Agora builds cloud service technology worldwide in accordance with internationally recognized standards to ensure the information security of global customers. #### ISO/IEC 27001:2022 [#isoiec-27001] ISO/IEC 27001:2022 is the most authoritative, widely accepted, and applied system certification standard in the field of information security. This system covers information security management during the global operation and maintenance of Agora's cloud platform for real-time engagement. Download the certificate: [ISO/IEC 27001:2022](https://docs.agora.io/en/assets/files/Agora_ISO_27001-d408e0245f77e7e09e0eef8527103483.pdf) #### ISO/IEC 27017:2015 [#isoiec-27017] ISO/IEC 27017:2015 is a supplementary standard to ISO 27001, which provides information security implementation specifications for cloud service providers. Download the certificate: [ISO/IEC 27017:2015](https://docs.agora.io/en/assets/files/Agora_ISO_27017-b32474dd5f07aff909b250691670509c.pdf) #### ISO/IEC 27018:2019 [#isoiec-27018] ISO/IEC 27018:2019 is a supplementary standard to ISO 27001, which provides implementation specifications for the protection of personal information for cloud service providers. Download the certificate: [ISO/IEC 27018:2019](https://docs.agora.io/en/assets/files/Agora_ISO_27018-0f20342310eefc1424b81ff98caf707d.pdf) #### ISO/IEC 27701:2019 [#isoiec-27701] ISO/IEC 27701:2019 is a privacy extension to ISO/IEC 27001 information security management and ISO/IEC 27002 security controls. It is an international management system standard that provides guidance on the protection of personal privacy, including how organizations should manage personal information. Download the certificate: [ISO/IEC 27701:2019](https://docs.agora.io/en/assets/files/Agora_ISO_27701-7b06782b01b6c2cdcc97d610ccbd802f.pdf) ## Security best practices [#security-best-practices] Security and compliance are essential for real-time engagements through technology. In order to provide safe and reliable cloud services, Agora adheres to the compliance requirements of different countries, regions, and industries, in addition to being certified to ISO/IEC 27001. For more information, see [ISO certificates](#iso-certificates). Agora products and services are designed and built with multiple protection measures against attacks commonly seen in the real-time engagement industry. This article describes some of the security best practices that Agora has adopted, as well as security tools it provides for developers, as follows: | Protection measures | Applied by default | Recommended use cases | | -------------------------- | ------------------ | ---------------------------------------------------------------------------------------------------------- | | Channel separation | Yes | All real-time use cases. | | Token-based authentication | No | All real-time apps in a production environment should use Token-based end-user authentication. | | Encryption | No | Real-time use cases that require confidentiality. | | Network geofencing | No | Real-time use cases where customers wish to restrict access to Agora servers to within a specified region. | ### Level 1 - Channel separation [#level-1---channel-separation] The channel architecture is the first built-in layer of protection. Agora creates an independent and isolated channel for each audio, video, or messaging data transmission. All channels are logically separated, and only authenticated users from the same App ID can join the same channel. ### Level 2 - Token-based authentication [#level-2---token-based-authentication] The second layer of protection is the authentication layer. This is implemented through dynamic Token-based authentication. The Token is a short-lived access key that is generated by the app backend and allows users to access the Agora platform after the user has been properly validated by the app. A Token is generated with important information such as App ID, user ID (`uid`), channel name, and expiration date. ![Authentication workflow](https://assets-docs.agora.io/images/chat/chat-authentication.svg) The app developer can enable Token-based authentication (App Certificate) on [Agora Console](https://console.agora.io/v2). When enabled, all users' requests to join a channel must use a valid Token. ![img](https://assets-docs.agora.io/images/certificate-enable.png) As a security best practice, set the Token expiration time (24 hours by default). A Token has three expiration timestamps: * Token expiration: How long a Token is valid for, that is, how long a user can stay in a channel. * Privilege expiration: * Join a channel: How long a Token can be used to join a channel. * Streaming privilege: Whether a user can send audio, video, or messaging to the channel, or join as an audience member. This privilege is not enabled by default. ### Level 3 - Encryption [#level-3---encryption] The next layer of security in the Agora platform is encryption. Agora supports transmission encryption and data encryption. To guarantee data confidentiality during transmission, Agora uses the AUT (Agora Universal Transport) encryption protocol, Agora's proprietary secured transport layer. Data encryption encrypts all the audio and video streams with a symmetric key and encryption controlled by the app developer. At this level, the app provides a symmetric encryption key to the local SDK libraries. The SDK encrypts all the captured media using the key and the configured AES-128/256 encryption. The data is sent encrypted to Agora SDRTN® and from there to the other endpoints in the channel. The receiving endpoint uses the key provided by the app layer to decrypt the media streams and send them to the renderers. With this method, only the application knows the keys. In the Native SDK (iOS, Android, macOS, Windows), the keys are not sent to Agora servers. ![Token encryption](https://assets-docs.agora.io/images/common/token-encryption.svg) When using other Agora services like Web SDK, Cloud Recording, Content Moderation, and Transcoding, encryption is done a bit differently and it is not end-to-end. In this case, media is still encrypted, but the Agora service needs to be aware of the key to be able to connect to the channel and provide the service. For example, in Web SDK, user/browser protection is provided through web server protection (HTTPS) as well as WebRTC standard security practices, such as encryption and key management. More information on WebRTC security can be found [here](https://webrtc-security.github.io/). Media encryption in Web SDK is based on the WebRTC standard, but interoperability with Agora is handled using Agora’s encryption engine, and the encryption key is passed securely to the Web SDK servers through APIs. The key is required because the Agora edge server converts from WebRTC protocols to Agora’s protocol and allows interoperability with Native SDK. A similar situation occurs with Cloud Recording, Content Moderation, and other services, where RESTful APIs are used to securely pass the key for the channel. ![img](https://assets-docs.agora.io/images/common/media-encryption.svg) ### Level 4 - Network geofencing [#level-4---network-geofencing] To conform to the laws and regulations of different countries and regions, the Agora Video SDK and the Signaling SDK support network geofencing, which limits the transmission of data to within a specified region. These SDKs support network geofencing in the following regions: global (default), North America, Europe, Asia (excluding Mainland China), Japan, India, and Mainland China. Once a customer specifies a region using geofencing, no audio, video, or message can access Agora servers outside that region. ### Security best practice checklist [#security-best-practice-checklist] Use this list to quickly check what measures you have or have not taken to best protect the security of your app and users: 1. [Enable Token-based authentication](/en/introduction/projects#manage-app-certificates) on [Agora Console](https://console.agora.io/v2). 2. Disable *No certificate* in your project management page. Once this is done, your app authenticates users with Tokens only. 3. [Deploy a Token server](/en/realtime-media/video/build/authenticate-users/deploy-token-server) in your backend services. 4. Protect the Token server and only allow the app backend server to connect to the Token server. 5. Set the Token expiration time to a reasonable value. See [Deploy a Token server](/en/realtime-media/video/build/authenticate-users/deploy-token-server). 6. If needed, contact [support@agora.io](mailto\:support@agora.io) to enable Token privilege and set Token per role. 7. For additional security, work with the Agora SA team if needed to customize the Token server and modify the Token join privilege to a short time, such as 5 minutes. This is more advanced and recommended after app development is completed. 8. Channel encryption key management: Generate a random key (secret) per channel at the time of creating the channel; do not reuse encryption keys. 9. Pass the channel encryption key (secret) securely outside the Agora engine to authenticated endpoints that are allowed to join the channel. 10. For additional security, you can also do the following: 1. Set the “Agora channel name” to a one-time random string managed by your app. 2. Set the Agora `uid` to a one-time numeric ID that is mapped to the user on the app level. Do not use users' real IDs in your app as the Agora `uid`. ## Agora bug bounty program [#agora-bug-bounty-program] Agora is always looking to better protect our system and customers. Therefore, we invite security researchers to report any bugs or vulnerabilities they discover to us. Once we receive notice of a bug or vulnerability, Agora customer service and security teams will respond quickly to address the issue. In addition to our gratitude, those who report a vulnerability may be eligible for a monetary “bounty” based on the risk associated with the vulnerability and the importance of the affected system. ### System importance classifications [#system-importance-classifications] The following table shows system importance classifications (in descending order), along with some example Agora assets: | Classification | Asset Examples | | -------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------- | | Core system | Agora SDRTN®, the mail system, and official websites such as [www.agora.io](http://www.agora.io/), [sso.agora.io](http://sso.agora.io/), and `api.agora.io` | | General system | Forums and the Developer Portal ([docs.agora.io](https://docs.agora.io/)) | | Fringe system | Test sites | ### Vulnerability severity classifications [#vulnerability-severity-classifications] The following list outlines detailed examples of how various vulnerabilities would be classified: **Critical** 1. Unauthorized system privileges Examples include command injection (execution), code injection (execution), web shell execution, SQL injection, and buffer overflow that gains system privileges on the core system. 2. Direct denial of service Examples include actions that make service unavailable, reduce service quality, and so on. 3. Sensitive information leakage Examples include SQL injection of the core database (identity, order), unauthorized disclosure of sensitive information relating to a user, product order, or payment method, and so on. 4. Serious logic design flaws and process defects Examples include the ability to send batches of fraudulent messages, account consumption through a business interface, and large-scale modification of account passwords, and so on. **High** 1. Sensitive information leakage Examples include unauthorized disclosure of sensitive information relating to source code, hardcoded passwords, and so on. 2. Unauthorized access to sensitive information Examples include bypassing authentication or backend password checks, leading to unauthorized access to sensitive intranet information. 3. Unauthorized sensitive operations Examples include manipulating important information without authorization, such as orders, major business configurations, and so on. **Medium** 1. Vulnerabilities that require interaction and affect users Examples include stored XSS (cross-site scripting). 2. General unauthorized operations Examples include incorrect direct object references, unauthorized access to orders, unauthorized access to user information, and so on. 3. General information leakage Examples include client-side stored plaintext passwords, system path traversal, and so on. 4. General logic design flaws and process defects **Low** 1. Local denial of service vulnerabilities, CSRF (cross-site request forgery), reflected XSS, and so on. 2. Minor information leakage, such as path information, SVN information, exception information, the local SQL injection of a client-side application (limited to database name, field name, log print), and so on. 3. Vulnerabilities that are difficult to exploit but still have security implications, such as plaintext transmission of passwords. **Info** 1. Exposure of software banners, internal IP addresses, some public email addresses or phone numbers, and so on. 2. Using outdated versions of a system, supporting outdated versions of encryption protocols, such as SSL (Secure Sockets Layer) or TLS (Transport Layer Security) 1.0, supporting low-strength encryption algorithms, and so on. ### Bug Bounties [#bug-bounties] Please report any potential risks of Agora services that you may notice to [security@agora.io](mailto\:security@agora.io). Any bounties awarded will conform roughly to the following ranges (based on the severity and system location of the bug or vulnerability, payable in USD): * Critical: $300 to $5000 * High: $200 to $2000 * Medium: $150 * Low: $50 * Info: $0 Note: Agora reserves the sole right to determine any reward amount given. # Start with AI (/en/introduction/start-with-ai) Use this page when you want your coding assistant to help you get started without guessing. The steps are simple: install the CLI, add the Agora skill, then give the assistant a prompt with the product, platform, language, and goal. If you also want live docs lookup while the assistant works, pair [Agora Skills](/en/introduction/agora-skills) with [Agora MCP](/en/introduction/agora-mcp). Skills handles workflow guidance, and MCP handles current documentation. ## Quick start [#quick-start] ### 1. Install CLI [#1-install-cli] macOS and Linux Windows (PowerShell) ```bash curl -fsSL https://agoraio.github.io/cli/install.sh | sh -s -- --add-to-path agora --help ``` ```powershell irm https://agoraio.github.io/cli/install.ps1 | iex agora --help ``` If your execution policy blocks inline scripts, download `install.ps1` and run: ```powershell powershell -ExecutionPolicy Bypass -File .\install.ps1 ``` After installation, confirm `agora --help` works in the same shell. ### 2. Add skill [#2-add-skill] Install [Agora Skills](/en/introduction/agora-skills) so your assistant can choose the right Agora workflow, starter, and setup sequence before it edits code. Skills CLI Manual installation ```bash npx skills add AgoraIO/skills ``` Clone the repository and point your coding assistant to the Agora skill files: ```bash git clone https://github.com/AgoraIO/skills.git ~/agora-skills ``` Use `skills/agora/` as the directory entry point, or load `SKILL.md` directly if your tool expects a single root file. If your tool prefers local instruction files, point it at `skills/agora/` or the top-level `SKILL.md`. If you only want docs lookup and not workflow guidance, install [Agora MCP](/en/introduction/agora-mcp) instead. Many teams use both. ### 3. Prompt [#3-prompt] Now tell the assistant what you want it to build and how to approach the setup. A strong prompt includes the Agora product, the target platform, the language or starter preference, and the end goal. ```text Use Agora Skills to help me build a web-based conversational AI demo in Python. ``` From there, let the assistant choose the starter, run `agora login`, initialize the project, and explain any environment values it writes locally. ## What success looks like [#what-success-looks-like] You are on the right path when the assistant can: * choose an official starter instead of inventing one * explain why that starter matches your product and platform * use the Agora CLI for setup instead of ad hoc manual steps * stay grounded in current Agora docs when platform details matter # Support (/en/introduction/support) ## Submit a ticket [#submit-a-ticket] To ask Agora Support a question: 1. Log in to Agora Console. 2. Click **Get Support** > **Create Support Ticket**. 3. Type your question or keywords to see whether the issue has already been answered. 4. If you cannot find an answer, select a category and submit a ticket to customer support. You can track ticket status under **Tickets**. ## Status page [#status-page] Agora Status Page provides up-to-date information about product and service status, including: * historical service stability for the past 90 days * real-time quality data for the past 24 hours * maintenance events and disruptions * RSS event subscriptions You can access the status page directly at [status.agora.io](https://status.agora.io/). The status dashboard currently provides status information for: * Real-Time Communication * Cloud Recording * Agora Chat * Interactive Whiteboard # Usage Analytics (/en/introduction/usage-analytics) ![Analytics overview](https://assets-docs.agora.io/images/analytics/analytics-overview.png) Agora Analytics tracks and analyzes the usage, quality, and performance of real-time voice, video, live streaming, chat, and other Agora products. ## Features [#features] * **Call Inspector**: identify, analyze, and respond to performance quality issues with detailed call and user metrics. * **Data Insights**: periodic call usage and quality statistics. * **Real Time Monitoring**: visualized data for multiple call metrics in real time. * **Alert notifications**: alerts when abnormal metrics or events are detected. * **RESTful APIs**: retrieve call statistics and quality metrics for your own application or DataOps workflow. * **Embedding**: embed Analytics pages in internal web portals using a low-code approach. ## Console usage page [#console-usage-page] The Console **Usage** page lets you check usage for the past 12 months. You can: * Select a time frame and data granularity. * Filter by all projects or a specific project. * Select a product. * Turn on **View Breakdown**. Only **Admin**, **Product/Operation**, and authorized custom teams can access the **Usage** page. # Foundational realtime capabilities (/en/realtime-media/foundation-realtime)
## RTC [#rtc] Owns audio, video, and channel sessions as the realtime base for calling, live streaming, meetings, and classrooms.
## RTM [#rtm] Provides low-latency, reliable messaging plus state synchronization for channel messages, presence, and workflow coordination.
## Instant Messaging IM [#instant-messaging-im] Fits broader cloud-chat workloads where a full messaging service is needed beyond lightweight realtime coordination.
## Realtime Transcription & Translation [#realtime-transcription--translation] Provides ultra-low-latency, high-accuracy speech transcription and translation for captions, multilingual communication, and content understanding.
## RTSA [#rtsa] Provides realtime media-stream and signaling transport for device-oriented and terminal-style workloads. # Overview (/en/realtime-media) Realtime & Media is the capability domain for products that need something to happen live. Use this section when you are deciding how people, devices, backend services, and media streams should connect, stay in sync, and keep working while a session is in progress. Instead of organizing docs around a single product story, this tab is organized around the functional layers most teams assemble when building a realtime experience: live interaction, session intelligence, media routing, large-scale delivery, and device or server participation. ## What you can build from this section [#what-you-can-build-from-this-section] ### Create live interaction surfaces [#create-live-interaction-surfaces] Use these docs when you are defining how users join a session, exchange media, coordinate actions, or collaborate in the same live space. * [Voice & Video](/en/realtime-media/rtc): calls, meetings, co-hosting, interactive live rooms, and media quality control * [Signaling](/en/realtime-media/rtm): channel messaging, presence, state sync, and realtime coordination * [Chat](/en/realtime-media/im): persistent and full-featured messaging systems beyond lightweight room coordination * [Whiteboard](/en/realtime-media/whiteboard): shared visual collaboration inside live sessions ### Understand, capture, and transform session content [#understand-capture-and-transform-session-content] Use these docs when media inside the session needs to become text, records, composites, or downstream assets. * [Transcription & Translation](/en/realtime-media/speech-to-text): captions, speech understanding, and multilingual live experiences * [Cloud Recording](/en/realtime-media/cloud-recording): archive, replay, compliance, QA, and post-session review workflows * [Transcoding](/en/realtime-media/transcoding): mixing, layout composition, and output transformation ### Bridge realtime sessions with external media systems [#bridge-realtime-sessions-with-external-media-systems] Use these docs when your product must ingest outside streams, publish session media to other systems, or interoperate with existing streaming infrastructure. * [Media Push](/en/realtime-media/media-push): send RTC channel media to CDN pipelines or downstream media systems * [Media Pull](/en/realtime-media/media-pull): bring online media streams into an interactive realtime session * [RTMP Gateway](/en/realtime-media/rtmp-gateway): connect RTMP-based devices and systems to Agora ### Deliver playback to larger audiences [#deliver-playback-to-larger-audiences] Use these docs when interactive participation and large-scale viewing are separate concerns in your architecture. * [Fusion CDN](/en/realtime-media/fusion-cdn): multi-CDN distribution for live playback at audience scale ### Extend the session to devices and backend services [#extend-the-session-to-devices-and-backend-services] Use these docs when participants are not just mobile or web clients, but also embedded devices, operator consoles, or server-side workers. * [IoT & Edge](/en/realtime-media/rtsa): device and edge connectivity for cameras, terminals, and embedded endpoints * [RTC Server SDK](/en/realtime-media/rtc-server-sdk): backend participation in media send, receive, subscribe, and control flows ## How to navigate this tab [#how-to-navigate-this-tab] * Start with [Voice & Video](/en/realtime-media/rtc), [Signaling](/en/realtime-media/rtm), [Chat](/en/realtime-media/im), or [Whiteboard](/en/realtime-media/whiteboard) when your main question is how users interact inside a live session. * Start with [Transcription & Translation](/en/realtime-media/speech-to-text), [Cloud Recording](/en/realtime-media/cloud-recording), or [Transcoding](/en/realtime-media/transcoding) when your main question is what should happen to the media during or after the session. * Start with [Media Push](/en/realtime-media/media-push), [Media Pull](/en/realtime-media/media-pull), [RTMP Gateway](/en/realtime-media/rtmp-gateway), or [Fusion CDN](/en/realtime-media/fusion-cdn) when your main question is how to move media across systems or out to larger audiences. * Start with [IoT & Edge](/en/realtime-media/rtsa) or [RTC Server SDK](/en/realtime-media/rtc-server-sdk) when your main question is how devices or backend services participate in the realtime workflow. ## Common reading paths [#common-reading-paths] * Interactive live streaming with large audience playback: [Voice & Video](/en/realtime-media/rtc) -> [Media Push](/en/realtime-media/media-push) -> [Fusion CDN](/en/realtime-media/fusion-cdn) * Meeting archive and searchable records: [Voice & Video](/en/realtime-media/rtc) -> [Cloud Recording](/en/realtime-media/cloud-recording) -> [Transcription & Translation](/en/realtime-media/speech-to-text) * External stream into an interactive session: [Media Pull](/en/realtime-media/media-pull) or [RTMP Gateway](/en/realtime-media/rtmp-gateway) -> [Voice & Video](/en/realtime-media/rtc) * Smart device connectivity with backend media control: [IoT & Edge](/en/realtime-media/rtsa) -> [RTC Server SDK](/en/realtime-media/rtc-server-sdk) # Media processing and distribution (/en/realtime-media/media-processing-and-distribution)
## Recording [#recording] Recording captures live interaction content into your storage flow for archive, replay, review, and compliance-oriented workflows. ### Cloud Recording [#cloud-recording] Fits teams that want a managed recording path from Agora.
### Local Server Recording [#local-server-recording] Fits teams that need self-managed recording servers and finer-grained control over the recording workflow.
## Media Push [#media-push] Pushes interactive channel content into CDN or broadcast-style distribution paths.
## Media Pull [#media-pull] Brings external online media streams into an existing realtime scenario.
## Transcoding [#transcoding] Transcoding handles media transformation and composition for live streaming and multi-layout output scenarios. ### Cloud Transcoding [#cloud-transcoding] Fits teams that need hosted stream composition and output transformation.
## RTMP Gateway [#rtmp-gateway] Bridges standard live protocol flows with realtime interaction systems.
## Fusion CDN Live Streaming [#fusion-cdn-live-streaming] Optimizes end-to-end playback through multi-CDN scheduling and distribution.
Additional note: the broader Agora content system also includes PPT Conversion. For this IA pass, it can stay as an extension of transcoding rather than a first-class left-nav item. # RTC overview (/en/realtime-media/overview) ## Build live interaction [#build-live-interaction] ## Process session content [#process-session-content] ## Connect external media [#connect-external-media] ## Deliver to audiences [#deliver-to-audiences] # Server-side and extensions (/en/realtime-media/server-and-extensions)
## RTC Server SDK [#rtc-server-sdk] Lets the backend participate directly in RTC media send, receive, and control flows.
## SDK extension plugins [#sdk-extension-plugins] Support capability enhancement and customized audio-video effects on top of the existing SDK stack.
## Marketplace [#marketplace] Provides a faster path to selecting, purchasing, and connecting external realtime modules into your product. # Set up service and credentials (/en/realtime-media/setup-service-and-credentials) ## Why this always comes first [#why-this-always-comes-first] No matter whether you end up building AI, RTC, RTM, recording, or transcoding, the first step is rarely "write code." It is usually confirming Console access, project state, App ID, credentials, and authentication. ## What you should prepare [#what-you-should-prepare] * project and App ID * customer ID and customer secret * token generation strategy * service enablement status * the boundary between test and production environments ## Recommended practice [#recommended-practice] Prepare the control plane and credential flow first, then move into product-specific docs. That prevents expensive rework when permissions, quotas, or auth assumptions diverge later. # API reference (/en/api-reference/api-ref) # Channel Management REST API (/en/api-reference/api-ref/iot-channel-management-rest-api) In addition to the SDK that you integrate into the app client, Agora provides server-side RESTful APIs to manage real-time channels. The IoT Channel Management REST API uses the shared RTC REST API reference. Use these APIs to query channel information, manage user privileges, and query Agora Notifications service IP addresses. ## REST API reference [#rest-api-reference] * [RTC REST API overview](/en/api-reference/api-ref/rtc) * [How to call RESTful APIs](/en/api-reference/api-ref/rtc/how-to-call-api) * [RESTful authentication](/en/api-reference/api-ref/rtc/authentication) * [Response status codes](/en/api-reference/api-ref/rtc/response-status-codes) ## Endpoints [#endpoints] **Channel information** * [Query the channel list](/en/api-reference/api-ref/rtc/query-channel-list) * [Query the user list](/en/api-reference/api-ref/rtc/query-user-list) * [Query the host list](/en/api-reference/api-ref/rtc/query-host-list) * [Query the user status](/en/api-reference/api-ref/rtc/query-user-status) **User privilege banning** * [Create a banning rule](/en/api-reference/api-ref/rtc/create-ban-rule) * [Delete a banning rule](/en/api-reference/api-ref/rtc/delete-ban-rule) * [Get the banning rule list](/en/api-reference/api-ref/rtc/get-ban-rule-list) * [Update the banning rule expiration](/en/api-reference/api-ref/rtc/update-ban-expiration) **Message notification service** * [Query the IP address](/en/api-reference/api-ref/rtc/query-ip-address) ## Supporting topics [#supporting-topics] * [Ban user privileges best practices](/en/api-reference/api-ref/rtc/ban-user-privileges-best-practices) * [Ensure service reliability](/en/api-reference/api-ref/rtc/ensure-service-reliability) * [Channel event types](/en/api-reference/api-ref/rtc/channel-event-types) # Fastboard API (/en/api-reference/api-ref/uikit-sdk) This page provides the API reference for the Fastboard SDK. <_PlatformTabsGroup groupMode="structured" canonicalPlatform="web" platforms="["android","ios","web"]" showTabs="true"> <_PlatformPanel platform="android"> <_PlatformProcessedMarker groupMode="structured" canonicalPlatform="web" platform="android" /> ## Understand the tech [#understand-the-tech] This section provides the API reference for the Fastboard SDK on Android. ## Reference [#reference] ## FastboardView class [#fastboardview-class] ### getFastboard [#getfastboard] ```java public Fastboard getFastboard() ``` Get the `Fastboard` object. The Fastboard SDK does not support initializing a `Fastboard` instance directly. To get the `Fastboard` object, you need to add the `FastboardView` object to the app's layout, and then call the `getFastboard` method. ## Fastboard class [#fastboard-class] ### createFastRoom [#createfastroom] ```java public FastRoom createFastRoom(FastRoomOptions roomOptions) ``` ### preloadWhiteboardView [#preloadwhiteboardview] ```java public static void preloadWhiteboardView() { WhiteboardViewManager.get().preload(); } ``` ### destroy [#destroy] ```java public void destroy() ``` ### setConfig [#setconfig] ```java public static void setConfig(FastboardConfig config) ``` ### FastRoomOptions [#fastroomoptions] ```java public class FastRoomOptions { private final String appId; private final String uuid; private final String token; private final String uid; private final boolean writable; private final FastRegion fastRegion; } ``` ### FastRegion [#fastregion] * `CN_HZ` * `US_SV` * `SG` * `IN_MUM` * `EU` ### FastUserPayload [#fastuserpayload] ```java public class FastUserPayload { private final String nickName; private final String avatar; } ``` ### FastboardConfig [#fastboardconfig] ```java public class FastboardConfig { private final boolean enablePreload; private final int preloadCount; private final boolean autoPreload; } ``` ## FastRoom class [#fastroom-class] ### join \[1/2] [#join-12] ```java public void join() ``` ### join \[2/2] [#join-22] ```java public void join(@Nullable OnRoomReadyCallback onRoomReadyCallback) ``` ### setWritable [#setwritable] ```java public void setWritable(boolean writable) ``` ### registerApp [#registerapp] ```java public void registerApp(FastRegisterAppParams params, FastResult result) ``` ### insertImage [#insertimage] ```java public void insertImage(String url, int width, int height) ``` ### insertVideo [#insertvideo] ```java public void insertVideo(String url, String title) ``` ### insertStaticDoc [#insertstaticdoc] ```java public void insertStaticDoc(DocPage[] pages, String title, FastResult result) ``` ### insertPptx [#insertpptx] ```java public void insertPptx(String taskUuid, String prefixUrl, String title, FastResult result) ``` ### insertDocs [#insertdocs] ```java public void insertDocs(FastInsertDocParams params, FastResult result) ``` <_PlatformProcessedMarker close="true" /> <_PlatformPanel platform="ios"> <_PlatformProcessedMarker groupMode="structured" canonicalPlatform="web" platform="ios" /> ## Understand the tech [#understand-the-tech-1] This section provides the API reference for the Fastboard SDK on iOS. ## Reference [#reference-1] ## Fastboard class [#fastboard-class-1] ### createFastRoom [#createfastroom-1] ```swift public class func createFastRoom(withFastRoomConfig config: FastRoomConfiguration) ``` Create a `FastRoom` instance. ### FastRoomConfiguration [#fastroomconfiguration] ```swift public init(appIdentifier: String, roomUUID: String, roomToken: String, region: Region, userUID: String, userPayload: FastUserPayload? = nil) ``` ### Region [#region] * `CN` * `US` * `SG` * `IN` * `EU` ### FastUserPayload [#fastuserpayload-1] ```swift public class FastUserPayload: NSObject { let nickName: String? let avatar: String? } ``` ## FastRoom class [#fastroom-class-1] ### joinRoom [#joinroom] ```swift public func joinRoom(completionHandler: ((Result)->Void)? = nil) ``` ### disconnectRoom [#disconnectroom] ```swift public func disconnectRoom() ``` ### insertImg [#insertimg] ```swift public func insertImg(_ src: URL, imageSize: CGSize) ``` ### insertMedia [#insertmedia] ```swift public func insertMedia(_ src: URL, title: String, completionHandler: ((String)->Void)? = nil) ``` ### insertPptx [#insertpptx-1] ```swift public func insertPptx( uuid: String, url: String, title: String, completionHandler: ((String)->Void)? = nil ) ``` ### insertStaticDocument [#insertstaticdocument] ```swift public func insertStaticDocument(_ pages: [WhitePptPage], title: String, completionHandler: ((String)->Void)? = nil) ``` ### followSystemPencilBehavior [#followsystempencilbehavior] ```swift public static var followSystemPencilBehavior ``` ## FastRoomThemeManager class [#fastroomthememanager-class] ### apply [#apply] ```swift public func apply(_ theme: FastRoomThemeAsset) ``` <_PlatformProcessedMarker close="true" /> <_PlatformPanel platform="web"> <_PlatformProcessedMarker groupMode="structured" canonicalPlatform="web" platform="web" /> ## Understand the tech [#understand-the-tech-2] This section provides the API reference for the Fastboard SDK on Web. ## Reference [#reference-2] ## createFastboard [#createfastboard] ```typescript createFastboard(options: FastboardOptions): Promise; ``` ## createUI [#createui] ```typescript createUI(app?: FastboardApp | null, div?: Element): UI; ``` ## createReplayUI [#createreplayui] ```typescript createReplayUI(player?: FastboardPlayer | null, div?: Element): ReplayUI; ``` ### canOperate [#canoperate] ```typescript get canOperate(): boolean ``` ## mount [#mount] ```typescript mount(app: FastboardApp, div: HTMLDivElement, options?: MountProps): { update(props?: MountProps | undefined): void; destroy(): void; }; ``` ## dispatchDocsEvent [#dispatchdocsevent] ```typescript export function dispatchDocsEvent( fastboard: FastboardApp | WindowManager, event: "prevPage" | "nextPage" | "prevStep" | "nextStep" | "jumpToPage", options: DocsEventOptions = {} ) ``` ## FastboardApp class [#fastboardapp-class] ### undo [#undo] ```typescript undo(): void ``` ### redo [#redo] ```typescript redo(): void ``` ### moveCamera [#movecamera] ```typescript moveCamera(camera: Partial): void ``` ### moveCameraToContain [#movecameratocontain] ```typescript moveCameraToContain(rect: Rectangle): void; ``` ### setAppliance [#setappliance] ```typescript setAppliance(appliance: ApplianceNames, shape?: ShapeType): void; ``` ### setStrokeWidth [#setstrokewidth] ```typescript setStrokeWidth(strokeWidth: number): void ``` ### setStrokeColor [#setstrokecolor] ```typescript setStrokeColor(strokeColor: Color): void ``` ### setTextColor [#settextcolor] ```typescript setTextColor(textColor: Color): void ``` ### setTextSize [#settextsize] ```typescript setTextSize(textSize: number): void ``` ### insertImage [#insertimage-1] ```typescript async insertImage(url: string, crossOrigin?: boolean | string) ``` ### insertMedia [#insertmedia-1] ```typescript insertMedia(title: string, src: string): Promise ``` ### insertDocs \[1/2] [#insertdocs-12] ```typescript insertDocs(title: string, response: ProjectorResponse): Promise; ``` ### insertDocs \[2/2] [#insertdocs-22] ```typescript insertDocs(params: InsertDocsParams): Promise; ``` ### removePage [#removepage] ```typescript removePage(index?): Promise ``` ### jumpPage [#jumppage] ```typescript jumpPage(index: number) ``` <_PlatformProcessedMarker close="true" /> # FAQ (/en/api-reference/faq) # Optimize audio quality (/en/ai/best-practices/audio-setup) In real-time audio interactions, the rhythm, continuity, and intonation of conversations between humans and AI often differ from those between humans. To improve the AI–human conversation experience, it's important to optimize audio settings. When using the Android, iOS, or Web RTC SDK with the Conversational AI Engine, follow the best practices in this guide to improve conversation fluency and reliability, especially in complex network environments. ## Server configuration [#server-configuration] When calling the server API to create a conversational AI agent, use the default values for audio-related parameters to ensure the best audio experience. ## Client configuration [#client-configuration] To configure the client app, implement the following: ### Integrate the required dynamic libraries [#integrate-the-required-dynamic-libraries] For the best Conversational AI Engine audio experience, integrate and load the following dynamic libraries in your project: Android iOS Web * AI noise suppression plugin: `libagora_ai_noise_suppression_extension.so` * AI echo cancellation plug-in: `libagora_ai_echo_cancellation_extension.so` For integration details, refer to [App size optimization](/en/realtime-media/voice/build/optimize-and-operate/app-size-optimization/android). * AI noise suppression plugin: `AgoraAiNoiseSuppressionExtension.xcframework` * AI echo cancellation plug-in: `AgoraAiEchoCancellationExtension.xcframework` For integration details, refer to [App size optimization](/en/realtime-media/voice/build/optimize-and-operate/app-size-optimization/ios). * Integrate the `agora-extension-ai-denoiser` extension. Refer to [AI Noise Suppression](/en/realtime-media/voice/build/enhance-the-audio-experience/ai-noise-suppression/web). Info Optimizing audio uses AI Noise Suppression, which is a paid feature. ## Optimize audio for optimal performance [#optimize-audio-for-optimal-performance] You can optimize audio settings in the following ways: * **(Recommended) Use the Toolkit APIs** Supported in Video/Voice SDK version 4.5.1 and above. * **Use the Video/Voice SDK APIs directly** Supported in SDK version 4.3.1 and above. ### Use the toolkit APIs [#use-the-toolkit-apis] In this solution, you use the toolkit APIs to optimize audio settings. Android iOS Web 1. **Integrate the toolkit** Copy the [`convoaiApi`](https://github.com/AgoraIO-Community/Conversational-AI-Demo/tree/main/Android/scenes/convoai/src/main/java/io/agora/scene/convoai/convoaiApi) folder to your project and import the toolkit before calling the toolkit API. 2. **Create a toolkit instance** ```kotlin val config = ConversationalAIAPIConfig( rtcEngine = rtcEngineInstance, rtmClient = rtmClientInstance, enableLog = true ) val api = ConversationalAIAPIImpl(config) ``` 3. **Set optimal audio settings** Before joining the RTC channel, call `loadAudioSettings()` to apply the optimal audio parameters. ```kotlin api.loadAudioSettings() rtcEngine.joinChannel(token, channelName, null, userId) ``` 4. **Release resources** ```kotlin api.destroy() ``` 1. **Integrate the toolkit** Copy the [`ConversationalAIAPI`](https://github.com/AgoraIO-Community/Conversational-AI-Demo/tree/main/iOS/Scenes/ConvoAI/ConvoAI/ConvoAI/Classes/ConversationalAIAPI) folder to your project and import the toolkit before calling the toolkit APIs. 2. **Create a toolkit instance** ```swift let config = ConversationalAIAPIConfig( rtcEngine: rtcEngine, rtmEngine: rtmEngine, enableLog: true ) convoAIAPI = ConversationalAIAPIImpl(config: config) ``` 3. **Set optimal audio settings** Before joining the RTC channel, call `loadAudioSettings()` to apply the optimal audio parameters. ```swift convoAIAPI.loadAudioSettings() rtcEngine.joinChannel(rtcToken: token, channelName: channelName, uid: uid, isIndependent: independent) ``` 4. **Release resources** ```swift convoAIAPI.destroy() ``` * **Integrate the transcript processing toolkit** Copy the [`conversational-ai-api`](https://github.com/AgoraIO-Community/Conversational-AI-Demo/tree/main/Web/Scenes/VoiceAgent/src/conversational-ai-api) file to your project and import the toolkit before calling its API. Refer to [Folder structure](#folder-structure) to understand the role of each file. ### Use the SDK APIs [#use-the-sdk-apis] In this solution, you use the Voice/Video SDK to optimize audio settings. #### Set audio parameters [#set-audio-parameters] The settings in this section apply to Video/Voice SDK versions 4.3.1 and above. If you are using an earlier version, upgrade to version 4.5.1 or above or [Contact Technical Support](mailto\:support@agora.io). Android iOS Web For the best conversational AI audio experience, apply the following settings: 1. **Set the audio scenario**: When initializing the engine, set the audio scenario to the AI client scenario. You can also set the scenario before joining a channel by calling the `setAudioScenario` method. 2. **Configure audio parameters**: Call `setParameters` before joining a channel and whenever the `onAudioRouteChanged` callback is triggered. This configuration sets audio 3A plug-ins (acoustic echo cancellation, noise suppression, and automatic gain control), the audio sampling rate, the audio processing mode, and other settings. For recommended parameter values, refer to the sample code. Info Since Video/Voice SDK versions 4.3.1 to 4.5.0 do not support the AI client audio scenario, set the scenario to `AUDIO_SCENARIO_CHORUS` to improve the audio experience. However, the audio experience cannot be aligned with versions 4.5.1 and above. To get the best audio experience, upgrade the SDK to version 4.5.1 or higher. The following sample code defines a `setAudioConfigParameters` function to configure audio parameters. Call this function before joining a channel and whenever the audio route changes. ```kotlin private var rtcEngine: RtcEngineEx? = null private var mAudioRouting = Constants.AUDIO_ROUTE_DEFAULT // highlight-start // Set audio configuration parameters private fun setAudioConfigParameters(routing: Int) { mAudioRouting = routing rtcEngine?.apply { setParameters("{\"che.audio.aec.split_srate_for_48k\":16000}") setParameters("{\"che.audio.sf.enabled\":true}") setParameters("{\"che.audio.sf.stftType\":6}") setParameters("{\"che.audio.sf.ainlpLowLatencyFlag\":1}") setParameters("{\"che.audio.sf.ainsLowLatencyFlag\":1}") setParameters("{\"che.audio.sf.procChainMode\":1}") setParameters("{\"che.audio.sf.nlpDynamicMode\":1}") if (routing == Constants.AUDIO_ROUTE_HEADSET // 0 || routing == Constants.AUDIO_ROUTE_EARPIECE // 1 || routing == Constants.AUDIO_ROUTE_HEADSETNOMIC // 2 || routing == Constants.AUDIO_ROUTE_BLUETOOTH_DEVICE_HFP // 5 || routing == Constants.AUDIO_ROUTE_BLUETOOTH_DEVICE_A2DP) { // 10 setParameters("{\"che.audio.sf.nlpAlgRoute\":0}") } else { setParameters("{\"che.audio.sf.nlpAlgRoute\":1}") } setParameters("{\"che.audio.sf.ainlpModelPref\":10}") setParameters("{\"che.audio.sf.nsngAlgRoute\":12}") setParameters("{\"che.audio.sf.ainsModelPref\":10}") setParameters("{\"che.audio.sf.nsngPredefAgg\":11}") setParameters("{\"che.audio.agc.enable\":false}") } } // highlight-end // Create and initialize the RTC engine fun createRtcEngine(rtcCallback: IRtcEngineEventHandler): RtcEngineEx { val config = RtcEngineConfig() config.mContext = AgentApp.instance() config.mAppId = ServerConfig.rtcAppId config.mChannelProfile = Constants.CHANNEL_PROFILE_LIVE_BROADCASTING // highlight-start // Set the audio scene to AI dialogue scene (supported by 4.5.1 and above) // Version 4.3.1 ~ 4.5.0 is set to chorus scene AUDIO_SCENARIO_CHORUS config.mAudioScenario = Constants.AUDIO_SCENARIO_AI_CLIENT // Register audio route change callback config.mEventHandler = object : IRtcEngineEventHandler() { override fun onAudioRouteChanged(routing: Int) { super.onAudioRouteChanged(routing) // Set audio related parameters setAudioConfigParameters(routing) } } // highlight-end try { rtcEngine = (RtcEngine.create(config) as RtcEngineEx).apply { // highlight-start // Load the audio plugin loadExtensionProvider("ai_echo_cancellation_extension") loadExtensionProvider("ai_noise_suppression_extension") // highlight-end } } catch (e: Exception) { Log.e("CovAgoraManager", "createRtcEngine error: $e") } return rtcEngine!! } // Join the channel fun joinChannel(rtcToken: String, channelName: String, uid: Int, isIndependent: Boolean = false) { // highlight-start // Initialize audio configuration parameters setAudioConfigParameters(mAudioRouting) // highlight-end // Configure channel options and join the channel val options = ChannelMediaOptions() options.clientRoleType = CLIENT_ROLE_BROADCASTER options.publishMicrophoneTrack = true options.publishCameraTrack = false options.autoSubscribeAudio = true options.autoSubscribeVideo = false val ret = rtcEngine?.joinChannel(rtcToken, channelName, uid, options) } ``` For the best conversational AI audio experience, apply the following settings: 1. **Set the audio scenario**: When initializing the engine, set the audio scenario to the AI client scenario. You can also set the scenario before joining a channel by calling the `setAudioScenario` method. 2. **Configure audio parameters**: Call `setParameters` before joining a channel and whenever the `rtcEngine:didAudioRouteChanged:` callback is triggered. This configuration sets audio 3A plug-ins (acoustic echo cancellation, noise suppression, and automatic gain control), the audio sampling rate, the audio processing mode, and other settings. For recommended parameter values, refer to the sample code. Info Since Video/Voice SDK versions 4.3.1 to 4.5.0 do not support the AI client audio scenario, set the scenario to `AgoraAudioScenarioChorus` to improve the audio experience. However, the audio experience cannot be aligned with versions 4.5.1 and above. To get the best audio experience, upgrade the SDK to version 4.5.1 or higher. The following sample code defines a `setAudioConfigParameters` function to configure audio parameters. Call this function before joining a channel and whenever the audio route changes. ```swift class RTCManager: NSObject { private var rtcEngine: AgoraRtcEngineKit! private var audioDumpEnabled: Bool = false private var audioRouting = AgoraAudioOutputRouting.default // highlight-start // Set audio related parameters private func setAudioConfigParameters(routing: AgoraAudioOutputRouting) { audioRouting = routing rtcEngine.setParameters("{\"che.audio.aec.split_srate_for_48k\":16000}") rtcEngine.setParameters("{\"che.audio.sf.enabled\":true}") rtcEngine.setParameters("{\"che.audio.sf.stftType\":6}") rtcEngine.setParameters("{\"che.audio.sf.ainlpLowLatencyFlag\":1}") rtcEngine.setParameters("{\"che.audio.sf.ainsLowLatencyFlag\":1}") rtcEngine.setParameters("{\"che.audio.sf.procChainMode\":1}") rtcEngine.setParameters("{\"che.audio.sf.nlpDynamicMode\":1}") if routing == .headset || routing == .earpiece || routing == .headsetNoMic || routing == .bluetoothDeviceHfp || routing == .bluetoothDeviceA2dp { rtcEngine.setParameters("{\"che.audio.sf.nlpAlgRoute\":0}") } else { rtcEngine.setParameters("{\"che.audio.sf.nlpAlgRoute\":1}") } rtcEngine.setParameters("{\"che.audio.sf.ainlpModelPref\":10}") rtcEngine.setParameters("{\"che.audio.sf.nsngAlgRoute\":12}") rtcEngine.setParameters("{\"che.audio.sf.ainsModelPref\":10}") rtcEngine.setParameters("{\"che.audio.sf.nsngPredefAgg\":11}") rtcEngine.setParameters("{\"che.audio.agc.enable\":false}") } // highlight-end } extension RTCManager: RTCManagerProtocol { func createRtcEngine(delegate: AgoraRtcEngineDelegate) -> AgoraRtcEngineKit { let config = AgoraRtcEngineConfig() config.appId = AppContext.shared.appId config.channelProfile = .liveBroadcasting // highlight-start // Set the audio scene to AI dialogue scene (supported by 4.5.1 and above) // Versions 4.3.1 ~ 4.5.0 support chorus scenes .chorus config.audioScenario = .aiClient rtcEngine = AgoraRtcEngineKit.sharedEngine(with: config, delegate: delegate) // Register audio route change callback rtcEngine.addDelegate(self) // highlight-end return rtcEngine } func joinChannel(rtcToken: String, channelName: String, uid: String) { // highlight-start // Initialize audio configuration parameters setAudioConfigParameters(routing: audioRouting) // highlight-end // Configure channel options and join the channel let options = AgoraRtcChannelMediaOptions() options.clientRoleType = .broadcaster options.publishMicrophoneTrack = true options.publishCameraTrack = false options.autoSubscribeAudio = true options.autoSubscribeVideo = false let ret = rtcEngine.joinChannel(byToken: rtcToken, channelId: channelName, uid: UInt(uid) ?? 0, mediaOptions: options) } } // highlight-start // Implement the AgoraRtcEngineDelegate interface to handle audio route change callbacks extension RTCManager: AgoraRtcEngineDelegate { public func rtcEngine(_ engine: AgoraRtcEngineKit, didAudioRouteChanged routing: AgoraAudioOutputRouting) { setAudioConfigParameters(routing: routing) } } // highlight-end ``` Integrate the `agora-extension-ai-denoiser` extension. Refer to [AI Noise Suppression](/en/realtime-media/voice/build/enhance-the-audio-experience/ai-noise-suppression/web). Info Use Web SDK version 4.15.0 or later. ## Reference [#reference] This section contains content that completes the information on this page, or points you to documentation that explains other aspects to this product. ### Sample project [#sample-project] Refer to the following open-source sample code to set audio-related parameters. Android iOS * [`CovRtcManager.kt`](https://github.com/AgoraIO-Community/Conversational-AI-Demo/blob/main/Android/scenes/convoai/src/main/java/io/agora/scene/convoai/rtc/CovRtcManager.kt) * [`RTCManager.swift`](https://github.com/AgoraIO-Community/Conversational-AI-Demo/blob/main/iOS/Scenes/ConvoAI/ConvoAI/ConvoAI/Classes/Manager/RTCManager.swift) ### Folder structure [#folder-structure] Android iOS Web * `IConversationalAIAPI.kt`: API interface and related data structures and enumerations * `ConversationalAIAPIImpl.kt`: ConversationalAI API main implementation logic * `ConversationalAIUtils.kt`: Tool functions and event callback management * `subRender/` * `v3/`: Transcript module * `TranscriptionController.kt`: Transcript Controller * `MessageParser.kt`: Message Parser * `ConversationalAIAPI.swift`: API interface and related data structures and enumerations * `ConversationalAIAPIImpl.swift`: ConversationalAI API main implementation logic * `Transcription/` * `TranscriptionController.swift`: Transcript Controller * `index.ts`: API Class * `type.ts`: API interface and related data structures and enumerations * `utils/` * `index.ts`: API utility functions * `events.ts`: Event management class, which can be extended to easily implement event monitoring and broadcasting * `sub-render.ts`: Transcript module ### API reference [#api-reference] Android iOS * SDK * `setAudioScenario` * `setParameters` * `onAudioRouteChanged` * Toolkit * [`loadAudioSettings`](/en/api-reference/api-ref/conversational-ai/client-toolkit/android#loadaudiosettings) * [`destroy`](/en/api-reference/api-ref/conversational-ai/client-toolkit/android#destroy) * SDK * `setAudioScenario` * `setParameters` * `rtcEngine:didAudioRouteChanged:` * Toolkit * [`loadAudioSettings`](/en/api-reference/api-ref/conversational-ai/client-toolkit/ios#loadaudiosettings12) * [`destroy`](/en/api-reference/api-ref/conversational-ai/client-toolkit/ios#destroy) * Toolkit * [`IConversationalAIAPIEventHandlers interface`](/en/api-reference/api-ref/conversational-ai/client-toolkit/web#iconversationalaiapieventhandlers-interface) * [`EConversationalAIAPIEvents`](/en/api-reference/api-ref/conversational-ai/client-toolkit/web#econversationalaiapievents) * [`subscribeMessage`](/en/api-reference/api-ref/conversational-ai/client-toolkit/web#subscribemessage) * [`unsubscribeMessage`](/en/api-reference/api-ref/conversational-ai/client-toolkit/web#unsubscribe) * [`destroy`](/en/api-reference/api-ref/conversational-ai/client-toolkit/web#destroy) # Get agent state (/en/ai/best-practices/get-agent-state) If you need to track agent state changes in your client app, you can use either Signaling (RTM) Message or RTM Presence. This guide explains how to get agent state using RTM Message and describes [how the two approaches differ](#rtm-message-vs-rtm-presence). Info * To monitor state changes using client components, see [Listen to agent events](/en/ai/build/handle-runtime-events/webhooks). * To get agent state through RTM Presence, continue using your existing Presence implementation. ## Understand the tech [#understand-the-tech] The Conversational AI Engine pushes agent state change events to the client as individual RTM messages through the `onMessageEvent` callback. Unlike RTM Presence, which synchronizes state as key-value pairs, RTM Message wraps each state change in a separate event message. This makes it easier to record state change timestamps, process events sequentially, or consume state events together with other RTM Message events. The inner `message` field of each RTM message is a JSON string that requires a second parse to access the state event data. Use the `event_type` field in the inner message to identify the event type. The following agent states are pushed through RTM Message: * `state.listening` * `state.thinking` * `state.speaking` The flow works as follows: ```text Agent state change ├─ listening -> true / false ├─ thinking -> true / false └─ speaking -> true / false ↓ RTM Message ↓ Client parses inner payload and identifies event by event_type ``` ### State event types [#state-event-types] The Conversational AI Engine sends an RTM Message each time the agent state changes. Each message corresponds to a single state change event. | `event_type` | Description | | :---------------- | :-------------------------------------------------- | | `state.listening` | The agent starts or stops listening for user input. | | `state.thinking` | The agent starts or stops processing user input. | | `state.speaking` | The agent starts or stops speaking. | ### Message structure [#message-structure] The inner `message` field is a JSON string. After deserialization, the structure is as follows: ```json { "event_id": "xxxx", "event_type": "state.listening", "event_ms": 1611566412672, "payload": { "value": true, "timestamp": 1611566412600 } } ``` | Field | Type | Description | | :------------------ | :------ | :-------------------------------------------------------------------------------------- | | `event_id` | string | Unique event ID. | | `event_type` | string | State event type. One of `state.listening`, `state.thinking`, or `state.speaking`. | | `event_ms` | integer | Timestamp when the event was sent, in milliseconds. | | `payload.value` | boolean | Current state value. `true` means the agent entered the state; `false` means it exited. | | `payload.timestamp` | integer | Timestamp when the state change actually occurred, in milliseconds. | ## Prerequisites [#prerequisites] Before you begin, ensure that you have: * Completed the basic agent integration. See [Quickstart](/en/ai/get-started/quickstart). * Integrated the RTM SDK and implemented message listener logic. See [RTM quickstart](https://docs.agora.io/en/signaling/get-started/sdk-quickstart). * Enabled the RTM service and set `data_channel` to `rtm` when starting the agent. ## Implementation [#implementation] ### Enable RTM when starting the agent [#enable-rtm-when-starting-the-agent] When calling [Start a conversational AI agent](/en/api-reference/api-ref/conversational-ai/join), include the following configuration: ```json { "properties": { "advanced_features": { "enable_rtm": true }, "parameters": { "data_channel": "rtm" } } } ``` ### Listen for and parse state messages [#listen-for-and-parse-state-messages] After logging in to RTM and subscribing to the channel with the same name as the agent, parse state messages in the RTM message callback. The handling logic consists of the following steps: 1. Deserialize the RTM `message` field into a state event object. 2. Check that the deserialized result contains `event_type` and `payload`. 3. Dispatch handling based on `event_type`, and use `payload.value` to update your app state. In the following examples: * The inner `message` field is a JSON string and must be deserialized again to obtain `event_type` and `payload`. * `event_type` identifies the specific state type. * `payload.value` indicates whether the agent is entering (`true`) or exiting (`false`) the state. * `payload.timestamp` can be used for precise ordering or logging. Android iOS Web ```kotlin val rtmConfig = RtmConfig() rtmConfig.eventListener = object : RtmEventListener { override fun onMessageEvent(event: MessageEvent) { // 1. Deserialize the inner message (JSON string) val payload = JSONObject(event.message.data) if (!payload.has("event_type") || !payload.has("payload")) { return } val eventType = payload.getString("event_type") val statePayload = payload.getJSONObject("payload") val value = statePayload.getBoolean("value") val timestamp = statePayload.getLong("timestamp") // 2. Dispatch based on event_type when (eventType) { "state.listening" -> handleListeningChanged(value, timestamp) "state.thinking" -> handleThinkingChanged(value, timestamp) "state.speaking" -> handleSpeakingChanged(value, timestamp) } } } mRtmClient = RtmClient.create(rtmConfig) ``` ```swift func rtmKit(_ rtmKit: AgoraRtmClientKit, didReceiveMessageEvent event: AgoraRtmMessageEvent) { // 1. Deserialize the inner message (JSON string) guard let jsonString = event.message.stringData, let data = jsonString.data(using: .utf8), let payload = try? JSONSerialization.jsonObject(with: data) as? [String: Any], let eventType = payload["event_type"] as? String, let statePayload = payload["payload"] as? [String: Any] else { return } let value = statePayload["value"] as? Bool ?? false let timestamp = statePayload["timestamp"] as? Int ?? 0 // 2. Dispatch based on event_type switch eventType { case "state.listening": handleListeningChanged(value, timestamp) case "state.thinking": handleThinkingChanged(value, timestamp) case "state.speaking": handleSpeakingChanged(value, timestamp) default: break } } ``` ```typescript rtm.addEventListener("message", (event) => { // 1. Deserialize the inner message (JSON string) const payload = JSON.parse(event.message as string); if (!payload.event_type || !payload.payload) { return; } const { event_type, payload: statePayload } = payload; // 2. Dispatch based on event_type switch (event_type) { case "state.listening": handleListeningChanged(statePayload.value, statePayload.timestamp); break; case "state.thinking": handleThinkingChanged(statePayload.value, statePayload.timestamp); break; case "state.speaking": handleSpeakingChanged(statePayload.value, statePayload.timestamp); break; } }); ``` ### Update app state [#update-app-state] Use the received events to update your UI or internal state machine. For example: * When `state.listening` with `payload.value = true` is received, display "Listening". * When `state.thinking` with `payload.value = true` is received, display "Thinking". * When `state.speaking` with `payload.value = true` is received, display "Speaking". ## RTM Message vs RTM Presence [#rtm-message-vs-rtm-presence] The Conversational AI Engine supports getting agent state through both RTM Message and RTM Presence. The key differences are: | | RTM Presence | RTM Message | | :---------- | :------------------------------- | :------------------------------------------------------- | | Return form | State key-value pairs | State event messages | | Granularity | Better for reading current state | Better for handling each state change | | Timing info | Relies on message arrival time | Includes explicit `event_ms` and `payload.timestamp` | | Use case | Simple UI state display | Event-driven logic, logging, unified message consumption | Choose based on your needs: * If you only need to update the UI based on whether the agent is currently `listening`, `thinking`, or `speaking`, RTM Presence is simpler to integrate. * If you want to process state changes together with captions, interruptions, or performance metrics in a unified RTM Message handler, use RTM Message. * If you need to record the timestamp of each state change or analyze state transitions in sequence, use RTM Message. ## Considerations [#considerations] Keep the following in mind when implementing this feature: * **Double deserialization**: The inner `message` field is a JSON string and must be deserialized a second time to access `event_type` and `payload`. * **Avoid duplicate UI updates**: If you are already driving UI state through RTM Presence, avoid having both implementations update the same UI state simultaneously, as this can cause redundant updates or state flickering. * **`state.speaking` end timing**: A `state.speaking` value of `false` does not guarantee that the client has finished playing audio. The server estimates the speaking end time based on TTS audio length, so there may be a delay between the state switching to `false` and the client completing playback. * **Scope of state messages**: State messages cover only `listening`, `thinking`, and `speaking` agent state events. Manual SoS and EoS signals and their callbacks are not included. # Manually control start and end of speech (/en/ai/best-practices/manual-turn-control) When interacting with a conversational AI agent, you may need to explicitly control when a user's speech starts and ends on the client side. This supports scenarios with strict turn-boundary requirements, such as AI interviews, interactive quizzes, or walkie-talkie style push-to-talk. This guide explains how to send manual Start of Speech (SoS) and End of Speech (EoS) requests directly through raw Signaling (RTM) messages, without integrating the Conversational AI client components, and how to handle the results returned by the server. This approach is suitable when: * You want to implement your own RTM send and receive logic instead of relying on client components. * Your business already has an independent RTM message dispatch layer and needs to integrate Conversational AI events into that existing pipeline. * You need to explicitly control when a user's speech starts and ends. ## Understand the tech [#understand-the-tech] Manual turn control consists of two independent capabilities: * **Manual SoS**: The client explicitly declares that the user has started speaking. * **Manual EoS**: The client explicitly declares that the user has finished speaking. You can enable either capability independently: * Set only [`end_of_speech.mode`](/en/api-reference/api-ref/conversational-ai/join#properties-turn-detection-config-end-of-speech-mode) to `manual`: Suitable when start of speech is still determined by VAD or semantic detection, but end of speech is submitted explicitly by the user. * Set both [`start_of_speech.mode`](/en/api-reference/api-ref/conversational-ai/join#properties-turn-detection-config-start-of-speech-mode) and `end_of_speech.mode` to `manual`: Suitable for a complete push-to-talk experience. ## Prerequisites [#prerequisites] Before you begin, ensure that you have: * Completed the basic steps for interacting with an agent. See [Quickstart](/en/ai/get-started/quickstart). * Integrated the RTM SDK and implemented basic RTM login, subscription, and message listening logic. See [RTM quickstart](https://docs.agora.io/en/signaling/get-started/sdk-quickstart). * Set `advanced_features.enable_rtm = true` when creating the agent. ## Implement manual control [#implement-manual-control] ### Enable manual turn control when starting the agent [#enable-manual-turn-control-when-starting-the-agent] When calling [Start a conversational AI agent](/en/api-reference/api-ref/conversational-ai/join), first enable RTM, then decide whether to manually control SoS, EoS, or both, depending on your use case. The following `curl` examples show two common configurations: Manual EoS only Manual SoS + EoS ```shell curl --request POST \ --url https://api.agora.io/api/conversational-ai-agent/v2/projects//join \ --header 'Authorization: agora token="007abcxxxxxxx123"' \ --data ' { "name": "manual-eos-agent", "properties": { "channel": "channel_name", "token": "token", "agent_rtc_uid": "0", "remote_rtc_uids": [ "123" ], "advanced_features": { "enable_rtm": true }, "asr": { "language": "en-US" }, "llm": { "url": "https://api.xxxx/v1/xxxx", "api_key": "xxx", "system_messages": [ { "role": "system", "content": "You are a helpful chatbot." } ], "greeting_message": "Hello, how can I help you?", "failure_message": "Sorry, I am unable to answer that question.", "max_history": 10, "params": { "model": "xxxx" } }, "tts": { "vendor": "minimax", "params": { "key": "your-minimax-key", "model": "speech-01-turbo", "voice_setting": { "voice_id": "female-shaonv", "speed": 1, "vol": 1, "pitch": 0, "emotion": "happy" }, "audio_setting": { "sample_rate": 16000 } } }, "turn_detection": { "mode": "default", "config": { "start_of_speech": { "mode": "vad" }, "end_of_speech": { "mode": "manual" } } } } } ' ``` ```shell curl --request POST \ --url https://api.agora.io/api/conversational-ai-agent/v2/projects//join \ --header 'Authorization: agora token="007abcxxxxxxx123"' \ --data ' { "name": "manual-sos-eos-agent", "properties": { "channel": "channel_name", "token": "token", "agent_rtc_uid": "0", "remote_rtc_uids": [ "123" ], "advanced_features": { "enable_rtm": true }, "asr": { "language": "en-US" }, "llm": { "url": "https://api.xxxx/v1/xxxx", "api_key": "xxx", "system_messages": [ { "role": "system", "content": "You are a helpful chatbot." } ], "greeting_message": "Hello, how can I help you?", "failure_message": "Sorry, I am unable to answer that question.", "max_history": 10, "params": { "model": "xxxx" } }, "tts": { "vendor": "minimax", "params": { "key": "your-minimax-key", "model": "speech-01-turbo", "voice_setting": { "voice_id": "female-shaonv", "speed": 1, "vol": 1, "pitch": 0, "emotion": "happy" }, "audio_setting": { "sample_rate": 16000 } } }, "turn_detection": { "mode": "default", "config": { "start_of_speech": { "mode": "manual" }, "end_of_speech": { "mode": "manual" } } } } } ' ``` After you start the agent, save the following information for use in later steps: * `channelName`: The name of the channel used for this session. * `userId`: The RTM login ID of the current user. * `agentUserId`: The RTM user ID of the agent, which typically matches `agent_rtc_uid`. ### Log in to RTM and subscribe to channel messages [#log-in-to-rtm-and-subscribe-to-channel-messages] Before sending manual SoS/EoS requests, the client must complete RTM login and subscribe to `channelName`. This ensures you receive the agent's processing results promptly. Android ```kotlin val rtmConfig = RtmConfig.Builder(appId, userId.toString()).build() val rtmClient = RtmClient.create(rtmConfig) rtmClient.addEventListener(object : RtmEventListener { override fun onMessageEvent(event: MessageEvent?) { event ?: return val message = event.message ?: return val rawJson = when (message.type) { RtmConstants.RtmMessageType.BINARY -> { val bytes = message.data as? ByteArray ?: return String(bytes, Charsets.UTF_8) } else -> message.data as? String ?: return } handleAgentRtmMessage( publisherId = event.publisherId.orEmpty(), channelName = event.channelName.orEmpty(), customType = event.customType.orEmpty(), rawJson = rawJson ) } }) rtmClient.login(rtmToken, object : ResultCallback { override fun onSuccess(responseInfo: Void?) { val options = SubscribeOptions().apply { withMessage = true withPresence = true } rtmClient.subscribe(channelName, options, object : ResultCallback { override fun onSuccess(responseInfo: Void?) { // RTM subscription succeeded. } override fun onFailure(errorInfo: ErrorInfo) { // Handle subscription failure. } }) } override fun onFailure(errorInfo: ErrorInfo) { // Handle RTM login failure. } }) ``` Info Login to RTM and subscribe to the channel before starting the agent to avoid missing callback messages sent during the agent's startup phase. ### Send manual SoS when the user starts speaking [#send-manual-sos-when-the-user-starts-speaking] When your business logic determines that the user has started speaking, send an RTM USER peer-to-peer message to `agentUserId`. Use `RtmConstants.RtmChannelType.USER` as the channel type, and set `customType` to `user.manual_sos`. Android ```kotlin fun publishManualSos( rtmClient: RtmClient, agentUserId: String, completion: (String, ErrorInfo?) -> Unit ) { val requestId = "sos-req-${System.currentTimeMillis()}-${UUID.randomUUID()}" val body = JSONObject(mapOf("request_id" to requestId)).toString() val options = PublishOptions().apply { setChannelType(RtmConstants.RtmChannelType.USER) setCustomType("user.manual_sos") } rtmClient.publish(agentUserId, body, options, object : ResultCallback { override fun onSuccess(responseInfo: Void?) { completion(requestId, null) } override fun onFailure(errorInfo: ErrorInfo) { completion(requestId, errorInfo) } }) } ``` The `publish` call sends the following RTM message to the agent. ```json { "customType": "user.manual_sos", "publisher": "123", "message": "{\"request_id\":\"sos-req-20260612-001\"}" } ``` Info * A successful RTM `publish` call only confirms that the message was sent. * Whether the agent actually accepted the SoS request is confirmed by the `user.manual_sos.result` callback. ### Send manual EoS when the user finishes speaking [#send-manual-eos-when-the-user-finishes-speaking] When your business logic determines that the user has finished speaking, send an RTM USER peer-to-peer message to `agentUserId`. Use `RtmConstants.RtmChannelType.USER` as the channel type, and set `customType` to `user.manual_eos`. Android ```kotlin fun publishManualEos( rtmClient: RtmClient, agentUserId: String, completion: (String, ErrorInfo?) -> Unit ) { val requestId = "eos-req-${System.currentTimeMillis()}-${UUID.randomUUID()}" val body = JSONObject(mapOf("request_id" to requestId)).toString() val options = PublishOptions().apply { setChannelType(RtmConstants.RtmChannelType.USER) setCustomType("user.manual_eos") } rtmClient.publish(agentUserId, body, options, object : ResultCallback { override fun onSuccess(responseInfo: Void?) { completion(requestId, null) } override fun onFailure(errorInfo: ErrorInfo) { completion(requestId, errorInfo) } }) } ``` The `publish` call sends the following RTM message to the agent. ```json { "customType": "user.manual_eos", "publisher": "123", "message": "{\"request_id\":\"eos-req-20260612-001\"}" } ``` Info * A successful RTM `publish` call only confirms that the message was sent. * Whether the agent actually accepted the EoS request is confirmed by the `user.manual_eos.result` callback. ### Listen for and handle callback messages [#listen-for-and-handle-callback-messages] The results of manual SoS/EoS requests are returned as RTM channel messages. After receiving a message, parse the message body as JSON, then use the `event_type` field in the inner `message` to distinguish the event type. #### Identify callback message types [#identify-callback-message-types] Common callback types include: * `user.manual_sos.result`: The server's processing result for a client manual SoS request. * `user.manual_eos.result`: The server's processing result for a client manual EoS request. * `assistant.manual_eos.result`: In manual mode, the server automatically triggers EoS due to a protective policy, such as a maximum speaking duration limit. user.manual_sos.result user.manual_eos.result assistant.manual_eos.result ```json { "event_type": "user.manual_sos.result", "event_id": "evt_xxx", "event_ms": 1773901235435, "payload": { "success": true, "request_id": "sos-req-1773901235000-a1b2c3d4", "turn_id": 12 } } ``` ```json { "event_type": "user.manual_eos.result", "event_id": "evt_xxx", "event_ms": 1773901240000, "payload": { "success": true, "request_id": "eos-req-1773901240000-b2c3d4e5", "turn_id": 12 } } ``` ```json { "event_type": "assistant.manual_eos.result", "event_id": "evt_xxx", "event_ms": 1773901250000, "payload": { "reason": "max_duration_reached", "max_duration_ms": 60000, "turn_id": 12 } } ``` #### Handle result callbacks using `request_id` [#handle-result-callbacks-using-request_id] In your RTM message callback, Agora recommends consistently parsing the `event_type`, `request_id`, `success`, and `turn_id` fields, and updating your app state based on the result. Android ```kotlin data class ManualResult( val type: String, val requestId: String, val success: Boolean, val turnId: Long?, val errorMessage: String? ) fun handleAgentRtmMessage( publisherId: String, channelName: String, customType: String, rawJson: String ) { val json = JSONObject(rawJson) val type = json.optString("event_type").ifEmpty { json.optString("object") } val payload = json.optJSONObject("payload") when (type) { "user.manual_sos.result", "user.manual_eos.result" -> { if (payload == null) return val result = ManualResult( type = type, requestId = payload.optString("request_id"), success = payload.optBoolean("success", false), turnId = payload.optLongOrNull("turn_id"), errorMessage = payload.optString("error_message").ifEmpty { null } ) onUserManualResult(publisherId, result) } "assistant.manual_eos.result" -> { if (payload == null) return val reason = payload.optString("reason") val maxDurationMs = payload.optLong("max_duration_ms") val turnId = payload.optLong("turn_id") onAgentManualEos( agentUserId = publisherId, reason = reason, maxDurationMs = maxDurationMs, turnId = turnId ) } } } fun JSONObject.optLongOrNull(name: String): Long? { return if (has(name) && !isNull(name)) optLong(name) else null } ``` Pay particular attention to the following when implementing this logic: * Use `request_id` to correlate the result with the SoS/EoS request you previously sent. * Use `publisherId` to identify the agent's RTM user ID. * Use the `event_type` field in the message body to distinguish event types. Do not rely on the outer RTM message's `customType`. ## See also [#see-also] * [Get agent state](/en/ai/best-practices/get-agent-state) * [Interrupt agent](/en/ai/build/shape-the-conversation/interrupt-agent) * [Receive webhook notifications](/en/ai/build/handle-runtime-events/webhooks) * [Start a conversational AI agent](/en/api-reference/api-ref/conversational-ai/join) # Optimize latency (/en/ai/best-practices/optimize-latency) Latency is a key factor affecting user experience in conversational AI agent scenarios. This guide helps you understand and optimize conversation latency to improve user experience. ## How latency works [#how-latency-works] Understanding latency composition helps you identify optimization opportunities. ### Cascaded architecture latency [#cascaded-architecture-latency] When using the cascaded architecture with ASR, LLM, and TTS components, end-to-end latency consists of the following: ```text End-to-end latency = RTC latency + Algorithm preprocessing latency + ASR latency + LLM latency + TTS latency (+ Avatar latency) ``` ### Latency for each component [#latency-for-each-component] The following table shows typical latency ranges for each component based on actual test data: | Component | Latency metric | Description | Typical latency range (ms) | | ----------------------- | ----------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------- | | RTC | Audio and video latency | Includes audio capture, encoding, network transmission, decoding, and playback | 150-300 | | Algorithm preprocessing | Preprocessing latency | Includes VAD (Voice Activity Detection), intelligent interruption handling (AIVAD), and other algorithm processing time | 720-940\* | | ASR | `asr_ttlw` | Time To Last Word. The latency from when the user stops speaking to when ASR outputs the last word. | 400-700 | | LLM | `llm_ttfb` / `llm_ttfs` | TTFB: Time To First Byte, the first byte latency.
TTFS: Time To First Sentence, the first sentence latency. | 250-1000 | | TTS | `tts_ttfb` | Time To First Byte. The response latency from when the TTS request starts to when the first byte is received. | 100-350 | | Avatar rendering | Rendering latency | The latency from when the avatar module receives the first frame of TTS audio to when it generates and synchronizes the first frame of audio and video (if enabled) | 50-200 | \* Algorithm preprocessing latency is based on `silence_duration_ms` set to 640 ms. Adjusting this parameter affects preprocessing latency. Info Test data shows that LLM typically contributes the most to overall latency. Optimizing LLM selection and configuration is key to reducing end-to-end latency. ### Real-world latency example [#real-world-latency-example] The following example shows latency data from three conversation turns in an actual conversation. This data comes from the [`111 agent metrics`](../reference/event-types#111-agent-metrics) event in Agora's [message notification service](../build/handle-runtime-events/webhooks): ```json { "metrics": [ { "turn_id": 1, "tts_ttfb": 61 }, { "turn_id": 2, "asr_ttlw": 141, "llm_ttfb": 270, "llm_ttfs": 482, "tts_ttfb": 90 }, { "turn_id": 3, "asr_ttlw": 103, "llm_ttfb": 306, "llm_ttfs": 948, "tts_ttfb": 106 } ] } ``` {/* ### MLLM architecture latency When using MLLM (Multimodal Large Language Model), the model processes audio input directly and generates audio output without separate ASR, LLM, and TTS components. This results in simpler latency composition: ```text End-to-end latency = RTC latency + Algorithm preprocessing latency + MLLM latency ``` You can obtain the `mllm_ttfb` metric by listening to the `111 agent metrics` event. This metric represents the response latency from when the MLLM request starts to when the first byte is received. :::info[Info] * MLLM architecture typically achieves lower end-to-end latency by reducing data conversion and transmission between components. * When MLLM is enabled, the graceful interruption (AIVAD) feature is disabled, which reduces algorithm preprocessing latency. ::: */} ## Monitor latency metrics [#monitor-latency-metrics] The Conversational AI engine provides two ways to monitor latency metrics for each conversation turn: ### Use client components [#use-client-components] If you use client components (Android, iOS, or Web), you can listen to agent performance metrics in real time by registering the `onAgentMetrics` callback. Android iOS Web ```kotlin api.addHandler(object : IConversationalAIAPIEventHandler { override fun onAgentMetrics(agentUserId: String, metric: Metric) { when (metric.type) { ModuleType.ASR -> { Log.d("Metrics", "ASR TTLW: \${metric.value}ms") } ModuleType.LLM -> { // metric.name can be "ttfb" or "ttfs" Log.d("Metrics", "LLM \${metric.name}: \${metric.value}ms") } ModuleType.TTS -> { Log.d("Metrics", "TTS TTFB: \${metric.value}ms") } ModuleType.TOTAL -> { Log.d("Metrics", "Total Delay: \${metric.value}ms") } else -> { Log.d("Metrics", "\${metric.type}: \${metric.name} = \${metric.value}ms") } } } }) ``` ```swift func onAgentMetrics(agentUserId: String, metrics: Metric) { switch metrics.type { case .asr: print("ASR TTLW: \(metrics.value)ms") case .llm: print("LLM \(metrics.name): \(metrics.value)ms") case .tts: print("TTS TTFB: \(metrics.value)ms") case .total: print("Total Delay: \(metrics.value)ms") case .unknown: print("Unknown metric: \(metrics.name) = \(metrics.value)ms") } } ``` ```ts conversationalAIAPI.on( EConversationalAIAPIEvents.AGENT_METRICS, (agentUserId: string, metrics: Metric) => { console.log(`[\${metrics.type}] \${metrics.name}: \${metrics.value}ms`); if (metrics.type === 'TOTAL') { console.log(`Total delay for turn: \${metrics.value}ms`); } } ); ``` For detailed integration steps and API reference, see [Webhooks](../build/handle-runtime-events/webhooks). ### Use Message Notification Service [#use-message-notification-service] If you have enabled Agora's Message Notification Service, you can obtain agent performance metrics by receiving the `agent metrics` event where `eventType` is `111`. **Event callback example** ```json { "noticeId": "2000001428:4330:107", "productId": 17, "eventType": 111, "notifyMs": 1611566412672, "payload": { "agent_id": "A42AC47Hxxxxxxxx4PK27ND25E", "start_ts": 1000, "stop_ts": 1672531200, "channel": "test-channel", "metrics": [ { "turn_id": 1, "tts_ttfb": 61 }, { "turn_id": 2, "asr_ttlw": 141, "llm_ttfb": 270, "llm_ttfs": 482, "tts_ttfb": 90, } ] } } ``` For detailed event field descriptions, see [Notification event 111 agent metrics](../reference/event-types#111-agent-metrics). ## Optimize cascaded architecture latency [#optimize-cascaded-architecture-latency] To reduce latency in cascaded architecture, focus on optimizing individual components, geographic deployment, and RTC settings. ### Optimize LLM, ASR, and TTS components [#optimize-llm-asr-and-tts-components] LLM is typically the component that contributes the most to latency. Optimizing LLM can significantly reduce overall latency. **Choose low-latency vendors** LLM, ASR, and TTS vendors and models vary significantly in response speed. Refer to the [Conversational AI Performance Lab](https://www.agora.io/en/conversational-ai-performance-lab) to compare performance metrics across vendors. **Optimize parameter configuration** When creating an agent, read the vendor documentation for ASR, LLM, and TTS to understand available parameters and tune them for your use case. The following are general optimization approaches: * **LLM** * **Choose smaller models**: Models such as `gpt-4o-mini` and `gemini-2.5-flash` typically respond faster than larger models. * **Limit `max_tokens`**: Reducing the maximum number of tokens generated can lower TTFS (first sentence latency). * **Enable streaming response**: Ensure `stream: true` so the agent can start speaking as soon as possible. * **ASR** * **Use vendor-recommended sampling rate**: Use the sampling rate recommended by the vendor, such as 16 kHz, to avoid unnecessary resampling. * **Limit language model**: Use the `phrases` or `context` parameter to provide domain-specific vocabulary and improve recognition speed. * **Disable non-essential features**: Some ASR vendors provide advanced parameters such as punctuation and tone output. You can disable these based on your use case to improve response speed. * **TTS** * **Choose faster modes**: Some TTS providers offer modes such as turbo or low-latency, which typically respond faster than default mode. * **Choose simpler voices**: Some TTS providers offer voices with varying complexity. Choosing less complex voices can reduce generation time. * **Disable non-essential features**: Some TTS vendors provide advanced parameters such as profanity\_filter, punctuation\_filter, and diarization. You can disable these based on your use case to improve response speed. The following example shows how to optimize LLM parameters: ```json { "properties": { "llm": { "url": "https://api.openai.com/v1/chat/completions", "api_key": "your_api_key", "params": { "model": "gpt-4o-mini", // Select the faster-responding model "temperature": 0.7, "max_tokens": 150, // Limit the generation length to reduce latency "stream": true // Enable streaming response } } } } ``` {/* ### Optimize geographic deployment **Understand the default location selection strategy** When regional access restrictions are not enabled or when using the `GLOBAL` region, the Conversational AI engine automatically selects the nearest server to deploy the agent based on the IP address of the configured LLM URL. The system includes the following capabilities: 1. **Smart deployment**: The system deploys the Conversational AI engine service in the region that corresponds to the IP address of the LLM URL. 2. **Proximity selection**: If no servers are available in the corresponding region, the system selects the nearest available region. 3. **Failover**: When the service in a region becomes unavailable, the system switches to another available region. **Optimize deployment region** To minimize network latency, Agora recommends: 1. **Deploy ASR, LLM, and TTS in the same region**: Choose vendors that support the same region to reduce cross-region network latency. 2. **Use regional access restrictions**: If your users are concentrated in a specific region, use the `geofence` parameter to restrict the Conversational AI engine to servers in that region: ```json { "properties": { "geofence": { "area": "NORTH_AMERICA" // Or EUROPE, ASIA, etc. }, "llm": { "url": "https://api.openai.com/v1/chat/completions" // Use LLM endpoint in North America region }, "tts": { "url": "wss://api.elevenlabs.io/v1/text-to-speech/{voice_id}/stream-input" // Use TTS endpoint in North America region } } } ``` For detailed regional access restriction configuration, see [Restrict agent zones](regional-restrictions). */} ### Optimize RTC latency [#optimize-rtc-latency] RTC latency includes audio capture, encoding, network transmission, decoding, and playback. To optimize RTC latency, configure audio settings on the client side. **Use AI conversation scenario** Agora RTC SDK 4.5.1 and later supports the AI conversation scenario (`AUDIO_SCENARIO_AI_CLIENT`), which is specifically optimized for AI conversations and includes: * Optimized audio 3A algorithms (echo cancellation, noise reduction, and gain control) * Lower audio capture and playback latency * Audio processing tailored to AI voice characteristics Android iOS **Use the client-side component API (recommended)** ```kotlin val config = ConversationalAIAPIConfig( rtcEngine = rtcEngineInstance, rtmClient = rtmClientInstance, enableLog = true ) val api = ConversationalAIAPIImpl(config) // Load optimal audio settings api.loadAudioSettings() ``` **Configure the RTC SDK directly** ```kotlin val config = RtcEngineConfig() config.mAudioScenario = Constants.AUDIO_SCENARIO_AI_CLIENT rtcEngine = RtcEngine.create(config) ``` **Use the client-side component API (recommended)** ```swift let config = ConversationalAIAPIConfig( rtcEngine: rtcEngine, rtmEngine: rtmEngine, enableLog: true ) convoAIAPI = ConversationalAIAPIImpl(config: config) // Load optimal audio settings convoAIAPI.loadAudioSettings() ``` **Configure the RTC SDK directly** ```swift let config = AgoraRtcEngineConfig() config.audioScenario = .aiClient rtcEngine = AgoraRtcEngineKit.sharedEngine(with: config, delegate: delegate) ``` For detailed audio setting optimization, see [Optimize audio](audio-setup). {/* ## Optimize MLLM architecture latency If you use an MLLM (Multimodal Large Language Model) architecture, optimizing latency primarily relies on choosing low-latency MLLM vendors. MLLM vendors vary in real-time performance. The Conversational AI engine currently supports: * OpenAI Realtime API: Provides low-latency real-time audio processing capabilities * Google Gemini Live: Supports multimodal real-time interaction For detailed MLLM integration guides, see the MLLM provider pages under `AI > Models > MLLM`. */} ## Latency optimization checklist [#latency-optimization-checklist] Use the following checklist to systematically optimize your conversational AI agent latency: ### Server-side optimization [#server-side-optimization] * **Choose low-latency LLM models**: Refer to [Conversational AI Performance Lab](https://www.agora.io/en/conversational-ai-performance-lab) to select models with excellent TTFT and throughput performance. * **Enable streaming response**: Ensure `stream: true`. * **Choose low-latency ASR and TTS vendors**: Enable low-latency or turbo modes when available. Refer to [Conversational AI Performance Lab](https://www.agora.io/en/conversational-ai-performance-lab) to select models with low latency and high throughput performance. * **Optimize geographic deployment**: Deploy ASR, LLM, and TTS in the same region. ### Client-side optimization [#client-side-optimization] * **Use AI conversation scenario**: Set `AUDIO_SCENARIO_AI_CLIENT` (RTC SDK 4.5.1 and later). * **Load optimal audio settings**: Call the `loadAudioSettings()` method of the client component. * **Integrate required audio plugins**: Ensure integration of AI noise reduction and AI echo cancellation plugins. * **Optimize network conditions**: Ensure stable user network connection and consider using SD-RTN™ to optimize network transmission. ### Monitoring and analysis [#monitoring-and-analysis] * **Monitor latency metrics in real time**: Obtain latency data for each conversation turn through client components or NCS. * **Identify latency bottlenecks**: Analyze which component contributes the most to latency. * **Continuously optimize**: Adjust configuration based on actual data and conduct A/B testing. ## Balance latency and quality [#balance-latency-and-quality] When optimizing latency, find a balance between response speed and conversation quality: | Optimization strategy | Latency impact | Quality impact | Recommended scenario | | ---------------------------- | ----------------------- | -------------------------- | ---------------------------------------------------------------- | | Use smaller LLM models | ✅ Significantly reduces | ⚠️ May reduce | Latency-sensitive scenarios with relatively simple conversations | | Limit `max_tokens` | ✅ Moderately reduces | ⚠️ May affect completeness | Scenarios requiring short responses | | Regional access restrictions | ✅ Moderately reduces | No impact | Users concentrated in a specific region | | Optimize RTC settings | ✅ Moderately reduces | No impact | All scenarios | Tip Monitor latency metrics to identify bottlenecks, then optimize the highest-contributing component rather than pursuing minimal latency across all components. ## References [#references] Refer to the following resources for further details. * [Webhooks](../build/handle-runtime-events/webhooks) * [Optimize audio](audio-setup) * [Notification event types](../reference/event-types) * [Conversational AI Performance Lab](https://www.agora.io/en/conversational-ai-performance-lab) {/* * MLLM provider pages under `AI > Models > MLLM` */} # Record agent conversations (/en/ai/best-practices/record-agent-conversation) This guide explains how to configure Agora [Cloud Recording](/en/api-reference/api-ref/cloud-recording) for Voice Agent conversations and avoid audio-scene conflicts when recording starts. ## Why this needs special handling [#why-this-needs-special-handling] When you run a Voice Agent and Cloud Recording in the same channel, recording startup can briefly interrupt audio. The issue is usually caused by the agent audio scene falling back to a default mode when Cloud Recording joins the channel. To keep audio stable, set the agent `audio_scenario` to `"chorus"` when you start the agent. ## Configure the agent [#configure-the-agent] Set the agent's RTC audio scenario to `chorus` when you configure the agent: Python SDK TypeScript SDK Go SDK REST API ```python from agora_agent import Agent # client is your configured Agora client agent = ( Agent(client) .with_stt(...) # configure your STT vendor .with_llm(...) # configure your LLM vendor .with_tts(...) # configure your TTS vendor .with_audio_scenario('chorus') ) ``` ```typescript import { Agent } from 'agora-agents'; // client is your configured Agora client const agent = new Agent({ client }) .withStt(/* configure your STT vendor */) .withLlm(/* configure your LLM vendor */) .withTts(/* configure your TTS vendor */) .withAudioScenario('chorus'); ``` ```go import "github.com/AgoraIO/agora-agents-go/v2/agentkit" // client is your configured *agentkit.AgoraClient agent := agentkit.NewAgent(client). WithStt(/* configure your STT vendor */). WithLlm(/* configure your LLM vendor */). WithTts(/* configure your TTS vendor */). WithAudioScenario(agentkit.AudioScenarioChorus) ``` Add or modify the `audio_scenario` field in the agent startup parameters: ```json { "properties": { "parameters": { "audio_scenario": "chorus" } } } ``` ## Complete example [#complete-example] The following example shows an agent launch request with the recommended recording-safe configuration: ```json { "name": "TestConvoAgent", "properties": { "channel": "{{AccessChannel}}", "token": "{{token}}", "agent_rtc_uid": "0", "remote_rtc_uids": [ "*" ], "idle_timeout": 30, "advanced_features": { "enable_bhvs": true }, "llm": { "url": "https://api.openai.com/v1/chat/completions", "api_key": "", "system_messages": [ { "role": "system", "content": "You are a helpful chatbot." } ], "greeting_message": "Hello, how can I help you?", "failure_message": "Sorry, I don't know how to answer this question.", "max_history": 10, "params": { "model": "gpt-4o-mini" } }, "tts": { "vendor": "microsoft", "params": { "key": "", "region": "eastus", "voice_name": "en-US-AndrewMultilingualNeural" } }, "turn_detection": { "silence_duration_ms": 640 }, "parameters": { "enable_dump": true, "enable_error_message": true, "audio_scenario": "chorus", "enable_delay": true } } } ``` ## Start recording [#start-recording] After the agent is running, call the [Start Cloud Recording API](/en/api-reference/api-ref/cloud-recording/start) to begin recording. Example Cloud Recording request body: ```json { "cname": "{{AccessChannel}}", "uid": "{{RecordingUID}}", "clientRequest": { "token": "{{token}}", "recordingConfig": { "maxIdleTime": 120, "streamTypes": 0, "audioProfile": 1, "channelType": 1 }, "recordingFileConfig": { "avFileType": [ "hls", "mp4" ] }, "storageConfig": { "vendor": "{{Vendor}}", "region": "{{Region}}", "bucket": "{{Bucket}}", "accessKey": "{{AccessKey}}", "secretKey": "{{SecretKey}}" } } } ``` ## Related pages [#related-pages] * [Start and stop an agent](../build/start-stop-agent) * [Cloud Recording API reference](/en/api-reference/api-ref/cloud-recording) # Restrict where agent can run (/en/ai/best-practices/regional-restrictions) import { Parameter, ParameterList } from '@/components/mdx'; To comply with laws and regulations in different countries and regions, the Conversational AI Engine supports regional access restrictions. When you enable regional access restrictions, the Conversational AI Engine only accesses Agora servers in the designated region, regardless of the user's location. For example, if you specify North America as the access region, two users who initiate calls from North America and Singapore have different experiences: | Specified access region | User location | Actual access region | User experience | |:-------|:-------|:-------|:-------| | North America | North America | North America | Normal quality. | | North America | Singapore | North America | Quality may be significantly impacted. Cross-regional public internet connections between the designated region and the user's location can result in poor network quality. If all servers in the specified region are unavailable, the service returns an error. | ## Understand the tech When regional access restrictions are disabled, the Conversational AI Engine automatically selects the nearest server to deploy the agent based on the IP address of the LLM URL and supports failover: 1. **Intelligent deployment**: The system automatically deploys the Conversational AI Engine service in the region that corresponds to the IP address of the configured LLM URL. 1. **Nearest available region**: If no server is available in the corresponding region, the system automatically selects the nearest available region. 1. **Failover**: When a service becomes unavailable in a region, the system automatically switches to another available region. :::caution[Caution] When you enable regional access restrictions, the system strictly limits access to the specified region and does not perform cross-regional failover. ::: ## Implementation ### Configure regional access To configure region access restrictions, set `properties.geofence` when you [Start a conversational AI agent](/en/api-reference/api-ref/conversational-ai/join). The allowed region for server access. The excluded region. Only available when `area` is set to `GLOBAL`. ### Configuration examples Refer to the following examples to understand `geofence` configuration. #### Restrict access to North America only Python SDK TypeScript SDK Go SDK REST API ```python from agora_agent import Agent from agora_agent.agentkit import GeofenceConfig # client is your configured Agora client agent = ( Agent(client) .with_llm(...) # configure your LLM vendor .with_tts(...) # configure your TTS vendor .with_geofence(GeofenceConfig( area='NORTH_AMERICA', )) ) ``` ```typescript import { Agent } from 'agora-agents'; // client is your configured Agora client const agent = new Agent({ client }) .withLlm(/* configure your LLM vendor */) .withTts(/* configure your TTS vendor */) .withGeofence({ area: 'NORTH_AMERICA', }); ``` ```go import "github.com/AgoraIO/agora-agents-go/v2/agentkit" // client is your configured *agentkit.AgoraClient agent := agentkit.NewAgent(client). WithLlm(/* configure your LLM vendor */). WithTts(/* configure your TTS vendor */). WithGeofence(&agentkit.GeofenceConfig{ Area: agentkit.GeofenceAreaNorthAmerica, }) ``` Use the following request body when you [Start a conversational AI agent](/en/api-reference/api-ref/conversational-ai/join): ```json { "name": "customer_service", "properties": { "channel": "test_channel", "token": "your_rtc_token", "agent_rtc_uid": "123", "remote_rtc_uids": ["456"], "geofence": { "area": "NORTH_AMERICA" }, "llm": { // ... LLM configuration }, "tts": { // ... TTS configuration } } } ``` #### Allow global access but exclude India Python SDK TypeScript SDK Go SDK REST API ```python from agora_agent import Agent from agora_agent.agentkit import GeofenceConfig # client is your configured Agora client agent = ( Agent(client) .with_llm(...) # configure your LLM vendor .with_tts(...) # configure your TTS vendor .with_geofence(GeofenceConfig( area='GLOBAL', exclude_area='INDIA', )) ) ``` ```typescript import { Agent } from 'agora-agents'; // client is your configured Agora client const agent = new Agent({ client }) .withLlm(/* configure your LLM vendor */) .withTts(/* configure your TTS vendor */) .withGeofence({ area: 'GLOBAL', exclude_area: 'INDIA', }); ``` ```go import "github.com/AgoraIO/agora-agents-go/v2/agentkit" // client is your configured *agentkit.AgoraClient agent := agentkit.NewAgent(client). WithLlm(/* configure your LLM vendor */). WithTts(/* configure your TTS vendor */). WithGeofence(&agentkit.GeofenceConfig{ Area: agentkit.GeofenceAreaGlobal, ExcludeArea: agentkit.GeofenceExcludeAreaIndia.Ptr(), }) ``` Use the following request body when you [Start a conversational AI agent](/en/api-reference/api-ref/conversational-ai/join): ```json { "name": "customer_service", "properties": { "channel": "test_channel", "token": "your_rtc_token", "agent_rtc_uid": "123", "remote_rtc_uids": ["456"], "geofence": { "area": "GLOBAL", "exclude_area": "INDIA" }, "llm": { // ... LLM configuration }, "tts": { // ... TTS configuration } } } ``` ### Configure RTC regional access The Conversational AI Engine and Agora RTC services are independent, and their regional access restriction configurations are separate. To implement complete end-to-end regional access restrictions and ensure that the entire call chain with the agent is confined to a specified region, you must also configure RTC regional access restrictions. For more information, see [Restrict media zones](../../realtime-media/video/build/manage-connection-and-quality/geofencing). ## Data residency In addition to configuring regional access restrictions for the Conversational AI Engine, some LLM, TTS, and ASR vendors offer data residency services in different regions. You can ensure that data is not transferred across regions by configuring the URLs used by the LLM, TTS, and ASR modules. For example, ElevenLabs supports selecting data processing regions through different URL endpoints. For more information, see the [ElevenLabs documentation](https://elevenlabs.io/docs/overview/administration/data-residency). The following example shows how to configure the ElevenLabs TTS service using a URL endpoint for the European region: ```json { "properties": { "tts": { "vendor": { "name": "elevenlabs" }, "url": "wss://api.eu.elevenlabs.io/v1/text-to-speech/{voice_id}/stream-input" } } } ``` ## Best practices To ensure that data remains entirely within the designated region, follow these recommendations: 1. **Configure regional restrictions for the Conversational AI Engine**: Use the `geofence` object to restrict server access regions. 2. **Configure RTC regional restrictions**: Configure the corresponding regional access restrictions in the RTC SDK. 3. **Select regional AI services**: When configuring LLM, TTS, and ASR services, select a provider that supports regional data residency and use the corresponding regional URL endpoint. 4. **Verify the configuration**: Before deploying to production, thoroughly test that regional access restrictions work correctly and that data is not transferred across regions. ## Considerations Keep the following in mind when implementing regional access restrictions: - **Performance impact**: Regional access restrictions may affect user experience. If a user's location is far from the designated region, network latency may increase. - **Service availability**: When regional access restrictions are enabled and the server in the specified region is unavailable or lacks sufficient resources, the service returns an error and does not automatically switch to another region. - **Configuration consistency**: Ensure that the regional configurations of the Conversational AI Engine, RTC SDK, and LLM, TTS, and ASR services are consistent to prevent data transfer across regions. - **Compliance requirements**: Before configuring regional access restrictions, understand the legal and regulatory requirements of the target region to ensure that the configuration complies with local data protection and privacy regulations. # Architecture (/en/ai/build/architecture) The TEN framework uses a graph structure of interconnected nodes to define data flow between extensions. Subgraphs extend this architecture by providing a powerful reuse mechanism that allows you to break down complex graph structures into modular, reusable components. Subgraphs improve code organization and maintainability by enabling you to create self-contained graph modules that can be shared across different applications or used multiple times within the same application. This modular approach simplifies the development of complex systems while maintaining the same runtime performance characteristics as regular graphs. ## Understand the tech [#understand-the-tech] Subgraphs preserve the fundamental principle of TEN graphs: defining how data flows between extensions. Rather than introducing new runtime complexity, subgraphs act as a development-time abstraction that gets flattened into the larger graph structure during deployment. This design approach provides the organizational benefits of modularity while using the same graph execution mechanism that powers all TEN applications. ### Design principles [#design-principles] Subgraphs are built on the following foundational principles that ensure they integrate seamlessly with the TEN framework while providing meaningful development benefits: * **Independence**: Every subgraph is a complete, self-contained graph that can run independently or be embedded as a component within larger graphs. This independence means you can develop, test, and validate subgraphs in isolation before integrating them into complex applications. * **Tool-friendly design**: Subgraphs provide structured metadata through `exposed_messages` and `exposed_properties` that development tools can use to offer intelligent suggestions, validation, and debugging capabilities. This enhanced tooling support improves the development experience without adding runtime overhead. * **Flattening mechanism**: At deployment time, subgraphs are flattened into standard graph structures, ensuring that runtime performance remains identical to regular graphs. This approach provides development-time modularity benefits while maintaining the proven performance characteristics of the core TEN graph engine. * **Simplicity**: Subgraph design prioritizes simplicity. Rather than providing patching mechanisms or complex configuration options at reference points, subgraphs use straightforward mapping and exposure patterns. If a subgraph needs modification, you modify the original definition directly, avoiding complexity cascades and maintenance overhead. This design approach allows you to treat subgraphs as black boxes. You can use them without understanding their internal complexity, while the framework handles the technical details of integration and execution. ### Key features [#key-features] Subgraphs provide four essential features that enable modular graph development and seamless integration within TEN applications: * **Message exposure**: Subgraphs declare their external interfaces through the `exposed_messages` field, specifying which message types (commands, data, audio, video) can be sent to or received from the subgraph. This exposure mechanism allows development tools to provide intelligent autocomplete and validation while hiding internal implementation details from external consumers. * **Property exposure**: Through the `exposed_properties` field, subgraphs can selectively expose internal extension properties to external configuration. This controlled exposure enables customization of subgraph behavior without revealing the entire internal structure, maintaining encapsulation while providing necessary flexibility. * **Namespace management**: Each referenced subgraph creates its own namespace, preventing naming conflicts between elements in different subgraphs. When you reference a subgraph with name `subgraph_1`, all internal elements become accessible through the namespace syntax (e.g., `subgraph_1:ext_name`), ensuring global uniqueness across complex graph hierarchies. * **Cross-graph connections**: Subgraphs enable sophisticated connection patterns that span multiple graph boundaries. You can connect directly to subgraph interfaces, connect to specific elements within subgraphs, or even establish connections between elements in different subgraphs. This flexibility supports complex data flow architectures while maintaining clear organizational boundaries. These features work together to provide a powerful abstraction layer that simplifies complex graph development without sacrificing the flexibility and performance that TEN applications require. ## Creating subgraphs [#creating-subgraphs] ### Subgraph definition [#subgraph-definition] A subgraph is defined in a standard JSON file that contains the same basic structure as any TEN graph, with additional metadata for external interfaces. Here's a basic example: ```json { "nodes": [ { "type": "extension", "name": "audio_processor", "addon": "audio_enhancement" }, { "type": "extension", "name": "audio_filter", "addon": "noise_reduction" } ], "connections": [ { "extension": "audio_processor", "audio_frame": [ { "name": "processed_audio", "dest": [ { "extension": "audio_filter" } ] } ] } ], "exposed_messages": [ { "extension": "audio_processor", "type": "audio_frame_in", "name": "raw_audio" }, { "extension": "audio_filter", "type": "audio_frame_out", "name": "clean_audio" } ], "exposed_properties": [ { "extension": "audio_processor", "name": "sample_rate", "alias": "input_sample_rate" }, { "extension": "audio_filter", "name": "noise_threshold", "alias": "filter_sensitivity" } ] } ``` Following is the structure breakdown: * **Nodes section**: Defines the extensions that make up the subgraph, just like in a regular graph. Each extension specifies its type, name, and addon identifier. * **Connections section**: Establishes data flow between extensions within the subgraph. These internal connections define how the subgraph processes data internally. * **Exposed messages section**: Declares which message interfaces are available for external connections. This acts as the subgraph's public API, allowing other graphs to send data to or receive data from the subgraph without knowing its internal structure. * **Exposed properties section**: Specifies which extension properties can be configured externally. The `alias` field provides user-friendly names for external configuration, abstracting away internal implementation details. This sample subgraph can function as a standalone audio processing pipeline or be embedded within larger applications that need audio enhancement capabilities. ### Exposed interfaces [#exposed-interfaces] Subgraphs use two interface types to control what external graphs can access and configure. These interfaces act as the subgraph's public API, providing controlled access while maintaining internal encapsulation. #### exposed\_messages [#exposed_messages] The `exposed_messages` field defines which message interfaces are available for external connections. This allows other graphs to interact with your subgraph without needing to understand its internal structure. **Message exposure structure:** ```json { "exposed_messages": [ { "extension": "internal_extension_name", "type": "message_direction_and_type", "name": "message_name" } ] } ``` **Message types:** * `cmd_in` / `cmd_out`: Command messages for control and responses * `data_in` / `data_out`: General data messages * `audio_frame_in` / `audio_frame_out`: Audio stream data * `video_frame_in` / `video_frame_out`: Video stream data **Example:** ```json { "exposed_messages": [ { "extension": "speech_recognizer", "type": "audio_frame_in", "name": "audio_input" }, { "extension": "text_processor", "type": "data_out", "name": "transcribed_text" }, { "extension": "speech_recognizer", "type": "cmd_in", "name": "configure_language" } ] } ``` This example exposes an audio input, text output, and configuration command, allowing external graphs to send audio data and receive transcribed text while configuring the recognition language. #### exposed\_properties [#exposed_properties] The `exposed_properties` field specifies which internal extension properties can be configured from outside the subgraph. This enables customization without exposing the entire internal configuration. **Property exposure structure:** ```json { "exposed_properties": [ { "extension": "internal_extension_name", "name": "internal_property_name", "alias": "external_property_name" } ] } ``` * `extension`: The internal extension that owns the property * `name`: The actual property name within the extension * `alias`: The external name used when referencing the subgraph (optional) **Example:** ```json { "exposed_properties": [ { "extension": "video_encoder", "name": "bitrate_kbps", "alias": "output_quality" }, { "extension": "frame_buffer", "name": "max_buffer_size", "alias": "buffer_limit" }, { "extension": "video_encoder", "name": "codec_type" } ] } ``` In this example, external graphs can configure video quality through the `output_quality` alias, set buffer limits through `buffer_limit`, and specify codec type directly. The aliases provide user-friendly names while hiding implementation details. Exposed interfaces provide the following benefits: * **Encapsulation**: Hide internal complexity while providing clean external APIs * **Selective exposure**: Allow control over which interfaces and properties can be accessed externally * **Interface documentation**: Serve as documentation for subgraph capabilities ## Using subgraphs [#using-subgraphs] ### Referencing subgraphs [#referencing-subgraphs] To use a subgraph in your main graph, reference it as a node with type `subgraph`. The subgraph becomes part of your graph's namespace and can be connected like any other component. **Basic subgraph reference:** ```json { "nodes": [ { "type": "extension", "name": "input_source", "addon": "audio_input" }, { "type": "extension", "name": "output_sink", "addon": "audio_output" }, { "type": "subgraph", "name": "audio_pipeline", "source_uri": "./ten_packages/extension/audio_processing/subgraph.json" } ], "connections": [ { "extension": "input_source", "audio_frame": [ { "name": "raw_audio", "dest": [ { "subgraph": "audio_pipeline" } ] } ] }, { "subgraph": "audio_pipeline", "audio_frame": [ { "name": "processed_audio", "dest": [ { "extension": "output_sink" } ] } ] } ] } ``` **Subgraph with properties:** You can configure exposed properties when referencing a subgraph: ```json { "type": "subgraph", "name": "video_processor", "source_uri": "./subgraphs/video_enhancement.json", "property": { "output_quality": "high", "buffer_limit": 1024, "codec_type": "h264" } } ``` The properties you specify must exist in the subgraph's `exposed_properties` field, otherwise the configuration is rejected. ### Connection patterns [#connection-patterns] Subgraphs support multiple connection patterns that provide flexibility in how you route data between components. #### Direct connections to subgraphs [#direct-connections-to-subgraphs] Connect directly to a subgraph using its name when the subgraph exposes the appropriate message interface: ```json { "connections": [ { "extension": "data_source", "cmd": [ { "name": "process_data", "dest": [ { "subgraph": "data_processor" } ] } ] } ] } ``` This pattern works when `data_processor` exposes a `cmd_in` interface for `process_data`. The connection is clean and hides the subgraph's internal structure. #### Connections to elements within subgraphs [#connections-to-elements-within-subgraphs] For more granular control, connect directly to specific extensions within a subgraph using namespace syntax: ```json { "connections": [ { "extension": "external_source", "data": [ { "name": "raw_input", "dest": [ { "extension": "video_pipeline:frame_buffer" } ] } ] }, { "extension": "video_pipeline:encoder", "video_frame": [ { "name": "encoded_output", "dest": [ { "extension": "external_sink" } ] } ] } ] } ``` The `subgraph_name:extension_name` syntax allows you to bypass the subgraph's exposed interfaces and connect directly to internal elements. Use this pattern when you need fine-grained control over data flow. #### Advanced connection syntax [#advanced-connection-syntax] You can mix different connection patterns within the same graph to create sophisticated data flow architectures: ```json { "connections": [ { "extension": "input_hub", "data": [ { "name": "sensor_data", "dest": [ { "subgraph": "processing_pipeline" }, { "extension": "backup_pipeline:data_logger" }, { "extension": "monitoring_extension" } ] } ] }, { "subgraph": "processing_pipeline", "cmd": [ { "name": "status_update", "dest": [ { "extension": "monitoring_extension" }, { "subgraph": "alert_system" } ] } ] } ] } ``` **Connection pattern guidelines:** * **Use direct subgraph connections** when working with well-defined interfaces * **Use namespace syntax** when you need to access specific internal functionality * **Mix patterns** when building complex data flow architectures * **Prefer exposed interfaces** for maintainability and clarity These patterns give you the flexibility to build everything from simple linear pipelines to complex multi-path processing architectures while maintaining clear organizational boundaries. ## Flattening process [#flattening-process] ### How flattening works [#how-flattening-works] Before your application runs, the TEN framework automatically flattens all subgraph references into a single, standard graph structure. This flattening process converts the modular development structure into the standard graph format that the TEN runtime executes. **Flattening transformations:** 1. **Namespace resolution**: Subgraph references are resolved and internal elements are prefixed with the subgraph name to ensure global uniqueness. 2. **Node expansion**: Subgraph nodes are replaced with their constituent extension nodes, maintaining all original properties and configurations. 3. **Connection mapping**: References using namespace syntax (`subgraph:extension`) are converted to direct extension names, and subgraph-level connections are mapped to specific internal extensions. 4. **Property inheritance**: Properties specified when referencing a subgraph are applied to the appropriate internal extensions based on the `exposed_properties` mapping. 5. **Interface removal**: The `exposed_messages` and `exposed_properties` metadata is removed since it's only needed during graph composition. **Naming convention:** The colon (`:`) is replaced with an underscore (`_`) to create valid extension names in the flattened structure. * Original: `subgraph_name:extension_name` * Flattened: `subgraph_name_extension_name` ### Flattened graph example [#flattened-graph-example] Following is a complete example that shows how a graph with subgraphs transforms during the flattening process: **Before flattening:** ```json { "nodes": [ { "type": "extension", "name": "input_source", "addon": "microphone_input" }, { "type": "extension", "name": "output_sink", "addon": "speaker_output" }, { "type": "subgraph", "name": "audio_pipeline", "source_uri": "./subgraphs/audio_processing.json", "property": { "sample_rate": 48000, "noise_threshold": 0.3 } }, { "type": "subgraph", "name": "effects_chain", "source_uri": "./subgraphs/audio_effects.json" } ], "connections": [ { "extension": "input_source", "audio_frame": [ { "name": "raw_audio", "dest": [ { "subgraph": "audio_pipeline" } ] } ] }, { "extension": "audio_pipeline:processor", "audio_frame": [ { "name": "clean_audio", "dest": [ { "extension": "effects_chain:reverb" } ] } ] }, { "subgraph": "effects_chain", "audio_frame": [ { "name": "final_audio", "dest": [ { "extension": "output_sink" } ] } ] } ] } ``` **After flattening:** ```json { "nodes": [ { "type": "extension", "name": "input_source", "addon": "microphone_input" }, { "type": "extension", "name": "output_sink", "addon": "speaker_output" }, { "type": "extension", "name": "audio_pipeline_enhancer", "addon": "audio_enhancement", "property": { "sample_rate": 48000 } }, { "type": "extension", "name": "audio_pipeline_processor", "addon": "noise_reduction", "property": { "noise_threshold": 0.3 } }, { "type": "extension", "name": "effects_chain_reverb", "addon": "reverb_effect" }, { "type": "extension", "name": "effects_chain_compressor", "addon": "dynamic_compressor" } ], "connections": [ { "extension": "input_source", "audio_frame": [ { "name": "raw_audio", "dest": [ { "extension": "audio_pipeline_enhancer" } ] } ] }, { "extension": "audio_pipeline_enhancer", "audio_frame": [ { "name": "enhanced_audio", "dest": [ { "extension": "audio_pipeline_processor" } ] } ] }, { "extension": "audio_pipeline_processor", "audio_frame": [ { "name": "clean_audio", "dest": [ { "extension": "effects_chain_reverb" } ] } ] }, { "extension": "effects_chain_reverb", "audio_frame": [ { "name": "reverb_audio", "dest": [ { "extension": "effects_chain_compressor" } ] } ] }, { "extension": "effects_chain_compressor", "audio_frame": [ { "name": "final_audio", "dest": [ { "extension": "output_sink" } ] } ] } ] } ``` **Key changes during flattening:** * **Subgraph nodes removed**: `audio_pipeline` and `effects_chain` subgraph nodes are replaced with their constituent extensions * **Names prefixed**: Internal extensions get prefixed names (`audio_pipeline_processor`, `effects_chain_reverb`) * **Properties distributed**: Subgraph properties are applied to the correct internal extensions * **Connections resolved**: All namespace references and subgraph connections are converted to direct extension connections * **Internal connections preserved**: Connections that existed within the original subgraphs are included in the flattened result The flattened graph contains only standard extensions and direct connections, maintaining the organizational benefits you gained during development while using the standard TEN graph execution model. ## Advanced features [#advanced-features] ### Message conversion with subgraphs [#message-conversion-with-subgraphs] Subgraphs support the TEN framework's message conversion mechanism, allowing you to transform message formats when routing data between components. **Message conversion syntax:** You can apply message conversion rules when connecting to subgraphs, internal subgraph elements, or when subgraphs connect to external components: ```json { "connections": [ { "extension": "data_source", "cmd": [ { "name": "user_request", "dest": [ { "subgraph": "ai_processor", "msg_conversion": { "type": "per_property", "rules": [ { "path": "request_type", "conversion_mode": "fixed_value", "value": "chat_completion" }, { "path": "model_config.temperature", "conversion_mode": "from_original", "original_path": "settings.creativity" } ], "keep_original": true } }, { "extension": "nlp_pipeline:tokenizer", "msg_conversion": { "type": "per_property", "rules": [ { "path": "input_text", "conversion_mode": "from_original", "original_path": "user_message" } ], "keep_original": false } } ] } ] } ] } ``` **Conversion during flattening:** Message conversion rules are preserved during the flattening process, ensuring that your data transformations work correctly at runtime: ```json { "connections": [ { "extension": "data_source", "cmd": [ { "name": "user_request", "dest": [ { "extension": "ai_processor_chat_handler", "msg_conversion": { "type": "per_property", "rules": [ { "path": "request_type", "conversion_mode": "fixed_value", "value": "chat_completion" } ], "keep_original": true } }, { "extension": "nlp_pipeline_tokenizer", "msg_conversion": { "type": "per_property", "rules": [ { "path": "input_text", "conversion_mode": "from_original", "original_path": "user_message" } ], "keep_original": false } } ] } ] } ] } ``` ### Multi-graph connections [#multi-graph-connections] Beyond subgraphs, TEN supports connections between separate graphs running in the same application or across different applications. This enables sophisticated distributed architectures. #### Connecting to predefined graphs [#connecting-to-predefined-graphs] You can reference and connect to other predefined graphs within the same TEN application: ```json { "nodes": [ { "type": "extension", "name": "local_processor", "addon": "data_processor" }, { "type": "graph", "graph_name": "analytics_pipeline", "singleton": true }, { "type": "graph", "graph_name": "monitoring_system", "singleton": false } ], "connections": [ { "extension": "local_processor", "data": [ { "name": "processed_data", "dest": [ { "extension": "analytics_pipeline:data_ingester" }, { "graph": "monitoring_system" } ] } ] }, { "graph": "analytics_pipeline", "cmd": [ { "name": "analysis_complete", "dest": [ { "extension": "local_processor" } ] } ] } ] } ``` #### Remote graph connections [#remote-graph-connections] Connect to graphs running in different TEN applications across the network: ```json { "nodes": [ { "type": "extension", "name": "local_service", "addon": "api_handler" }, { "type": "graph", "graph_name": "ml_inference", "singleton": true, "app": "msgpack://ml-server.example.com:8002/" }, { "type": "graph", "graph_name": "data_warehouse", "singleton": false, "app": "msgpack://data-cluster.example.com:8003/" } ], "connections": [ { "extension": "local_service", "cmd": [ { "name": "inference_request", "dest": [ { "graph": "ml_inference" } ] } ] }, { "graph": "ml_inference", "data": [ { "name": "inference_result", "dest": [ { "extension": "local_service" }, { "graph": "data_warehouse" } ] } ] } ] } ``` #### Graph node types [#graph-node-types] TEN supports three types of graph nodes for different connection scenarios: **Subgraph nodes:** ```json { "type": "subgraph", "name": "processing_module", "source_uri": "./subgraphs/data_processing.json", "property": { "batch_size": 100 } } ``` **Local graph nodes:** ```json { "type": "graph", "graph_name": "background_service", "singleton": true } ``` ```json { "type": "graph", "graph_name": "worker_pool", "singleton": false } ``` **Remote graph nodes:** ```json { "type": "graph", "graph_name": "external_api", "singleton": true, "app": "msgpack://api.service.com:8001/" } ``` ```json { "type": "graph", "graph_name": "distributed_cache", "singleton": false, "app": "msgpack://cache-cluster.internal:8004/" } ``` **Graph node properties:** * **graph\_name**: Identifies the target graph within its application * **singleton**: Controls whether multiple instances can exist (`true` = single instance, `false` = multiple instances allowed) * **app**: Specifies the target application URL for remote connections (omit for local graphs) These advanced features enable you to build sophisticated distributed systems where subgraphs, local graphs, and remote services work together seamlessly while maintaining clear architectural boundaries. ## Reference [#reference] ### Graph node types [#graph-node-types-1] TEN graphs support three types of node references that enable different architectural patterns and connection scopes. #### Subgraph nodes [#subgraph-nodes] Reference external graph files as reusable components within your graph: **Basic subgraph node:** ```json { "type": "subgraph", "name": "unique_subgraph_name", "source_uri": "path/to/subgraph.json" } ``` **Subgraph node with properties:** ```json { "type": "subgraph", "name": "configurable_module", "source_uri": "./ten_packages/extension/module/subgraph.json", "property": { "param1": "value1", "param2": 42, "nested_config": { "enabled": true, "threshold": 0.85 } } } ``` **Remote subgraph node:** ```json { "type": "subgraph", "name": "shared_component", "source_uri": "https://cdn.example.com/subgraphs/v1.2/nlp_pipeline.json" } ``` **Required fields:** * `type`: Must be `"subgraph"` * `name`: Unique identifier within the current graph (serves as namespace) * `source_uri`: Path or URL to the subgraph definition file **Optional fields:** * `property`: Configuration values for exposed properties #### Local graph nodes [#local-graph-nodes] Reference other predefined graphs within the same TEN application: **Singleton local graph:** ```json { "type": "graph", "graph_name": "predefined_graph_id", "singleton": true } ``` **Multi-instance local graph:** ```json { "type": "graph", "graph_name": "worker_template", "singleton": false } ``` **Required fields:** * `type`: Must be `"graph"` * `graph_name`: Identifier of the predefined graph in the same application * `singleton`: Boolean controlling instance multiplicity **Singleton behavior:** * `true`: Only one instance of the graph exists, shared across references * `false`: Each reference creates a separate graph instance #### Remote graph nodes [#remote-graph-nodes] Reference graphs running in different TEN applications across the network: **Singleton remote graph:** ```json { "type": "graph", "graph_name": "ml_service", "singleton": true, "app": "msgpack://ml-cluster.internal:8002/" } ``` **Multi-instance remote graph:** ```json { "type": "graph", "graph_name": "data_processor", "singleton": false, "app": "msgpack://processing-farm.example.com:8003/" } ``` **Required fields:** * `type`: Must be `"graph"` * `graph_name`: Identifier of the target graph in the remote application * `singleton`: Boolean controlling instance behavior * `app`: Network address of the target TEN application **Supported protocols:** * `msgpack://host:port/`: MessagePack-based communication * Additional protocols may be supported in future versions ### Connection syntax reference [#connection-syntax-reference] TEN provides flexible syntax for establishing connections between different types of graph nodes and their internal elements. #### subgraph\_name:element_name [#subgraph_name] Access specific elements within subgraphs using namespace syntax: **Extension within subgraph:** ```json { "extension": "audio_pipeline:noise_filter", "audio_frame": [ { "name": "filtered_audio", "dest": [ { "extension": "output_device" } ] } ] } ``` **Connecting to subgraph element:** ```json { "extension": "microphone", "audio_frame": [ { "name": "raw_audio", "dest": [ { "extension": "processing_chain:input_buffer" } ] } ] } ``` **Syntax rules:** * Format: `subgraph_name:extension_name` * The colon (`:`) separates namespace from element name * Element name must exist within the referenced subgraph * Bypasses subgraph's exposed interface constraints #### Graph references [#graph-references] Connect to graphs using their identifiers: **Direct graph connection:** ```json { "subgraph": "local_processor", "cmd": [ { "name": "process_complete", "dest": [ { "graph": "analytics_engine" } ] } ] } ``` **Graph element connection:** ```json { "graph": "background_service", "data": [ { "name": "status_update", "dest": [ { "extension": "monitor:logger" } ] } ] } ``` **Local graph with namespace:** ```json { "extension": "coordinator", "cmd": [ { "name": "task_assignment", "dest": [ { "extension": "worker_pool:task_manager" } ] } ] } ``` #### Advanced patterns [#advanced-patterns] Combine multiple connection types for sophisticated data flow architectures: **Multi-destination routing:** ```json { "extension": "data_ingester", "data": [ { "name": "incoming_data", "dest": [ { "subgraph": "validation_pipeline" }, { "extension": "processing_farm:load_balancer" }, { "graph": "audit_logger" }, { "extension": "backup_storage" } ] } ] } ``` **Cross-application message flow:** ```json { "graph": "local_analytics", "cmd": [ { "name": "analysis_request", "dest": [ { "extension": "ml_cluster:inference_engine" } ] } ] } ``` **Conditional routing with message conversion:** ```json { "extension": "request_router", "cmd": [ { "name": "user_request", "dest": [ { "subgraph": "fast_processing", "msg_conversion": { "type": "per_property", "rules": [ { "path": "priority", "conversion_mode": "fixed_value", "value": "high" } ] } }, { "extension": "slow_processing:queue_manager", "msg_conversion": { "type": "per_property", "rules": [ { "path": "priority", "conversion_mode": "fixed_value", "value": "normal" } ] } } ] } ] } ``` **Connection target types:** * `extension`: Direct extension name * `subgraph`: Subgraph name (uses exposed interfaces) * `graph`: Graph name (uses exposed interfaces) * `extension` with namespace: `namespace:extension_name` **Best practices:** * Use exposed interfaces when possible for better encapsulation * Use namespace syntax only when you need direct access to internal elements * Prefer subgraphs over direct graph references for reusable components * Document complex connection patterns for maintainability # Send images to the agent (/en/ai/build/send-multimodal-messages) When interacting with an agent, you may need to upload images or send image messages from the client to help the agent better understand the user's intent. This page describes how to use the Conversational AI Engine toolkit to send image messages to the large language model from your app. The LLM can then automatically reference the image content in subsequent conversations and generate more relevant responses. ## Understand the tech [#understand-the-tech] Agora provides a flexible, scalable, and standardized conversational AI engine toolkit. The toolkit supports **iOS**, **Android**, and **Web** platforms, and encapsulates scenario-based APIs. You can use these APIs to integrate the capabilities of the [Agora Signaling SDK](/en/realtime-media/rtm) and [Agora Video SDK](../../realtime-media/video/quickstart) to enable the following features: * [Interrupt agents](shape-the-conversation/interrupt-agent) * [Display live transcripts](transcripts) * [Client-side events](handle-runtime-events/event-notifications) * [Set optimal audio parameters](../best-practices/audio-setup) for iOS and Android * [Send picture messages](send-multimodal-messages) Call the toolkit's `chat` API to send a picture message, and listen to `onMessageReceiptUpdated` to receive the picture message receipt. ## Prerequisites [#prerequisites] Before you begin, ensure the following: * You have implemented the Conversational AI Engine [quickstart](../get-started/quickstart). * Your app integrates Agora Video SDK v4.5.1 or later and includes the [Video SDK quickstart](../../realtime-media/video/quickstart). * You have enabled Signaling in the Agora Console and completed the [Signaling quickstart](/en/realtime-media/rtm/quickstart) for basic messaging. * You maintain active and authenticated RTC and Signaling instances that persist beyond the component lifecycle. The toolkit does not manage RTC or Signaling initialization, lifecycle, or authentication. Info * Picture messaging is currently in beta and free for a limited time. * Image processing depends on the capabilities of the integrated LLM. Make sure the LLM you connect to Conversational AI Engine supports image input. ## Implementation [#implementation] This section explains how to send a picture message from your app. Android iOS Web 1. **Integrate the toolkit** Copy the [`convoaiApi`](https://github.com/AgoraIO-Community/Conversational-AI-Demo/tree/main/Android/scenes/convoai/src/main/java/io/agora/scene/convoai/convoaiApi) folder to your project and import the toolkit before calling the toolkit API. Refer to [Folder structure](#reference) to understand the role of each file. 2. **Create a toolkit instance** Create a configuration object with the Video SDK and Signaling engine instances. Use the configuration to create a toolkit instance. ```kotlin // Create configuration objects for the RTC and RTM instances val config = ConversationalAIAPIConfig( rtcEngine = rtcEngineInstance, rtmClient = rtmClientInstance, enableLog = true ) // Create component instance val api = ConversationalAIAPIImpl(config) ``` 3. **Register callback** ```kotlin api.addHandler(covEventHandler) ``` 4. **Subscribe to the channel** Agent-related events are delivered through Signaling channel messages. To receive these events, call `subscribeMessage` before starting the agent session. ```kotlin api.subscribeMessage("channelName") { error -> if (error != null) { // Handle error } } ``` 5. **Add a conversational AI agent to the channel** To [start a conversational AI agent](/en/api-reference/api-ref/conversational-ai/join), configure the following parameters in your `POST` request: | Parameter | Description | Required | | --------------------------------------- | -------------------------------------------------- | -------- | | `advanced_features.enable_rtm: true` | Starts the Signaling service | Yes | | `parameters.data_channel: "rtm"` | Enables Signaling as the data transmission channel | Yes | | `parameters.enable_metrics: true` | Enables agent performance data collection | Optional | | `parameters.enable_error_message: true` | Enables reporting of agent error events | Optional | After a successful response, the agent joins the specified RTC channel and is ready to interact with the user. 6. **Send an image** ```kotlin val uuid = "unique-image-id-123" val imageUrl = "https://example.com/image.jpg" api.chat("agentUserId", ImageMessage(uuid = uuid, imageUrl = imageUrl)) { error -> if (error != null) { Log.e("Chat", "Failed to send image: ${error.errorMessage}") } else { Log.d("Chat", "Image send request successful") } } ``` Info The `chat` completion callback only indicates whether the sending request is successful, and does not reflect the actual processing status of the message. 7. **Handle image sending status** Image send success is confirmed by `onMessageReceiptUpdated`. If sending fails, `onMessageError` is triggered. Use the `uuid` value in the callback to identify the uploaded picture. Image sent successfully: ```kotlin override fun onMessageReceiptUpdated(agentUserId: String, receipt: MessageReceipt) { if (receipt.chatMessageType == ChatMessageType.Image) { try { val json = JSONObject(receipt.message) if (json.has("uuid")) { val receivedUuid = json.getString("uuid") if (receivedUuid == "your-sent-uuid") { Log.d("ImageSend", "Image sent successfully: $receivedUuid") } } } catch (e: Exception) { Log.e("ImageSend", "Failed to parse message receipt: ${e.message}") } } } ``` Image sending failed: ```kotlin override fun onMessageError(agentUserId: String, error: MessageError) { if (error.chatMessageType == ChatMessageType.Image) { try { val json = JSONObject(error.message) if (json.has("uuid")) { val failedUuid = json.getString("uuid") if (failedUuid == "your-sent-uuid") { Log.e("ImageSend", "Image send failed: $failedUuid") } } } catch (e: Exception) { Log.e("ImageSend", "Failed to parse error message: ${e.message}") } } } ``` 8. **Unsubscribe from the channel** ```kotlin api.unsubscribeMessage("channelName") { error -> if (error != null) { // Handle the error } } ``` 9. **Release resources** ```kotlin api.destroy() ``` 1. **Integrate the toolkit** Copy the [`ConversationalAIAPI`](https://github.com/AgoraIO-Community/Conversational-AI-Demo/tree/main/iOS/Scenes/ConvoAI/ConvoAI/ConvoAI/Classes/ConversationalAIAPI) folder to your project and import the toolkit before calling the toolkit APIs. Refer to [Folder structure](#reference) to understand the role of each file. 2. **Create a toolkit instance** Create a configuration object with the Video SDK and Signaling engine instances, then use the configuration to create a toolkit instance. ```swift let config = ConversationalAIAPIConfig( rtcEngine: rtcEngine, rtmEngine: rtmEngine, enableLog: true ) convoAIAPI = ConversationalAIAPIImpl(config: config) ``` 3. **Subscribe to the channel** Agent-related events are delivered through Signaling channel messages. To receive these events, call `subscribeMessage` before starting the agent session. ```swift convoAIAPI.subscribeMessage(channelName: channelName) { error in if let error = error { print("Subscription failed: \(error.message)") } else { print("Subscription successful") } } ``` 4. **Register callback** ```swift convoAIAPI.addHandler(handler: self) ``` 5. **Add a conversational AI agent to the channel** To [start a conversational AI agent](/en/api-reference/api-ref/conversational-ai/join), configure the following parameters in your `POST` request: | Parameter | Description | Required | | --------------------------------------- | -------------------------------------------------- | -------- | | `advanced_features.enable_rtm: true` | Starts the Signaling service | Yes | | `parameters.data_channel: "rtm"` | Enables Signaling as the data transmission channel | Yes | | `parameters.enable_metrics: true` | Enables agent performance data collection | Optional | | `parameters.enable_error_message: true` | Enables reporting of agent error events | Optional | 6. **Send an image** ```swift let uuid = UUID().uuidString let imageUrl = "https://example.com/image.jpg" let message = ImageMessage(uuid: uuid, url: imageUrl) self.convoAIAPI.chat(agentUserId: "\(agentUid)", message: message) { [weak self] error in if let error = error { print("send image failed, error: \(error.message)") } else { print("send image success") } } ``` Info The `chat` completion callback only indicates whether the sending request is successful, and does not reflect the actual processing status of the message. 7. **Receive image sending response** Success is confirmed by `onMessageReceiptUpdated`. If sending fails, `onMessageError` is triggered. Image sent successfully: ```swift struct PictureInfo: Codable { let uuid: String } public func onMessageReceiptUpdated(agentUserId: String, messageReceipt: MessageReceipt) { if messageReceipt.type == .context { guard let messageData = messageReceipt.message.data(using: .utf8) else { return } do { let imageInfo = try JSONDecoder().decode(PictureInfo.self, from: messageData) self.messageView.viewModel.updateImageMessage(uuid: imageInfo.uuid, state: .success) } catch { print("Failed to decode PictureInfo: \(error)") } } } ``` Image sending failed: ```swift struct ImageUploadError: Codable { let code: Int let message: String } struct ImageUploadErrorResponse: Codable { let uuid: String let success: Bool let error: ImageUploadError? } public func onMessageError(agentUserId: String, error: MessageError) { if let messageData = error.message.data(using: .utf8) { do { let errorResponse = try JSONDecoder().decode(ImageUploadErrorResponse.self, from: messageData) if !errorResponse.success { DispatchQueue.main.async { [weak self] in self?.messageView.viewModel.updateImageMessage(uuid: errorResponse.uuid, state: .failed) } } } catch { addLog("Failed to parse error message JSON: \(error)") } } } ``` 8. **Unsubscribe from the channel** ```swift convoAIAPI.unsubscribeMessage(channelName: channelName) { error in if let error = error { print("Unsubscription failed: \(error.message)") } else { print("Unsubscribed successfully") } } ``` 9. **Release resources** ```swift convoAIAPI.destroy() ``` 1. **Integrate the toolkit** Copy the [`conversational-ai-api`](https://github.com/AgoraIO-Community/Conversational-AI-Demo/tree/main/Web/Scenes/VoiceAgent/src/conversational-ai-api) file to your project and import the toolkit before calling its API. Refer to [Folder structure](#reference) to understand the role of each file. 2. **Create a toolkit instance** Before joining an RTC channel, create Video SDK and Signaling engine instances and pass them into the toolkit instance. ```ts ConversationalAIAPI.init({ rtcEngine, rtmEngine, }) const conversationalAIAPI = ConversationalAIAPI.getInstance() ``` 3. **Subscribe to the channel** Agent-related events are delivered through Signaling messages. Before starting an agent session, call `subscribeMessage` to receive these events. ```ts conversationalAIAPI.subscribeMessage(channel_name) ``` 4. **Register callbacks** ```ts conversationalAIAPI.on(EConversationalAIAPIEvents.MESSAGE_RECEIPT_UPDATED, handleMessageReceiptUpdated) conversationalAIAPI.on(EConversationalAIAPIEvents.MESSAGE_ERROR, onAgentError) ``` 5. **Add a conversational AI agent to the channel** To [start a conversational AI agent](/en/api-reference/api-ref/conversational-ai/join), configure the following parameters in your `POST` request: | Parameter | Description | Required | | --------------------------------------- | -------------------------------------------------- | -------- | | `advanced_features.enable_rtm: true` | Starts the Signaling service | Yes | | `parameters.data_channel: "rtm"` | Enables Signaling as the data transmission channel | Yes | | `parameters.enable_metrics: true` | Enables agent performance data collection | Optional | | `parameters.enable_error_message: true` | Enables reporting of agent error events | Optional | 6. **Send an image** ```ts import { EChatMessageType } from '@/conversational-ai-api/type' await conversationalAIAPI.chat(`${agent_rtc_uid}`, { messageType: EChatMessageType.IMAGE, url: 'https://example.com/image.jpg', uuid: genUUID() }) ``` Info The `chat` completion callback only indicates whether the sending request is successful, and does not reflect the actual processing status of the message. 7. **Receive image sending response** Image sent successfully: ```ts conversationalAIAPI.on(EConversationalAIAPIEvents.MESSAGE_RECEIPT_UPDATED, (agentUserId: string, messageReceipt: TMessageReceipt) => { if (messageReceipt.moduleType !== EModuleType.CONTEXT) { return } try { const receiptMessage = JSON.parse(messageReceipt.message) const uuid = receiptMessage.uuid if (!uuid) return console.log(`Message sent successfully, UUID: ${uuid}`) } catch (error) { console.error('Failed to parse message:', error) } }) ``` Image sending failed: ```ts conversationalAIAPI.on(EConversationalAIAPIEvents.MESSAGE_ERROR, (agentUserId, error) => { console.error(`Message error for agent ${agentUserId}:`, error) if (error.type === EChatMessageType.IMAGE) { try { const errorData = JSON.parse(error.message) if (errorData?.uuid) { console.warn(`Image error for agent ${agentUserId} with UUID: ${errorData.uuid}`) } } catch (e) { console.error(`Failed to handle image error for agent ${agentUserId}:`, e) } } }) ``` 8. **Unsubscribe from the channel** ```ts conversationalAIAPI.unsubscribeMessage(channel_name) ``` 9. **Release resources** ```ts conversationalAIAPI.destroy() ``` ## Reference [#reference] This section contains supporting information that completes the guidance on this page. Android iOS Web **Folder structure** * `IConversationalAIAPI.kt`: API interface, related data structures, and enumerations * `ConversationalAIAPIImpl.kt`: main implementation logic * `ConversationalAIUtils.kt`: utility functions and callback management * `v3/` * `TranscriptionController.kt`: transcript rendering and synchronization * `MessageParser.kt`: transcription and message parsing **API reference** * [`addHandler`](/en/api-reference/api-ref/conversational-ai/client-toolkit/android#addhandler) * [`subscribeMessage`](/en/api-reference/api-ref/conversational-ai/client-toolkit/android#subscribemessage) * [`unsubscribeMessage`](/en/api-reference/api-ref/conversational-ai/client-toolkit/android#unsubscribemessage) * [`destroy`](/en/api-reference/api-ref/conversational-ai/client-toolkit/android#destroy) * [`onMessageReceiptUpdated`](/en/api-reference/api-ref/conversational-ai/client-toolkit/android#onmessagereceiptupdated) * [`onMessageError`](/en/api-reference/api-ref/conversational-ai/client-toolkit/android#onmessageerror) * [`chat`](/en/api-reference/api-ref/conversational-ai/client-toolkit/android#chat) **Folder structure** * `ConversationalAIAPI.swift`: API interface, data structures, and enumerations * `ConversationalAIAPIImpl.swift`: main implementation logic * `Transcription/` * `TranscriptionController.swift`: transcript rendering and transcription control **API reference** * [`addHandler`](/en/api-reference/api-ref/conversational-ai/client-toolkit/ios#addhandler) * [`subscribeMessage`](/en/api-reference/api-ref/conversational-ai/client-toolkit/ios#subscribemessage) * [`unsubscribeMessage`](/en/api-reference/api-ref/conversational-ai/client-toolkit/ios#unsubscribemessage) * [`destroy`](/en/api-reference/api-ref/conversational-ai/client-toolkit/ios#destroy) * [`onMessageReceiptUpdated`](/en/api-reference/api-ref/conversational-ai/client-toolkit/ios#onmessagereceiptupdated) * [`onMessageError`](/en/api-reference/api-ref/conversational-ai/client-toolkit/ios#onmessageerror) * [`chat`](/en/api-reference/api-ref/conversational-ai/client-toolkit/ios#chat) **Folder structure** * `index.ts`: main API class * `type.ts`: API interfaces, data structures, and enumerations * `utils/` * `index.ts`: general utility functions * `events.ts`: event management * `sub-render.ts`: transcript rendering module **API reference** * [`IConversationalAIAPIEventHandler`](/en/api-reference/api-ref/conversational-ai/client-toolkit/web#iconversationalaiapieventhandlers-interface) * [`EConversationalAIAPIEvents`](/en/api-reference/api-ref/conversational-ai/client-toolkit/web#econversationalaiapievents) * [`subscribeMessage`](/en/api-reference/api-ref/conversational-ai/client-toolkit/web#subscribemessage) * [`unsubscribeMessage`](/en/api-reference/api-ref/conversational-ai/client-toolkit/web#unsubscribe) * [`destroy`](/en/api-reference/api-ref/conversational-ai/client-toolkit/web#destroy) * [`chat`](/en/api-reference/api-ref/conversational-ai/client-toolkit/web#chat) # Start and stop an agent (/en/ai/build/start-stop-agent) This page describes how to call the Conversational AI Engine APIs to start and stop an AI agent. ## Understand the tech [#understand-the-tech] A user joins a channel and triggers your business server to start an agent via the Conversational AI Engine API. The agent joins the same channel and communicates with the user through voice, using the specified LLM, TTS service, and Agora’s low-latency Software-Defined Real-Time Network (SDRTN®). When the conversation ends, the business server stops the agent and the user leaves the channel.
Conversational AI Engine workflow ![Conversational AI Engine workflow](https://assets-docs.agora.io/images/conversational-ai/ai-agent-tech.svg)
## Prerequisites [#prerequisites] Use the [Agora CLI](/en/introduction/agora-cli) to create or select an Agora project, enable RTC and Conversational AI, verify project readiness, and export local credentials: ```bash agora login agora project create conv-ai-tutorial --feature rtc --feature convoai agora project use conv-ai-tutorial agora project doctor --feature convoai agora project env --format shell --with-secrets ``` You also need: * A client app that can join an Agora RTC channel, such as the [Voice Calling](../../realtime-media/voice/quickstart) or [Video Calling](../../realtime-media/video/quickstart) quickstart. ## Install the SDK [#install-the-sdk] Select your preferred language and install the corresponding SDK. Python SDK TypeScript SDK Go SDK ```sh pip install agora-agents ``` ```sh npm install agora-agents ``` ```sh go get github.com/AgoraIO/agora-agents-go/v2@v2.2.0 ``` ## Implementation [#implementation] This section introduces the basic RESTful API requests you use to start and stop a conversational AI agent. In a production environment, implement these requests on your business server. ### Start a conversational AI agent [#start-a-conversational-ai-agent] Call the `join` endpoint to create an agent instance that joins an Agora channel. Python SDK TypeScript SDK Go SDK REST API ```python from agora_agent import Agent, Agora, Area, DeepgramSTT, OpenAI, MicrosoftTTS client = Agora( area=Area.US, app_id='your-app-id', app_certificate='your-app-certificate', ) agent = ( Agent(client) .with_stt(DeepgramSTT(model='nova-3', language='en-US')) .with_llm(OpenAI( api_key='your-llm-api-key', base_url='https://api.openai.com/v1/chat/completions', model='gpt-4o-mini', system_messages=[{'role': 'system', 'content': 'You are a helpful chatbot.'}], greeting_message='Hello, how can I help you?', failure_message="Sorry, I don't know how to answer this question.", max_history=10, )) .with_tts(MicrosoftTTS( key='your-tts-api-key', region='eastus', voice_name='en-US-AndrewMultilingualNeural', )) ) session = agent.create_session( channel='your_channel_name', agent_uid='0', remote_uids=['1002'], name='unique_name', idle_timeout=120, ) agent_id = session.start() print(f'Agent started: {agent_id}') ``` ```typescript import { Agent, AgoraClient, Area, DeepgramSTT, OpenAI, MicrosoftTTS } from 'agora-agents'; const client = new AgoraClient({ area: Area.US, appId: 'your-app-id', appCertificate: 'your-app-certificate', }); const agent = new Agent({ client }) .withStt(new DeepgramSTT({ model: 'nova-3', language: 'en-US' })) .withLlm(new OpenAI({ apiKey: 'your-llm-api-key', url: 'https://api.openai.com/v1/chat/completions', model: 'gpt-4o-mini', systemMessages: [{ role: 'system', content: 'You are a helpful chatbot.' }], greetingMessage: 'Hello, how can I help you?', failureMessage: 'Sorry, I don't know how to answer this question.', maxHistory: 10, })) .withTts(new MicrosoftTTS({ key: 'your-tts-api-key', region: 'eastus', voiceName: 'en-US-AndrewMultilingualNeural', })); const session = agent.createSession({ channel: 'your_channel_name', agentUid: '0', remoteUids: ['1002'], name: 'unique_name', idleTimeout: 120, }); const agentId = await session.start(); console.log('Agent started:', agentId); ``` ```go package main import ( "context" "fmt" "log" Agora "github.com/AgoraIO/agora-agents-go/v2" "github.com/AgoraIO/agora-agents-go/v2/agentkit" "github.com/AgoraIO/agora-agents-go/v2/agentkit/vendors" "github.com/AgoraIO/agora-agents-go/v2/option" ) func main() { ctx := context.Background() idleTimeout := 120 client := agentkit.NewAgoraClient(agentkit.AgoraClientOptions{ Area: option.AreaUS, AppID: "your-app-id", AppCertificate: "your-app-certificate", }) agent := agentkit.NewAgent(client).WithStt( vendors.NewDeepgramSTT(vendors.DeepgramSTTOptions{ Model: "nova-3", Language: "en-US", }), ).WithLlm( vendors.NewOpenAI(vendors.OpenAIOptions{ APIKey: "your-llm-api-key", URL: "https://api.openai.com/v1/chat/completions", Model: "gpt-4o-mini", SystemMessages: []map[string]interface{}{ {"role": "system", "content": "You are a helpful chatbot."}, }, GreetingMessage: "Hello, how can I help you?", FailureMessage: "Sorry, I don't know how to answer this question.", MaxHistory: Agora.Int(10), }), ).WithTts( vendors.NewMicrosoftTTS(vendors.MicrosoftTTSOptions{ Key: "your-tts-api-key", Region: "eastus", VoiceName: "en-US-AndrewMultilingualNeural", }), ) session := agent.CreateSession(agentkit.CreateSessionOptions{ Channel: "your_channel_name", AgentUID: "0", RemoteUIDs: []string{"1002"}, Name: "unique_name", IdleTimeout: &idleTimeout, }) agentID, err := session.Start(ctx) if err != nil { log.Fatal(err) } fmt.Println("Agent started:", agentID) } ``` For the Curl request, pass in the `channel` name and `token` for agent authentication. To generate your base64-encoded credentials, see [RESTful authentication](/en/api-reference/api-ref/conversational-ai/authentication). ```bash curl --request POST \ --url https://api.agora.io/api/conversational-ai-agent/v2/projects/:appid/join \ --header 'Authorization: Basic ' \ --header 'Content-Type: application/json' \ --data ' { "name": "unique_name", "properties": { "channel": "", "token": "", "agent_rtc_uid": "0", "remote_rtc_uids": ["1002"], "enable_string_uid": false, "idle_timeout": 120, "llm": { "url": "https://api.openai.com/v1/chat/completions", "api_key": "", "system_messages": [ { "role": "system", "content": "You are a helpful chatbot." } ], "greeting_message": "Hello, how can I help you?", "failure_message": "Sorry, I don't know how to answer this question.", "max_history": 10, "params": { "model": "gpt-4o-mini" } }, "asr": { "language": "en-US" }, "tts": { "vendor": "microsoft", "params": { "key": "", "region": "eastus", "voice_name": "en-US-AndrewMultilingualNeural" } } } }' ``` If the request is successful, you receive the following response: ```json // 200 OK { "agent_id": "1NT29X10YHxxxxxWJOXLYHNYB", "create_ts": 1737111452, "status": "RUNNING" } ``` If the request is successful, the agent ID is returned. Store the agent ID to manage the agent in subsequent calls. ### Stop the conversational AI agent [#stop-the-conversational-ai-agent] To end the conversation with the AI agent, call the `leave` endpoint. This causes the agent to leave the Agora channel. Python SDK TypeScript SDK Go SDK REST API ```python session.stop() ``` ```typescript await session.stop(); ``` ```go if err := session.Stop(ctx); err != nil { log.Fatal(err) } ``` ```bash curl --request POST \ --url https://api.agora.io/api/conversational-ai-agent/v2/projects/:appid/agents/:agentId/leave \ --header 'Authorization: Basic ' ``` If the request is successful, the server responds with a `200 OK` status and an empty JSON object. ```json // 200 OK {} ``` Info The number of Peak Concurrent Users (PCU) allowed to call the server API under a single App ID is limited to 20. If you need to increase this limit, please [contact technical support](mailto\:support@agora.io). ## Reference [#reference] This section contains content that completes the information on this page, or points you to documentation that explains other aspects of this product. ### API reference [#api-reference] * [Start a conversational AI agent](/en/api-reference/api-ref/conversational-ai/join) * [Stop a conversational AI agent](/en/api-reference/api-ref/conversational-ai/leave) * [Update agent configuration](/en/api-reference/api-ref/conversational-ai/update) * [Query agent status](/en/api-reference/api-ref/conversational-ai/query) * [Retrieve a list of agents](/en/api-reference/api-ref/conversational-ai/list) # Display live transcripts (/en/ai/build/transcripts) When interacting with conversational AI in real time, you can enable live transcripts to display the conversation content. This page explains how to implement live transcripts in your app. ## Understand the tech [#understand-the-tech] Agora provides a flexible, scalable, and standardized conversational AI engine toolkit. The toolkit supports **iOS**, **Android**, and **Web** platforms, and encapsulates scenario-based APIs. You can use these APIs to integrate the capabilities of the [Agora Signaling SDK](/en/realtime-media/rtm) and [Agora Video SDK](../../realtime-media/video/quickstart) to enable the following features: * [Interrupt agents](shape-the-conversation/interrupt-agent) * [Display live transcripts](transcripts) * [Client-side events](handle-runtime-events/event-notifications) * [Set optimal audio parameters](../best-practices/audio-setup) for iOS and Android * [Send picture messages](send-multimodal-messages) The toolkit receives transcript content through the `onTranscriptUpdated` callback and supports monitoring the following transcript data types: * **Agent transcript**: The agent's speech, including streaming updates and final results. * **User transcript**: The user's speech, including real-time display and status management. * **Transcript status**: State changes such as in progress, completed, or interrupted. The following diagram outlines the step-by-step process to integrate live transcript functionality into your application:
Transcript rendering workflow ![](https://assets-docs.agora.io/images/conversational-ai/transcript-workflow.svg)
## Prerequisites [#prerequisites] Before you begin, ensure the following: * You have implemented the Conversational AI Engine [quickstart](../get-started/quickstart). * Your app integrates Agora Video SDK v4.5.1 or later and includes the [Video SDK quickstart](../../realtime-media/video/quickstart). * You have enabled Signaling in the Agora Console and completed the [Signaling quickstart](/en/realtime-media/rtm/quickstart) for basic messaging. * You maintain active and authenticated RTC and Signaling instances that persist beyond the component lifecycle. The toolkit does not manage RTC or Signaling initialization, lifecycle, or authentication. ## Implementation [#implementation] This section describes how to receive transcript content from the transcript processing module and display it in your app UI. Android iOS Web 1. **Integrate the toolkit** Copy the [`convoaiApi`](https://github.com/AgoraIO-Community/Conversational-AI-Demo/tree/main/Android/scenes/convoai/src/main/java/io/agora/scene/convoai/convoaiApi) folder to your project and import the toolkit before calling the toolkit API. Refer to [Folder structure](#reference) to understand the role of each file. 2. **Create a toolkit instance** Create a configuration object with the Video SDK and Signaling engine instances. Set the transcript rendering mode, then use the configuration to create a toolkit instance. ```kotlin // Create configuration objects for the RTC and RTM instances val config = ConversationalAIAPIConfig( rtcEngine = rtcEngineInstance, rtmClient = rtmClientInstance, // Set the transcript rendering mode. Options: // - TranscriptRenderMode.Word: render transcript word by word. // - TranscriptRenderMode.Text: render the full sentence at once. renderMode = TranscriptRenderMode.Word, enableLog = true ) // Create component instance val api = ConversationalAIAPIImpl(config) ``` 3. **Subscribe to the channel** Transcript data is delivered through Signaling channel messages. To receive transcript data, call `subscribeMessage` before starting the agent session. ```kotlin api.subscribeMessage("channelName") { error -> if (error != null) { // Handle error } } ``` 4. **Receive transcript** Call the `addHandler` method to register your implementation of the transcription callback. ```kotlin api.addHandler(covEventHandler) ``` 5. **Implement UI rendering logic** Inherit your UI module from the `IConversationalAIAPIEventHandler` interface. Implement the `onTranscriptUpdated` method to handle transcript rendering to the UI. ```kotlin private val covEventHandler = object : IConversationalAIAPIEventHandler { override fun onTranscriptUpdated(agentUserId: String, transcript: Transcript) { // Handle transcript data and update the UI here } } ``` 6. **Add a conversational AI agent to the channel** To [start a conversational AI agent](/en/api-reference/api-ref/conversational-ai/join), configure the following parameters in your `POST` request: | Parameter | Description | Required | | --------------------------------------- | -------------------------------------------------- | -------- | | `advanced_features.enable_rtm: true` | Starts the Signaling service | Yes | | `parameters.data_channel: "rtm"` | Enables Signaling as the data transmission channel | Yes | | `parameters.enable_metrics: true` | Enables agent performance data collection | Optional | | `parameters.enable_error_message: true` | Enables reporting of agent error events | Optional | After a successful response, the agent joins the specified RTC channel and is ready to interact with the user. 7. **Unsubscribe from the channel** After an agent session ends, unsubscribe from channel messages to release transcription resources. ```kotlin api.unsubscribeMessage("channelName") { error -> if (error != null) { // Handle the error } } ``` 8. **Release resources** At the end of each call, use the `destroy` method to clean up the cache. ```kotlin api.destroy() ``` 1. **Integrate the toolkit** Copy the [`ConversationalAIAPI`](https://github.com/AgoraIO-Community/Conversational-AI-Demo/tree/main/iOS/Scenes/ConvoAI/ConvoAI/ConvoAI/Classes/ConversationalAIAPI) folder to your project and import the toolkit before calling the toolkit APIs. Refer to [Folder structure](#reference) to understand the role of each file. 2. **Create a toolkit instance** Create a configuration object with the Video SDK and Signaling engine instances. Set the transcript rendering mode, then use the configuration to create a toolkit instance. ```swift // Create a configuration object for the RTC and RTM instances let config = ConversationalAIAPIConfig( rtcEngine: rtcEngine, rtmEngine: rtmEngine, /** * Set the transcript rendering mode. Available options: * - .words: Word-by-word rendering mode. * - .text: Sentence-by-sentence rendering mode. */ renderMode: .words, enableLog: true ) // Create the component instance convoAIAPI = ConversationalAIAPIImpl(config: config) ``` 3. **Subscribe to the channel** Transcript data is delivered through Signaling channel messages. To receive transcript data, call `subscribeMessage` before starting the agent session. ```swift convoAIAPI.subscribeMessage(channelName: channelName) { error in if let error = error { print("Subscription failed: \(error.message)") } else { print("Subscription successful") } } ``` 4. **Receive transcript** Call the `addHandler` method to register and implement the transcript callback. ```swift convoAIAPI.addHandler(handler: self) ``` 5. **Implement UI rendering logic** Implement the `ConversationalAIAPIEventHandler` protocol in your UI module, and use `onTranscriptUpdated` to handle and render transcript updates. ```swift extension ChatViewController: ConversationalAIAPIEventHandler { public func onTranscriptUpdated(agentUserId: String, transcript: Transcript) { // Handle transcript data and update the UI here } } ``` 6. **Add a conversational AI agent to the channel** To [start a conversational AI agent](/en/api-reference/api-ref/conversational-ai/join), configure the following parameters in your `POST` request: | Parameter | Description | Required | | --------------------------------------- | -------------------------------------------------- | -------- | | `advanced_features.enable_rtm: true` | Starts the Signaling service | Yes | | `parameters.data_channel: "rtm"` | Enables Signaling as the data transmission channel | Yes | | `parameters.enable_metrics: true` | Enables agent performance data collection | Optional | | `parameters.enable_error_message: true` | Enables reporting of agent error events | Optional | After a successful response, the agent joins the specified RTC channel and is ready to interact with the user. 7. **Unsubscribe from the channel** After each agent session ends, unsubscribe from channel messages to release transcript-related resources. ```swift convoAIAPI.unsubscribeMessage(channelName: channelName) { error in if let error = error { print("Unsubscription failed: \(error.message)") } else { print("Unsubscribed successfully") } } ``` 8. **Release resources** At the end of each call, use the `destroy` method to clean up the cache. ```swift convoAIAPI.destroy() ``` 1. **Integrate the toolkit** Copy the [`conversational-ai-api`](https://github.com/AgoraIO-Community/Conversational-AI-Demo/tree/main/Web/Scenes/VoiceAgent/src/conversational-ai-api) file to your project and import the toolkit before calling its APIs. Refer to [Folder structure](#reference) to understand the role of each file. 2. **Create a toolkit instance** Before joining an RTC channel, create Video SDK and Signaling engine instances and pass them into the toolkit instance. ```ts // Initialize the component ConversationalAIAPI.init({ rtcEngine, rtmEngine, /** * Set the rendering mode for transcript. Available options: * - ESubtitleHelperMode.WORD: render transcript word by word. * - ESubtitleHelperMode.TEXT: render the full transcript at once. */ renderMode: ESubtitleHelperMode.WORD, }) // Get the API instance (singleton) const conversationalAIAPI = ConversationalAIAPI.getInstance() ``` 3. **Set audio parameters** In word-by-word rendering mode, you must receive audio timestamp metadata from RTC to synchronize subtitles with speech. Before creating the client object, configure the following parameter: ```ts AgoraRTC.setParameter('ENABLE_AUDIO_PTS_METADATA', true) const client = AgoraRTC.createClient({ mode: 'rtc', codec: 'vp8' }) ``` 4. **Subscribe to the channel** Agent-related events are delivered through Signaling messages. Before starting an agent session, call `subscribeMessage` to receive these events: ```ts conversationalAIAPI.subscribeMessage(channel_name) ``` 5. **Receive transcript** Register an event listener to receive transcript updates: ```tsx import * as React from 'react' import { type IUserTranscription, type IAgentTranscription, type ISubtitleHelperItem, EConversationalAIAPIEvents, } from '@/conversational-ai-api/type' import { ConversationalAIAPI } from '@/conversational-ai-api' export const ChatHistory = () => { const [chatHistory, setChatHistory] = React.useState< ISubtitleHelperItem>[] >([]) const conversationalAIAPI = ConversationalAIAPI.getInstance() conversationalAIAPI.on( EConversationalAIAPIEvents.TRANSCRIPT_UPDATED, setChatHistory ) return ( <> {chatHistory.map((message) => (
{message.uid}: {message.text}
))} ) } ``` 6. **Add a conversational AI agent to the channel** To [start a conversational AI agent](/en/api-reference/api-ref/conversational-ai/join), configure the following parameters in your `POST` request: | Parameter | Description | Required | | --------------------------------------- | -------------------------------------------------- | -------- | | `advanced_features.enable_rtm: true` | Starts the Signaling service | Yes | | `parameters.data_channel: "rtm"` | Enables Signaling as the data transmission channel | Yes | | `parameters.enable_metrics: true` | Enables agent performance data collection | Optional | | `parameters.enable_error_message: true` | Enables reporting of agent error events | Optional | After a successful response, the agent joins the specified RTC channel and is ready to interact with the user. 7. **Unsubscribe from the channel** After each agent session ends, unsubscribe from channel messages to release resources associated with callback events. ```ts conversationalAIAPI.unsubscribeMessage(channel_name) ``` 8. **Release resources** At the end of each call, use the `destroy` method to clean up the cache. ```ts conversationalAIAPI.destroy() ```
## Reference [#reference] This section contains supporting information that completes the guidance on this page. Android iOS Web **Folder structure** * `IConversationalAIAPI.kt`: API interface and related data structures and enumerations * `ConversationalAIAPIImpl.kt`: ConversationalAI API main implementation logic * `ConversationalAIUtils.kt`: tool functions and event callback management * `subRender/` * `v3/`: transcription module * `TranscriptionController.kt`: transcription controller * `MessageParser.kt`: message parser **API reference** * [`addHandler`](/en/api-reference/api-ref/conversational-ai/client-toolkit/android#addhandler) * [`subscribeMessage`](/en/api-reference/api-ref/conversational-ai/client-toolkit/android#subscribemessage) * [`unsubscribeMessage`](/en/api-reference/api-ref/conversational-ai/client-toolkit/android#unsubscribemessage) * [`destroy`](/en/api-reference/api-ref/conversational-ai/client-toolkit/android#destroy) * [`onTranscriptUpdated`](/en/api-reference/api-ref/conversational-ai/client-toolkit/android#ontranscriptupdated) **Folder structure** * `ConversationalAIAPI.swift`: API interface and related data structures and enumerations * `ConversationalAIAPIImpl.swift`: ConversationalAI API main implementation logic * `Transcription/` * `TranscriptionController.swift`: transcription controller **API reference** * [`addHandler`](/en/api-reference/api-ref/conversational-ai/client-toolkit/ios#addhandler) * [`subscribeMessage`](/en/api-reference/api-ref/conversational-ai/client-toolkit/ios#subscribemessage) * [`unsubscribeMessage`](/en/api-reference/api-ref/conversational-ai/client-toolkit/ios#unsubscribemessage) * [`destroy`](/en/api-reference/api-ref/conversational-ai/client-toolkit/ios#destroy) * [`onTranscriptUpdated`](/en/api-reference/api-ref/conversational-ai/client-toolkit/ios#ontranscriptupdated) **Folder structure** * `index.ts`: API class * `type.ts`: API interface and related data structures and enumerations * `utils/` * `index.ts`: API utility functions * `events.ts`: event management class * `sub-render.ts`: transcription module **API reference** * [`IConversationalAIAPIEventHandler interface`](/en/api-reference/api-ref/conversational-ai/client-toolkit/web#iconversationalaiapieventhandlers-interface) * [`EConversationalAIAPIEvents`](/en/api-reference/api-ref/conversational-ai/client-toolkit/web#econversationalaiapievents) * [`subscribeMessage`](/en/api-reference/api-ref/conversational-ai/client-toolkit/web#subscribemessage) * [`unsubscribeMessage`](/en/api-reference/api-ref/conversational-ai/client-toolkit/web#unsubscribe) * [`destroy`](/en/api-reference/api-ref/conversational-ai/client-toolkit/web#destroy) # Convo AI Device Kit (/en/ai/device-kit) Convo AI Device Kit R1 is a device-oriented product built on top of Agora's IoT stack and Conversational AI Engine. It packages hardware, connectivity, audio processing, and software resources into a faster path for smart device teams. Use this product space when you need a hardware-first developer workflow instead of the general Conversational AI build path. ## Start here [#start-here] * [Quickstart](start-here/quickstart) ## Build [#build] * [Run the R1 demo](build/run-the-r1-demo) * [Run the demo server](build/run-the-demo-server) * [Configure device network](build/configure-device-network) * [Build and flash firmware](build/build-and-flash-firmware) ## Understand the system [#understand-the-system] * [Architecture overview](build/architecture-overview) * [Specifications and compatibility](build/specifications-and-compatibility) ## Plan rollout [#plan-rollout] * [Enable services](reference/enable-services) * [Pricing](reference/pricing) * [Release notes](reference/release-notes) # Integrate with MCP (/en/ai/get-started/mcp-integrate) ## Connect to the Agora Doc MCP Server [#connect-to-the-agora-doc-mcp-server] Choose one of the following ways to connect to the MCP server: #### Trae [#trae] **Manual configuration:** Add the following configuration manually in Trae under Settings > MCP: ```json title= { "mcpServers": { "shengwang-docs": { "type": "http", "url": "https://mcp.agora.io" } } } ``` #### Claude Code [#claude-code] **Add with the `CLI`:** ```bash claude mcp add --transport http shengwang-docs https://mcp.agora.io ``` **Or configure it manually:** Create or edit `.mcp.json` in the project root, then add the following section: ```json { "mcpServers": { "shengwang-docs": { "type": "http", "url": "https://mcp.agora.io" } } } ``` #### Codex [#codex] **Add with the `CLI`:** ```bash codex mcp add shengwang-docs --url https://mcp.agora.io ``` **Or configure it manually:** Create or edit `.codex/config.toml` in the project root, then add the following section: ```text [mcp_servers.shengwang-docs] url = "https://mcp.agora.io" ``` #### Cursor [#cursor] **Manual configuration:** Create or edit `.cursor/mcp.json` in the project root, then add the following section: ```json { "mcpServers": { "shengwang-docs": { "url": "https://mcp.agora.io" } } } ``` #### Kiro [#kiro] **Manual configuration:** Create or edit `.kiro/settings/mcp.json` in the project root, then add the following section: ```json { "mcpServers": { "shengwang-docs": { "url": "https://mcp.agora.io" } } } ``` #### VS Code & Copilot [#vs-code--copilot] **Add with the `CLI`:** ```bash code --add-mcp '{"name":"shengwang-docs","type":"http","url":"https://mcp.agora.io"}' ``` **Or configure it manually:** Create or edit `.vscode/mcp.json` in the root directory, then add the following section: ```json { "servers": { "shengwang-docs": { "type": "http", "url": "https://mcp.agora.io" } } } ``` ## MCP tools [#mcp-tools] * `search-docs`: Search for relevant documentation. * `list-docs`: Browse documentation categories and document lists. * `get-doc-content`: Read the full content of a specified document. ## Use MCP tools [#use-mcp-tools] **Automatically called by the coding agent** In Agent Mode, you can describe your requirement directly and let the coding agent decide when to call MCP tools. For example: ```text Search the relevant Agora documentation, read the document content, and tell me how to build the simplest possible 1v1 real-time audio and video call. ``` **Specify tool calls in the prompt** You can also specify which MCP tools to use in the prompt through the tool-calling syntax supported by your AI IDE. For example: ```text #search-docs #get-doc-content Use search-docs to find relevant Agora documentation, then use get-doc-content to read the full content of the matched document and tell me how to build an interactive experience with an AI agent. ``` # Voice agent quickstart (/en/ai/get-started/quickstart) Set up a working voice agent in under five minutes. This page walks you through installing the Agora skills to give your AI coding assistant the official quickstarts and Agora CLI workflows. You then use the CLI to log in, clone the official starter, and run it locally. You can follow the CLI steps yourself or paste the sample prompt and let your assistant handle setup for you. ### Install Agora skills [#install-agora-skills] Agora skills teaches your AI coding assistant how to work with Conversational AI projects using official starter repos and the Agora CLI, including signing in, project binding, generating environment files, and running diagnostics. Install the CLI in the next section, or ask your assistant to run the installer for you. ```bash npx skills add https://github.com/agoraio/skills --skill agora ``` Paste the following prompt into your assistant's chat. You can replace Python with TypeScript or Go depending on your language preference: ```text Set up and run the Agora Conversational AI Python starter project locally. ``` Your assistant installs the CLI, signs you in, scaffolds the official starter, and guides you through the remaining steps. Follow the manual CLI steps below if you prefer to run each command yourself. ### Install the Agora CLI [#install-the-agora-cli] The Agora CLI is a native Go binary available at [AgoraIO-Community/cli](https://github.com/AgoraIO-Community/cli). macOS and Linux Windows (PowerShell) ```bash curl -fsSL https://agoraio.github.io/cli/install.sh | sh agora --help ``` If the `agora` command is not found after installation, re-run with `--add-to-path` or manually add the install directory to your shell profile. ```powershell irm https://agoraio.github.io/cli/install.ps1 | iex agora --help ``` If your execution policy blocks inline scripts, download `install.ps1` and run: ```powershell powershell -ExecutionPolicy Bypass -File .\install.ps1 ``` ## Sign in, scaffold, and run [#sign-in-scaffold-and-run] Sign in with the Agora CLI, clone the starter project, and configure it for your chosen language. 1. Sign in to Agora Console. ```bash agora login ``` 2. Use `agora init` to clone the official starter for your chosen template, bind it to your Agora project, and write the runtime-specific environment file. Python TypeScript Go ```bash agora init my-python-demo --template python cd my-python-demo bun install bun run dev ``` ```bash agora init my-nextjs-demo --template nextjs cd my-nextjs-demo pnpm install pnpm dev ``` ```bash agora init my-go-demo --template go cd my-go-demo make setup make dev ``` 3. Open `http://localhost:3000` and click **Start conversation**. If the agent does not join or transcripts do not appear, run `agora project doctor` to check credential validity, feature enablement, and network reachability. ## Next steps [#next-steps] Explore the following topics: * [Start and stop an agent](../build/start-stop-agent): Create and end a voice agent session with the REST API. * [Use managed mode](../build/custom-model-integration/managed-mode): Start with Agora-managed ASR, LLM, and TTS providers. * [Optimize conversation latency](../best-practices/optimize-latency): Tune ASR, LLM, TTS, and deployment choices. * [Optimize audio](../best-practices/audio-setup): Apply recommended client audio settings. * [REST API reference](/en/api-reference/api-ref/conversational-ai): Review exact request and response shapes. * Explore the SDK API references for [TypeScript](/en/api-reference/api-ref/server-sdk/typescript), [Python](/en/api-reference/api-ref/server-sdk/python), and [Go](/en/api-reference/api-ref/server-sdk/go). # Integrate with Skills (/en/ai/get-started/skills-integrate) ## Install Agora skills [#install-agora-skills] Choose one of the following ways to install the skills: #### Skills CLI [#skills-cli] Install with the `CLI`: ```bash npx skills add AgoraIO/skills ``` This is the most direct installation method. After the installation finishes, restart the session or refresh the skills list according to the instructions for your coding agent. #### Claude Code Plugin Marketplace [#claude-code-plugin-marketplace] Run the following command in `Claude Code`: ```bash plugin marketplace add AgoraIO/skills ``` #### OpenClaw [#openclaw] Install through `ClawHub`: ```bash clawhub install voice-ai-integration clawhub update voice-ai-integration ``` Use `install` the first time and `update` for later upgrades. ## Repository [#repository] * [Agora skills](https://github.com/AgoraIO/skills) ## Best practices [#best-practices] After configuring the skills, you can directly describe tasks such as integration, code generation, or troubleshooting in your coding agent. The skills help the agent choose the right integration workflow and generation rules, and supplement responses with the latest Agora documentation, making generated code, configuration guidance, and issue diagnosis more accurate. * Clearly specify the Agora product, target platform, development language, and desired feature in your prompt. * Provide as much context as possible, such as your current tech stack, existing code, expected interaction flow, whether you need a token, and whether you need server-side examples. This helps the AI produce solutions that are easier to use directly. * When troubleshooting, include the observed issue, reproduction steps, logs, error codes, screenshots, or call stack details whenever possible so the coding agent can narrow down the cause faster. * For complex tasks, prefer an AI IDE with stronger model support and more complete tooling for more stable code generation and problem analysis. ## Prompt example [#prompt-example] After configuring the skills, you can send a prompt like the following to your coding agent to generate an AI voice conversation demo with real-time captions: ```markdown Build me an AI voice conversation demo. After the user opens the web page and clicks "Start conversation", they should be able to talk with the AI by voice, and the page should display real-time captions. ## Tech stack - Frontend: Next.js + React + TypeScript + Tailwind CSS - Backend: FastAPI (Python) ## Requirements 1. Start and end the voice conversation with one click 2. Show real-time captions for both the user and the AI 3. Mute and unmute the microphone 4. Display the AI status (listening, thinking, speaking) 5. Include a system log panel ``` # Enable service (/en/ai/reference/enable-conversational-ai) Before you build with Conversational AI, enable the service for your Agora project in the Agora Console. ## Basic flow [#basic-flow] 1. Sign in to the [Agora Console](https://console.agora.io). 2. Open **Projects** and select the target project. 3. Under **Conversational AI**, enable the service for the project. 4. Generate the credentials you need for REST API or SDK access. # Notification event types (/en/ai/reference/event-types) After enabling Agora message notifications, the Agora notification server sends channel event notifications to your server through HTTPS POST requests. The data format is `JSON`, the character encoding is `UTF-8`, and the signature algorithm can be either `HMAC/SHA1` or `HMAC/SHA256`. This page explains the types of events returned in channel event callbacks and their meanings. If you are starting from a product goal instead of an event number, begin with [Monitor agent status, errors, and performance](../build/handle-runtime-events/monitor-agent-runtime). For webhook setup, request format, and signature verification, see [Webhooks](../build/handle-runtime-events/webhooks). ## Choose event types by goal [#choose-event-types-by-goal] | Goal | Use these event types | What they help you answer | | --------------------------------------------------- | ---------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Confirm that a session started or ended | [`101 agent joined`](#101-agent-joined), [`102 agent left`](#102-agent-left) | Did the agent enter the channel? How did the session end? | | Retrieve conversation history after a session | [`103 agent history`](#103-agent-history) | What was said during the session? See [Retrieve conversation history after a session ends](../build/handle-runtime-events/retrieve-session-history). | | Investigate failures and alert on production issues | [`110 agent error`](#110-agent-error), [`102 agent left`](#102-agent-left) | Which module failed? Did the session terminate because of that failure? See [Debug agent failures with runtime events](../build/handle-runtime-events/debug-agent-failures). | | Measure latency and runtime health | [`111 agent metrics`](#111-agent-metrics) | How long do LLM and TTS stages take? Are there regressions? | | Track telephony call progress | [`201 inbound call state`](#201-inbound-call-state), [`202 outbound call state`](#202-outbound-call-state) | Where is the SIP call in its lifecycle? | ## Request header [#request-header] The message notification callback header contains the following fields: Application/json The signature value generated by Agora using the customer key and `HMAC/SHA1` algorithm. Use the customer key and `HMAC/SHA1` algorithm to verify the signature value. See Verify the signature for details. The signature value generated by Agora using the customer key and `HMAC/SHA256` algorithm. Use the customer key and `HMAC/SHA256` algorithm to verify the signature value. See Verify the signature for details. ## Request body [#request-body] The message notification callback request body contains the following fields: Notification ID. Identifies an event notification from the Agora server. Business ID. A value of `17` indicates a Conversational AI Engine notification. The event type of the notification. See [Event types](#event-types) for details. The Unix timestamp (ms) indicating when the Agora message server sent the event notification to your server. This value is updated when the notification is retried. The session ID. The specific content of the notification event. `payload` varies depending on the event type. For details, see [Event types](#event-types). Following is an example of a message notification callback request body: ```json { "sid": "C866467GVJJ54687", "noticeId": "2000001428:4330:107", "productId": 17, "eventType": 101, "notifyMs": 1611566412672, "payload": {} } ``` ## Event types [#event-types] The Agora message notification service notifies the following Conversational AI Engine events: | Event Type | Event name | Event description | | :------------------------------ | :------------------ | :---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | [101](#101-agent-joined) | agent joined | The agent joins the channel. | | [102](#102-agent-left) | agent left | The agent leaves the channel. | | [103](#103-agent-history) | agent history | After an agent stops, this event notifies the stored history, which includes messages exchanged between the user and the agent and timestamps indicating when the agent was created and stopped. The maximum number of entries is determined by the `llm.max_history` parameter you can set when starting the agent. The default value is `32`. | | [104](#104-agent-expire) | agent expire | The agent's RTC token is about to expire. | | [110](#110-agent-error) | agent error | Agent error. | | [111](#111-agent-metrics) | agent metrics | The performance metrics of an agent. | | [112](#112-turns-finished) | turns finished | Batch callback of dialogue round data after the session ends. | | [201](#201-inbound-call-state) | inbound call state | Changes in the status of an incoming call. | | [202](#202-outbound-call-state) | outbound call state | Changes in the status of an outgoing call. | ### 101 agent joined [#101-agent-joined] An `eventType` of `101` indicates that an agent has joined a channel. The `payload` contains the following fields: Unique identifier of the agent. The agent name provided when calling [Start a conversational AI agent](/en/api-reference/api-ref/conversational-ai/join). Unique within a channel. Timestamp indicating when the agent was created. The name of the channel the agent was in. Custom labels in key-value pair format. Contains the same labels that were set when starting the agent. #### Payload example [#payload-example] ```json { "agent_id": "1NT29X10YHxxxxxWJOXLYHNYB", "name": "my-agent", "start_ts": 1737111452, "channel": "xxxxx", "labels": { "campaign_id": "test_campaign", "customer_group": "vip" } } ``` ### 102 agent left [#102-agent-left] An `eventType` of `102` indicates that an agent has left a channel. The `payload` contains the following fields: Unique identifier of the agent. The agent name provided when calling [Start a conversational AI agent](/en/api-reference/api-ref/conversational-ai/join). Unique within a channel. Timestamp indicating when the agent was created. Timestamp indicating when the agent left the channel. The name of the channel the agent was in. Agent status. The reason why the agent left the channel. Custom labels in key-value pair format. Contains the same labels that were set when starting the agent. #### Payload examples [#payload-examples] Following are some examples of the `payload` when an agent leaves a channel for different reasons. Agent manually stopped Agent idle timeout Agent error RTC connection error Task lifetime limit exceeded ```json { "agent_id": "1NT29X10YHxxxxxWJOXLYHNYB", "name": "my-agent", "start_ts": 1737111452, "stop_ts": 1737111455, "channel": "xxxxx", "status": "STOPPED", "message": "OK", "labels": { "campaign_id": "test_campaign", "customer_group": "vip" } } ``` ```json { "agent_id": "1NT29X10YHxxxxxWJOXLYHNYB", "name": "my-agent", "start_ts": 1737111452, "stop_ts": 1737111455, "channel": "xxxxx", "status": "STOPPED", "message": "Idle for too long", "labels": { "campaign_id": "test_campaign", "customer_group": "vip" } } ``` ```json { "agent_id": "1NT29X10YHxxxxxWJOXLYHNYB", "name": "my-agent", "start_ts": 1737111452, "stop_ts": 1737111455, "channel": "xxxxx", "status": "FAILED", "message": "Connecting for too long", "labels": { "campaign_id": "test_campaign", "customer_group": "vip" } } ``` ```json { "agent_id": "1NT29X10YHxxxxxWJOXLYHNYB", "name": "my-agent", "start_ts": 1737111452, "stop_ts": 1737111455, "channel": "xxxxx", "status": "FAILED", "message": "RTC connection error", "labels": { "campaign_id": "test_campaign", "customer_group": "vip" } } ``` ```json { "agent_id": "1NT29X10YHxxxxxWJOXLYHNYB", "name": "my-agent", "start_ts": 1737111452, "stop_ts": 1737111455, "channel": "xxxxx", "status": "STOPPED", "message": "Task lifetime limit exceeded", "labels": { "campaign_id": "test_campaign", "customer_group": "vip" } } ``` ### 103 agent history [#103-agent-history] An `eventType` of `103` notifies the history of a user and agent dialogue. The notification `payload` contains the following fields: Unique identifier of the agent. The agent name provided when calling [Start a conversational AI agent](/en/api-reference/api-ref/conversational-ai/join). Unique within a channel. The name of the channel the agent was in. The timestamp indicating when the agent started and joined the channel. The timestamp indicating when the agent stopped and left the channel. Agent history. The message sender. * `user`: User * `assistant`: AI agent Message content. Unix timestamp in milliseconds indicating when the user started speaking or the agent started TTS playback. Only returned when `llm.vendor` is `custom`. Unix timestamp in milliseconds indicating when the user stopped speaking, or when TTS playback completed or was interrupted. Only returned when `llm.vendor` is `custom`. The total delay in milliseconds introduced by audio processing algorithms, including noise reduction, background voice suppression, and voiceprint locking, after audio is captured from the user's microphone. Use this value to align timestamps with cloud recording audio. Only returned when: * `llm.vendor` is `custom` * `contents[].role` is `user` * Actual voice input is present Custom labels in key-value pair format. Contains the same labels that were set when starting the agent. `speech_start_ms`, `speech_end_ms`, and `speech_algorithmic_delay` are returned only in `103 agent history` events for sessions that use `llm.vendor` set to `custom`. #### Payload examples [#payload-examples-1] Basic Custom LLM ```json { "agent_id": "xxxx", "name": "my-agent", "channel": "xxxx", "start_ts": 123, "stop_ts": 123, "contents": [ { "role": "user", "content": "hello." }, { "role": "assistant", "content": "hi, how can I help you?" } ], "labels": { "campaign_id": "test_campaign", "customer_group": "vip" } } ``` ```json { "agent_id": "xxxx", "name": "my-agent", "channel": "xxxx", "start_ts": 123, "stop_ts": 123, "contents": [ { "role": "user", "content": "Hello, I need help with my recent order.", "speech_start_ms": 1715000001200, "speech_end_ms": 1715000003400, "speech_algorithmic_delay": 120 }, { "role": "assistant", "content": "Sure, could you please provide your order number?", "speech_start_ms": 1715000004100, "speech_end_ms": 1715000005000 }, { "role": "user", "content": "I'd like to check the status of my package, tracking number 12345.", "speech_start_ms": 1715000005200, "speech_end_ms": 1715000007600, "speech_algorithmic_delay": 120 }, { "role": "assistant", "content": "Your package with tracking number 12345 is currently out for delivery.", "speech_start_ms": 1715000009500, "speech_end_ms": 1715000013200 }, { "role": "user", "content": "[think API injected] User level: VIP" } ], "labels": { "campaign_id": "test_campaign", "customer_group": "vip" } } ``` ### 104 agent expire [#104-agent-expire] An `eventType` of `104` indicates that the agent's RTC token is about to expire. The `payload` contains the following fields: Unique identifier of the agent. The agent name provided when calling [Start a conversational AI agent](/en/api-reference/api-ref/conversational-ai/join). Unique within a channel. Timestamp indicating when the agent was created. The name of the channel the agent is in. Token expiry warning message. Fixed value: `Task exceeded maximum lifetime`. Info This event is a warning only. The agent does not exit the channel or interrupt the session when this event is triggered. Upon receiving this event, call the [Update agent](/en/api-reference/api-ref/conversational-ai/update) API with a new `properties.token` to refresh the token before it expires. #### Payload example [#payload-example-1] ```json { "agent_id": "1NT29X10YHxxxxxWJOXLYHNYB", "name": "my-agent", "start_ts": 1737111452, "channel": "xxxxx", "message": "Task exceeded maximum lifetime" } ``` ### 110 agent error [#110-agent-error] An `eventType` of `110` indicates that an agent has encountered an error. The `payload` contains the following fields: Unique identifier of the agent. The agent name provided when calling [Start a conversational AI agent](/en/api-reference/api-ref/conversational-ai/join). Unique within a channel. Timestamp indicating when the agent was created. The name of the channel the agent was in. Transcript conversation turn. For details, see [Display live transcripts](../build/transcripts#reference). An array of error messages. Each object contains the following fields: The module where the error occurred. Transcript dialogue turn. Error code. Refer to the error code document of the vendor corresponding to the error module for detailed information. The error message. Custom labels in key-value pair format. Contains the same labels that were set when starting the agent. #### Payload examples [#payload-examples-2] LLM module error Greeting audio URL error {/* SIP module error */} ```json { "agent_id": "1NT29X10YHxxxxxWJOXLYHNYB", "name": "my-agent", "start_ts": 1737111452, "channel": "xxxxx", "turn_id": 2, "errors": [{"module": "llm", "turn_id":2, "code":503, "message": "Insufficient balance."}], "labels": { "campaign_id": "test_campaign", "customer_group": "vip" } } ``` When the greeting audio file fails to download, times out, is in an unsupported format, or fails to decode, the system reports the error via a `110 agent error` event and automatically falls back to TTS synthesis using `greeting_message`. In these errors, `errors[].module` is `agent`. ```json { "agent_id": "1NT29X10YHxxxxxWJOXLYHNYB", "name": "outbound_voice_agent", "start_ts": 1737111452, "channel": "outbound_call_channel_001", "turn_id": 0, "errors": [ { "module": "agent", "turn_id": 0, "code": 4001, "message": "Greeting audio URL unreachable, fallback to TTS. audio_url: https://cdn.example.com/audio/welcome.mp3" } ], "labels": {} } ``` {/* ```json { "agent_id": "1NT29X10YHxxxxxWJOXLYHNYB", "name": "my-agent", "start_ts": 1737111452, "channel": "xxxxx", "turn_id": "", "errors": [{"module": "sip", "turn_id": "", "code":50, "message": "Insufficient balance."}], "labels": { "campaign_id": "test_campaign", "customer_group": "vip" } } ``` */} Info If `parameters.enable_error_message` is set to `true` when starting the agent, greeting audio URL errors are also delivered to the client via the `onMessageError` RTM callback, in addition to the server-side webhook event. ### 111 agent metrics [#111-agent-metrics] An `eventType` of `111` notifies the performance metrics of an agent. The `payload` contains the following fields: Unique identifier of the agent. The agent name provided when calling [Start a conversational AI agent](/en/api-reference/api-ref/conversational-ai/join). Unique within a channel. Timestamp indicating when the agent was created and joined the channel. Timestamp indicating when the agent stopped and left the channel. The name of the channel where the agent is located. An array of performance metrics. Each object contains the following fields: Dialogue round ID. The ttlw (Time To Last Word) metric of the ASR module. Represents the delay (in milliseconds) from when the user finishes speaking until the last word is output by the ASR. The ttfb (Time To First Byte) metric for the LLM module. Represents the latency (in milliseconds) from the start of an LLM request to the receipt of the first byte of the response. The ttfs (Time To First Sentence) metric for the LLM module. Represents the time (in milliseconds) elapsed from the start of an LLM request to the receipt of the first complete sentence response. The ttfb (Time To First Byte) metric for the TTS module. Represents the latency (in milliseconds) from the start of a TTS request to the receipt of the first byte of the response. Custom labels in key-value pair format. Contains the same labels that were set when starting the agent. #### Payload example [#payload-example-2] ```json { "agent_id": "A42AC47Hxxxxxxxx4PK27ND25E", "name": "my-agent", "start_ts": 1000, "stop_ts": 1672531200, "channel": "test-channel", "metrics": [ { "turn_id": 1, "asr_ttlw": 503, "llm_ttfb": 1104, "tts_ttfb": 85 }, { "turn_id": 2, "asr_ttlw": 2385, "llm_ttfb": 980, "tts_ttfb": 78 } ], "labels": { "campaign_id": "test_campaign", "customer_group": "vip" } } ``` ### 112 turns finished [#112-turns-finished] An `eventType` of `112` indicates a batch callback of conversation turn data after the session ends. This event returns the conversation turns for the current session in a single callback, making it convenient for post-session analysis, archiving, or offline processing. When a session has more than 200 turns, the callback data may be truncated. Use the `is_truncated` field to check whether the current callback contains the complete turn data. The `payload` contains the following fields: Unique identifier of the agent. The agent name provided when calling [Start a conversational AI agent](/en/api-reference/api-ref/conversational-ai/join). Unique within a channel. The name of the channel where the agent is located. Timestamp indicating when the agent was created and joined the channel. Timestamp indicating when the agent stopped and left the channel. The total number of conversation turns in the session. Whether the turn data in the current callback is truncated. A value of `true` indicates that the current callback did not return all turns. Custom labels in key-value pair format. Contains the same labels that were set when starting the agent. Only returned if labels were passed in when creating the agent. A list of conversation turns for the session. Unique identifier of the agent. The name of the channel where the agent is located. The sequential index of the turn within the session. Starts at `1`. Details about the start of the turn. The Unix timestamp in milliseconds (UTC time) when the turn started. The type of event that initiated the turn. Additional context about the turn start event. Included fields depend on the value of the `type` field. The duration of the user's voice input in milliseconds. Included only when `type` is `voice_input`. The minimum voice duration in milliseconds required to trigger an interruption. Included only when `type` is `voice_input`. Details about the end of the turn. The Unix timestamp in milliseconds (UTC time) when the turn ended. The type of event that ended the turn. Additional context about the turn end event. Latency metrics for the turn. The end-to-end latency in milliseconds for the turn. Represents the time from when the user finishes speaking to when the agent starts speaking. A breakdown of latency by segment. Algorithm processing delay in milliseconds. The ASR Time To Last Word (TTLW) in milliseconds. Represents the delay from when the user finishes speaking to when the ASR module outputs the last word. The LLM Time To First Token (TTFT) in milliseconds. Represents the delay from when the LLM receives the request to when it outputs the first token. The LLM First Token To First Sentence (FTFS) in milliseconds. Represents the delay from when the LLM outputs the first token to when it outputs the first complete sentence. The TTS Time To First Byte (TTFB) in milliseconds. Represents the delay from when the TTS module receives a text request to when it outputs the first audio byte. Network transmission delay in milliseconds. Not returned when the user is connected using the RTC Web SDK. #### Payload example [#payload-example-3] ```json { "agent_id": "xxxx", "name": "support_agent_001", "channel": "xxxx", "start_ts": 1715000000000, "stop_ts": 1715000015000, "total_turn_count": 250, "is_truncated": true, "labels": { "campaign_id": "test_campaign", "customer_group": "vip" }, "turns": [ { "agent_id": "xxxx", "channel": "xxxx", "turn_id": 1, "start": { "start_at": 1774579820147, "type": "voice_input" }, "end": { "end_at": 1774579822412, "type": "ok" }, "metrics": { "e2e_latency_ms": 1500, "segmented_latency_ms": { "algorithm_processing": 300, "asr_ttlw": 200, "llm_ttft": 600, "tts_ttfb": 200, "transport": 300 } } } ] } ``` ### 201 inbound call state [#201-inbound-call-state] An `eventType` of `201` notifies changes in the status of an incoming call. The `payload` contains the following fields: Unique identifier of the agent. The agent name provided when calling [Start a conversational AI agent](/en/api-reference/api-ref/conversational-ai/join). Unique within a channel. The name of the channel where the agent is located. Incoming call status: * `START`: Call received * `ANSWERED`: Call answered * `TRANSFERED`: Transfer to human operator * `HANGUP`: Call hung up Timestamp (in milliseconds) of the state change. Custom labels in key-value pair format. Contains the same labels that were set when starting the agent. #### Payload example [#payload-example-4] ```json { "agent_id": "1NT29X10YHxxxxxWJOXLYHNYB", "name": "my-agent", "channel": "xxxxx", "state": "START", "report_ms": 1737111452000, "labels": { "campaign_id": "test_campaign", "customer_group": "vip" } } ``` ### 202 outbound call state [#202-outbound-call-state] An `eventType` of `202` notifies changes in the status of an outgoing call. The `payload` contains the following fields: Unique identifier of the agent. The agent name provided when calling [Start a conversational AI agent](/en/api-reference/api-ref/conversational-ai/join). Unique within a channel. The name of the channel where the agent is located. Outgoing call status: * `START`: Call begins * `CALLING`: Making a phone call * `RINGING`: The telephone is ringing * `ANSWERED`: The call was connected * `HANGUP`: Call hung up Timestamp (in milliseconds) of the state change. Custom labels in key-value pair format. Contains the same labels that were set when starting the agent. #### Payload example [#payload-example-5] ```json { "agent_id": "1NT29X10YHxxxxxWJOXLYHNYB", "name": "my-agent", "channel": "xxxxx", "state": "CALLING", "report_ms": 1737111452000, "labels": { "campaign_id": "test_campaign", "customer_group": "vip" } } ``` # OpenAI Realtime integration (/en/ai/reference/openai-realtime-integration) This guide explains how to combine Agora's realtime audio transport with the OpenAI Realtime API. Use this integration when you want Agora to handle the live audio session and network transport, while OpenAI Realtime handles the model-side conversational behavior. This is not the same as building on Agora's managed Conversational AI Engine. If you want Agora to manage the voice-agent runtime, use the main Conversational AI build path instead. ## When to use this path [#when-to-use-this-path] * You already know you want to build around the OpenAI Realtime API. * You need Agora RTC or Video SDK transport in front of that model stack. * You are building a browser demo, device-oriented flow, or backend-mediated prototype around OpenAI Realtime. ## When not to use this path [#when-not-to-use-this-path] * You want the default Agora-managed voice-agent stack. * You are looking for the main onboarding path for Conversational AI Engine. * You want Agent Studio or the Device Kit route instead of a custom integration. ## Core pieces [#core-pieces] * **Agora App ID**: Identifies your Agora project. * **App certificate and token model**: Used for secure join flows where needed. * **Channel and user ID**: Define the live session and participants. * **SD-RTN**: Provides Agora's low-latency transport layer. * **OpenAI Realtime API**: Provides the model-side realtime conversation flow. ## Recommended quickstart flow [#recommended-quickstart-flow] 1. Prepare your Agora account and project. 2. Decide how your client joins the Agora session and how the OpenAI-side session is coordinated. 3. Configure the OpenAI Realtime model and audio flow. 4. Run the round-trip voice path locally and validate latency, auth, and media behavior. ## Security and network checks [#security-and-network-checks] * Keep Agora credentials and provider API keys on the server side. * Separate local, staging, and production credentials. * Review firewall requirements before testing in restricted networks. * Re-test the full join and media path after changing transport or security settings. ## Related pages [#related-pages] * [Conversational AI quickstart](/en/ai/get-started/quickstart) * [MCP integration](/en/ai/get-started/mcp-integrate) * [Agora skills](/en/ai/get-started/skills-integrate) * [Security and privacy](/en/introduction/security-privacy) * [Firewall requirements](/en/introduction/firewall) * [Glossary](/en/introduction/glossary) # Pricing (/en/ai/reference/pricing) This page describes how Agora calculates and bills for Conversational AI Engine usage. When you use Conversational AI Engine in your project, Agora charges a monthly fee based on your usage across all projects under your developer account. At the end of each month, the free quota is subtracted from your total usage, and the remaining minutes are multiplied by the unit price to calculate your bill, rounded to two decimal places. For general billing information, see [Billing](../../introduction/billing). Info If you have signed a sales contract with Agora, your billing follows the terms in your contract. ## Unit price [#unit-price] Creating a Conversational AI Engine instance using the REST API and joining a channel incurs an audio task fee at the following rate: | Usage Type | Pricing (USD / minute) | Free Minutes | | ----------------------------------- | ---------------------: | -------------------------- | | Conversational AI Engine Audio Task | 0.10\* | First 300 minutes are free | \* The unit price includes usage of selected ASR, LLM, and TTS models. You are charged the same price even if you bring your own key (BYOK). Usage of ASR, LLM, and TTS providers is included in the unit price when using an Agora managed key. Agora provides and manages the API keys for the following providers: * **ASR** * ARES * Deepgram nova-2 * Deepgram nova-3 * **LLM** * OpenAI GPT-4o-mini * OpenAI GPT-4.1-mini * OpenAI GPT-5-nano * OpenAI GPT-5-mini * **TTS** * MiniMax 2.6 Turbo * MiniMax 2.8 Turbo * OpenAI TTS-1 ## Examples [#examples] The following examples demonstrate how billing is calculated for different Conversational AI Engine configurations. ### Using ASR, LLM, and TTS managed by Agora [#using-asr-llm-and-tts-managed-by-agora] User A joins a channel and starts a voice conversation with an instance created by Conversational AI Engine. The interaction lasts for 10 minutes. User A and the Conversational AI Engine instance exit the channel at the same time. Agora calculates the cost for this session as follows: | Usage Type | Duration (minutes) | Unit Price | Service Cost (USD) | | ---------------------------------------------------------- | -----------------: | ---------: | -----------------: | | User A: Audio RTC | 10 | 0.00099 | 0.0099 | | Conversational AI Engine Audio Task | 10 | 0.10 | 1.00 | | ASR: ARES, LLM: OpenAI GPT-4o-mini, TTS: MiniMax 2.8 Turbo | 10 | 0.00 | 0.00 | Total cost: 1.0099 USD # Release notes (/en/ai/reference/release-notes) This document tracks notable changes and improvements to the Conversational AI Engine. ## Releases [#releases] ### v2.9 [#v29] Released on July 1, 2026. #### New features [#new-features] Included in this release: * **Manual turn control** You can now explicitly control when a user's speech starts and ends by sending manual Start of Speech (SoS) and End of Speech (EoS) requests over Signaling (RTM), instead of relying on automatic detection. This supports scenarios with strict turn-boundary requirements, such as AI interviews, interactive quizzes, or push-to-talk. Set [`start_of_speech.mode`](/en/api-reference/api-ref/conversational-ai/join#properties-turn-detection-config-start-of-speech-mode) and [`end_of_speech.mode`](/en/api-reference/api-ref/conversational-ai/join#properties-turn-detection-config-end-of-speech-mode) to `manual` when starting the agent. For details, see [Manually control start and end of speech](/en/ai/best-practices/manual-turn-control). * **Agent state notifications over RTM Message** You can now track agent state changes in real time using Signaling (RTM) Message events. The agent pushes `state.listening`, `state.thinking`, and `state.speaking` events as individual RTM messages, making it easier to record state change timestamps, process events sequentially, or consume state events alongside other RTM Message events. For details, see [Get agent state](/en/ai/best-practices/get-agent-state). * **Pre-recorded greeting audio** You can now play a pre-recorded audio file as the agent's greeting instead of synthesizing it with TTS. Set [`properties.llm.greeting_audio_url`](/en/api-reference/api-ref/conversational-ai/join#properties-llm-greeting-audio-url) to the URL of an `mp3`, `wav`, or `pcm` file when starting the agent. If the audio fails to download, decode, or is in an unsupported format, the agent automatically falls back to TTS synthesis using `greeting_message`. * **Managed mode** You can now use supported ASR, LLM, and TTS providers with Agora-managed credentials, without supplying your own API keys. Set [`credential_mode`](/en/api-reference/api-ref/conversational-ai/join#properties-asr-credential-mode) to `"managed"` within the `asr`, `llm`, or `tts` block when starting an agent. Managed mode replaces the deprecated `preset` parameter. For details, see [Use managed mode](/en/ai/build/custom-model-integration/managed-mode). #### Improvements [#improvements] This release includes the following enhancements. * **Speech captured during an uninterruptible greeting** When [`properties.llm.greeting_configs.interruptable`](/en/api-reference/api-ref/conversational-ai/join#properties-llm-greeting-configs-interruptable) is set to `false`, the agent now silently collects the user's ASR results while the greeting plays. After the greeting ends, the system concatenates the collected speech segments into a single user message and sends it to the LLM for a unified response, instead of discarding speech that occurs during playback. #### API changes [#api-changes] This release introduces the following changes to the RESTful API. * Changes to [**Start a conversational AI agent**](/en/api-reference/api-ref/conversational-ai/join) * **New parameters** * [`properties.llm.greeting_audio_url`](/en/api-reference/api-ref/conversational-ai/join#properties-llm-greeting-audio-url): The URL of a pre-recorded audio file (`mp3`, `wav`, or `pcm`) to play as the agent's greeting instead of synthesizing it with TTS. * [`properties.llm.greeting_configs.audio_download_timeout_ms`](/en/api-reference/api-ref/conversational-ai/join#properties-llm-greeting-configs-audio-download-timeout-ms): The timeout in milliseconds for downloading the greeting audio file. Defaults to `1000`. * [`properties.llm.greeting_configs.audio_pcm_sample_rate`](/en/api-reference/api-ref/conversational-ai/join#properties-llm-greeting-configs-audio-pcm-sample-rate): The sample rate in Hz of the greeting audio file when the format is PCM. Defaults to `16000`. * [`properties.asr.credential_mode`](/en/api-reference/api-ref/conversational-ai/join#properties-asr-credential-mode), [`properties.llm.credential_mode`](/en/api-reference/api-ref/conversational-ai/join#properties-llm-credential-mode), and [`properties.tts.credential_mode`](/en/api-reference/api-ref/conversational-ai/join#properties-tts-credential-mode): Set to `"managed"` to use Agora-managed credentials for the provider without supplying your own API key. * **Updated parameters** * [`properties.turn_detection.config.start_of_speech.mode`](/en/api-reference/api-ref/conversational-ai/join#properties-turn-detection-config-start-of-speech-mode): Now accepts `manual`, which disables automatic start-of-speech detection so the client can explicitly signal the start of user speech over RTM. * [`properties.turn_detection.config.end_of_speech.mode`](/en/api-reference/api-ref/conversational-ai/join#properties-turn-detection-config-end-of-speech-mode): Now accepts `manual`, which disables automatic end-of-speech detection so the client can explicitly signal the end of user speech over RTM. * [`properties.llm.greeting_message`](/en/api-reference/api-ref/conversational-ai/join#properties-llm-greeting-message): In addition to serving as a standalone greeting, this field is now required as a fallback when `greeting_audio_url` is configured, and is used to maintain conversation context when an audio greeting is interrupted. * **Deprecated parameters** * [`preset`](/en/api-reference/api-ref/conversational-ai/join#preset): Deprecated in favor of setting `credential_mode` to `"managed"` within the `asr`, `llm`, or `tts` block. See [Use managed mode](/en/ai/build/custom-model-integration/managed-mode). * Changes to [**Notification event types**](/en/ai/reference/event-types) * **Updated events** * [`110 agent error`](/en/ai/reference/event-types#110-agent-error): Now reports greeting audio URL errors when the greeting audio file fails to download, times out, is in an unsupported format, or fails to decode. The agent automatically falls back to TTS synthesis using `greeting_message`. When `parameters.enable_error_message` is `true`, these errors are also delivered to the client via the `onMessageError` RTM callback. ### v2.8 [#v28] Released on June 11, 2026. #### New features [#new-features-1] Included in this release: * **Session data retention control** By default, session interaction text and audio are temporarily retained for the minimum necessary period to support service operation, agent optimization, and troubleshooting. To disable data retention for a session, set [`properties.parameters.opt_out`](/en/api-reference/api-ref/conversational-ai/join#properties-parameters-opt-out) to `true` when starting an agent. * **RTC token expiry warning** A new [`104 agent expire`](event-types#104-agent-expire) event is triggered when the agent's RTC token is about to expire. Upon receiving this event, call the [Update agent](/en/api-reference/api-ref/conversational-ai/update) API with a new token to refresh it. The agent does not exit the channel or interrupt the session when this event is triggered. * **New ASR provider** * [xAI](../models/asr/xai) * **New LLM provider** * [xAI Grok](../models/llm/xai) * **New TTS provider** * [xAI](../models/tts/xai) #### Improvements [#improvements-1] This release includes the following enhancements. Important These changes may affect your existing integration. Review each change and update your integration as needed. * **Idle timeout behavior change** Setting `properties.idle_timeout` to `0` no longer means the agent runs indefinitely. From this release, `0` only disables the channel idle timeout. Regardless of the `idle_timeout` value, the maximum running time for a single session is 72 hours, after which the agent automatically exits. If your integration previously relied on `idle_timeout = 0` to keep an agent running until manually stopped, update your task scheduling and lifecycle management logic to account for the 72-hour limit. #### API changes [#api-changes-1] This release introduces the following changes to the RESTful API. * Changes to [**Start a conversational AI agent**](/en/api-reference/api-ref/conversational-ai/join) * **New parameters** * [`properties.parameters.opt_out`](/en/api-reference/api-ref/conversational-ai/join#properties-parameters-opt-out): Set to `true` to disable data retention for the current session. * **Updated parameters** * [`properties.idle_timeout`](/en/api-reference/api-ref/conversational-ai/join#properties-idle-timeout): Valid range is now `0` to `259200` seconds (72 hours). Regardless of the value set, the maximum running time for a single session is 72 hours. * Changes to [**Retrieve agent history**](/en/api-reference/api-ref/conversational-ai/history) * **New response fields** * [`contents[].speech_start_ms`](/en/api-reference/api-ref/conversational-ai/history#contents-speech-start-ms): Unix timestamp in milliseconds indicating when the user started speaking or the agent started TTS playback. Only returned when `llm.vendor` is `custom`. * [`contents[].speech_end_ms`](/en/api-reference/api-ref/conversational-ai/history#contents-speech-end-ms): Unix timestamp in milliseconds indicating when the user stopped speaking, or when TTS playback completed or was interrupted. Only returned when `llm.vendor` is `custom`. * [`contents[].speech_algorithmic_delay`](/en/api-reference/api-ref/conversational-ai/history#contents-speech-algorithmic-delay): The total delay in milliseconds introduced by audio processing algorithms after audio is captured from the user's microphone. Only returned when `llm.vendor` is `custom`, `contents[].role` is `user`, and actual voice input is present. * Changes to [**Notification event types**](event-types) * **New events** * [`104 agent expire`](event-types#104-agent-expire): Triggered when the agent's RTC token is about to expire. Call the Update agent API with a new token upon receiving this event. * **Updated events** * [`102 agent left`](event-types#102-agent-left): Added `Task lifetime limit exceeded` as a new exit reason when the 72-hour session limit is reached. * [`103 agent history`](event-types#103-agent-history): Added `contents[].speech_start_ms`, `contents[].speech_end_ms`, and `contents[].speech_algorithmic_delay`. Only returned when `llm.vendor` is `custom`. * Changes to [**Retrieve a list of agents**](/en/api-reference/api-ref/conversational-ai/list) * **Updated parameters** * `state`: Now accepts a comma-separated list of values to filter by multiple agent states in a single request. For example, `state=0,1,2`. ### v2.7 [#v27] Released on May 20, 2026. #### New features [#new-features-2] Included in this release: * **New MLLM provider** * [xAI Grok](../models/mllm/xai) * **New Avatar providers** * [Generic avatar (Beta)](../models/avatar/generic) * **Greeting interruption control** The `properties.llm.greeting_configs` object in the [Start a conversational AI agent](/en/api-reference/api-ref/conversational-ai/join) API now supports an `interruptable` parameter, which controls whether user speech can interrupt a greeting during playback. #### Improvements [#improvements-2] This release includes the following enhancements: Important These changes may affect your existing integration. Review each change and update your integration as needed. * **Response status codes updated** This release adjusts the HTTP response status codes and `reason` values returned by the RESTful API. New status codes `401`, `429`, and `500` are added. New `reason` values `ServiceNotEnabled`, `AccountSuspended`, and `ResourceAllocationFailed` are introduced. The existing `InvalidRequest` reason is replaced by `InvalidRequestBody`, `MissingRequiredField`, and `InvalidFieldValue`. For details, see [Status codes and error messages](/en/api-reference/api-ref/conversational-ai/status-codes). If your integration relies on `reason` values for error handling, retry logic, or alerting, update your logic to handle the new values accordingly. * **Query conversation turns API now returns paginated results** The [Query conversation turn information](/en/api-reference/api-ref/conversational-ai/turns) API no longer returns all turns in a single response. Results are now paginated, defaulting to page 1 with up to 50 turns per page. This affects sessions with more than 50 conversation turns. Update your integration to handle the new `total_turn_count` and `pagination` fields in the response, and retrieve subsequent pages as needed. This release also adds the [112 turns finished](event-types#112-turns-finished) notification event, which delivers conversation turn data in batches after a session ends as an alternative way to retrieve turn data. * **`on_listening_action` default behavior change** The default value of `on_listening_action` in the [Send a custom instruction](/en/api-reference/api-ref/conversational-ai/think) API has changed from `inject` to `interrupt`. When set to `interrupt`, the API immediately interrupts the current flow and initiates a new round of dialogue. If you rely on the previous default behavior, explicitly set `on_listening_action` to `inject` in your requests. #### API changes [#api-changes-2] This release introduces the following changes to the RESTful API. * Changes to [**Start a conversational AI agent**](/en/api-reference/api-ref/conversational-ai/join) * **New parameters** * [`properties.llm.greeting_configs.interruptable`](/en/api-reference/api-ref/conversational-ai/join#properties-llm-greeting-configs-interruptable): Controls whether user speech can interrupt a greeting during playback. * Changes to [**Query conversation turn information**](/en/api-reference/api-ref/conversational-ai/turns) * **New query parameters** Adds `page_index` and `page_size` fields to [query parameters](/en/api-reference/api-ref/conversational-ai/turns#query-parameters). * **New response parameters** The [response](/en/api-reference/api-ref/conversational-ai/turns#response) now includes additional top-level fields: * `agent_id`: The unique identifier of the agent. * `name`: The name of the agent. * `channel`: The name of the RTC channel the agent joined. * `total_turn_count`: The total number of dialogue turns in the session. * `pagination`: Pagination details for the response, including `page_index`, `total_pages`, and `is_last_page`. * Changes to [**Send a custom instruction**](/en/api-reference/api-ref/conversational-ai/think) * **Behavior changes** * [`on_listening_action`](/en/api-reference/api-ref/conversational-ai/think#on-listening-action): The default value has changed from `inject` to `interrupt`. When set to `interrupt`, the API immediately interrupts the current flow and initiates a new round of dialogue. * Changes to [**Notification event types**](event-types) * **New events** * `112 turns finished`: Triggered after a session ends, delivering conversation turn data in batches for post-session analysis and processing. ### v2.6 [#v26] Released on April 22, 2026. #### New features [#new-features-3] Included in this release: * **New TTS provider** * [Deepgram](../models/tts/deepgram) * **New endpoint: Send a custom instruction** Adds a new endpoint to inject a custom text instruction into the agent's current conversation pipeline. The instruction is processed as user input, enabling scenarios such as implicit instruction injection, client-side event triggering, and voice and text collaboration. For details, see [Send a custom instruction](/en/api-reference/api-ref/conversational-ai/think). * **MLLM turn detection refactoring** Introduces a new [`mllm.turn_detection`](/en/api-reference/api-ref/conversational-ai/join#properties-mllm-turn-detection) object for configuring turn detection when using MLLM. When defined, this overrides the top-level `turn_detection` object. Supported modes vary by vendor: * OpenAI Realtime API: `agora_vad`, `server_vad`, `semantic_vad` * Google Gemini Live: `agora_vad`, `server_vad` * **Interruption control** Adds a new top-level [`interruption`](/en/api-reference/api-ref/conversational-ai/join#properties-interruption) object for unified management of agent interruption behavior. Supports `start_of_speech` and `keywords` trigger modes, and configurable handling strategies when interruption is disabled. #### Improvements [#improvements-3] This release includes the following enhancements: * **New agent state callbacks** Adds three callbacks to the [Android](/en/api-reference/api-ref/conversational-ai/client-toolkit/android#onagentlisteningchanged), [iOS](/en/api-reference/api-ref/conversational-ai/client-toolkit/ios#onagentlisteningchanged), and [web](/en/api-reference/api-ref/conversational-ai/client-toolkit/web#econversationalaiapievents) Toolkit event handler interface: * `onAgentListeningChanged`: Listen for changes in the agent's listening state to monitor when the agent starts or stops listening to user input. * `onAgentThinkingChanged`: Listen for changes in the agent's thinking state to monitor when the agent starts or stops processing a request. * `onAgentSpeakingChanged`: Listen for changes in the agent's speaking state to monitor when the agent starts or stops playing back speech. * **Added `name` field to all [NCS event](event-types) payloads** Returns the agent name provided when starting the agent. #### API changes [#api-changes-3] This release introduces the following changes to the RESTful API. * **New endpoint** * [Send a custom instruction](/en/api-reference/api-ref/conversational-ai/think): `POST /v2/projects/{appid}/agents/{agentId}/think` * Changes to [**Start a conversational AI agent**](/en/api-reference/api-ref/conversational-ai/join) * **New parameters** * [`interruption`](/en/api-reference/api-ref/conversational-ai/join#properties-interruption): Unified interruption control configuration. Supports `start_of_speech` and `keywords` trigger modes, and configurable handling strategies when interruption is disabled. * [`greeting_configs.delay_ms`](/en/api-reference/api-ref/conversational-ai/join#properties-llm-greeting-configs-delay-ms): Specifies the delay in milliseconds before the agent plays the greeting message after the user joins the channel. * [`llm.headers`](/en/api-reference/api-ref/conversational-ai/join#properties-llm-headers): Allows you to pass custom headers in LLM requests, such as business-specific fields or tenant identifiers. * [`mllm.enable`](/en/api-reference/api-ref/conversational-ai/join#properties-mllm-enable): Enables the MLLM module. Replaces the deprecated `advanced_features.enable_mllm`. * [`mllm.turn_detection`](/en/api-reference/api-ref/conversational-ai/join#properties-mllm-turn-detection): Turn detection configuration for the MLLM module. When defined, overrides the top-level `turn_detection` object. * [`mllm.turn_detection.server_vad_config.idle_timeout_ms`](/en/api-reference/api-ref/conversational-ai/join#properties-mllm-turn-detection-server-vad-config-idle-timeout-ms): Idle timeout in milliseconds for server VAD mode. Applicable to OpenAI Realtime API only. * [`mllm.vendor`](/en/api-reference/api-ref/conversational-ai/join#properties-mllm-vendor) value `gemini`: Enables Google Gemini Live integration using the Gemini Developer API. * **Behavior changes** * `turn_detection` now handles SoS and EoS detection only. Interruption handling strategies have moved to the new top-level `interruption` field. * `turn_detection.config.start_of_speech.mode` values `keywords` and `disabled` are deprecated. Use the new `interruption` field instead. * When `turn_detection.end_of_speech.mode` is set to `semantic`, EOS detection supports English and Chinese only. For unsupported languages, the engine automatically falls back to VAD. * **Deprecated fields** * `advanced_features.enable_mllm`: Use `mllm.enable` instead. * `turn_detection.config.start_of_speech.mode` value `keywords`: Use `interruption.mode = "keywords"` instead. * `turn_detection.config.start_of_speech.mode` value `disabled`: Use `interruption.enable = false` with `interruption.disabled_config.strategy` instead. * **Removed fields** * `mllm.style` * `mllm.create_response` and `mllm.interrupt_response` for OpenAI Realtime API. * Changes to [**Notification event types**](event-types) * **New fields** * Added `name` to all [NCS event payloads](event-types). Returns the agent name provided when starting the agent. ### v2.5 [#v25] Released on March 31, 2026. #### New features [#new-features-4] Included in this release: * **Preset model configurations** Adds a [`preset`](/en/api-reference/api-ref/conversational-ai/join#preset) parameter to the Join API. Pass a comma-separated list of preset names to apply predefined configurations for ASR, LLM, and TTS providers. When you use a preset, you do not need to provide the endpoint URL, API key, or model for the preset provider. Use the `asr`, `llm`, and `tts` fields to configure additional settings. When you use a preset, the corresponding ASR, LLM, or TTS service is provided through Agora-managed accounts. For details, see the [pricing](pricing) page. Available presets: * ASR: `deepgram_nova_2`, `deepgram_nova_3` * LLM: `openai_gpt_4o_mini`, `openai_gpt_4_1_mini`, `openai_gpt_5_nano`, `openai_gpt_5_mini` * TTS: `minimax_speech_2_6_turbo`, `minimax_speech_2_8_turbo`, `openai_tts_1` * **Pause state detection** When enabled, the agent uses semantic understanding to detect when a user intends to pause the conversation. For example, if the user says "hold on" or "just a moment", the agent waits for further input instead of forwarding the utterance to the LLM. Controlled by the new [`pause_state_enabled`](/en/api-reference/api-ref/conversational-ai/join#properties-turn-detection-config-end-of-speech--mode-config-pause-state-enabled) parameter. * **Conversation turn insights** You can now query per-turn start, end, and latency metrics for completed agent sessions. See [Query conversation turn information](/en/api-reference/api-ref/conversational-ai/turns). * **Real-time TTS parameter updates with custom LLM** In scenarios with a custom LLM integrated, this release adds support for real-time TTS parameter updates during conversations, enabling a more natural conversational experience. This feature applies to the following use cases: * The custom LLM detects a user request for the agent to change its voice or speaking style. * The custom LLM identifies a user's positive emotion and adjusts TTS volume, pitch, speech rate, and other parameters in real time to better match the user's mood. For implementation details, refer to [Real-time TTS parameter updates](../build/custom-model-integration/custom-llm#update-tts-parameters-in-real-time). * **New TTS provider** * [Murf (Beta)](../models/tts/murf) * **New Avatar provider** * [Anam (Beta)](../models/avatar/anam) #### Improvements [#improvements-4] This release includes the following enhancements: * **Migration from HeyGen API to LiveAvatar (Beta) API** Updates the HeyGen avatar model configuration to use the [LiveAvatar (Beta)](../models/avatar/heygen) API. #### API changes [#api-changes-4] This release introduces the following changes to the RESTful API. * **New endpoint: Query conversation turns** Adds a new GET endpoint to query conversation turn information for an agent session. After a conversation ends, use this endpoint to retrieve the start event, end event, and latency metrics for each turn within the session. For details, see [Query conversation turn information](/en/api-reference/api-ref/conversational-ai/turns). * Changes to [**Start a conversational AI agent**](/en/api-reference/api-ref/conversational-ai/join) * **New parameters added** * [`preset`](/en/api-reference/api-ref/conversational-ai/join#preset) * [`pause_state_enabled`](/en/api-reference/api-ref/conversational-ai/join#properties-turn-detection-config-end-of-speech--mode-config-pause-state-enabled) * **Deprecated fields deleted** The following deprecated fields are deleted: * `properties.silence_timeout` * `properties.advanced_features.enable_aivad` * `properties.llm.silence_message` ### v2.4 [#v24] Released on February 2, 2026. #### New features [#new-features-5] Included in this release: * **MCP integration** Adds a [`llm.mcp_servers`](/en/api-reference/api-ref/conversational-ai/join#properties-llm-mcp-servers) field to the [Start a conversational AI agent](/en/api-reference/api-ref/conversational-ai/join) API to connect the LLM to an MCP (Model Context Protocol) server. Set [`advanced_features.enable_tools`](/en/api-reference/api-ref/conversational-ai/join#properties-advanced-features-enable-tools) to `true` to enable tool calls. This allows the agent to call tools provided by external services to extend functionality. * **Filler phrases** Adds a [`properties.filler_words`](/en/api-reference/api-ref/conversational-ai/join#properties-filler-words) field to insert pre-set or LLM-generated filler phrases during conversations to fill periods of silence while waiting for LLM output. This feature smooths dialogue flow, reduces user anxiety, and makes agent speech sound more natural. #### Improvements [#improvements-5] This release includes the following enhancements: * **Turn detection configuration optimization** The `turn_detection` parameter structure has been updated to provide more flexible conversation turn detection. The new structure uses a `mode` + `config` pattern with separate Start of Speech (SoS) and End of Speech (EoS) detection configurations. **Basic configuration** * `mode`: Conversation turn detection mode (currently supports `default`) * `config.speech_threshold`: Voice activity detection sensitivity **Start of Speech (SoS) detection** Detects when the user begins speaking. When SoS is detected while the agent is speaking, an interruption occurs. The [`config.start_of_speech`](/en/api-reference/api-ref/conversational-ai/join#properties-turn-detection-config-start-of-speech) parameter supports three detection modes: * **VAD mode** (`vad`): Triggered by voice activity detection. Supports configuration of interruption threshold, prefix padding, and ignore word list. * **Keyword mode (Beta)** (`keywords`): Based on keyword triggering. The agent begins conversation after detecting a specified keyword. * **Disabled mode** (`disabled`): Disables interruption. Supports append or ignore strategies. **End of Speech (EoS) detection** Detects when the user finishes speaking. When EoS is detected, the agent generates a response. The [`config.end_of_speech`](/en/api-reference/api-ref/conversational-ai/join#properties-turn-detection-config-end-of-speech) parameter supports two detection modes: * **VAD mode** (`vad`): Determines conversation end based on silence duration. * **Semantic mode** (`semantic`): Determines conversation end based on semantic understanding. Supports configurable maximum wait time. #### API changes [#api-changes-5] This release introduces the following changes to the RESTful API. * Changes to [**Start a conversational AI agent**](/en/api-reference/api-ref/conversational-ai/join) * **[`turn_detection`](/en/api-reference/api-ref/conversational-ai/join#properties-turn-detection) has a revamped structure** All previous fields under `turn_detection` have been deprecated and replaced with a new data structure. * Deprecated * `turn_detection.interrupt_mode` * `turn_detection.interrupt_keywords` * `turn_detection.interrupt_duration_ms` * `turn_detection.prefix_padding_ms` * `turn_detection.silence_duration_ms` * `turn_detection.threshold` * **AIVAD activation** The `advanced_features.enable_aivad` field is now deprecated; to configure AIVAD activation use [`turn_detection.config.end_of_speech.mode = "sematic"`](/en/api-reference/api-ref/conversational-ai/join#properties-turn-detection-config-end-of-speech-mode). * **New parameters added** * [`advanced_features.enable_tools`](/en/api-reference/api-ref/conversational-ai/join#properties-advanced-features-enable-tools) * [`llm.mcp_servers`](/en/api-reference/api-ref/conversational-ai/join#properties-llm-mcp-servers) * [`properties.filler_words`](/en/api-reference/api-ref/conversational-ai/join#properties-filler-words) * [`turn_detection.mode`](/en/api-reference/api-ref/conversational-ai/join#properties-turn-detection-mode) * [`turn_detection.config`](/en/api-reference/api-ref/conversational-ai/join#properties-turn-detection-config) ### v2.3 [#v23] Released on January 7, 2026. #### New features [#new-features-6] Included in this release: * **Control LLM response interruption in custom LLM scenarios** Adds a `metadata.interruptable` field in the first chunk of the `chat.completion.custom_metadata` object to control whether user speech can interrupt the agent's TTS output when using custom LLMs with streaming response (SSE). Use this to prevent interruptions when delivering critical information such as regulations, policies, or pricing. For more information, see [Configure LLM response interruption](../build/custom-model-integration/custom-llm#configure-llm-response-interruption). * **RTC media content encryption** Adds an `rtc` parameter to the [Start a conversational AI agent](/en/api-reference/api-ref/conversational-ai/join) API for configuring RTC encryption. * **New ASR provider** * [Sarvam (Beta)](../models/asr/sarvam) #### Improvements [#improvements-6] This release includes the following enhancements: * **ElevenLabs TTS compatibility Optimization** Supports SSML parsing and forwarding to ensure ElevenLabs TTS correctly renders intended speech effects such as pauses, emphasis, and pronunciation. * **Latency optimization best practices** Adds a guide to [Optimize conversation latency](../best-practices/optimize-latency). * **Cloud Recording best practices** Adds a guide to [Record conversations with Cloud Recording](../best-practices/record-agent-conversation). #### API changes [#api-changes-6] This release introduces the following changes to the RESTful API. * **Stop agent API is now asynchronous** The [Stop a conversational AI agent](/en/api-reference/api-ref/conversational-ai/leave) API now responds immediately after request parameters are validated. The request is processed asynchronously after the API returns. * Changes to [**Start a conversational AI agent**](/en/api-reference/api-ref/conversational-ai/join) * **New parameters added** * `properties.rtc` ### v2.2 [#v22] Released on December 15, 2025. #### New features [#new-features-7] Included in this release: * **Geofencing** Use `geofence` configuration when you [Start a conversational AI agent](/en/api-reference/api-ref/conversational-ai/join) to limit which Agora servers the Conversational AI Engine can access based on geographic regions. See [Restrict agent zones](../best-practices/regional-restrictions) for details. * **Agent greeting mode** Adds a field `llm.greeting_configs.mode` to the [Start a conversational AI agent](/en/api-reference/api-ref/conversational-ai/join) request to set the agent's greeting broadcast mode. The following modes are supported: * `single_every` (Default): The agent broadcasts a greeting every time a user joins a channel where there are no other users. * `single_first`: The agent broadcasts a greeting only when the first user joins a channel. * **New TTS providers** * [Sarvam (Beta)](../models/tts/sarvam) #### Improvements [#improvements-7] This release includes the following enhancements: * **TTS parameter update** Adds `base_url` field to the TTS parameters for [OpenAI](../models/tts/openai) and [ElevenLabs](../models/tts/elevenlabs). This field specifies the endpoint URL for the TTS service. #### API changes [#api-changes-7] This release introduces the following changes to the RESTful API. * Changes to [**Start a conversational AI agent**](/en/api-reference/api-ref/conversational-ai/join) * **New parameters added** * `properties.geofence` * `llm.greeting_configs.mode` ### v2.1 [#v21] Released on December 5, 2025. #### New features [#new-features-8] Included in this release: * **Template variables** This version adds the `llm.template_variables` field to the [Start a conversational AI agent](/en/api-reference/api-ref/conversational-ai/join) API, used to insert variables into the agent's `system_messages`, `greeting_message`, `failure_message`, and `parameters.silence_config.content` text. By configuring these variables, the Conversational AI engine automatically replaces them with the corresponding values defined in `llm.template_variables`. Template variables, combined with prompt customization and SIP outbound calling functionality, enable you to dynamically inject content to automate processes such as automatic hang-up, voicemail recognition, automatic message leaving, and call transfer. * **Custom labels** This version adds a `labels` field to the [Start a conversational AI agent](/en/api-reference/api-ref/conversational-ai/join) API, enabling agents to carry custom labels. These labels are bound to the agent and returned in the `payload` field of all message notification callbacks from the Conversational AI engine, allowing you to implement custom business logic, such as tagging activity IDs, customer groups, and business scenarios. In SIP outbound call scenarios, you can pass a custom label in the `properties.labels` field when calling the outbound call interface to mark the call. #### API changes [#api-changes-8] This release introduces the following changes to the RESTful API. * Changes to [**Start a conversational AI agent**](/en/api-reference/api-ref/conversational-ai/join) * **New parameters added** * `properties.labels` * `llm.template_variables` ### v2.0 [#v20] Released on November 15, 2025. #### New features [#new-features-9] Included in this release: * **Telephony (Beta)** This version adds an outbound calling feature that enables the conversational AI agent to initiate an outbound call to a specified number through a POST API. After the call is answered, the agent can engage in real-time dialogue with the callee. This feature supports a wide range of outbound AI call scenarios. A set of phone number management APIs is also added for handling numbers connected to Conversational AI Engine. Refer to the API changes section for details. Telephony pricing The telephony feature is currently in **Beta** and is provided free of charge. Pricing terms may change upon official release. * **Selective attention locking (Beta)** This version adds the selective attention lock feature. Register voiceprints to enable the agent to identify specific speakers and suppress background voices and environmental noise, ensuring clearer, more focused conversations. * **Graceful exit** This version adds a new `farewell_config` field to [Start a conversational AI agent](/en/api-reference/api-ref/conversational-ai/join) API to configure the graceful exit feature. When enabled, calling the [Stop a Conversational Agent](/en/api-reference/api-ref/conversational-ai/leave) API causes the agent to enter an `IDLE` state before leaving the channel. * **Keyword interruption mode** This release adds a new option to the `turn_detection.interrupt_mode` field in the [Start a conversational AI agent](/en/api-reference/api-ref/conversational-ai/join) API. Set this field to `"keyword"` to enable keyword interruption mode. When this mode is enabled, the agent stops its current behavior after detecting any of the keywords specified in the `turn_detection.interrupt_keywords` field. * **Adaptive interruption mode** This release adds a new option to the `turn_detection.interrupt_mode` field in the [Start a conversational AI agent](/en/api-reference/api-ref/conversational-ai/join) API. Set this field to `"adaptive"` to enable adaptive interruption mode. When this mode is enabled, the agent dynamically increases the voice continuity threshold while speaking to reduce accidental interruptions. * **New ASR, LLM, MLLM, and TTS providers** * ASR * [OpenAI (Beta)](../models/asr/openai) * [Speechmatics](../models/asr/speechmatics) * [Google (Beta)](../models/asr/google) * Amazon Transcribe (Beta) * [AssemblyAI (Beta)](../models/asr/assembly-ai) * LLM * [Groq](../models/llm/groq) * [Amazon Bedrock](../models/llm/amazon) * MLLM * [Google Gemini Live](../models/mllm/gemini) * TTS * [Rime (Beta)](../models/tts/rime) * [Fish Audio (Beta)](../models/tts/fish-audio) * [Google (Beta)](../models/tts/google) * [Amazon Polly (Beta)](../models/tts/amazon) * **New webhook notification events** This release adds three new webhook notification event types to support metrics reporting and call-state monitoring: * `111`: [agent metrics](event-types#111-agent-metrics) Notifies real-time performance metrics for each dialogue turn, including ASR, LLM, and TTS latency measurements. * `201`: [inbound call state](event-types#201-inbound-call-state) Reports state changes for incoming calls, such as when a call starts, is answered, transferred, or hung up. * `202`: [outbound call state](event-types#202-outbound-call-state) Reports state changes for outbound calls initiated by the agent, including call start, dialing, ringing, answer, and hang-up events. #### Improvements [#improvements-8] This release includes the following enhancements: * **Support for avatars with MLLMs** Added support for using avatars with MLLMs. #### API changes [#api-changes-9] This release introduces the following changes to the RESTful API. * Changes to [**Start a conversational AI agent**](/en/api-reference/api-ref/conversational-ai/join) * **New parameters added** * `properties.parameters.farewell_config` * `properties.advanced_features.enable_sal` * `properties.sal` * `properties.sal.sal_mode` * `properties.sal.sample_urls` * `properties.turn_detection.interrupt_mode` (supports `adaptive` and `keyword` values) * `properties.turn_detection.interrupt_keywords` * `properties.turn_detection.interrupt_duration_ms` (migrated from `vad.interrupt_duration_ms`) * `properties.turn_detection.prefix_padding_ms` (migrated from `vad.prefix_padding_ms`) * `properties.turn_detection.silence_duration_ms` (migrated from `vad.silence_duration_ms`) * `properties.turn_detection.threshold` (migrated from `vad.threshold`) * **Deprecated** * The `vad` interface is deprecated. All configuration items have been moved to the `turn_detection` field. * **New APIs** * Telephony (Beta) * Initiate an outbound call * Retrieve call records * Hang-up a call * Retrieve call status * Phone number management (Beta) * Retrieve list of numbers * Import number * Retrieve number information * Update number configuration * Delete number #### Toolkit API [#toolkit-api] This release renames all APIs and parameters containing the word `transcription` in the client-side subtitle API to use `transcript`, as shown below: #### Android [#android] * `onTranscriptionUpdated` renamed to `onTranscriptUpdated` * `TranscriptionRenderMode` renamed to `TranscriptRenderMode` * `TranscriptionType` renamed to `TranscriptType` * `TranscriptionStatus` renamed to `TranscriptStatus` * `Transcription` renamed to `Transcript` #### iOS [#ios] * `onTranscriptionUpdated` renamed to `onTranscriptUpdated` * `TranscriptionRenderMode` renamed to `TranscriptRenderMode` * `TranscriptionType` renamed to `TranscriptType` * `TranscriptionStatus` renamed to `TranscriptStatus` * `Transcription` renamed to `Transcript` #### Web [#web] * `TRANSCRIPTION_UPDATED` renamed to `TRANSCRIPT_UPDATED` ### v1.7 [#v17] Released on July 31, 2025. #### New features [#new-features-10] * **AI avatars** Create visual avatar representations for your conversational agents using third-party avatar providers. AI avatars provide a visual presence during voice interactions, making conversations feel more natural and engaging. Enable AI avatars by setting `avatar.enable` to `true` and configuring the `avatar.vendor` and `avatar.params` fields when calling [Start a conversational AI agent](/en/api-reference/api-ref/conversational-ai/join) to create your agent. Info AI avatars require video streaming and incur additional charges. See [video calling pricing](../../realtime-media/video/reference/pricing) for details. * **Selective attention locking (Beta)** This version introduces the selective attention locking feature, which uses voiceprint recognition technology to identify and filter out the speaker while suppressing background noise. This enhances the efficiency of conversational AI, particularly improving speech recognition accuracy. To experience this feature, contact [technical support](mailto\:support@agora.io). * **Send picture messages (Beta)** The toolkit now includes an API for [sending picture messages](../build/send-multimodal-messages). You can send image URLs to the main model, which automatically references the image in future interactions to generate more relevant responses. A new callback is available to receive image message receipt details after successful transmission. Info * The picture messaging feature is currently in Beta and free for a limited time. * Image processing depends on the capabilities of the integrated LLM. Ensure the LLM you connect to the Conversational AI Engine supports image input. #### API changes [#api-changes-10] This release introduces the following modifications to the RESTful API. * [Start a conversational AI agent](/en/api-reference/api-ref/conversational-ai/join) * **New parameters added** * `avatar.enable` * `avatar.vendor` * `avatar.params` ##### Toolkit API [#toolkit-api-1] * Android: * `chat` * `ImageMessage` * `onMessageReceiptUpdated` * `MessageReceipt` * iOS: * `chat` * `ChatMessage` * `ChatMessageType` * `ImageMessage` * `onMessageReceiptUpdated` * `MessageReceipt` * Web: * `chat` * `TMessageReceipt` * `EChatMessagePriority` * `EChatMessageType` * `IChatMessageBase` * `IChatMessageImage` ### v1.6 [#v16] Released on July 15, 2025. #### New features [#new-features-11] * **Support for OpenAI realtime API** Integrate Multimodal Large Language Models (MLLMs) with Conversational AI Engine to enable end-to-end real-time audio and text interactions. See [OpenAI Realtime API](../models/mllm/openai) for integration details. * **Support for more TTS vendors** Conversational AI Engine now supports the following additional TTS vendors: * [Cartesia](../models/tts/cartesia) * [OpenAI](../models/tts/openai) * **Custom ASR provider support** To improve flexibility in configuring conversational agents, this release allows you to select a custom automatic speech recognition (ASR) provider. The [Start a conversational AI agent](/en/api-reference/api-ref/conversational-ai/join) API now includes the following new parameters: * `asr.vendor`: Specify the ASR provider * `asr.params`: Configure ASR parameters The following ASR providers are supported: * **ARES** (default) * **Microsoft Azure** * **Deepgram** **Billing update:** In earlier versions, the service fee included the cost of the Ares ASR provider. Starting in v1.6, the pricing is restructured as follows: * If you use **ARES ASR**, the total price remains unchanged: ***Total cost = Conversational AI Engine Audio Basic Task + ARES ASR Task*** * If you use **a different ASR provider**, you are charged **only** the new **Conversational AI Engine Audio Basic Task** fee. For further details, see [Pricing](pricing). * **Multi-platform toolkit** Agora now offers a toolkit to help you quickly build conversational agent apps. The toolkit is available for [**iOS**](/en/api-reference/api-ref/conversational-ai/client-toolkit/ios), [**Android**](/en/api-reference/api-ref/conversational-ai/client-toolkit/android), and [**Web**](/en/api-reference/api-ref/conversational-ai/client-toolkit/web), and includes APIs for common scenarios. Call these APIs to combine the capabilities of the Agora Voice SDK and Signaling SDK to achieve the following functions: * [**Display live transcript**](../build/transcripts) Display real-time text output of user-agent conversations. The transcript component is now more robust, with better error handling, session management, and extensibility. * [**Interrupt the agent**](../build/shape-the-conversation/interrupt-agent) Stop the agent from speaking or thinking mid-conversation. * [**Client-side events**](../build/handle-runtime-events/event-notifications) Track changes in conversation state, performance metrics, and error events. * [**Optimize audio settings**](../best-practices/audio-setup) Quickly apply best-practice audio configurations to improve agent responsiveness and clarity. #### API changes [#api-changes-11] ##### REST API [#rest-api] This release introduces several important modifications to the RESTful API. * [Start a conversational AI agent](/en/api-reference/api-ref/conversational-ai/join) * **New parameters added** * `asr.vendor` * `asr.params` * `advanced_features.enable_mllm` * `properties.mllm` * `turn_detection.type` * `turn_detection.interrupt_duration_ms` * `turn_detection.prefix_padding_ms` * `turn_detection.silence_duration_ms` * `turn_detection.threshold` * `turn_detection.create_response` * `turn_detection.interrupt_response` * `turn_detection.eagerness` * `parameters.enable_metrics` * `parameters.data_channel` * `parameters.enable_error_message` ##### Toolkit APIs [#toolkit-apis] * [Android SDK](/en/api-reference/api-ref/conversational-ai/client-toolkit/android) * [iOS SDK](/en/api-reference/api-ref/conversational-ai/client-toolkit/ios) * [Web SDK](/en/api-reference/api-ref/conversational-ai/client-toolkit/web) ### v1.5 [#v15] Released on Jun 9, 2025. #### New features [#new-features-12] * **Voice interruption mode** This release adds the `turn_detection.interrupt_mode` parameter to the [Start a conversational AI agent](/en/api-reference/api-ref/conversational-ai/join) API, allowing you to control how the agent handles human voice interruptions. The following modes are supported: * **`interrupt`**: (Default) The human voice immediately interrupts the agent. The agent terminates the current interaction and processes the new human voice input. * **`append`**: The human voice does not interrupt the agent. The agent processes the newly received human voice request after the current interaction ends. * **`ignore`**: The agent ignores human voice requests received during speaking or thinking. These requests are discarded and not stored in the context. * **TTS filtering** This release adds the `tts.skip_patterns` parameter to the [Start a conversational AI agent](/en/api-reference/api-ref/conversational-ai/join) API. This parameter controls whether the TTS module skips bracketed content when reading LLM response text. This prevents the agent from vocalizing structural prompt information like tone indicators, action descriptions, and system prompts, creating a more natural and immersive listening experience. #### API changes [#api-changes-12] This release introduces several important modifications to the RESTful API. * [Start a conversational AI agent](/en/api-reference/api-ref/conversational-ai/join) * **New parameters added** * `turn_detection.interrupt_mode` * `parameters.silence_config` * `tts.skip_patterns` ### v1.4 [#v14] Released on May 29, 2025. #### New features [#new-features-13] * **Metadata support for LLM requests** This release adds the `llm.vendor` parameter to the [Start a conversational AI agent](/en/api-reference/api-ref/conversational-ai/join) API. When set to `"custom"`, the agent includes additional metadata when calling the LLM, such as `turn_id` and `timestamp`. * **Support for Anthropic** Conversational AI Engine now supports `anthropic` as a request style for chat completion. Refer to the `llm.style` parameter in [Start a conversational AI agent](/en/api-reference/api-ref/conversational-ai/join). #### Improvements [#improvements-9] This release includes the following enhancements: * **Advanced LLM configuration**: The [Update agent configuration](/en/api-reference/api-ref/conversational-ai/update) API now supports: * `llm.system_messages` for updating system prompts * `llm.params` for modifying configuration parameters used when calling the large language model #### API changes [#api-changes-13] This release introduces several important modifications to the RESTful API. * [Start a conversational AI agent](/en/api-reference/api-ref/conversational-ai/join) * **New parameters added** * `llm.vendor` * **Removed parameters:** * `agent_rtm_uid` * [Update agent configuration](/en/api-reference/api-ref/conversational-ai/update) * **New parameters added** * `llm.system_messages` * `llm.params` ### v1.3 [#v13] Released on April 16, 2025. #### New features [#new-features-14] * **Agent conversation history**: This version adds two methods to retrieve an agent's history. The history includes messages exchanged between the user and the agent and timestamps of agent creation and exit. * Call the RESTful API `history` endpoint to [Retrieve agent history](/en/api-reference/api-ref/conversational-ai/history). * Subscribe to the [agent history event](event-types#103-agent-history) through the [Agora message notification service](../build/handle-runtime-events/webhooks). When the agent stops, Agora automatically sends the agent's history to your business server through a Webhook callback. #### Improvements [#improvements-10] * **Customize the priority of broadcast information**: This version upgrades the [Broadcast a message using TTS](/en/api-reference/api-ref/conversational-ai/speak) interface and adds two new configuration parameters related to broadcast interruption logic: * `priority`: Sets the priority of the message broadcast. Supports setting the following priorities: * `INTERRUPT` High priority * `APPEND`: Medium priority * `IGNORE`: Low priority * `interruptable`: Configure whether to allow human voice to interrupt the agent's broadcast. #### API changes [#api-changes-14] * Adds the [Retrieve agent history](/en/api-reference/api-ref/conversational-ai/history) method. * Adds `priority` and `interruptable` parameters to the [Broadcast a message using TTS](/en/api-reference/api-ref/conversational-ai/speak) method. ### v1.2 [#v12] Released on April 10, 2025. #### New features [#new-features-15] * **Broadcast a message using TTS**: A new message broadcast interface enables a specified agent to deliver a custom message. When interacting with an agent, calling this interface interrupts the agent's speech and thinking process, allowing the TTS module to immediately broadcast the custom message. * **Interrupt the agent**: The interrupt agent endpoint allows you to stop the specified agent's speech and thinking process. #### API changes [#api-changes-15] This version adds the following APIs: * [Broadcast a message using TTS](/en/api-reference/api-ref/conversational-ai/speak) * [Interrupt the agent](/en/api-reference/api-ref/conversational-ai/interrupt) ### v1.1 [#v11] Released on March 27, 2025. #### New features [#new-features-16] The [Start a conversational agent](/en/api-reference/api-ref/conversational-ai/join) API adds the `enable_rtm` and `agent_rtm_uid` parameters to enable Signaling integration with the conversational AI agent. When this feature is enabled, the agent can leverage the Signaling SDK to obtain a user's custom context information such as speaking status, selected text, signature, and score, and pass this data to the agent to generate more relevant content. For details, see [Transmit custom information](../build/shape-the-conversation/custom-information). #### Improvements [#improvements-11] To help you quickly integrate a custom large language model (LLM), this version adds documentation for [Custom LLMs](../build/custom-model-integration/custom-llm). Refer to the sample code in the documentation to integrate your custom model into the Conversational AI Engine and enable advanced capabilities such as Retrieval-Augmented Generation (RAG), multi-modal processing, and tool invocation. #### API changes [#api-changes-16] The `POST` method to [Start a conversational AI agent](/en/api-reference/api-ref/conversational-ai/join) now includes the `enable_rtm` and `agent_rtm_uid` parameters. ### v1.0 (Public Beta) [#v10-public-beta] This version, released on March 4, 2025, adds pricing information for the Agora Conversational AI Engine. For more information, see [Pricing](pricing). #### Integration guide [#integration-guide] To achieve the best conversation experience, use Agora Conversational AI Engine with the following Agora SDKs: * Agora RTC Native SDK, v4.5.1 or later. * Agora RTC Web SDK, version 4.23.2 or later. #### New features [#new-features-17] * **Live transcripts**: Supports real-time text output of conversations between users and the AI agent for transcript display in your app's UI. Agora provides an open-source transcript processing module. Simply integrate the module and call its API to implement live transcript. For details, see [Display live transcripts](../build/transcripts). * **Message Notification Service**: Introduces a new Conversational AI Engine message notification service. Configure it in the Agora console and subscribe to agent creation, stop, and error events. When a subscribed event occurs, Agora sends the details to your specified callback address. See [Receive event notifications](../build/handle-runtime-events/webhooks). * **Keywords**: Enhances recognition accuracy of Conversational AI Engine for proprietary words by adding keywords. This feature is currently in Beta stage. For details, [contact technical support](https://agoraio.zendesk.com/hc/en-us). ### v1.0 (Private Beta) [#v10-private-beta] Released on February 18, 2025. The first beta release of the Conversational AI Engine brings natural, smooth, low-latency, and highly reliable real-time voice conversations with AI agents to Agora channels. It enables you to efficiently build intelligent and immersive interactive experiences. See [Product overview](/en/ai) for details. #### Core Features [#core-features] * **Real-time voice conversation** Supports natural and smooth real-time voice conversations with AI. It delivers a low-latency, ultra-responsive interactive experience as if the user is communicating with a real person. * **Intelligent noise suppression** Intelligently identifies and suppresses background noise, ensuring clear sound transmission even in noisy environments to provide users with a high-quality audio experience. * **Background human voice suppression** Suppresses background voices and noise while accurately preserving the primary speaker's voice. This ensures a clear and focused interactive experience in multi-speaker environments. * **Intelligent interruption handling** Allows users to interrupt AI at any time to ensure quick and natural responses. This feature enables smooth transitions and avoids mechanical interactions. * **Intelligent transmission** An AI-optimized transmission algorithm ensures stable voice data delivery even in weak network conditions where packet loss reaches 80%. This guarantees conversation continuity and reliability across diverse network environments. * **Flexible arrangement** Supports multiple Large Language Model (LLM) and Text-to-Speech (TTS) providers, enabling flexible orchestration to meet diverse business needs and deliver highly customizable AI dialogue solutions. * **Multi-platform support** Compatible with iOS, Android, Web, and various embedded hardware platforms, providing a seamless and consistent cross-platform experience. #### Integration guide [#integration-guide-1] * For the best conversational experience, Agora recommends using Conversational AI Engine with specific Agora Video/Voice SDK versions. For details, [contact technical support](mailto\:support@agora.io). * The number of Peak Concurrent Users (PCU) allowed to call the server API under a single App ID is limited to 20. If you need to increase this limit, please [contact technical support](mailto\:support@agora.io). # TEN Framework core concepts (/en/ai/ten-agent/core-concepts) ## TEN runtime [#ten-runtime] The TEN runtime is an execution environment designed for running applications within the TEN Framework. It provides a flexible architecture that supports extensions developed in multiple programming languages and handles various types of data streams. By managing the lifecycle of extensions, data flows, and threads, the TEN runtime offers developers a powerful tool for building a wide range of applications and services. ![TEN Framework](https://assets-docs.agora.io/images/ten-framework/ten-architecture.png) ## App [#app] An app in the TEN Framework can operate as a standalone process or as a thread within an existing process. This flexibility allows for versatile deployment options depending on the needs of the application. ## Extension group [#extension-group] An extension group is a mechanism that designates a specific thread for execution. All extensions within a given group are executed on the same thread, ensuring consistent and synchronized processing. ## Extension [#extension] An extension is the fundamental building block of the TEN Framework. Developers can create extensions in various programming languages and combine them to build different applications and scenarios. The framework's design supports cross-language collaboration, enabling extensions written in different languages to work together seamlessly within the same application or service. For example, a developer might use C++ to create an extension for real-time communication (RTC) due to its performance benefits in handling audio and video data, while developing an AI extension in Python to take advantage of its robust libraries for data analysis and machine learning. These extensions can then be integrated into a single application, leveraging the strengths of each language. ## Graph [#graph] A graph in the TEN Framework describes the data flow between extensions. It orchestrates how data moves from one extension to another, defining the participants and the flow of data between them. For example, you can route the output of a speech-to-text (STT) extension to a large language model (LLM) extension for further processing. The TEN Framework supports four main types of data flows between extensions: * Command * Data * Video frame * Audio frame By defining these data flows within a graph, you can create inter-extension communication and unidirectional data streams, particularly useful for handling audio and video data. ## Component hierarchy [#component-hierarchy] The TEN Framework uses a nested structure where apps contain graphs, graphs contain extension groups, and extension groups contain extensions. Understanding this hierarchy is essential for building and configuring TEN applications. ![Hierarchical relationship of concepts](https://assets-docs.agora.io/images/ten-framework/hierarchical_relationship_of_concepts.png) * **App**: An app can execute multiple graphs, which can be either statically predefined or dynamically assembled. * **Graph**: A graph is formed by multiple extensions working together to create a meaningful scenario. Each graph instance operates as a session within the app. * **Extension group**: The concept of an extension group is analogous to a thread. Extensions written in the same language and within the same extension group run on the same thread during runtime. You do not need to manage threads directly; simply declare the group to which each extension belongs. * **Extension**: Each extension within the framework is assigned a unique ID, structured as: `app-uri/graph-name/group-name/extension-name` ## TEN cloud store [#ten-cloud-store] The TEN cloud store functions similarly to Google Play Store or Apple's App Store, providing a marketplace for extensions. You can share your own extensions or download those created by others. Integrate these extensions into TEN apps to facilitate development and expand functionality. ![TEN cloud store](https://assets-docs.agora.io/images/ten-framework/ten_cloud_store.png) ## TEN manager [#ten-manager] The TEN manager is a tool that simplifies the management of extensions. It handles tasks such as uploading, sharing, and installing extensions, automatically managing dependencies between them and their environment. This makes the installation and publication of extensions convenient and efficient, and streamlines the development process within the TEN Framework. # TEN Framework overview (/en/ai/ten-agent/framework-overview) The TEN Framework, or Transformative Extensions Network, is an open-source framework for building real-time multimodal AI agents. It helps you create applications that process voice, video, data streams, images, and text at the same time. TEN is designed for agents that need to think, listen, see, and respond in real time. It provides a graph-based runtime, reusable extensions, and multi-language development support so you can assemble complex AI systems without rebuilding every capability from scratch. ## What you can build [#what-you-can-build] Use the TEN Framework to build real-time AI applications such as: * Voice and video AI agents * Simultaneous interpretation systems * Speech-to-text and text-to-speech pipelines * Multilingual communication experiences * Audio-visual assistants and virtual companions * AI-generated meeting minutes * Language tutoring and coaching applications ## Key capabilities [#key-capabilities] ### Real-time multimodal interactions [#real-time-multimodal-interactions] TEN optimizes interaction between extensions to support low-latency AI applications. A graph can process multiple data types, including audio, video, commands, structured data, images, and text. ### Multi-language and cross-platform development [#multi-language-and-cross-platform-development] You can build modular extensions in C++, Go, and Python. TEN applications can run across Windows, macOS, Linux, and mobile environments, which gives you flexibility when choosing where each capability should execute. ### Modular extension architecture [#modular-extension-architecture] TEN applications are composed from extensions. Each extension owns a focused capability, such as audio capture, ASR, LLM processing, TTS, tool calling, or output rendering. This modular architecture makes it easier to prototype, replace providers, and add new capabilities over time. ### Edge-cloud integration [#edge-cloud-integration] TEN supports applications that combine edge and cloud components. You can run latency-sensitive or privacy-sensitive work near the user while using cloud-based models for heavier reasoning or generation tasks. ### Flexible orchestration beyond model limits [#flexible-orchestration-beyond-model-limits] TEN lets you combine AI models with databases, retrieval systems, monitoring tools, and external services. Instead of depending on one model to handle the whole workflow, you can orchestrate multiple specialized components through the graph. ### Real-time agent state management [#real-time-agent-state-management] TEN graphs can manage changing agent state so applications can adapt behavior during a session. This is useful for interactive agents that need to respond to user context, media state, or tool results while the conversation is still running. ## How it relates to TEN Agent [#how-it-relates-to-ten-agent] TEN Agent is built on the TEN Framework. The framework provides the runtime, graph model, extension system, schema system, and package mechanics. TEN Agent applies those framework capabilities to conversational AI workflows. Start with [Project overview](./project-overview) when you want to understand the TEN Agent repository layout, runtime configuration, extension folders, and local server API. For implementation details, see: * [Subgraphs](./architecture/subgraphs) * [Schema system](./architecture/schema-system) * [Type system](./architecture/type-system) * [Development workflow](./develop/development-workflow) ## Samples [#samples] Explore TEN Framework examples in the [TEN Framework repository](https://github.com/TEN-framework/ten-framework/tree/main/ai_agents/agents/examples). # Project overview (/en/ai/ten-agent/project-overview) The TEN Agent project is built on the TEN Framework and follows a modular architecture. This page describes the project structure, configuration system, and web server API. TEN Agent uses a graph-based configuration system built on TEN Framework concepts. The project organizes code into modular extensions that can be orchestrated through configuration files without modifying source code. The folder structure follows this pattern: ```text ├── agents/ │ └── ten_packages/ │ ├── extension/ │ │ ├── openai_chatgpt_python/ │ │ │ ├── extension.py │ │ │ ├── .. │ │ │ └── requirements.txt │ │ ├── elevenlabs_tts │ │ └── .. │ └── property.json ├── playground ├── demo └── server ``` It contains the following important folders and files: * `property.json`: This file contains the orchestration of extensions. It is the main runtime configuration file. * `ten_packages/extension`: This folder contains the extension modules. Each extension module is a separate Python/Golang/C++ package. * `server`: This folder contains the web server code. It is responsible for handling the incoming requests and start/stop of agent processes. * `playground`: This folder contains the UI code for the playground. It is a web-based interface to interact with the agent. ## Runtime configuration [#runtime-configuration] The main runtime configuration file is `property.json`. It contains the orchestration of extensions. The file is structured as follows: ```json "predefined_graphs": [ { "name": "va_openai_azure_fashionai", "auto_start": false, "connections": [ // ... ], "nodes": [ // ... ] }, // ... ] ``` The file contains the following orchestration info: * **Graphs**: Collections of nodes and connections that determine agent behavior * **Nodes**: Individual extension instances with specific configurations * **Connections**: Data flow paths between nodes ```json { "predefined_graphs": [ { "name": "va_openai_azure_fashionai", "auto_start": false, "connections": [...], "nodes": [...] } ] } ``` ### Graphs [#graphs] The `predefined_graphs` property contains a list of available graphs. Each graph defines how the agent behaves in a specific scenario. Each graph contains: * `name`: Unique identifier for the graph * `auto_start`: Whether to start this graph automatically * `nodes`: List of extension instances * `connections`: Data routing between nodes ### Nodes [#nodes] The `nodes` section contains the list of extensions that are part of the graph. Each node includes the following: * `name`: Unique identifier of a node within the graph * `addon`: Specifies the extension module to use. You can create multiple instances of the same extension within a graph. For example, you might include several `chatgpt_openai_python` nodes, each using the same `addon` property but with unique `name` properties. This allows you to run parallel instances of an extension with different configurations or behaviors. * `property`: Extension-specific configuration ```json { "name": "chatgpt_openai_python", "addon": "chatgpt_openai_python", "property": { "api_key": "${env:OPENAI_API_KEY|}", "model": "gpt-3.5-turbo", "temperature": 0.5, "max_tokens": 100, "prompt": "You are a helpful assistant" } } ``` #### Node properties [#node-properties] The `property` section of a node configures the extension's behavior. Each extension defines its available properties in the `manifest.json` file within its folder. You can customize an extension by setting these runtime properties. #### Read environment variables [#read-environment-variables] Many extensions require API keys to function. Instead of hardcoding sensitive keys in `property.json`, use environment variables with the syntax `${env:|}`. The following example shows how to read and use the `OPENAI_API_KEY` environment variable: ```json { "name": "chatgpt_openai_python", "addon": "chatgpt_openai_python", "property": { "api_key": "${env:OPENAI_API_KEY|}" } } ``` ### Connections [#connections] The connections section defines how data flows between nodes. Each connection specifies: * **Source node**: Identified by `extension_group` and `extension` properties * **Data protocols**: Types of data that can flow: `audio_frame`, `video_frame`, `data`, or `cmd` * **Destinations**: Where each data type is routed For each protocol, you define destinations with: * `name`: The data property key * `dest`: List of target nodes The following example connects `agora_rtc` extension to `deepgram_asr` extension. The `agora_rtc` extension sends `pcm_frame` data to `deepgram_asr` extension. ```json { "extension_group": "default", "extension": "agora_rtc", "audio_frame": [ { "name": "pcm_frame", "dest": [ { "extension_group": "default", "extension": "deepgram_asr" } ] } ] } ``` ## Extension folder [#extension-folder] The `ten_packages/extension` folder contains extension modules written in Python, Go, or C++. Each extension is a self-contained package with its own structure. Extension folder names typically match the module name, though the definitive module name is specified in the extension's `manifest.json` file. Use this module name as the `addon` property value in your configuration. The extension folder is structured as follows: ```text project_root/ ├── ten_packages/ │ └── extension/ │ ├── sample_python_extension/ │ │ ├── ... │ │ ├── extension.py │ │ ├── manifest.json │ │ └── property.json │ ├── sample_go_extension/ │ │ ├── ... │ │ ├── example_extension.go │ │ ├── manifest.json │ │ └── property.json │ └── sample_cpp_extension/ │ ├── ... │ ├── src/ │ │ ├── main.cc │ │ └── ... │ ├── BUILD.gn │ ├── manifest.json │ └── property.json ├── manifest.json └── property.json ``` ### Extension common files [#extension-common-files] * `manifest.json`: Defines the extension's metadata including name, version, available properties, and supported APIs such as `data`, `audio_frame`, and `video_frame` * `property.json`: Specifies default configuration values for the extension ### Extension-specific files [#extension-specific-files] Each language has its own set of required files for extension development: | Language | Main logic file | Dependencies/Build file | | -------- | ---------------------- | ------------------------------------------------------------------------------------------------------------------------------------ | | Python | `extension.py` | `requirements.txt`: Python dependencies required by the extension. Dependencies are installed automatically when you run `task use`. | | Go | `example_extension.go` | `go.mod`: Go module definition that specifies the module name and dependencies. | | C++ | `src/main.cc` | `BUILD.gn`: Build configuration that specifies the target name and dependencies. | The main logic file contains the core implementation of the extension for each language. ## Web server [#web-server] The Go-based web server manages agent processes and handles HTTP requests from clients. Both the playground and demo interfaces communicate with this server, though you can also interact directly using curl or any HTTP client. ### API endpoints [#api-endpoints] The server exposes three endpoints for agent management: * [POST /start](#post-start): Start an agent with specified configuration * [POST /stop](#post-stop): Stop a running agent * [POST /ping](#post-ping): Maintain agent connection #### POST /start [#post-start] Starts an agent with a specified graph and property overrides. The agent joins the specified RTC channel and subscribes to the user's audio stream. | Parameter | Description | | -------------- | --------------------------------------------------------------------------------------------------------------------- | | `request_id` | UUID for request tracking | | `channel_name` | RTC channel name (must match the channel your client joins) | | `user_uid` | User's RTC identifier for audio subscription | | `bot_uid` | Optional agent RTC identifier | | `graph_name` | Graph configuration name from `property.json` | | `properties` | Property overrides (temporary, doesn't modify `property.json`) | | `timeout` | Inactivity timeout in seconds (`-1` disables timeout, default: 60s, configurable using `WORKER_QUIT_TIMEOUT_SECONDS`) | Example: ```bash curl 'http://localhost:8080/start' \ -H 'Content-Type: application/json' \ --data-raw '{ "request_id": "c1912182-924c-4d15-a8bb-85063343077c", "channel_name": "test", "user_uid": 176573, "graph_name": "camera_va_openai_azure", "properties": { "openai_chatgpt": { "model": "gpt-4o" } } }' ``` #### POST /stop [#post-stop] Stops a running agent instance. | Parameter | Description | | -------------- | ----------------------------------------- | | `request_id` | UUID for request tracking | | `channel_name` | Channel name used when starting the agent | Example: ```bash curl 'http://localhost:8080/stop' \ -H 'Content-Type: application/json' \ --data-raw '{ "request_id": "c1912182-924c-4d15-a8bb-85063343077c", "channel_name": "test" }' ``` #### POST /ping [#post-ping] Maintains the agent connection by sending a keepalive signal. Not required if the agent was started with `timeout: -1`. Without pings, agents terminate after the configured timeout period. | Param | Description | | -------------- | -------------------------------- | | `request_id` | UUID for request tracking | | `channel_name` | Channel name of the active agent | Example: ```bash curl 'http://localhost:8080/ping' \ -H 'Content-Type: application/json' \ --data-raw '{ "request_id": "c1912182-924c-4d15-a8bb-85063343077c", "channel_name": "test" }' ``` # Studio overview (/en/ai/studio) Agent Studio is a visual workspace in the Agora Console for building and operating conversational AI voice agents. You configure agent behavior, connect telephony, and monitor production calls from a single interface, without writing integration code. Studio abstracts the underlying AI stack. You configure your ASR, LLM, and TTS providers; Studio handles the orchestration between them. Instead of managing API integrations across multiple services, you focus on what your agent should say and do. ## How Studio is organized [#how-studio-is-organized] Studio is structured around three areas that map to the agent lifecycle: **Build:** Create and configure agents. Start from a use-case template or a blank template, then define the system prompt, greeting, voice, and behavior. Connect credentials, knowledge bases, and MCP servers through the **Integration** interface. Studio stores these resources centrally so they can be reused across multiple agents. See [Customize your agent](build/customize-agent) and [Manage integrations](build/integrations). **Deploy:** Connect agents to telephony. Assign phone numbers using Elastic SIP Trunk and configure inbound routing or outbound campaigns. Studio supports providers such as Twilio, Exotel, and Telnyx. See [Publish your agent](deploy/deploy-agent), [Phone Numbers](deploy/import), [Handle inbound calls](deploy/inbound), and [Campaigns](deploy/campaign). **Observe:** Monitor production usage. Review call history, transcripts, and analytics to understand agent performance and debug issues. See [Analytics](observe/analytics) and [Call History](observe/call-history). ## How it works [#how-it-works] Studio sits between your telephony provider and the AI models that power your agent. For inbound calls, Studio receives the call from your carrier via Elastic SIP Trunk, processes the conversation through your configured ASR, LLM, and TTS providers, and delivers the agent's response back to the caller. For outbound calls, Studio initiates the call through the same SIP trunk, connects to the recipient, and handles the conversation in the same way. In both cases, Studio manages the full call lifecycle. No SIP servers to manage, no voice recognition to train, no telephony protocols to debug. ## Key concepts [#key-concepts] ### Agents [#agents] Agents are the primary objects in Studio. You create an agent from a template or from scratch, then configure it across the following areas: * **Live Audio Input**: Voice Activity Detection (VAD), turn detection, and interruption mode * **LLM**: The provider and the prompt that defines conversational behavior * **ASR**: The speech recognition provider and language settings * **TTS**: The voice synthesis provider and voice parameters After configuring, test the agent in the built-in playground and activate it when ready. ### Integration [#integration] The **Integration** section provides a centralized resource library for your Studio workspace. Store API credentials, knowledge bases, and MCP server connections here once, then attach them to any agent. This eliminates repeated setup when you create new agents or update provider settings. ### Phone numbers [#phone-numbers] Import phone numbers from your SIP trunk provider and assign them to agents for inbound calls or use them as caller IDs for outbound campaigns. ### Campaigns [#campaigns] For outbound use cases, create campaigns with contact lists and scheduling. See [Set up a campaign](deploy/campaign). ## Who should use Studio [#who-should-use-studio] Studio is designed for developers and technical users who want to build and operate voice agents without managing the underlying AI infrastructure. If you need capabilities beyond what Studio exposes, such as advanced API parameters or custom orchestration logic, you can use the [Conversational AI REST API](/en/api-reference/api-ref/conversational-ai) for full programmatic control. ## Get started [#get-started] To get started, follow the [Quickstart](quickstart), which walks you through creating and testing your first agent in about 10 minutes using a pre-built template. # Quickstart (/en/ai/studio/quickstart) This page guides you through creating and testing your first AI voice agent using Agent Studio. In less than 10 minutes, you will have a working agent that you can speak to in the browser. No telephony setup is required. ## Prerequisites [#prerequisites] Make sure you have the following: * An active Agora account and project * [Enabled Conversational AI](../reference/enable-conversational-ai) for your Agora project ## Create an agent [#create-an-agent] To create your first agent: 1. Log into [Agora Console](https://console.agora.io) and open [Agent Studio](https://console.agora.io/studio). ![](https://assets-docs.agora.io/images/conversational-ai/studio/create-agent.png) 2. Select **Agents** from the sidebar, then click **Create Agent**. 3. Select an Agora project from the dropdown. The agent uses this project's credentials to connect to Agora, and usage is billed to this project. ![](https://assets-docs.agora.io/images/conversational-ai/studio/create-agent-template.png) 4. Under **Choose a template**, select the **Blank Template**. 5. Enter a name for your agent. 6. Click **Create Agent**. Studio opens the agent editor with a system prompt and pre-configured model settings. ![](https://assets-docs.agora.io/images/conversational-ai/studio/agent-editor-prompt.png) ## Configure your agent [#configure-your-agent] The agent editor is organized into tabs. For the quickstart, you only need to review the **Prompt** tab and the **Models** tab. ### Review the prompt [#review-the-prompt] The agent editor opens with the following prompts: 1. **System prompt**: It defines how your agent behaves. 2. **Greeting message**: It is the first thing your agent says when a session starts. Type in a greeting such as `Hello, how can I help you today?` 3. **Failure message**: It is used when the LLM encounters an error or fails to respond. ### Review selected models [#review-selected-models] Models determine which ASR, LLM, and TTS services your agent uses to process speech, generate responses, and synthesize voice output. 1. Click the **Models** tab. ![](https://assets-docs.agora.io/images/conversational-ai/studio/agent-editor-models.png) 2. Review the pre-selected models for **Automatic Speech Recognition (ASR)**, **Large Language Model (LLM)**, and **Text-to-Speech (TTS)**. You can use the defaults or select a different vendor-model from the dropdowns. The dropdowns list vendor-model combinations that support Agora Managed Key. When you use a managed key, Agora provides the API credentials for your selected vendor. You do not need to obtain an API key directly from the vendor. See [Pricing](../reference/pricing) for details. To use your own API key or use a vendor that does not support a managed key, see [Customize your agent](build/customize-agent). ## Test the agent [#test-the-agent] 1. In the right panel, click **Start Call** to start a test call. 2. Allow microphone access when prompted. 3. Speak to the agent to verify it responds correctly. For guidance on what to test and how to troubleshoot issues, see [Test your agent](build/test-agent). ## Next steps [#next-steps] You now have a working voice agent. From here you can: * [Customize your agent](build/customize-agent): Fully configure models, prompts, and advanced settings for your use case * [Manage integrations](build/integrations): Manage API credentials, knowledge bases, and MCP servers for reuse across agents # Account settlement (/en/introduction/billing/account-settlement) To ensure billing transparency and smooth service continuity, Agora implements real-time account balance reservation based on estimated usage across all Agora products for SSP customers. This does not apply to customers with a signed contract. For uninterrupted access to Agora's products and services, ensure at least one of the following: * Add a valid credit card for auto-recharge to your Agora account, **or** * Maintain sufficient funds in your Agora Console balance to cover your estimated usage. Agora employs the following policies to manage your account balance and ensure uninterrupted access to its products and services. ### Real-time usage and estimated bill [#real-time-usage-and-estimated-bill] You can view your real-time usage and estimated monthly bill at any time directly in the Agora Console. ![bill estimate preview](https://assets-docs.agora.io/images/console/bill-estimate-preview.png) ### Real-time reservation of balance [#real-time-reservation-of-balance] Your reserved balance is updated continuously based on your real-time bill estimation. The corresponding estimated amount is reserved from your available balance at all times. ![Available and reserved balance](https://assets-docs.agora.io/images/console/available-reserved-balance.png) ### Auto-recharge and account suspension [#auto-recharge-and-account-suspension] On the **7th, 14th, 21st, and 28th** of each calendar month (excluding February 28), if your available balance is in arrears by more than **$50**: * If a valid credit card is linked, the system automatically recharges your balance to zero. * If no credit card is linked, or if auto-recharge fails, a notification email is sent to you. * Top up your wallet within the **24-hour grace period** to avoid account suspension. * If your balance remains in arrears after 24 hours, your account is suspended until the balance is restored to zero or above. ### Monthly billing [#monthly-billing] Your final bill is issued monthly. After bill finalization: * Any overpaid amount is refunded to your balance. * Any undercharged amount is deducted accordingly. ### Balance withdrawal [#balance-withdrawal] You can withdraw your available balance for payments completed within the past 90 days directly through the Agora Console. Funds will be returned to the original payment method. For further inquiries, please email [billing@agora.io](mailto\:billing@agora.io). ![Withdraw balance](https://assets-docs.agora.io/images/console/withdraw-balance.png) # Billing policies (/en/introduction/billing/billing-policies) This page explains billing, account settlement, end-of-life policies, and any applicable free-of-charge policies for this product. ## Billing and account policies [#billing-and-account-policies] Agora Console provides billing information, fee deduction details, and account suspension notices based on your account type. If you have signed a contract with Agora, the contract terms override all billing, deduction, and suspension details described on this page. ### Paid accounts [#paid-accounts] Your account is a paid account if you have registered with Agora and completed any of the following: * Added a credit card to your account or topped up your balance using a bank account. * Made a recent payment. * Signed a contract with Agora. ### Billing cycle [#billing-cycle] Agora provides each account with [10,000 free minutes](#free-of-charge-policy) every month. On the first day of each month, Agora issues your bill for the previous calendar month. To view billing information for your projects: 1. In [Agora Console](https://console.agora.io/v2), click home. 2. Click **Billing**. You see the detailed billing information for your projects, including billing period, due date, and amount. ![View bills](https://assets-docs.agora.io/images/video-sdk/bills.png) #### Additional charges [#additional-charges] This section describes the additional charges applicable to your account. #### Singapore goods and services tax [#singapore-goods-and-services-tax] As of November 2021, Agora Singapore charges [9% Singapore Goods and Services Tax (GST)](https://www.iras.gov.sg/taxes/goods-services-tax-\(gst\)/basics-of-gst/goods-and-services-tax-\(gst\)-what-it-is-and-how-it-works) on invoices for all Agora services provided to accounts located in Singapore. Agora determines the account location based on the tax identification number, contact address, or billing address that you have provided. All GST collected from Singapore accounts is paid to the Singapore tax authority. If you have any questions, contact [support@agora.io](mailto\:support@agora.io). #### Fee deduction [#fee-deduction] On the sixth day of each month, Agora automatically deducts the fee for the previous month and notifies you by email. No fee is deducted if you meet both of the following conditions: * Your monthly usage does not exceed the free quota. * You do not use any other charged Agora services or products. If your account balance is negative after the deduction, Agora sends you an email, reminding you to top up your account at your earliest convenience and avoid account suspension. #### Account suspension [#account-suspension] If your account balance remains negative for 5 days after the deduction date, Agora suspends your account and notifies you by email. During suspension, none of your projects can access Agora services. To restore access, top up your account as soon as possible. Once your balance is zero or greater, Agora unfreezes your account. ### Free accounts [#free-accounts] Your account is a free account if you have registered with Agora and have not completed any of the following: * Added a credit card to your account or topped up your balance using a bank account. * Made a recent payment. * Signed a contract with Agora. Free accounts include 10,000 free minutes per month. For details, see [Free-of-charge policy](#free-of-charge-policy). #### Free account suspension [#free-account-suspension] Agora suspends your account on the second day after any of the following occur: * Your total usage exceeds the free quota of a service or product. * You use Agora services or products not included in the free quota. After account suspension, none of your projects can access Agora services. To restore access: 1. Add a credit card to your account or top up your balance using a bank account. This unfreezes your account and upgrades it to a paid account. 2. Purchase a [pre-paid monthly package](/en/introduction/billing/subscription-packages) or [top-up package](/en/introduction/billing/subscription-packages#top-up-package). This option is highly recommended. ## Free-of-charge policy [#free-of-charge-policy] Agora offers a new pricing model. If your account uses this model, you receive a monthly free usage package by default instead of the 10,000 free minutes described below. Agora provides each account with 10,000 free minutes per month and deducts them in the following order: * Broadcast Streaming audio minutes * Voice call, video call, and Interactive Live Streaming audio minutes * On-premise recording audio minutes * Cloud recording audio non-transcoding minutes * Cloud recording audio minutes * Web page recording audio minutes * Cloud Proxy audio minutes * Broadcast Streaming HD video minutes * Voice call, video call, and Interactive Live Streaming HD video minutes * On-premise recording HD video minutes * Cloud recording HD video minutes * Web page recording HD video minutes * Cloud Proxy HD video minutes * Broadcast Streaming Full HD\* video minutes * Voice call, video call, and Interactive Live Streaming Full HD\* video minutes * On-premise recording Full HD\* video minutes * Cloud recording Full HD\* video minutes * Web page recording Full HD\* video minutes * Cloud Proxy Full HD\* video minutes * Broadcast Streaming 2K video minutes * Voice call, video call, and Interactive Live Streaming 2K video minutes * On-premise recording 2K video minutes * Cloud recording 2K video minutes * Broadcast Streaming 2K+ video minutes * Voice call, video call, Interactive Live Streaming 2K+ video minutes * On-premise recording 2K+ video minutes * Cloud recording 2K+ video minutes * Cloud proxy 2K+ video minutes \* HD+ minutes are also included in the shared 10,000 free minutes. If your total service minutes do not exceed 10,000 minutes, the service is free-of-charge. After the 10,000 free-of-charge minutes are fully deducted, Agora charges you for the additional service minutes. Agora clears any remaining free-of-charge minutes at the end of each calendar month. The 10,000 free-of-charge minutes policy does not apply to the Signaling SDK, the Chat SDK, or the IoT SDK. ### How service minutes are calculated [#how-service-minutes-are-calculated] Service minutes are calculated either by the number of users or by the number of streams. Agora calculates service minutes **by the number of users**. #### Approach 1: calculate by the number of users [#approach-1-calculate-by-the-number-of-users] Suppose N users talk for M minutes in a channel, the total service minutes = N \* M. * If two users talk for 10 minutes, the total service minutes are: 2 \* 10 = 20. * If five users talk for 10 minutes, the total service minutes are: 5 \* 10 = 50. * If 10 users talk for 10 minutes, the total service minutes are: 10 \* 10 = 100. In this approach, service minutes depend only on the number of users in the channel, regardless of how many streams each user subscribes to. #### Approach 2: calculate by the number of streams [#approach-2-calculate-by-the-number-of-streams] Suppose N users talk for M minutes in a channel, and each user subscribes to all remote streams in the channel, the total service minutes = N \* (N-1) \* M. * If two users talk for 10 minutes, the total service minutes are: 2 \* (2-1) \* 10 = 20. * If five users talk for 10 minutes, the total service minutes are: 5 \* (5-1) \* 10 = 200. * If 10 users talk for 10 minutes, the total service minutes are: 10 \* (10-1) \* 10 = 900. In this approach, every remote stream that a user subscribes to is counted separately. #### Comparison of calculation approaches [#comparison-of-calculation-approaches] The following table compares service minutes under each approach: | Use-case | Service minutes by the number of users | Service minutes by the number of streams | | ------------------------------- | -------------------------------------- | ---------------------------------------- | | Two users talk for 10 minutes. | 20 minutes | 20 minutes | | Five users talk for 10 minutes. | 50 minutes | 200 minutes | | 10 users talk for 10 minutes. | 100 minutes | 900 minutes | The difference between the two approaches increases as more users join the channel. ### Agora's calculation method [#agoras-calculation-method] Agora calculates service minutes **by the number of users**. Service minutes are also calculated based on aggregate video resolution. ## End-of-life policy [#end-of-life-policy] Agora is committed to providing regular updates to core products, extensions, and tools. These updates include new features, updated APIs, bug fixes, security patches, and documentation improvements. Agora strongly encourages developers to update to the latest product releases to benefit from new features, security enhancements, and other improvements. This section outlines the stages that an Agora product or service moves through, from pre-GA Beta to general availability to retirement, and the support Agora provides during each phase. ### Default introduction period [#default-introduction-period] Agora guarantees a minimum introduction period of 12 months from the release of a product on the Agora Developer Center. During this period, Agora does not initiate any end-of-support or end-of-life actions. After the 12-month period, Agora may transition a product into the Maintenance phase, which marks the beginning of the end-of-support phase. ### SDK lifecycle [#sdk-lifecycle] | Phase | Description | | ------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Beta | During this phase, SDKs are intended solely for early access and feedback purposes. They are not recommended for use in production environments. Beta products fall outside of the EOL policy and are intended for evaluation purposes only. For more information, see the Beta Service Agreement. | | General Availability (GA) | SDKs in this phase are fully supported by Agora. Agora provides support for new services, API updates, feature enhancements, bug fixes, and security patches. Agora guarantees a minimum of 12 months of support for GA SDKs. For details on client SDK support periods, see [Client SDK support periods](#client-sdk-support-periods). | | Retirement (EOL) SDKs: (Maintenance) | When an SDK enters the Retirement phase, Agora announces its retirement with a minimum notice period of 60 days, which is approximately 2 months. This announcement will include crucial timelines and guidelines to assist developers in migrating to the latest recommended SDK version. Agora communicates retirement announcements through email, the Agora Console notifications center, and SDK documentation, and may also post announcements on social media or the [Agora blog](https://www.agora.io/en/blog/). When a product or service reaches the end-of-support or end-of-life stage, Agora stops providing security updates, non-security updates, and assisted support. | ### Client SDK support periods [#client-sdk-support-periods] Agora provides the following support periods for client SDKs: | Release | Support | | ------------- | ------------------------------------------------------------------------------------------ | | Major release | Agora guarantees support for at least 12 months starting from the date of a major release. | | Minor release | For minor releases, Agora ensures support for at least 6 months from the release date. | ### Maintenance support for retired SDKs [#maintenance-support-for-retired-sdks] Occasionally, Agora may transition a product SDK into a maintenance support phase focused primarily on bug fixes and security patches. In such cases, Agora notifies customers at least 60 days, approximately 2 months, in advance of the retirement period for the affected SDK. During the Retirement phase, products entering the EOS/EOL phase will continue to be supported as follows: | Duration | Support | | ---------------------------------- | ------------------------------------------------------------------------------------------------------------------------------- | | Months 1–6 after EOL announcement | Agora provides bug fixes and security updates. New feature requests are not accepted. | | Months 7–12 after EOL announcement | Agora provides security updates only. | | After 12 months | Agora no longer supports the product or service, unless a support extension was arranged before the end of the 12-month period. | Using an SDK beyond its maintenance support cycle is not recommended and is entirely at the developer's discretion and risk. Agora is committed to delivering the best SDK experience to its developers. This policy ensures that you have access to the latest features and security updates with clear guidance on the support timelines. ### Support extensions [#support-extensions] Customers may apply to extend support for a retired SDK for a temporary period, subject to Agora's discretion and commercial terms. Agora reviews each request and decides whether to grant the customer a license to use the product or service for a limited period beyond the EOL timeline. # Billing (/en/introduction/billing) This is the single place to understand how Agora charges for usage and how to manage payment. For product-specific rates, see the **Pricing** page of each product. ## Billing center [#billing-center] In Agora Console, the **Billing** page lets you check your balance, add funds, and view or export invoices and transactions. Access is limited to members assigned to **Admin**, **Finance**, or an authorized custom team. ## How usage is billed [#how-usage-is-billed] Most Agora products bill by usage (for example, by minutes, message volume, storage, or peak concurrent users), with a free tier before charges begin. The exact unit and rate depend on the product, so check the product's own **Pricing** page for current numbers. Before you ship to production, confirm: * the billing unit makes sense for your expected usage pattern * the right people on your team can access billing and financial records ## Billing policies and management [#billing-policies-and-management] * [Billing policies](/en/introduction/billing/billing-policies) — free-usage allowance and product lifecycle rules. * [Account settlement](/en/introduction/billing/account-settlement) — balance reservation, auto-recharge, and suspension. * [Subscription packages](/en/introduction/billing/subscription-packages) — package tiers and upgrades. # Subscription packages (/en/introduction/billing/subscription-packages) A subscription package is a prepaid billing method. You can purchase a package in the [`Agora Console`](https://console.agora.io/) to cover usage for the current month. This page explains how to purchase, and manage subscription and top-up packages. ### Purchase and upgrade [#purchase-and-upgrade] By default, a new account may be assigned a Free package when the first eligible project is created. You can upgrade to a paid package at any time. Available package tiers, discounts, and unit prices vary by product. To upgrade: * Log in to [Agora Console](https://console.agora.io/). * In the sidebar, click **Subscriptions**. * Under **All Subscriptions**, select the product subscription you want to manage, such as RTC, Signaling, Chat, or Agora Analytics. * Select your package and click **Subscribe** or **Upgrade**. The following screenshot shows an RTC subscription package as an example: ![RTC subscription package example](https://assets-docs.agora.io/images/video-sdk/prepaid-packages.png) - For non-contracted customers, most packages can be purchased directly from the Agora Console. Some enterprise or contracted packages may require contacting [Agora sales](mailto\:sales@agora.io). - You can upgrade sequentially or skip levels when the product supports it. - Package upgrades take effect immediately and apply to the entire calendar month. If you make multiple upgrades on the same day, only the last upgrade is applied. - If your account has a negative balance, purchasing a prepaid package first charges your credit card for the outstanding balance, then for the package fee. #### Upgrade rules [#upgrade-rules] When you upgrade to a paid package, it becomes effective for the entire calendar month, regardless of the purchase date. You receive the full monthly allocation of the included quota. The cost of the previous package is deducted from the new package price, so you only pay the difference. If you request an upgrade after auto-renewal has triggered, you may upgrade for the next month as well. ### Renewal [#renewal] By default, your subscription package renews automatically 48 hours before it expires. Ensure your account has sufficient balance or enable auto-pay to avoid renewal failure. You can unsubscribe at any time to prevent renewal. See [Cancel subscription](#cancel-subscription). If renewal fails: * The system sends a notification and retries the renewal. * If renewal still fails by the time the package expires, the package downgrades to the Free package. * To restore the previous package, see [Purchase and Upgrade](#purchase-and-upgrade). ### Downgrade [#downgrade] You can downgrade by selecting and purchasing a lower-tier package. A downgrade takes effect when your current package expires if you request it before auto-renewal is triggered. You are charged for the new package when you request the downgrade. You can downgrade only once per month. * If your downgrade for the next month is successful, auto-renewal for the current package is not triggered. * If a package is already active, you must wait until the 1st day of that month to initiate a downgrade for the following month. #### Example [#example] * **Requested more than 48 hours before the current package expires**: The downgrade takes effect after the current package expires. * **Requested less than 48 hours before the current package expires**: Since a package is already active for the next month, you must wait until the new package starts to downgrade for the following month. ### Cancel subscription [#cancel-subscription] To cancel a prepaid package, downgrade to the Free package. Downgrade rules apply. ## Top-up package [#top-up-package] A top-up package is a pay-as-you-go plan. You can purchase a top-up package from the [`Agora Console`](https://console.agora.io/subscriptions/rtc-plans?tab=top-up) to cover usage that exceeds your monthly package quota. ### Purchase [#purchase] You can purchase multiple top-up packages. They are deducted in the order of purchase. #### Example: [#example-1] * Purchased 250,000-minutes top-up packages on **August 15** and **August 16**. On **September 1**, minutes consumed in August are deducted from the first package, then from the second. ### Validity [#validity] Each top-up package is valid for one year, starting from the purchase date to the end of the same month of the following year. #### Example: [#example-2] * Purchased on **August 15, 2025** → valid until **August 31, 2026** Unused minutes expire after the validity period and cannot be carried forward. ### Cancel top-up package [#cancel-top-up-package] Cancelling a top-up package is not supported. ## Free tier overage [#free-tier-overage] For Free tier users, exceeding the included minutes and any purchased top-up minutes results in service suspension. Service resumes when you purchase additional top-up minutes or upgrade to a prepaid subscription package. # Start building (/en/introduction/get-started/build-it-yourself) Most teams do not start by asking which Agora product to read first. They start by asking what kind of realtime problem they need to solve. Use this page to map your product goal to the most useful documentation path. ## When to use this path [#when-to-use-this-path] This is the right page when: * you already know the product problem you want to solve * you want to choose the right capability family before reading deep product docs * you do not need an AI-assisted setup workflow first * you want the shortest route from problem statement to relevant quickstart or product area If you want an AI agent to help choose the route and bootstrap a starter, use [Start with AI](/en/introduction/start-with-ai) instead. ## Start from the problem you are solving [#start-from-the-problem-you-are-solving] ### Voice-first AI interaction [#voice-first-ai-interaction] If users need to talk to an AI agent in real time, start with: * [Conversational AI](/en/introduction/conversational-ai) * [AI quickstart](/en/ai/get-started/quickstart) * [Voice agent quickstart](/en/ai/choose-your-path/quickstart-coding) This path is usually right for assistants, tutors, companions, service agents, and voice-enabled devices. ### Live voice, video, or interactive sessions [#live-voice-video-or-interactive-sessions] If users need to join the same session and interact through live media, start with: * [Real-Time Voice & Video](/en/introduction/realtime-audio-video) * [Audio & Video](/en/realtime-media/rtc) * [Voice & Video quickstart](/en/realtime-media/rtc/android/quick-start/build-from-scratch) This path is usually right for calling, meetings, classrooms, social rooms, telehealth, and interactive livestreaming. ### Room messaging, state sync, or workflow coordination [#room-messaging-state-sync-or-workflow-coordination] If the product depends on chat, presence, metadata, or event coordination, start with: * [Messaging & Presence](/en/introduction/messaging-presence) * [RTM / Signaling](/en/realtime-media/rtm) * [IM / Chat](/en/realtime-media/im) This path is usually right for in-room communication, control flows, collaborative state, and backend-triggered room actions. ### Cloud-side media workflows [#cloud-side-media-workflows] If the media must be captured, processed, routed, or redistributed beyond the live session itself, start with: * [Cloud Media Services](/en/introduction/cloud-media-services) * [Cloud Recording](/en/realtime-media/cloud-recording) * [Realtime Transcription & Translation](/en/realtime-media/speech-to-text) * [Transcoding](/en/realtime-media/transcoding) * [Server Media Processing](/en/realtime-media/rtc-server-sdk) This path is usually right for archive, replay, moderation, server-side automation, and downstream content pipelines. ## Common combinations [#common-combinations] Many production systems combine several of these paths: * voice or video sessions plus [Messaging & Presence](/en/introduction/messaging-presence) * AI interaction plus [Cloud Media Services](/en/introduction/cloud-media-services) * live sessions plus [Cloud Media Services](/en/introduction/cloud-media-services) * broadcast products plus [Cloud Media Services](/en/introduction/cloud-media-services) ## What to do next [#what-to-do-next] If you are still evaluating the platform, read [About Agora](/en/introduction/about-agora) first. If you want the shared mental model before choosing a path, read [Core Concepts](/en/introduction/core-concepts). If you want an AI-assisted setup workflow, switch to [Start with AI](/en/introduction/start-with-ai). # Service activation (/en/realtime-media/agora-analytics/activation) To activate Agora Analytics, follow these steps: 1. Contact your account executive, solutions architect, or email [support@agora.io](mailto\:support@agora.io) to learn about the available Agora Analytics plans. 2. The account team or your solutions architect will open a support ticket with [Technical support](mailto\:support@agora.io) to request activation. Make sure the ticket includes your Customer ID and the specific Agora Analytics plan you want activated. Activation typically takes up to 2 business days. # Agora Analytics overview (/en/realtime-media/agora-analytics/product-overview) Agora Analytics tracks and analyzes the usage, quality, and performance of real-time voice and video live streams and other Agora products. Designed as a companion to Agora's Voice Calling, Video Calling, Broadcast Streaming, Interactive Live Streaming, and Chat it helps locate quality issues, identify root causes, and resolve problems to enhance the end-user experience. Monitor sessions in real-time with Call Inspector or use RESTful APIs to analyze usage trends and performance. Extend Agora Analytics with the Datadog integration to bring Agora Analytics data into your Datadog account. Visualize and monitor metrics, set up custom alerts, and manage Agora data alongside other system metrics for a unified view of performance and quality. ## Start building [#start-building] ## Product Features [#product-features] Identify, analyze, and respond to performance quality issues with advanced search, detailed visibility into call and user metrics, and diagnostics. Provides periodic call usage and quality statistics. It is designed to help customers understand the usage and quality of calls in apps. Visualized data for multiple call metrics in real time. Sends alerts to you when abnormal metrics or events are detected in your Agora projects. Use RESTful APIs to retrieve call statistics and quality of experience metrics, use them in your own application or DataOps workflow. Easily embed Agora Analytics pages in internal web portals using a low-code approach. # Core concepts (/en/realtime-media/broadcast-streaming/core-concepts) RTC (Real-Time Communication) refers to real-time communication technology, which allows almost instant exchange of audio, video, and other data between the sender and the receiver. Agora SDKs provide real-time audio and video interaction services, with multi-platform and multi-device support. This includes high-definition video calls, voice-only calls, interactive live streaming, as well as one-on-one and multi-group chats. This guide introduces the key processes and concepts you need to know to use Video SDK. Agora relies on the following fundamental concepts to enable seamless real-time communication:
### Agora SDRTN® [#agora-sdrtn] Agora's core engagement services are powered by its Software-Defined Real-Time Network (SDRTN®), a global infrastructure accessible anytime, anywhere. Unlike traditional networks, Agora SDRTN® is not restricted by devices, phone numbers, or telecom coverage areas. With data centers in over 200 countries and regions, it ensures sub-second latency and high availability for real-time media. Agora SDRTN® enables live user engagement through real-time communication (RTC), offering: * Unmatched quality of service * High availability and accessibility * True scalability * Low cost ## Channel concepts [#channel-concepts] Agora uses channels to group users together, enabling seamless communication and interaction. Channels serve as the foundation for transmitting real-time data, whether audio, video, or signaling, and play a crucial role in connecting users and services. ### Channel [#channel] A channel organizes users into a group and is identified by a unique channel name. Users who connect to the same channel are able to communicate with each other. A channel is created when the first user joins and ceases to exist when the last user leaves. Channels are created by calling the methods for transmitting real-time data. Agora uses different channels to transmit different types of data: * A Video SDK channel is used for transmitting audio or video data. * A Signaling channel is used for transmitting messaging or signaling data. These channels are independent of each other. Additional services provided by Agora, such as Cloud Recording and Speech to Text, join the Video SDK channel to provide real-time recording, transmission acceleration, media playback, and content moderation. ### Channel profile [#channel-profile] The Video SDK applies different optimization methods according to the selected channel profile. Agora supports the following channel profiles: | Channel profile | Description | | --------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | **Communication** | This profile is suitable for one-on-one or group calls, where all users in the channel talk freely. | | **Live Broadcasting** | In a live streaming channel, users have two client roles: *host* and *audience*. The *host* sends and receives streams, while the *audience* only receives streams with the sending function disabled. | ### Stream [#stream] A stream is a sequence of digitally encoded, coherent signals that contain media data. Users in a channel [publish](#publish) local streams and [subscribe](#subscribe) to remote streams from other users. ### User role [#user-role] The user role defines whether a user in a channel has the permission to publish streams. There are two user roles: * **Host**: A user who can publish streams to a channel. * **Audience**: A User who can only subscribe to remote media streams. A user with this role cannot publish streams. ### Publish [#publish] Publishing is the act of sending a user’s audio or video data to the channel. Usually, the published stream is created by the audio data sampled from a microphone or the video data captured by a camera. You can also publish media streams from other sources, such as an online music file or the user’s screen. After successfully publishing a stream, the SDK uses it to send media data to other users in the channel. Users communicate with each other in real-time by publishing local streams and subscribing to remote streams. ### Subscribe [#subscribe] Subscribing is the act of receiving media streams published by remote users to the channel. A user receives audio and video data from other users by subscribing to one or more of their streams. You either directly play the subscribed streams or process incoming data for other purposes such as recording or capturing screenshots. ### User ID [#user-id] In Broadcast Streaming, the UID is an integer value that uniquely identifies a user within the context of a channel. When joining a channel, you have the option to either assign a specific UID to the user or pass `0` or `null` and allow Agora to automatically generate and assign a UID to the user. If two users attempt to join the same channel with the same UID, it can lead to unexpected behavior. The UID is used by Agora's services and components to identify and manage users within a channel. Ensure that UIDs are properly assigned to prevent conflicts. ### RTC connection [#rtc-connection] The connection between the SDK and the channel. When publishing or subscribing to multiple streams in multiple channels, a connection is used to specify the target channel. ## Credentials [#credentials] To ensure reliable access and secure communication, Agora uses credentials such as the App ID, App Certificate, and tokens to identify applications, authenticate their requests, and authorize their access on its platform. ### App ID [#app-id] The App ID is a unique key generated by Agora to identify each project and provide billing and other statistical data services. The App ID is critical for connecting users within your app. It is used to initialize the Agora Engine in your app, and as one of the required keys to create authentication tokens for secure communication. Retrieve the App ID for your project using the [Agora Console](https://console.agora.io/v2/project-management). App IDs are stored on the front-end client and do not provide access control. Projects using only an App ID allow any user with the App ID to join. For access control, especially in production environments, choose the **App ID + Token** mechanism for user authentication when creating a new project. Without authentication tokens, your environment is open to anyone with access to your App ID. ### App Certificate [#app-certificate] An App Certificate is a unique key generated by the Agora Console to secure projects through token authentication. It is required, along with the App ID, to generate a token that proves authorization between your systems and Agora's network. App Certificates are used to generate Broadcast Streaming authentication tokens. Store the App Certificate securely in your backend systems. If your App Certificate is compromised or to meet security compliance requirements, you can invalidate certificates and create new ones through the Agora Console. ### Tokens [#tokens] A token is a dynamic key generated using the App ID, App Certificate, user ID, and expiration timestamp. Tokens authenticate and secure access to Agora's services, ensuring only authorized users can join a channel and participate in real-time communication. Tokens are generated on your server and passed to the client for use in Broadcast Streaming. The token generation process involves digitally signing the App ID, App Certificate, user ID, and expiration timestamp using a specific algorithm, preventing tampering or forgery. During development and testing, use the Agora Console to generate temporary tokens. For production environments, implement a token server as part of your security infrastructure to control access to your channels. ### Agora Console [#agora-console] [Agora Console](https://console.agora.io/v2) is the main dashboard where you manage your Agora projects and services. Before you can use Agora's SDKs, you must first create a project in the Agora Console. See [Agora account management](manage-agora-account) for details. ![Create project in Agora Console](https://assets-docs.agora.io/images/common/create-project.svg) Agora Console provides an intuitive interface for developers to query and manage their Agora account. After registering an Agora account, you use the Agora Console to perform the following tasks: * Manage your account * Create and configure Agora projects and services * Get an App ID and the App certificate * Generate temporary tokens for development and testing * Manage members and roles * Check call quality and usage * Check bills and make payments * Access product resources See [Agora account management](manage-agora-account) for details on how to manage all aspects of your Agora account. Agora also provides RESTful APIs that you use to implement features such as creating a project and fetching usage numbers programmatically. ## Audio and video concepts [#audio-and-video-concepts] ### Audio and video interaction workflow [#audio-and-video-interaction-workflow] The following figure illustrates the workflow of using the Video SDK to implement basic audio and video interaction. ![orientation\_adaptive\_locked\_landscape](https://assets-docs.agora.io/images/common/basic-audio-and-video.svg) Agora relies on the following fundamental concepts to enable seamless real-time communication: ### Audio module [#audio-module] In audio interaction, the main functions of the audio module are as shown in the figure below: ![Audio module functions](https://assets-docs.agora.io/images/common/audio-module.svg) After you call `registerAudioFrameObserver`, you can obtain the raw audio data at the following observation points in the audio transmission process: 1. Obtain the raw audio data of ear monitoring through the `onEarMonitoringAudioFrame` callback. 2. Obtain the captured raw audio data through the `onRecordAudioFrame` callback. 3. Obtain the raw audio playback data of each individual stream through the `onPlaybackAudioFrameBeforeMixing` callback. 4. Obtain the raw audio playback data of all mixed streams through the `onPlaybackAudioFrame` callback. 5. Obtain the raw audio data after mixing the captured and playback audio through the `onMixedAudioFrame` callback. (5) `onMixedAudioFrame` = (2) `onRecordAudioFrame` + (4) `onPlaybackAudioFrame` ### Audio routing [#audio-routing] The audio output device used by the app when playing audio. Common audio routes include wired headphones, earpieces, speakers, Bluetooth headphones, and others. The APIs used by the audio module are as follows: * Enable local audio collection: `enableLocalAudio` * Set local playback device: `setPlaybackDevice` * Set up audio routing: `setDefaultAudioRouteToSpeakerphone` ### Video module [#video-module] The following diagram shows the main functions of the video module in video interaction: ![Video module functions](https://assets-docs.agora.io/images/common/video-module.svg) The figure shows the following observation points: 1. `POSITION_POST_CAPTURER_ORIGIN`. 2. `POSITION_POST_CAPTURER`, corresponds to the `onCaptureVideoFrame` callback. 3. `POSITION_PRE_ENCODER`, corresponds to the `onPreEncodeVideoFrame` callback. 4. `POSITION_PRE_RENDERER`, corresponds to the `onRenderVideoFrame` callback. The APIs used by the video module are as follows: * Enable local video collection: `enableLocalVideo` * Local preview: `setupLocalVideo` → `startPreview` * Video rendering shows: `setupRemoteVideo` # Broadcast Streaming overview (/en/realtime-media/broadcast-streaming) Agora's Broadcast Streaming API delivers low-latency, high-definition live video streaming at scale for one-to-many broadcast delivery. With support for cross-platform integration and adaptive streaming technologies, it ensures reliable, high-quality viewing experiences across a wide range of network conditions and devices. Reach global audiences with smooth, uninterrupted broadcasts for large-scale events such as concerts, sports, conferences, live shows, and other live production experiences. Enhance Agora's Video SDK with broadcast production capabilities such as recording, stream management, content moderation, and audience analytics, or leverage the Extensions Marketplace to enable AI-powered features like noise cancellation, video effects, and more. ## Start building [#start-building] ## Product Features [#product-features] Agora’s Software-Defined Real-Time Network (SDRTN®) supports video users in over 200 countries and regions. Consistent high-quality video from few to thousands of concurrent users, even under challenging network conditions. Publish multiple audio and video tracks to one or more channels from a single instance, with support for multi-channel capture cameras and microphones. Enable screen sharing or interactive whiteboards that allow users to draw, annotate, and share content from multiple devices simultaneously. Support for high quality audio with 3D spatial audio, AI noise suppression, and gain control to provide an immersive audio experience. Record video sessions in the cloud or on premises with control over the format, path of storage, and quality. # Agora account management (/en/realtime-media/broadcast-streaming/manage-agora-account) This page shows you how to sign up for an Agora account, create a new project, and get the app ID and app certificate to generate a temporary token. ## Get started with Agora [#get-started-with-agora] To join a Broadcast Streaming session, you need an Agora App ID. This section shows you how to set up an Agora account, create an Agora project and get the required information from [Agora Console](https://console.agora.io/v2). ### Sign up for an Agora account [#sign-up-for-an-agora-account] To use Agora products and services, create an Agora account with your email, phone number, or a third-party account. **Sign up with email** 1. Go to the [signup page](https://sso.agora.io/en/signup). 2. Fill in the required fields. 3. Carefully read the **Terms of Service**, **Privacy Policy**, and **Acceptable Use Policy**, and tick the checkbox. 4. Click **Continue**. 5. Enter your **verification code** and click **Confirm**. 6. Follow the on-screen instructions to provide your name, company name, and phone number, set a password, and click **Continue**. **Sign up with a third-party account** 1. On the [Agora Console](https://console.agora.io/v2) login page, select the third-party account you want to use. 2. Follow the on-screen instructions to complete verification. 3. Click **Create a new account**. 4. Carefully read the **Terms of Service**, **Privacy Policy**, and **Acceptable Use Policy**, and select the checkbox. 5. Click **Continue**. Once you sign up successfully, your account is automatically logged in. Follow the on-screen instructions to create your first project and test out real-time communications. For later visits, log in to [Agora Console](https://console.agora.io/v2) with your phone number, email address, or linked third-party account. ### Create an Agora project [#create-an-agora-project] To create an Agora project, do the following: 1. In [Agora Console](https://console.agora.io/v2), open the [Projects](https://console.agora.io/v2/project-management) page. 2. Click **Create New**. 3. Follow the on-screen instructions to enter a project name and use case, and check **Secured mode: APP ID + Token (Recommended)** as the authentication mechanism. ![configure\_project](https://assets-docs.agora.io/images/signaling/create_new_project.png) 4. Click **Submit**. You see the new project on the **Projects** page. ### Get the App ID [#get-the-app-id] Agora automatically assigns a unique identifier to each project, called an App ID. To copy this App ID, find your project on the [Projects](https://console.agora.io/v2/project-management) page in Agora Console, and click the copy icon in the **App ID** column. ![configure\_project](https://assets-docs.agora.io/images/signaling/app-id.png) ## Security and authentication [#security-and-authentication] Use the following features from your Agora account to implement security and authentication features in your apps. ### Get the App Certificate [#get-the-app-certificate] When generating an authentication token on your app server, you need an App Certificate, in addition to the App ID. To get an App Certificate, do the following: 1. On the [Projects](https://console.agora.io/v2/project-management) page, click the pencil icon to edit the project you want to use. ![Console project management page](https://assets-docs.agora.io/images/common/console-project-management-page.png) 2. Click the copy icon under **Primary Certificate**. ![Console primary certificate](https://assets-docs.agora.io/images/common/console-primary-certificate.png) ### Generate temporary tokens [#generate-temporary-tokens] To ensure communication security, best practice is to use tokens to authenticate the users who log in from your app. To generate a temporary RTC token for use in your Video SDK projects: 1. On the [Projects](https://console.agora.io/v2/project-management) page, click the pencil icon next to your project. 2. On the **Security** panel, click **Generate Temp Token**, enter a channel name in the pop-up box and click **Generate**. Copy the generated RTC token for use in your Broadcast Streaming projects. To generate a token for other Agora products: 1. In your browser, navigate to the [Agora token builder](https://agora-token-generator-demo.vercel.app/). 2. Choose the Agora product your user wants to log in to. Fill in **App ID** and **App Certificate** with the details of your project in Agora Console. 3. Customize the token for each user. The required fields are visible in the Agora token builder. 4. Click **Generate Token**. The token appears in Token Builder. 3. Copy the token and use it in your app. For more information on managing other aspects of your Agora account, see [Agora console overview](reference/console-overview). # Agora MCP (/en/realtime-media/broadcast-streaming/mcp) The Agora MCP server gives your AI assistant direct access to Agora's documentation, so it can look up APIs, SDK methods, and platform-specific details in real time. The Agora MCP server is included when you install Agora Skills. If you prefer to install only the MCP server, it is available at: ```text https://mcp.agora.io ``` ### Installation [#installation] Refer to the installation instructions for your coding assistant. Cursor Claude Codex Gemini CLI Manual installation Click the button below to install the MCP server in [Cursor](https://www.cursor.com/) Install Agora MCP Server in Cursor or add it manually with the following JSON: ```json { "mcpServers": { "agora-docs": { "url": "https://mcp.agora.io" } } } ``` * **Claude Code** Run the following command in your terminal to install the MCP server in [Claude Code](https://claude.com/product/claude-code): ```bash claude mcp add --transport http agora-docs https://mcp.agora.io ``` * **Claude Desktop** In **Settings**, select **Connectors** and then choose **Add custom connector**. Enter the following values and click **Add**: * **Name**: `agora-docs` * **Remote MCP server URL**: `https://mcp.agora.io` Run the following command in your terminal to install the server in [OpenAI Codex](https://openai.com/codex/): ```bash codex mcp add --url https://mcp.agora.io agora-docs ``` Run the following command in your terminal to install the server in [Gemini CLI](https://github.com/google-gemini/gemini-cli): ```bash gemini mcp add --transport http agora-docs https://mcp.agora.io ``` Add the server URL `https://mcp.agora.io` to your MCP client of choice. If prompted, set the transport to `http` or "Streamable HTTP". ### Getting started [#getting-started] Once installed, your coding assistant has access to Agora's documentation through the MCP server. The assistant will intelligently use this resource when relevant to your questions. For more targeted results, mention Agora along with your target product and platform, such as 'iOS', 'Web', 'Conversational AI', 'Video Calling' in your prompts. ### System prompt [#system-prompt] This MCP works with all LLMs that support MCP, but performs best when the assistant understands facet-based exploration. Add the following prompt to your LLMs custom instructions: **System prompt for LLMs** ```markdown # Agora MCP Markdown - System Prompt You have access to Agora's documentation search via three tools: - `algolia_search_index_docs_platform_aware_markdown` - Full-text search with facets - `algolia_search_for_facet_values` - Browse products/platforms - `algolia_recommendations` - Find related documentation ## Key Behaviors **1. Use facets for discovery** - When users ask about "what's available", explore facets first - Example: User asks "what video products exist?" → Use `algolia_search_for_facet_values(facetName: "product", facetQuery: "video")` → Shows video-calling is the main product **2. Platform-aware searching** - Include target platform in search queries (iOS, Android, Web, etc.) - Results are intelligently ranked by platform relevance **3. Interpret facet results for refinement** - Search results include facet breakdowns by product and platform - Use these to suggest filtering: "15 results are for video-calling, 8 for voice-calling" **4. Maintain session continuity** - Use same sessionId (UUID) across related searches in a conversation - Improves ranking for follow-up queries ## Available Products & Platforms **Key Products:** conversational-ai, video-calling, broadcast-streaming, interactive-live-streaming, agora-chat, voice-calling, signaling, extensions-marketplace, interactive-whiteboard **Key Platforms:** android, web, ios, windows, macos, unity, flutter, react-native, electron, unreal ## Workflow Example User: "I'm building an iOS app with Video Calling. Where do I start?" 1. Search: "iOS video calling quickstart" 2. Results show facets indicating video-calling is the best match 3. Suggest: "Most results are for Video Calling, here's the quickstart" 4. Follow-up searches: Keep same sessionId for context ``` #### What This Enables [#what-this-enables] * Intelligent facet exploration before deep searches * Platform-aware result refinement suggestions * Cross-search session continuity for better context Without the prompt, searches still work but won't leverage faceted browsing patterns. ## AGENTS.md [#agentsmd] To get the most out of Agora Skills and the MCP server, Agora recommends that you include an [AGENTS.md](https://agents) or similar file in your repository. This file ensures that your agent always consults Agora docs to produce reliable, working code. Add the following instructions to your own `AGENTS.md` file: ```markdown ## Agora Documentation Agora documentation is updated frequently. You should always refer to the latest documentation when working with Agora products. Agora offers an MCP server that you can use to browse and search its documentation. If the developer has not yet installed this server, recommend that they install it at https://mcp.agora.io. If the Agora Skills files are available in your context, use them for platform-specific API guidance and code examples. ``` # Broadcast Streaming overview (/en/realtime-media/broadcast-streaming/product-overview) Agora's Broadcast Streaming API delivers low-latency, high-definition live video streaming at scale for one-to-many broadcast delivery. With support for cross-platform integration and adaptive streaming technologies, it ensures reliable, high-quality viewing experiences across a wide range of network conditions and devices. Reach global audiences with smooth, uninterrupted broadcasts for large-scale events such as concerts, sports, conferences, live shows, and other live production experiences. Enhance Agora's Video SDK with broadcast production capabilities such as recording, stream management, content moderation, and audience analytics, or leverage the Extensions Marketplace to enable AI-powered features like noise cancellation, video effects, and more. ## Start building [#start-building] ## Product Features [#product-features] * **Global coverage** - Agora’s Software-Defined Real-Time Network (SDRTN®) supports video users in over 200 countries and regions. * **High-quality video at scale** - Consistent high-quality video from few to thousands of concurrent users, even under challenging network conditions. * **Multiple audio and video tracks** - Publish multiple audio and video tracks to one or more channels from a single instance, with support for multi-channel capture cameras and microphones. * **Screen sharing and collaboration** - Enable screen sharing or interactive whiteboards that allow users to draw, annotate, and share content from multiple devices simultaneously. * **AI-powered audio enhancement** - Support for high quality audio with 3D spatial audio, AI noise suppression, and gain control to provide an immersive audio experience. * **Recording** - Record video sessions in the cloud or on premises with control over the format, path of storage, and quality. # Quickstart (/en/realtime-media/broadcast-streaming/quickstart) <_PlatformTabsGroup groupMode="structured" canonicalPlatform="web" platforms="["android","ios","macos","web","windows","electron","flutter","react-native","javascript","unity","unreal","blueprint","python"]" showTabs="true"> <_PlatformPanel platform="android"> <_PlatformProcessedMarker groupMode="structured" canonicalPlatform="web" platform="android" /> This page provides a step-by-step guide on how to create a basic Broadcast Streaming app using the Agora Video SDK. ## Understand the tech [#understand-the-tech] To start a Broadcast Streaming session, implement the following steps in your app: * **Initialize the Agora Engine**: Before calling other APIs, create and initialize an Agora Engine instance. * **Join a channel**: Call methods to create and join a channel. * **Join as a host**: A live streaming event has one or more hosts. A host publishes audio and video to the channel. Hosts can also subscribe to streams from other hosts. * **Join as audience**: Audience members can only subscribe to streams published by hosts. * **Send and receive audio and video**: Hosts publish streams to the channel. Audience members subscribe to audio and video streams published by hosts. ![Streaming workflow](https://assets-docs.agora.io/images/video-sdk/get-started-ils-bs.svg) ## Prerequisites [#prerequisites] * [Android Studio](https://developer.android.com/studio) 4.2 or higher. * Android SDK API Level 21 or higher. * Two mobile devices running Android 5.0 or higher. * A camera and a microphone * A valid Agora account and project. Please refer to [Agora account management](manage-agora-account.md) for details. ## Set up your project [#set-up-your-project] This section shows you how to set up your Android project and install the Agora Video SDK. **Create a new project** 1. Create a [new project](https://developer.android.com/studio/projects/create-project). 1. Open Android Studio and select **File > New > New Project...**. 2. Select **Phone and Tablet** > **Empty Activity** and click **Next**. 3. Set the project name and storage path. 4. Select **Java** or **Kotlin** as the language, and click **Finish** to create the Android project. After you create a project, Android Studio automatically starts gradle sync. Ensure that the synchronization is successful before proceeding to the next step. **Add to an existing project** 1. Add a new activity to your project. 1. Open your project in Android Studio. 2. Right-click on the `app/src/main/java/` folder. 3. Select **New → Activity → Empty Activity**. 4. Enter an activity name and click **Finish**. This guide uses `MainActivity` as the activity name in the sample code. Replace it with your activity name where required. 2. Add a layout file for your activity. Set up two container elements in your activity to display local and remote video streams. Refer to [Create a user interface](#create-a-user-interface) to get a bare bones sample layout. ### Install the SDK [#install-the-sdk] Use either of the following methods to add Video SDK to your project. **Maven Central** 1. Open the `settings.gradle` file in the project's root directory and add the Maven Central dependency, if it doesn't already exist: ```text repositories { mavenCentral() } ``` If your Android project uses dependencyResolutionManagement, the method of adding the Maven Central dependency may differ. 2. To integrate the Video SDK into your Android project, add the following to the `dependencies` block in your project module `build.gradle` file: * Groovy `build.gradle` ```json implementation 'io.agora.rtc:full-sdk:x.y.z' ``` * Kotlin `build.gradle.kts` ```kotlin implementation("io.agora.rtc:full-sdk:x.y.z") ``` Replace `x.y.z` with the specific SDK version number, such as `4.5.0`. To get the latest version number, check the [Release notes](reference/release-notes). To integrate the Lite SDK, use `io.agora.rtc:lite-sdk` instead. 3. Prevent code obfuscation Open the `/app/proguard-rules.pro` file and add the following lines to prevent the Video SDK code from being obfuscated: ```java -keep class io.agora.** { *; } -dontwarn io.agora.** ``` **Manual integration** 1. Download the latest version of Video SDK from the [SDKs](/en/api-reference/sdks?product=video\&platform=android) page and unzip it. 2. Open the unzipped file and copy the following files or subfolders to your project path. | File or folder | Project path | | :----------------------------------- | :----------------------- | | `agora-rtc-sdk.jar` file | `/app/libs/` | | `arm64-v8a` folder | `/app/src/main/jniLibs/` | | `armeabi-v7a` folder | `/app/src/main/jniLibs/` | | `x86` folder | `/app/src/main/jniLibs/` | | `x86_64` folder | `/app/src/main/jniLibs/` | | `high_level_api` in `include` folder | `/app/src/main/jniLibs/` | 3. Select the file `/app/libs/agora-rtc-sdk.jar` in the left navigation bar of Android Studio project files, right-click, and select **add as a library** from the drop-down menu. 4. Prevent code obfuscation Open the `/app/proguard-rules.pro` file and add the following lines to prevent the Video SDK code from being obfuscated: ```java -keep class io.agora.** { *; } -dontwarn io.agora.** ``` ## Implement Broadcast Streaming [#implement-broadcast-streaming] This section guides you through the implementation of basic real-time audio and video interaction in your app. The following figure illustrates the essential steps: ![Quick start sequence](https://assets-docs.agora.io/images/video-sdk/quick-start-sequence.svg) This guide includes [complete sample code](#complete-sample-code) that demonstrates implementing basic real-time interaction. To understand the core API calls in the sample code, review the following implementation steps and use the code in your `MainActivity` file. ### Import Agora classes [#import-agora-classes] Import the relevant Agora classes and interfaces: Java Kotlin ```java import io.agora.rtc2.Constants; import io.agora.rtc2.IRtcEngineEventHandler; import io.agora.rtc2.RtcEngine; import io.agora.rtc2.RtcEngineConfig; import io.agora.rtc2.video.VideoCanvas; import io.agora.rtc2.ChannelMediaOptions; ``` ```kotlin import io.agora.rtc2.Constants import io.agora.rtc2.IRtcEngineEventHandler import io.agora.rtc2.RtcEngine import io.agora.rtc2.RtcEngineConfig import io.agora.rtc2.video.VideoCanvas import io.agora.rtc2.ChannelMediaOptions ``` ### Initialize the engine [#initialize-the-engine] For real-time communication, initialize an `RtcEngine` instance and set up event handlers to manage user interactions within the channel. Use `RtcEngineConfig` to specify the application context, [App ID](manage-agora-account.md), and custom [event handler](#subscribe-to--events), then call `RtcEngine.create(config)` to initialize the engine, enabling further channel operations. In your `MainActivity` file, add the following code: Java Kotlin ```java // Fill in the app ID from Agora Console private String myAppId = ""; private RtcEngine mRtcEngine; private void initializeAgoraVideoSDK() { try { RtcEngineConfig config = new RtcEngineConfig(); config.mContext = getBaseContext(); config.mAppId = myAppId; config.mEventHandler = mRtcEventHandler; mRtcEngine = RtcEngine.create(config); } catch (Exception e) { throw new RuntimeException("Error initializing RTC engine: " + e.getMessage()); } } ``` ```kotlin // Fill in the App ID obtained from the Agora Console private val myAppId = "" private var mRtcEngine: RtcEngine? = null private fun initializeRtcEngine() { try { val config = RtcEngineConfig().apply { mContext = applicationContext mAppId = myAppId mEventHandler = mRtcEventHandler } mRtcEngine = RtcEngine.create(config) } catch (e: Exception) { throw RuntimeException("Error initializing RTC engine: ${e.message}") } } ``` ### Join a channel [#join-a-channel] To join a channel, call `joinChannel` with the following parameters: * **Channel name**: The name of the channel to join. Clients that pass the same channel name join the same channel. If a channel with the specified name does not exist, it is created when the first user joins. * **Authentication token**: A dynamic key that authenticates a user when the client joins a channel. In a production environment, you obtain a token from a [token server](build/authenticate-users/deploy-token-server.mdx) in your security infrastructure. For the purpose of this guide [Generate a temporary token](manage-agora-account.md). * **User ID**: A 32-bit signed integer that identifies a user in the channel. You can specify a unique user ID for each user yourself. If you set the user ID to `0` when joining a channel, the SDK generates a random number for the user ID and returns the value in the `onJoinChannelSuccess` callback. * **Channel media options**: Configure `ChannelMediaOptions` to define publishing and subscription settings, optimize performance for your specific use-case, and set optional parameters. For Broadcast Streaming, set the `channelProfile` to `CHANNEL_PROFILE_LIVE_BROADCASTING`, the `clientRoleType` to `CLIENT_ROLE_BROADCASTER` (host) or `CLIENT_ROLE_AUDIENCE`, and the `audienceLatencyLevel` to `AUDIENCE_LATENCY_LEVEL_LOW_LATENCY`. Java Kotlin ```java // Fill in the channel name private String channelName = ""; // Fill in the temporary token generated from Agora Console private String token = ""; private void joinChannel() { // Create an instance of ChannelMediaOptions and configure it ChannelMediaOptions options = new ChannelMediaOptions(); // Set the user role to BROADCASTER or AUDIENCE according to the use-case options.clientRoleType = Constants.CLIENT_ROLE_BROADCASTER; // In the broadcast streaming use-case, set the channelProfile to BROADCASTING options.channelProfile = Constants.CHANNEL_PROFILE_LIVE_BROADCASTING; // Set the latency level for audience options.audienceLatencyLevel = Constants.AUDIENCE_LATENCY_LEVEL_LOW_LATENCY; // Publish local media options.publishCameraTrack = true; options.publishMicrophoneTrack = true; mRtcEngine.joinChannel(token, channelName, 0, options); } ``` ```kotlin // Fill in the channel name private val channelName = "" // Fill in the temporary token generated from Agora Console private val token = "" private fun joinChannel() { // Create an instance of ChannelMediaOptions and configure it val options = ChannelMediaOptions().apply { // Set the user role to BROADCASTER or AUDIENCE according to the use-case clientRoleType = Constants.CLIENT_ROLE_BROADCASTER // In the broadcast streaming use-case, set the channelProfile to BROADCASTING channelProfile = Constants.CHANNEL_PROFILE_LIVE_BROADCASTING // Set the latency level for audience audienceLatencyLevel = Constants.AUDIENCE_LATENCY_LEVEL_LOW_LATENCY // Publish local media publishMicrophoneTrack = true publishCameraTrack = true } mRtcEngine?.joinChannel(token, channelName, 0, options) } ``` ### Subscribe to Video SDK events [#subscribe-to-video-sdk-events] The Video SDK provides an interface for subscribing to channel events. To use it, create an instance of `IRtcEngineEventHandler` and implement the event methods you want to handle. To ensure that you receive all Video SDK events, set the Agora Engine event handler before joining a channel. Java Kotlin ```java private final IRtcEngineEventHandler mRtcEventHandler = new IRtcEngineEventHandler() { // Triggered when the local user successfully joins the specified channel. @Override public void onJoinChannelSuccess(String channel, int uid, int elapsed) { super.onJoinChannelSuccess(channel, uid, elapsed); showToast("Joined channel " + channel); } // Triggered when a remote user/host joins the channel. @Override public void onUserJoined(int uid, int elapsed) { super.onUserJoined(uid, elapsed); runOnUiThread(() -> { // Initialize and display remote video view for the new user. setupRemoteVideo(uid); showToast("User joined: " + uid); }); } // Triggered when a remote user/host leaves the channel. @Override public void onUserOffline(int uid, int reason) { super.onUserOffline(uid, reason); runOnUiThread(() -> { showToast("User offline: " + uid); }); } }; ``` ```kotlin private val mRtcEventHandler = object : IRtcEngineEventHandler() { override fun onJoinChannelSuccess(channel: String?, uid: Int, elapsed: Int) { super.onJoinChannelSuccess(channel, uid, elapsed) runOnUiThread { showToast("Joined channel $channel") } } override fun onUserJoined(uid: Int, elapsed: Int) { runOnUiThread { showToast("User joined: $uid") } } override fun onUserOffline(uid: Int, reason: Int) { super.onUserOffline(uid, reason) runOnUiThread { showToast("User offline: $uid") } } } ``` ### Enable the video module [#enable-the-video-module] Follow these steps to enable the video module: 1. Call `enableVideo` to enable the video module. 2. Call `startPreview` to enable local video preview. Java Kotlin ```java private void enableVideo() { mRtcEngine.enableVideo(); mRtcEngine.startPreview(); } ``` ```kotlin private fun enableVideo() { mRtcEngine?.apply { enableVideo() startPreview() } } ``` ### Display the local video [#display-the-local-video] Call `setupLocalVideo` to initialize the local view and set the local video display properties. Java Kotlin ```java private void setupLocalVideo() { FrameLayout container = findViewById(R.id.local_video_view_container); SurfaceView surfaceView = new SurfaceView(getBaseContext()); container.addView(surfaceView); mRtcEngine.setupLocalVideo(new VideoCanvas(surfaceView, VideoCanvas.RENDER_MODE_FIT, 0)); } ``` ```kotlin /** * Initializes the local video view and sets the display properties. * This method adds a SurfaceView to the local video container and configures it. */ private fun setupLocalVideo() { val container: FrameLayout = findViewById(R.id.local_video_view_container) val surfaceView = SurfaceView(baseContext) container.addView(surfaceView) mRtcEngine.setupLocalVideo(VideoCanvas(surfaceView, VideoCanvas.RENDER_MODE_FIT, 0)) } ``` ### Display remote video [#display-remote-video] When a remote user joins the channel, call `setupRemoteVideo` and pass in the remote user's `uid`, obtained from the `onUserJoined` callback, to display the remote video. Java Kotlin ```java private void setupRemoteVideo(int uid) { FrameLayout container = findViewById(R.id.remote_video_view_container); SurfaceView surfaceView = new SurfaceView(getBaseContext()); surfaceView.setZOrderMediaOverlay(true); container.addView(surfaceView); mRtcEngine.setupRemoteVideo(new VideoCanvas(surfaceView, VideoCanvas.RENDER_MODE_FIT, uid)); } ``` ```kotlin private fun setupRemoteVideo(uid: Int) { val container = findViewById(R.id.remote_video_view_container) val surfaceView = SurfaceView(baseContext).apply { setZOrderMediaOverlay(true) } container.addView(surfaceView) mRtcEngine.setupRemoteVideo(VideoCanvas(surfaceView, VideoCanvas.RENDER_MODE_FIT, uid)) } ``` ### Handle permissions [#handle-permissions] To access the camera and microphone on Android devices, declare the necessary permissions in the app's manifest and ensure that the user grants these permissions when the app starts. 1. Open your project's `AndroidManifest.xml` file and add the following permissions before ``: ```xml ``` 2. Use the following code to handle runtime permissions in your Android app. The logic ensures that the necessary permissions are granted before starting Broadcast Streaming. In your `MainActivity` file, add the following code: Java Kotlin ```java private boolean checkPermissions() { for (String permission : getRequiredPermissions()) { if (ContextCompat.checkSelfPermission(this, permission) != PackageManager.PERMISSION_GRANTED) { return false; } } return true; } private String[] getRequiredPermissions() { if (android.os.Build.VERSION.SDK_INT >= android.os.Build.VERSION_CODES.S) { return new String[]{ Manifest.permission.RECORD_AUDIO, Manifest.permission.CAMERA, Manifest.permission.READ_PHONE_STATE, Manifest.permission.BLUETOOTH_CONNECT }; } else { return new String[]{ Manifest.permission.RECORD_AUDIO, Manifest.permission.CAMERA }; } } @Override public void onRequestPermissionsResult(int requestCode, @NonNull String[] permissions, @NonNull int[] grantResults) { if (requestCode == PERMISSION_REQ_ID && checkPermissions()) { startBroadcastStreaming(); } } ``` ```kotlin private val PERMISSION_REQ_ID = 22 private fun requestPermissions() { ActivityCompat.requestPermissions(this, getRequiredPermissions(), PERMISSION_REQ_ID) } private fun checkPermissions(): Boolean { for (permission in getRequiredPermissions()) { if (ContextCompat.checkSelfPermission(this, permission) != PackageManager.PERMISSION_GRANTED) { return false } } return true } private fun getRequiredPermissions(): Array { return if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.S) { arrayOf( Manifest.permission.RECORD_AUDIO, Manifest.permission.CAMERA, Manifest.permission.READ_PHONE_STATE, Manifest.permission.BLUETOOTH_CONNECT ) } else { arrayOf( Manifest.permission.RECORD_AUDIO, Manifest.permission.CAMERA ) } } override fun onRequestPermissionsResult( requestCode: Int, permissions: Array, grantResults: IntArray ) { super.onRequestPermissionsResult(requestCode, permissions, grantResults) if (requestCode == PERMISSION_REQ_ID && checkPermissions()) { startBroadcastStreaming() } } ``` ### Start and close the app [#start-and-close-the-app] When a user launches your app, start real-time interaction. When a user closes the app, stop the interaction. 1. In the `onCreate` callback, check whether the app has been granted the required permissions. If the permissions have not been granted, request the required permissions from the user. If permissions are granted, initialize `RtcEngine` and join a channel. Java Kotlin ```java @Override protected void onCreate(Bundle savedInstanceState) { super.onCreate(savedInstanceState); setContentView(R.layout.activity_main); if (checkPermissions()) { startBroadcastStreaming(); } else { requestPermissions(); } } ``` ```kotlin override fun onCreate(savedInstanceState: Bundle?) { super.onCreate(savedInstanceState) setContentView(R.layout.activity_main) if (checkPermissions()) { startBroadcastStreaming() } else { requestPermissions() } } ``` 2. When a user closes the app, or switches the app to the background, call `stopPreview` to stop the video preview and then call `leaveChannel` to leave the current channel and release all session-related resources. Java Kotlin ```java private void cleanupAgoraEngine() { if (mRtcEngine != null) { mRtcEngine.stopPreview(); mRtcEngine.leaveChannel(); mRtcEngine = null; } } ``` ```kotlin private fun cleanupAgoraEngine() { mRtcEngine?.apply { stopPreview() leaveChannel() } mRtcEngine = null } ``` ### Complete sample code [#complete-sample-code] A complete code sample demonstrating the basic process of real-time interaction is provided for your reference. To use the sample code, copy the following lines into the `MainActivity` file in your project. Then, replace `` in package `com.example.` with your project's name. Java Kotlin ```java package com.example. import android.Manifest; import android.content.pm.PackageManager; import android.os.Bundle; import android.view.SurfaceView; import android.widget.FrameLayout; import android.widget.Toast; import androidx.annotation.NonNull; import androidx.appcompat.app.AppCompatActivity; import androidx.core.app.ActivityCompat; import androidx.core.content.ContextCompat; import io.agora.rtc2.ChannelMediaOptions; import io.agora.rtc2.Constants; import io.agora.rtc2.IRtcEngineEventHandler; import io.agora.rtc2.RtcEngine; import io.agora.rtc2.RtcEngineConfig; import io.agora.rtc2.video.VideoCanvas; public class MainActivity extends AppCompatActivity { private static final int PERMISSION_REQ_ID = 22; // Fill in the app ID from Agora Console private String myAppId = ""; // Fill in the channel name private String channelName = ""; // Fill in the temporary token generated from Agora Console private String token = ""; private RtcEngine mRtcEngine; private final IRtcEngineEventHandler mRtcEventHandler = new IRtcEngineEventHandler() { // Callback when successfully joining the channel @Override public void onJoinChannelSuccess(String channel, int uid, int elapsed) { super.onJoinChannelSuccess(channel, uid, elapsed); showToast("Joined channel " + channel); } // Callback when a remote user or host joins the current channel @Override public void onUserJoined(int uid, int elapsed) { super.onUserJoined(uid, elapsed); runOnUiThread(() -> { // When a remote user joins the channel, display the remote video stream for the specified uid setupRemoteVideo(uid); showToast("User joined: " + uid); // Show toast for user joining }); } // Callback when a remote user or host leaves the current channel @Override public void onUserOffline(int uid, int reason) { super.onUserOffline(uid, reason); runOnUiThread(() -> { showToast("User offline: " + uid); // Show toast for user going offline }); } }; @Override protected void onCreate(Bundle savedInstanceState) { super.onCreate(savedInstanceState); setContentView(R.layout.activity_main); if (checkPermissions()) { startBroadcastStreaming(); } else { requestPermissions(); } } private void requestPermissions() { ActivityCompat.requestPermissions(this, getRequiredPermissions(), PERMISSION_REQ_ID); } private boolean checkPermissions() { for (String permission : getRequiredPermissions()) { if (ContextCompat.checkSelfPermission(this, permission) != PackageManager.PERMISSION_GRANTED) { return false; } } return true; } private String[] getRequiredPermissions() { if (android.os.Build.VERSION.SDK_INT >= android.os.Build.VERSION_CODES.S) { return new String[]{ Manifest.permission.RECORD_AUDIO, Manifest.permission.CAMERA, Manifest.permission.READ_PHONE_STATE, Manifest.permission.BLUETOOTH_CONNECT }; } else { return new String[]{ Manifest.permission.RECORD_AUDIO, Manifest.permission.CAMERA }; } } @Override public void onRequestPermissionsResult(int requestCode, @NonNull String[] permissions, @NonNull int[] grantResults) { super.onRequestPermissionsResult(requestCode, permissions, grantResults); if (requestCode == PERMISSION_REQ_ID && checkPermissions()) { startBroadcastStreaming(); } } private void startBroadcastStreaming() { initializeAgoraVideoSDK(); enableVideo(); setupLocalVideo(); joinChannel(); } private void initializeAgoraVideoSDK() { try { RtcEngineConfig config = new RtcEngineConfig(); config.mContext = getBaseContext(); config.mAppId = myAppId; config.mEventHandler = mRtcEventHandler; mRtcEngine = RtcEngine.create(config); } catch (Exception e) { throw new RuntimeException("Error initializing RTC engine: " + e.getMessage()); } } private void enableVideo() { mRtcEngine.enableVideo(); mRtcEngine.startPreview(); } private void setupLocalVideo() { FrameLayout container = findViewById(R.id.local_video_view_container); SurfaceView surfaceView = new SurfaceView(getBaseContext()); container.addView(surfaceView); mRtcEngine.setupLocalVideo(new VideoCanvas(surfaceView, VideoCanvas.RENDER_MODE_FIT, 0)); } private void joinChannel() { // Create an instance of ChannelMediaOptions and configure it ChannelMediaOptions options = new ChannelMediaOptions(); // Set the user role to BROADCASTER or AUDIENCE according to the use-case options.clientRoleType = Constants.CLIENT_ROLE_BROADCASTER; // In the live broadcast use-case, set the channel profile to BROADCASTING (live broadcast use-case) options.channelProfile = Constants.CHANNEL_PROFILE_LIVE_BROADCASTING; // Set the audience latency level options.audienceLatencyLevel = Constants.AUDIENCE_LATENCY_LEVEL_LOW_LATENCY; // Publish local media options.publishCameraTrack = true; options.publishMicrophoneTrack = true; mRtcEngine.joinChannel(token, channelName, 0, options); } private void setupRemoteVideo(int uid) { FrameLayout container = findViewById(R.id.remote_video_view_container); SurfaceView surfaceView = new SurfaceView(getBaseContext()); surfaceView.setZOrderMediaOverlay(true); container.addView(surfaceView); mRtcEngine.setupRemoteVideo(new VideoCanvas(surfaceView, VideoCanvas.RENDER_MODE_FIT, uid)); } @Override protected void onDestroy() { super.onDestroy(); cleanupAgoraEngine(); } private void cleanupAgoraEngine() { if (mRtcEngine != null) { mRtcEngine.stopPreview(); mRtcEngine.leaveChannel(); mRtcEngine = null; } } private void showToast(String message) { runOnUiThread(() -> Toast.makeText(MainActivity.this, message, Toast.LENGTH_SHORT).show()); } } ``` ```kotlin package com.example. import android.Manifest import android.content.pm.PackageManager import android.os.Build import android.os.Bundle import android.view.SurfaceView import android.widget.FrameLayout import android.widget.Toast import androidx.appcompat.app.AppCompatActivity import androidx.core.app.ActivityCompat import androidx.core.content.ContextCompat import io.agora.rtc2.ChannelMediaOptions import io.agora.rtc2.Constants import io.agora.rtc2.IRtcEngineEventHandler import io.agora.rtc2.RtcEngine import io.agora.rtc2.RtcEngineConfig import io.agora.rtc2.video.VideoCanvas class MainActivity : AppCompatActivity() { private val PERMISSION_REQ_ID = 22 private val myAppId = "" private val channelName = "" private val token = "" private var mRtcEngine: RtcEngine? = null private val mRtcEventHandler = object : IRtcEngineEventHandler() { override fun onJoinChannelSuccess(channel: String?, uid: Int, elapsed: Int) { super.onJoinChannelSuccess(channel, uid, elapsed) runOnUiThread { showToast("Joined channel $channel") } } override fun onUserJoined(uid: Int, elapsed: Int) { runOnUiThread { setupRemoteVideo(uid) } } override fun onUserOffline(uid: Int, reason: Int) { super.onUserOffline(uid, reason) runOnUiThread { showToast("User offline: $uid") } } } override fun onCreate(savedInstanceState: Bundle?) { super.onCreate(savedInstanceState) setContentView(R.layout.activity_main) if (checkPermissions()) { startBroadcastStreaming() } else { requestPermissions() } } private fun requestPermissions() { ActivityCompat.requestPermissions(this, getRequiredPermissions(), PERMISSION_REQ_ID) } private fun checkPermissions(): Boolean { for (permission in getRequiredPermissions()) { if (ContextCompat.checkSelfPermission(this, permission) != PackageManager.PERMISSION_GRANTED) { return false } } return true } private fun getRequiredPermissions(): Array { return if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.S) { arrayOf( Manifest.permission.RECORD_AUDIO, Manifest.permission.CAMERA, Manifest.permission.READ_PHONE_STATE, Manifest.permission.BLUETOOTH_CONNECT ) } else { arrayOf( Manifest.permission.RECORD_AUDIO, Manifest.permission.CAMERA ) } } override fun onRequestPermissionsResult( requestCode: Int, permissions: Array, grantResults: IntArray ) { super.onRequestPermissionsResult(requestCode, permissions, grantResults) if (requestCode == PERMISSION_REQ_ID && checkPermissions()) { startBroadcastStreaming() } } private fun startBroadcastStreaming() { initializeAgoraVideoSDK() enableVideo() setupLocalVideo() joinChannel() } private fun initializeAgoraVideoSDK() { try { val config = RtcEngineConfig().apply { mContext = applicationContext mAppId = myAppId mEventHandler = mRtcEventHandler } mRtcEngine = RtcEngine.create(config) } catch (e: Exception) { throw RuntimeException("Error initializing RTC engine: ${e.message}") } } private fun enableVideo() { mRtcEngine?.apply { enableVideo() startPreview() } } private fun setupLocalVideo() { val container = findViewById(R.id.local_video_view_container) val surfaceView = SurfaceView(this) container.addView(surfaceView) mRtcEngine?.setupLocalVideo(VideoCanvas(surfaceView, VideoCanvas.RENDER_MODE_FIT, 0)) } private fun joinChannel() { val options = ChannelMediaOptions().apply { clientRoleType = Constants.CLIENT_ROLE_BROADCASTER channelProfile = Constants.CHANNEL_PROFILE_LIVE_BROADCASTING audienceLatencyLevel = Constants.AUDIENCE_LATENCY_LEVEL_LOW_LATENCY publishMicrophoneTrack = true publishCameraTrack = true } mRtcEngine?.joinChannel(token, channelName, 0, options) } private fun setupRemoteVideo(uid: Int) { val container = findViewById(R.id.remote_video_view_container) val surfaceView = SurfaceView(this).apply { setZOrderMediaOverlay(true) } container.addView(surfaceView) mRtcEngine?.setupRemoteVideo(VideoCanvas(surfaceView, VideoCanvas.RENDER_MODE_FIT, uid)) } override fun onDestroy() { super.onDestroy() cleanupAgoraEngine() } private fun cleanupAgoraEngine() { mRtcEngine?.apply { stopPreview() leaveChannel() } mRtcEngine = null } private fun showToast(message: String) { runOnUiThread { Toast.makeText(this, message, Toast.LENGTH_SHORT).show() } } } ``` For the `myAppId` and `token` variables, replace the placeholders with the values you obtained from Agora Console. Ensure you enter the same `channelName` you used when generating the temporary token. ### Create a user interface [#create-a-user-interface] To connect the sample code to your existing UI, ensure that your XML layout includes the container UI element IDs used to [Display the local video](#display-the-local-video) and [Display remote video](#display-remote-video). Alternatively, use the following sample code to generate a basic user interface. Replace the existing content in `/app/src/main/res/layout/activity_main.xml` with this code. ![UI design](https://assets-docs.agora.io/images/video-sdk/quickstart-ui-android-design.png) **Sample code to create the user interface** ```xml ``` ## Test the sample code [#test-the-sample-code] Take the following steps to test the sample code: 1. In `MainActivity` update the values for `myAppId`, and `token` with values from Agora Console. Fill in the same `channelName` you used to generate the token. 2. Enable developer options on your Android test device. Turn on USB debugging, connect the Android device to your development machine through a USB cable, and check that your device appears in the Android device options. 3. In Android Studio, click ![image](https://assets-docs.agora.io/images/video-sdk/icon_android_gradle_sync.png) **Sync Project with Gradle Files** to resolve project dependencies and update the configuration. 4. After synchronization is successful, click ![image](https://assets-docs.agora.io/images/video-sdk/icon_android_run.png) **Run app**. Android Studio starts compilation. After a few moments, the app is installed on your Android device. 5. Launch the App, grant recording and camera permissions. If you set the user role to host, you will see yourself in the local view. 6. On a second Android device, repeat the previous steps to install and launch the app. Alternatively, use the [Web demo](https://webdemo.agora.io/basicVideoCall/index.html) to join the same channel and test the following use-cases: * If users on both devices join the channel as hosts, they can see and hear each other. * If one user joins as host and the other as audience, the host can see themselves in the local video window; the audience can see the host in the remote video window and hear the host. ## Reference [#reference] This section contains content that completes the information on this page, or points you to documentation that explains other aspects to this product. * If a firewall is deployed in your network environment, refer to [Connect with Cloud Proxy](build/optimize-quality-and-connection/cloud-proxy.mdx) to use Agora services normally. ### Next steps [#next-steps] After implementing the quickstart sample, read the following documents to learn more: * To ensure communication security in a test or production environment, best practice is to obtain and use a token from an authentication server. For details, see [Secure authentication with tokens](build/authenticate-users/use-tokens.mdx). ### Sample project [#sample-project] Agora provides open source sample projects on [GitHub](https://github.com/AgoraIO/API-Examples) for your reference. Download or view the [JoinChannelVideo](https://github.com/AgoraIO-Community/Agora-RTC-QuickStart/tree/main/Android/Agora-RTC-QuickStart-Android) project for a more detailed example. ### API reference [#api-reference] * [`RtcEngineConfig`](https://api-ref.agora.io/en/video-sdk/android/4.x/API/class_rtcengineconfig.html) * [`create`](https://api-ref.agora.io/en/video-sdk/android/4.x/API/class_irtcengine.html#api_irtcengine_create) * [`ChannelMediaOptions`](https://api-ref.agora.io/en/video-sdk/android/4.x/API/class_channelmediaoptions.html) * [`joinChannel`](https://api-ref.agora.io/en/video-sdk/android/4.x/API/class_irtcengine.html#api_irtcengine_joinchannel2) * [`enableVideo`](https://api-ref.agora.io/en/video-sdk/android/4.x/API/class_irtcengine.html#api_irtcengine_enablevideo) * [`startPreview`](https://api-ref.agora.io/en/video-sdk/android/4.x/API/class_irtcengine.html#api_irtcengine_startpreview) * [`leaveChannel`](https://api-ref.agora.io/en/video-sdk/android/4.x/API/class_irtcengine.html#api_irtcengine_leavechannel) * [`IRtcEngineEventHandler`](https://api-ref.agora.io/en/video-sdk/android/4.x/API/class_irtcengineeventhandler.html#class_irtcengineeventhandler) ### Frequently asked questions [#frequently-asked-questions] * [How can I fix black screen issues?](/en/api-reference/faq/quality/video_blank) * [Why can't I turn on the camera?](/en/api-reference/faq/quality/video_camera) * [How can I listen for audience joining or leaving a channel?](/en/api-reference/faq/integration/audience_event) * [How can I solve channel-related issues?](/en/api-reference/faq/integration/channel) * [How can I set the log file?](/en/api-reference/faq/integration/set_log_file) * [Why do apps on some Android versions fail to capture audio and video after screen locking or switching to the background?](/en/api-reference/faq/quality/android_background) ### See also [#see-also] * [Error codes](reference/error-codes.md) * [Connection status management](build/optimize-quality-and-connection/connection-status-management.mdx) <_PlatformProcessedMarker close="true" /> <_PlatformPanel platform="ios"> <_PlatformProcessedMarker groupMode="structured" canonicalPlatform="web" platform="ios" /> This page provides a step-by-step guide on how to create a basic Broadcast Streaming app using the Agora Video SDK. ## Understand the tech [#understand-the-tech-1] To start a Broadcast Streaming session, implement the following steps in your app: * **Initialize the Agora Engine**: Before calling other APIs, create and initialize an Agora Engine instance. * **Join a channel**: Call methods to create and join a channel. * **Join as a host**: A live streaming event has one or more hosts. A host publishes audio and video to the channel. Hosts can also subscribe to streams from other hosts. * **Join as audience**: Audience members can only subscribe to streams published by hosts. * **Send and receive audio and video**: Hosts publish streams to the channel. Audience members subscribe to audio and video streams published by hosts. ![Streaming workflow](https://assets-docs.agora.io/images/video-sdk/get-started-ils-bs.svg) ## Prerequisites [#prerequisites-1] * Xcode 13.0 or higher. * An Apple developer account. * If you need to use CocoaPods integrated SDK, make sure [CocoaPods is installed](https://guides.cocoapods.org/using/getting-started.html#getting-started). * Two devices running iOS 14.0 or higher. * A camera and a microphone * A valid Agora account and project. Please refer to [Agora account management](manage-agora-account.md) for details. ## Set up your project [#set-up-your-project-1] This section shows you how to set up your iOS project and install the Agora Video SDK. **Create a new project** Follow these steps to create a project in Xcode: 1. Refer to [Create a project](https://help.apple.com/xcode/mac/current/#/dev07db0e578). Under **Application** select **App**. Use **Storyboard** for the user interface and choose **Swift** as the programming language. If you have not added the development team information, you see the **Add account...** button. Click the button and follow the on-screen prompts to log in to your Apple ID. Once login is complete, click **Next**, and choose your Apple account as the development team. 2. [Set up automatic signing](https://help.apple.com/xcode/mac/current/#/dev23aab79b4) for your projects. 3. [Set the target devices](https://help.apple.com/xcode/mac/current/#/deve69552ee5) where your app will be deployed. 4. Create a user interface for your app. Refer to [Create a user interface](#create-a-user-interface) to create a bare bones UI. **Add to an existing project** Follow these steps to add Broadcast Streaming to your Xcode project: 1. Open your project in Xcode. 2. [Set the target devices](https://help.apple.com/xcode/mac/current/#/deve69552ee5) where your app will be deployed. 3. Create a user interface for your app. Refer to [Create a user interface](#create-a-user-interface) to create a bare bones UI. ### Install the SDK [#install-the-sdk-1] Use one of the following methods to install Video SDK. **Swift Package Manager** 1. In Xcode, go to **File** > **Add Package Dependencies**. 2. In the search bar, past the following URL: ```json {`https://github.com/AgoraIO/AgoraRtcEngine_iOS.git`} ``` 3. Click **Add Package**, select the latest version, and click **Next**. 4. For basic Broadcast Streaming, select **RtcBasic**. If needed, also select: * **`SpatialAudio`** for spatial audio effects. * **`VirtualBackground`** for virtual background. 5. Under **Add to Target**, select your project and click **Add Package**. For further information, refer to [Apple's official documentation](https://help.apple.com/xcode/mac/current/#/devb83d64851). **CocoaPods** 1. Go to the project root directory in the terminal and run the `pod init` command. A text file named `Podfile` is generated in the project folder. 2. Open `Podfile` and modify the content as follows. Replace `Your App` with your target name. ```ruby platform :ios, '9.0' target 'Your App' do # For x.y.z fill in the specific SDK version number, such as 4.4.0. # Integrate the Full SDK pod 'AgoraRtcEngine_iOS', 'x.y.z' # To integrate the Lite SDK, use the following line instead # pod 'AgoraLite_iOS', '4.4.0' end ``` Obtain the latest version number from the [release notes](reference/release-notes). 3. Run the `pod install` command in the terminal to install the Video SDK. After successful installation, the terminal shows **Pod installation complete!**. 4. After successful installation, a file with the suffix `.xcworkspace` is generated in the project folder. Open the file through Xcode for subsequent operations. **Manual integration** 1. Download the latest version of the SDK from [SDKs download](/en/api-reference/sdks?product=video\&platform=ios) and extract the contents. 2. Copy the files in the `libs` folder of the SDK package to your project directory. 3. Open Xcode and [add the corresponding dynamic library](https://help.apple.com/xcode/mac/current/#/dev51a648b07). Make sure the **Embed** property of the added dynamic library is set to **Embed & Sign**. Agora SDK uses `libc++` (LLVM) by default. If you need to use `libstdc++` (GNU), please contact [support@agora.io](mailto\:support@agora.io). The library provided by the SDK is a FAT Image, which includes 32/64-bit simulator and 32/64-bit real machine versions. The [privacy updates for App Store submissions](https://developer.apple.com/news/?id=r1henawx) released by Apple, require developers to declare approved reasons for using a set of APIs in their app’s privacy manifest. Agora provides a [`PrivacyInfo.xcprivacy`](https://download.agora.io/sdk/release/PrivacyInfo.xcprivacy) file that you can include in your project. ## Implement Broadcast Streaming [#implement-broadcast-streaming-1] This section guides you through the implementation of basic real-time audio and video interaction in your app. The following figure illustrates the essential steps: ![Quick start sequence](https://assets-docs.agora.io/images/video-sdk/quick-start-sequence.svg) This guide includes [complete sample code](#complete-sample-code) that demonstrates implementing basic real-time interaction. To understand the core API calls in the sample code, review the following implementation steps. ### Import Agora framework [#import-agora-framework] Add the following import to your swift file: ```swift import AgoraRtcKit ``` ### Initialize the engine [#initialize-the-engine-1] Call `sharedEngine(withAppId:delegate:)` to create and initialize an `AgoraRtcEngineKit` instance. Provide your [App ID](manage-agora-account.md) and an `AgoraRtcEngineDelegate` implementation to [handle SDK events](#subscribe-to--events). ```swift var agoraKit: AgoraRtcEngineKit! let appId = "YOUR_AGORA_APP_ID" // Initialize the Agora engine func initializeAgoraVideoSDK() { agoraKit = AgoraRtcEngineKit.sharedEngine(withAppId: appId, delegate: self) } ``` ### Join a channel [#join-a-channel-1] To join a channel, call `joinChannel` with the following parameters: * **Channel name**: The name of the channel to join. Clients that pass the same channel name join the same channel. If a channel with the specified name does not exist, it is created when the first user joins. * **Authentication token**: A dynamic key that authenticates a user when the client joins a channel. In a production environment, you obtain a token from a [token server](build/authenticate-users/deploy-token-server.mdx) in your security infrastructure. For the purpose of this guide [Generate a temporary token](manage-agora-account.md). * **User ID**: A 32-bit signed integer that identifies a user in the channel. You can specify a unique user ID for each user yourself. If you set the user ID to `0` when joining a channel, the SDK generates a random number for the user ID and returns the value in the `didJoinChannel` callback. * **Channel media options**: Configure `AgoraRtcChannelMediaOptions` to define publishing and subscription settings, optimize performance for your specific use-case, and set optional parameters. For Broadcast Streaming, set the `channelProfile` to `.liveBroadcasting`, the `clientRoleType` to `.broadcaster` (host) or `.audience`, and the `audienceLatencyLevel` to `lowLatency`. ```swift let channelName = "demo" // Replace with your actual channel name let token = "" // Replace with your token // Join the channel with specified options func joinChannel() { let options = AgoraRtcChannelMediaOptions() // In a live streaming use-case, set the channel use-case to liveBroadcasting options.channelProfile = .liveBroadcasting // Set the user role as broadcaster (default is audience) options.clientRoleType = .broadcaster // Publish audio captured by microphone options.publishMicrophoneTrack = true // Publish video captured by camera options.publishCameraTrack = true // Auto subscribe to all audio streams options.autoSubscribeAudio = true // Auto subscribe to all video streams options.autoSubscribeVideo = true // Set the audience latency level options.audienceLatencyLevel = .lowLatency // Use a temporary Token to join the channel agoraKit.joinChannel( byToken: token, channelId: channelName, uid: 0, mediaOptions: options ) } ``` You can also set the latency level after joining a channel by calling the `setClientRole` method with `role` set to `AgoraClientRole.audience` and `options.audienceLatencyLevel` set to `lowLatency`. ```swift // Set the user role to audience let role = AgoraClientRole.audience let options = AgoraClientRoleOptions() // Set the low latency level options.audienceLatencyLevel = .lowLatency agoraKit.setClientRole(role, options: options) ``` ### Subscribe to Video SDK events [#subscribe-to-video-sdk-events-1] The Video SDK provides a delegate for handling channel events. To use it, conform to the `AgoraRtcEngineDelegate` protocol in your class and implement the event methods you want to handle. The following code implements the `didJoinChannel`, `didOfflineOfUid`, and `didJoinedOfUid` callbacks: To ensure that you receive all Video SDK events, set the Agora Engine event handler before joining a channel. ```swift // Extension for handling Agora SDK callbacks extension ViewController: AgoraRtcEngineDelegate { // Triggered when the local user successfully joins a channel func rtcEngine(_ engine: AgoraRtcEngineKit, didJoinChannel channel: String, withUid uid: UInt, elapsed: Int) { print("Successfully joined channel: \(channel) with UID: \(uid)") } // Triggered when a remote user joins the channel func rtcEngine(_ engine: AgoraRtcEngineKit, didJoinedOfUid uid: UInt, elapsed: Int) { setupRemoteVideo(uid: uid, view: remoteView) } // Triggered when a remote user leaves the channel func rtcEngine(_ engine: AgoraRtcEngineKit, didOfflineOfUid uid: UInt, reason: AgoraUserOfflineReason) { setupRemoteVideo(uid: uid, view: nil) } } ``` To learn about the other SDK events, see [`AgoraRtcEngineDelegate`](https://api-ref.agora.io/en/video-sdk/ios/4.x/documentation/agorartckit/agorartcenginedelegate). ### Enable the video module [#enable-the-video-module-1] Follow the steps below to set up the video module. 1. To enable the video module, call `enableVideo`. 2. To enable local video preview, call `startPreview`. ```swift // Enable video functionality (audio is enabled by default) agoraKit.enableVideo() // Enable local video preview agoraKit.startPreview() ``` ### Display the local video [#display-the-local-video-1] Call `setupLocalVideo` to initialize the local view and set the local video display properties. ```swift // Configures and starts displaying the local video feed func setupLocalVideo() { let videoCanvas = AgoraRtcVideoCanvas() videoCanvas.view = localView videoCanvas.uid = 0 // UID 0 is assigned to the local user videoCanvas.renderMode = .hidden agoraKit.setupLocalVideo(videoCanvas) } ``` ### Display remote video [#display-remote-video-1] To initialize the remote user view, call `setupRemoteVideo` and set the local display properties for the remote user. Use the `didJoinedOfUid` callback to get the UID of the remote user. ```swift func setupRemoteVideo(uid: UInt, view: UIView?) { let videoCanvas = AgoraRtcVideoCanvas() videoCanvas.uid = uid videoCanvas.view = view // Assign view for joining, set to nil for leaving videoCanvas.renderMode = .hidden agoraKit.setupRemoteVideo(videoCanvas) } ``` ### Handle permissions [#handle-permissions-1] To access the camera and microphone on iOS devices, add the required permissions for real-time interaction. Open the `info.plist` file from the project navigation bar, [edit the property list](https://help.apple.com/xcode/mac/current/#/dev3f399a2a6), to add the required permissions. These permissions are optional. However, if you do not add these permissions, you will not be able to use the corresponding devices. | Key | Type | Value | | :------------------------------------- | :----- | :------------------------------------------------------------------------------------------------------ | | Privacy - Microphone Usage Description | String | For the purpose of using the microphone. For example, for a call or live interactive streaming session. | | Privacy - Camera Usage Description | String | For the purpose of using the camera. For example, for a call or live interactive streaming session. | * If your project depends on third-party plugins or libraries, such as a third-party camera library, and the signature of the plug-in or library is inconsistent with the signature of the project, check the **Hardened Runtime** settings. Specifically, review and potentially disable **Runtime Exceptions** and **Library Validation** in the project configuration. * For further information, refer to [Preparing your app for distribution](https://developer.apple.com/documentation/xcode/preparing_your_app_for_distribution). ### Start and close the app [#start-and-close-the-app-1] When the user launches the app, it joins the channel and starts Broadcast Streaming. When the user closes the app, it leaves the channel and ends Broadcast Streaming. 1. To start Broadcast Streaming, call the following methods: ```swift // Initialize the Agora engine initializeAgoraVideoSDK() // Start the local video preview setupLocalVideo() // Join an Agora channel joinChannel() ``` 2. To leave the channel and release SDK resources when the app is closed, call the following methods: ```swift // Stop local video preview agoraKit.stopPreview() // Leave the channel and release session-related resources agoraKit.leaveChannel(nil) // Release all resources used by the Agora SDK AgoraRtcEngineKit.destroy() ``` After destroying the engine, you can no longer use SDK methods and callbacks. To use the real-time interaction functions again, create a new engine. See [Initialize the engine](#initialize-the-engine) for details. ### Complete sample code [#complete-sample-code-1] A complete code sample demonstrating the basic process of real-time interaction is provided for your reference. Copy the following code into your `ViewController.swift` file: ```swift // ViewController.swift import UIKit import AgoraRtcKit class ViewController: UIViewController { let appId = "" // Replace with your actual App ID let channelName = "demo" // Replace with your actual channel name let token = "" // Replace with your token // UI view for displaying the local video stream var localView: UIView! // UI view for displaying the remote video stream var remoteView: UIView! // Instance of the Agora RTC engine var agoraKit: AgoraRtcEngineKit! override func viewDidLoad() { super.viewDidLoad() // Initialize the Agora engine initializeAgoraVideoSDK() // Set up the user interface setupUI() // Start the local video preview setupLocalVideo() // Join an Agora channel joinChannel() } // Clean up resources when the view controller is deallocated deinit { agoraKit.stopPreview() agoraKit.leaveChannel(nil) AgoraRtcEngineKit.destroy() } // Initializes the Video SDK instance func initializeAgoraVideoSDK() { // Create an instance of AgoraRtcEngineKit and set the delegate agoraKit = AgoraRtcEngineKit.sharedEngine(withAppId: appId, delegate: self) } // Sets up the UI layout for local and remote video views func setupUI() { // Create the local video view covering the full screen localView = UIView(frame: UIScreen.main.bounds) // Create the remote video view positioned in the top-right corner remoteView = UIView(frame: CGRect(x: self.view.bounds.width - 135, y: 50, width: 135, height: 240)) // Add video views to the main view self.view.addSubview(localView) self.view.addSubview(remoteView) } // Configures and starts displaying the local video feed func setupLocalVideo() { // Enable video functionality (audio is enabled by default) agoraKit.enableVideo() let videoCanvas = AgoraRtcVideoCanvas() videoCanvas.view = localView videoCanvas.uid = 0 // UID 0 is assigned to the local user videoCanvas.renderMode = .hidden agoraKit.setupLocalVideo(videoCanvas) agoraKit.startPreview() } // Join the channel with specified options func joinChannel() { let options = AgoraRtcChannelMediaOptions() // In a live streaming use-case, set the channel use-case to liveBroadcasting options.channelProfile = .liveBroadcasting // Set the user role as broadcaster (default is audience) options.clientRoleType = .broadcaster // Publish audio captured by microphone options.publishMicrophoneTrack = true // Publish video captured by camera options.publishCameraTrack = true // Auto subscribe to all audio streams options.autoSubscribeAudio = true // Auto subscribe to all video streams options.autoSubscribeVideo = true // Set the audience latency level options.audienceLatencyLevel = .lowLatency // If you set uid=0, the engine generates a uid internally; on success, it triggers didJoinChannel callback // Join the channel with a temporary token agoraKit.joinChannel( byToken: token, channelId: channelName, uid: 0, mediaOptions: options ) } func setupRemoteVideo(uid: UInt, view: UIView?) { let videoCanvas = AgoraRtcVideoCanvas() videoCanvas.uid = uid videoCanvas.view = view // Assign view for joining, set to nil for leaving videoCanvas.renderMode = .hidden agoraKit.setupRemoteVideo(videoCanvas) } } // Extension for handling Agora SDK callbacks extension ViewController: AgoraRtcEngineDelegate { // Triggered when the local user successfully joins a channel func rtcEngine(_ engine: AgoraRtcEngineKit, didJoinChannel channel: String, withUid uid: UInt, elapsed: Int) { print("Successfully joined channel: \(channel) with UID: \(uid)") } // Triggered when a remote user joins the channel func rtcEngine(_ engine: AgoraRtcEngineKit, didJoinedOfUid uid: UInt, elapsed: Int) { // Assign the remote user’s video stream to the remote view setupRemoteVideo(uid: uid, view: remoteView) } // Triggered when a remote user leaves the channel func rtcEngine(_ engine: AgoraRtcEngineKit, didOfflineOfUid uid: UInt, reason: AgoraUserOfflineReason) { // Remove the remote video feed when the user disconnects setupRemoteVideo(uid: uid, view: nil) } } ``` ### Create a user interface [#create-a-user-interface-1] To connect the sample code to your existing UI, ensure that your `ViewController.swift` file includes the `UIView`s used to [Display the local video](#display-the-local-video) and [Display remote video](#display-remote-video). Alternatively, use the following sample code to generate a basic user interface. To use this interface, replace the contents of the `ViewController.swift` file with the following code: **Sample code to create the user interface** ```swift // ViewController.swift import UIKit class ViewController: UIViewController { // UI view for displaying the local video stream var localView: UIView! // UI view for displaying the remote video stream var remoteView: UIView! override func viewDidLoad() { super.viewDidLoad() // Set up the user interface setupUI() } // Sets up the UI layout for local and remote video views func setupUI() { // Create the local video view covering the full screen localView = UIView(frame: UIScreen.main.bounds) // Create the remote video view positioned in the top-right corner remoteView = UIView(frame: CGRect(x: self.view.bounds.width - 135, y: 50, width: 135, height: 240)) // Add video views to the main view self.view.addSubview(localView) self.view.addSubview(remoteView) } } ``` ## Test the sample code [#test-the-sample-code-1] Take the following steps to test the sample code: 1. In your code update the `appId` and `token`, with the app ID and temporary token you obtained from Agora Console. Use the same `channelName` you filled in when generating the temporary token. 2. Connect your iOS device to your computer. 3. Click **Build** to run your project and wait a few seconds for the app installation to complete. 4. Allow the app to access the device's microphone and camera. 5. If an untrusted developer prompt pops up on the device, click **Cancel** to close the prompt, then open **Settings > General > VPN and Device Management** on the iOS device, and choose to trust the developer in the **Developer APP**. 6. On a second iOS device, repeat the previous steps to install and launch the app. Alternatively, use the [Web demo](https://webdemo.agora.io/basicVideoCall/index.html) to join the same channel and test the following use-cases: * If users on both devices join the channel as hosts, they can see and hear each other. * If one user joins as host and the other as audience, the host can see themselves in the local video window; the audience can see the host in the remote video window and hear the host. ## Reference [#reference-1] This section contains content that completes the information on this page, or points you to documentation that explains other aspects to this product. * If a firewall is deployed in your network environment, refer to [Connect with Cloud Proxy](build/optimize-quality-and-connection/cloud-proxy.mdx) to use Agora services normally. ### Next steps [#next-steps-1] After implementing the quickstart sample, read the following documents to learn more: * To ensure communication security in a test or production environment, best practice is to obtain and use a token from an authentication server. For details, see [Secure authentication with tokens](build/authenticate-users/use-tokens.mdx). ### Sample project [#sample-project-1] Agora provides an open source sample project on [GitHub](https://github.com/AgoraIO-Community/Agora-RTC-QuickStart/tree/main/iOS/Agora-RTC-QuickStart-iOS) for your reference. Download it or view the source code for a more detailed example. ### Add a privacy manifest file [#add-a-privacy-manifest-file] The Agora Video SDK for iOS provides the `PrivacyInfo.xcprivacy` file that contains the required reasons for the APIs used by the SDK. To add the privacy manifest to your app in Xcode, follow these steps: 1. Create a privacy manifest in your app project: 2. Choose **File > New File**. 3. Scroll down to the **Resource** section and select **App Privacy File** type. 4. Click **Next**. 5. Check your app in the **Targets** list. 6. Click **Create**. The default file name is `PrivacyInfo.xcprivacy` which is also the required file name for bundled privacy manifests. 7. Add the items in `PrivacyInfo.xcprivacy` file of the Video SDK to `PrivacyInfo.xcprivacy` of the app using the following source code: ```xml NSPrivacyTracking NSPrivacyCollectedDataTypes NSPrivacyAccessedAPITypes NSPrivacyAccessedAPIType NSPrivacyAccessedAPICategorySystemBootTime NSPrivacyAccessedAPITypeReasons 35F9.1 NSPrivacyAccessedAPIType NSPrivacyAccessedAPICategoryFileTimestamp NSPrivacyAccessedAPITypeReasons DDA9.1 NSPrivacyAccessedAPIType NSPrivacyAccessedAPICategoryDiskSpace NSPrivacyAccessedAPITypeReasons E174.1 ``` ### API reference [#api-reference-1] * [`sharedEngine`](https://api-ref.agora.io/en/video-sdk/ios/4.x/documentation/agorartckit/agorartcenginekit/sharedengine\(withappid\:delegate:\)) * [`joinChannel`](https://api-ref.agora.io/en/video-sdk/ios/4.x/documentation/agorartckit/agorartcenginekit/joinchannel\(bytoken\:channelid\:info\:uid\:joinsuccess:\)) * [`enableVideo`](https://api-ref.agora.io/en/video-sdk/ios/4.x/documentation/agorartckit/agorartcenginekit/enablevideo\(\)) * [`startPreview`](https://api-ref.agora.io/en/video-sdk/ios/4.x/documentation/agorartckit/agorartcenginekit/startpreview\(\)) * [`leaveChannel`](https://api-ref.agora.io/en/video-sdk/ios/4.x/documentation/agorartckit/agorartcenginekit/leavechannel\(_:\)) * [`enableMultiCamera`](https://api-ref.agora.io/en/video-sdk/ios/4.x/documentation/agorartckit/agorartcenginekit/enablemulticamera\(_\:config:\)) * [`AgoraRtcEngineDelegate`](https://api-ref.agora.io/en/video-sdk/ios/4.x/documentation/agorartckit/agorartcenginedelegate) ### Frequently asked questions [#frequently-asked-questions-1] * [How can I fix black screen issues?](/en/api-reference/faq/quality/video_blank) * [Why can't I turn on the camera?](/en/api-reference/faq/quality/video_camera) * [How can I listen for audience joining or leaving a channel?](/en/api-reference/faq/integration/audience_event) * [How can I solve channel-related issues?](/en/api-reference/faq/integration/channel) * [How can I set the log file?](/en/api-reference/faq/integration/set_log_file) * [How can I troubleshoot the issue of no sound?](/en/api-reference/faq/quality/audio_noaudio) * [What can I do if I get a pop-up warning saying 'the framework cannot be opened' when compiling an Xcode project?](/en/api-reference/faq/integration/framework_cannot_be_opened) * [How can I add a privacy manifest to my iOS app?](/en/api-reference/faq/other/ios_privacy_manifest) ### See also [#see-also-1] * [Error codes](reference/error-codes.md) * [Connection status management](build/optimize-quality-and-connection/connection-status-management.mdx) <_PlatformProcessedMarker close="true" /> <_PlatformPanel platform="macos"> <_PlatformProcessedMarker groupMode="structured" canonicalPlatform="web" platform="macos" /> This page provides a step-by-step guide on how to create a basic Broadcast Streaming app using the Agora Video SDK. ## Understand the tech [#understand-the-tech-2] To start a Broadcast Streaming session, implement the following steps in your app: * **Initialize the Agora Engine**: Before calling other APIs, create and initialize an Agora Engine instance. * **Join a channel**: Call methods to create and join a channel. * **Join as a host**: A live streaming event has one or more hosts. A host publishes audio and video to the channel. Hosts can also subscribe to streams from other hosts. * **Join as audience**: Audience members can only subscribe to streams published by hosts. * **Send and receive audio and video**: Hosts publish streams to the channel. Audience members subscribe to audio and video streams published by hosts. ![Streaming workflow](https://assets-docs.agora.io/images/video-sdk/get-started-ils-bs.svg) ## Prerequisites [#prerequisites-2] * Xcode 13.0 or higher. * An Apple developer account. * If you need to use CocoaPods integrated SDK, make sure [CocoaPods is installed](https://guides.cocoapods.org/using/getting-started.html#getting-started). * Two devices running macOS 10.10 or higher. * A camera and a microphone * A valid Agora account and project. Please refer to [Agora account management](manage-agora-account.md) for details. ## Set up your project [#set-up-your-project-2] This section shows you how to set up your macOS project and install the Agora Video SDK. **Create a new project** Follow these steps to create a project in Xcode: 1. Refer to [Create a project](https://help.apple.com/xcode/mac/current/#/dev07db0e578). Under **Application** select **App**. Use **Storyboard** for the user interface and choose **Swift** as the programming language. If you have not added the development team information, you see the **Add account...** button. Click the button and follow the on-screen prompts to log in to your Apple ID. Once login is complete, click **Next**, and choose your Apple account as the development team. 2. [Set up automatic signing](https://help.apple.com/xcode/mac/current/#/dev23aab79b4) for your projects. 3. [Set the target devices](https://help.apple.com/xcode/mac/current/#/deve69552ee5) where your app will be deployed. 4. Create a user interface for your app. Refer to [Create a user interface](#create-a-user-interface) to create a bare bones UI. **Add to an existing project** Follow these steps to add Broadcast Streaming to your Xcode project: 1. Open your project in Xcode. 2. [Set the target devices](https://help.apple.com/xcode/mac/current/#/deve69552ee5) where your app will be deployed. 3. Create a user interface for your app. Refer to [Create a user interface](#create-a-user-interface) to create a bare bones UI. ### Install the SDK [#install-the-sdk-2] Use one of the following methods to install Video SDK. **Swift Package Manager** 1. In Xcode, go to **File** > **Add Package Dependencies**. 2. In the search bar, past the following URL: ```json {`https://github.com/AgoraIO/AgoraRtcEngine_macOS.git`} ``` 3. Click **Add Package**, select the latest version, and click **Next**. 4. For basic Broadcast Streaming, select **RtcBasic**. If needed, also select: * **`SpatialAudio`** for spatial audio effects. * **`VirtualBackground`** for virtual background. 5. Under **Add to Target**, select your project and click **Add Package**. For further information, refer to [Apple's official documentation](https://help.apple.com/xcode/mac/current/#/devb83d64851). **CocoaPods** 1. Go to the project root directory in the terminal and run the `pod init` command. A text file named `Podfile` is generated in the project folder. 2. Open `Podfile` and modify the content as follows. Replace `Your App` with your target name. ```ruby platform :macos, '10.11' target 'Your App' do # For x.y.z fill in the specific SDK version number, such as 4.4.0. pod 'AgoraRtcEngine_macOS', 'x.y.z' end ``` Obtain the latest version number from the [release notes](reference/release-notes). 3. Run the `pod install` command in the terminal to install the Video SDK. After successful installation, the terminal shows **Pod installation complete!**. 4. After successful installation, a file with the suffix `.xcworkspace` is generated in the project folder. Open the file through Xcode for subsequent operations. **Manual integration** 1. Download the latest version of the SDK from [SDKs download](/en/api-reference/sdks?product=video\&platform=macos) and extract the contents. 2. Copy the files in the `libs` folder of the SDK package to your project directory. 3. Open Xcode and [add the corresponding dynamic library](https://help.apple.com/xcode/mac/current/#/dev51a648b07). Make sure the **Embed** property of the added dynamic library is set to **Embed & Sign**. Agora SDK uses `libc++` (LLVM) by default. If you need to use `libstdc++` (GNU), please contact [support@agora.io](mailto\:support@agora.io). The library provided by the SDK is a FAT Image, which includes 32/64-bit simulator and 32/64-bit real machine versions. The [privacy updates for App Store submissions](https://developer.apple.com/news/?id=r1henawx) released by Apple, require developers to declare approved reasons for using a set of APIs in their app’s privacy manifest. Agora provides a [`PrivacyInfo.xcprivacy`](https://download.agora.io/sdk/release/PrivacyInfo.xcprivacy) file that you can include in your project. ## Implement Broadcast Streaming [#implement-broadcast-streaming-2] This section guides you through the implementation of basic real-time audio and video interaction in your app. The following figure illustrates the essential steps: ![Quick start sequence](https://assets-docs.agora.io/images/video-sdk/quick-start-sequence-macos.svg) This guide includes [complete sample code](#complete-sample-code) that demonstrates implementing basic real-time interaction. To understand the core API calls in the sample code, review the following implementation steps. ### Import Agora framework [#import-agora-framework-1] Add the following import to your swift file: ```swift import AgoraRtcKit ``` ### Initialize the engine [#initialize-the-engine-2] Call `sharedEngine(withAppId:delegate:)` to create and initialize an `AgoraRtcEngineKit` instance. Provide your [App ID](manage-agora-account.md) and an `AgoraRtcEngineDelegate` implementation to [handle SDK events](#subscribe-to--events). ```swift var agoraKit: AgoraRtcEngineKit! let appId = "YOUR_AGORA_APP_ID" // Initialize the Agora engine func initializeAgoraVideoSDK() { agoraKit = AgoraRtcEngineKit.sharedEngine(withAppId: appId, delegate: self) } ``` ### Join a channel [#join-a-channel-2] To join a channel, call `joinChannel` with the following parameters: * **Channel name**: The name of the channel to join. Clients that pass the same channel name join the same channel. If a channel with the specified name does not exist, it is created when the first user joins. * **Authentication token**: A dynamic key that authenticates a user when the client joins a channel. In a production environment, you obtain a token from a [token server](build/authenticate-users/deploy-token-server.mdx) in your security infrastructure. For the purpose of this guide [Generate a temporary token](manage-agora-account.md). * **User ID**: A 32-bit signed integer that identifies a user in the channel. You can specify a unique user ID for each user yourself. If you set the user ID to `0` when joining a channel, the SDK generates a random number for the user ID and returns the value in the `didJoinChannel` callback. * **Channel media options**: Configure `AgoraRtcChannelMediaOptions` to define publishing and subscription settings, optimize performance for your specific use-case, and set optional parameters. For Broadcast Streaming, set the `channelProfile` to `.liveBroadcasting`, the `clientRoleType` to `.broadcaster` (host) or `.audience`, and the `audienceLatencyLevel` to `lowLatency`. ```swift let channelName = "demo" // Replace with your actual channel name let token = "" // Replace with your token // Join the channel with specified options func joinChannel() { let options = AgoraRtcChannelMediaOptions() // In a live streaming use-case, set the channel use-case to liveBroadcasting options.channelProfile = .liveBroadcasting // Set the user role as broadcaster (default is audience) options.clientRoleType = .broadcaster // Publish audio captured by microphone options.publishMicrophoneTrack = true // Publish video captured by camera options.publishCameraTrack = true // Auto subscribe to all audio streams options.autoSubscribeAudio = true // Auto subscribe to all video streams options.autoSubscribeVideo = true // Set the audience latency level options.audienceLatencyLevel = .lowLatency // Use a temporary Token to join the channel agoraKit.joinChannel( byToken: token, channelId: channelName, uid: 0, mediaOptions: options ) } ``` You can also set the latency level after joining a channel by calling the `setClientRole` method with `role` set to `AgoraClientRole.audience` and `options.audienceLatencyLevel` set to `lowLatency`. ```swift // Set the user role to audience let role = AgoraClientRole.audience let options = AgoraClientRoleOptions() // Set the low latency level options.audienceLatencyLevel = .lowLatency agoraKit.setClientRole(role, options: options) ``` ### Subscribe to Video SDK events [#subscribe-to-video-sdk-events-2] The Video SDK provides a delegate for handling channel events. To use it, conform to the `AgoraRtcEngineDelegate` protocol in your class and implement the event methods you want to handle. The following code implements the `didJoinChannel`, `didOfflineOfUid`, and `didJoinedOfUid` callbacks: To ensure that you receive all Video SDK events, set the Agora Engine event handler before joining a channel. ```swift // Extension for handling Agora SDK callbacks extension ViewController: AgoraRtcEngineDelegate { // Triggered when the local user successfully joins a channel func rtcEngine(_ engine: AgoraRtcEngineKit, didJoinChannel channel: String, withUid uid: UInt, elapsed: Int) { print("Successfully joined channel: \(channel) with UID: \(uid)") } // Triggered when a remote user joins the channel func rtcEngine(_ engine: AgoraRtcEngineKit, didJoinedOfUid uid: UInt, elapsed: Int) { setupRemoteVideo(uid: uid, view: remoteView) } // Triggered when a remote user leaves the channel func rtcEngine(_ engine: AgoraRtcEngineKit, didOfflineOfUid uid: UInt, reason: AgoraUserOfflineReason) { setupRemoteVideo(uid: uid, view: nil) } } ``` To learn about the other SDK events, see [`AgoraRtcEngineDelegate`](https://api-ref.agora.io/en/video-sdk/ios/4.x/documentation/agorartckit/agorartcenginedelegate). ### Enable the video module [#enable-the-video-module-2] Follow the steps below to set up the video module. 1. To enable the video module, call `enableVideo`. 2. To enable local video preview, call `startPreview`. ```swift // Enable video functionality (audio is enabled by default) agoraKit.enableVideo() // Enable local video preview agoraKit.startPreview() ``` ### Display the local video [#display-the-local-video-2] Call `setupLocalVideo` to initialize the local view and set the local video display properties. ```swift // Configures and starts displaying the local video feed func setupLocalVideo() { let videoCanvas = AgoraRtcVideoCanvas() videoCanvas.view = localView videoCanvas.uid = 0 // UID 0 is assigned to the local user videoCanvas.renderMode = .hidden agoraKit.setupLocalVideo(videoCanvas) } ``` ### Display remote video [#display-remote-video-2] To initialize the remote user view, call `setupRemoteVideo` and set the local display properties for the remote user. Use the `didJoinedOfUid` callback to get the UID of the remote user. ```swift func setupRemoteVideo(uid: UInt, view: UIView?) { let videoCanvas = AgoraRtcVideoCanvas() videoCanvas.uid = uid videoCanvas.view = view // Assign view for joining, set to nil for leaving videoCanvas.renderMode = .hidden agoraKit.setupRemoteVideo(videoCanvas) } ``` ### Handle permissions [#handle-permissions-2] To access the camera and microphone on macOS devices, add the required permissions for real-time interaction. Open the `info.plist` file from the project navigation bar, [edit the property list](https://help.apple.com/xcode/mac/current/#/dev3f399a2a6), to add the required permissions. These permissions are optional. However, if you do not add these permissions, you will not be able to use the corresponding devices. | Key | Type | Value | | :------------------------------------- | :----- | :------------------------------------------------------------------------------------------------------ | | Privacy - Microphone Usage Description | String | For the purpose of using the microphone. For example, for a call or live interactive streaming session. | | Privacy - Camera Usage Description | String | For the purpose of using the camera. For example, for a call or live interactive streaming session. | * If your project depends on third-party plugins or libraries, such as a third-party camera library, and the signature of the plug-in or library is inconsistent with the signature of the project, check the **Hardened Runtime** settings. Specifically, review and potentially disable **Runtime Exceptions** and **Library Validation** in the project configuration. * For further information, refer to [Preparing your app for distribution](https://developer.apple.com/documentation/xcode/preparing_your_app_for_distribution). Configure your macOS project settings by navigating to **TARGETS > Project Name > Signing & Capabilities**. Enable **App Sandbox and Hardened Runtime**, and add the necessary permissions as follows: | Capability | Category | Permission | | :--------------- | :-------------- | :-------------------------------------------------------------- | | App Sandbox | Network | * Incoming Connections (Server) * Outgoing Connections (Client) | | App Sandbox | Hardware | - Camera - Audio Input | | Hardened Runtime | Resource Access | * Camera * Audio Input | ### Start and close the app [#start-and-close-the-app-2] When the user launches the app, it joins the channel and starts Broadcast Streaming. When the user closes the app, it leaves the channel and ends Broadcast Streaming. 1. To start Broadcast Streaming, call the following methods: ```swift // Initialize the Agora engine initializeAgoraVideoSDK() // Start the local video preview setupLocalVideo() // Join an Agora channel joinChannel() ``` 2. To leave the channel and release SDK resources when the app is closed, call the following methods: ```swift // Stop local video preview agoraKit.stopPreview() // Leave the channel and release session-related resources agoraKit.leaveChannel(nil) // Release all resources used by the Agora SDK AgoraRtcEngineKit.destroy() ``` After destroying the engine, you can no longer use SDK methods and callbacks. To use the real-time interaction functions again, create a new engine. See [Initialize the engine](#initialize-the-engine) for details. ### Complete sample code [#complete-sample-code-2] A complete code sample demonstrating the basic process of real-time interaction is provided for your reference. Copy the following code into your `ViewController.swift` file: ```swift // ViewController.swift import Cocoa import AgoraRtcKit class ViewController: NSViewController { let appId = "" // Replace with your actual App ID let channelName = "demo" // Replace with your actual channel name let token = "" // Replace with your token // UI view for displaying the local video stream var localView: NSView! // UI view for displaying the remote video stream var remoteView: NSView! // Instance of the Agora RTC engine var agoraKit: AgoraRtcEngineKit! override func viewDidLoad() { super.viewDidLoad() // Initialize the Agora engine initializeAgoraVideoSDK() // Set up the user interface setupUI() // Start the local video preview setupLocalVideo() // Join an Agora channel joinChannel() } // Clean up resources when the view controller is deallocated deinit { agoraKit.stopPreview() agoraKit.leaveChannel(nil) AgoraRtcEngineKit.destroy() } // Initializes the Video SDK instance func initializeAgoraVideoSDK() { // Create an instance of AgoraRtcEngineKit and set the delegate agoraKit = AgoraRtcEngineKit.sharedEngine(withAppId: appId, delegate: self) } func setupUI() { guard let screenFrame = NSScreen.main?.frame else { return } // Ensure screen is available // Create the local video view covering the full screen localView = NSView(frame: screenFrame) // Create the remote video view positioned in the top-right corner remoteView = NSView(frame: CGRect(x: self.view.bounds.width - 135, y: 50, width: 250, height: 250)) // Add video views to the main view self.view.addSubview(localView) self.view.addSubview(remoteView) } // Configures and starts displaying the local video feed func setupLocalVideo() { // Enable video functionality (audio is enabled by default) agoraKit.enableVideo() let videoCanvas = AgoraRtcVideoCanvas() videoCanvas.view = localView videoCanvas.uid = 0 // UID 0 is assigned to the local user videoCanvas.renderMode = .hidden agoraKit.setupLocalVideo(videoCanvas) agoraKit.startPreview() } // Join the channel with specified options func joinChannel() { let options = AgoraRtcChannelMediaOptions() // In a live streaming use-case, set the channel use-case to liveBroadcasting options.channelProfile = .liveBroadcasting // Set the user role as broadcaster (default is audience) options.clientRoleType = .broadcaster // Publish audio captured by microphone options.publishMicrophoneTrack = true // Publish video captured by camera options.publishCameraTrack = true // Auto subscribe to all audio streams options.autoSubscribeAudio = true // Auto subscribe to all video streams options.autoSubscribeVideo = true // Set the audience latency level options.audienceLatencyLevel = .lowLatency // Use a temporary Token to join the channel // If you set uid=0, the engine generates a uid internally; on success, it triggers didJoinChannel callback agoraKit.joinChannel(byToken: "<#Your Token#>", channelId: "<#Your Channel Name#>", uid: 0, mediaOptions: options) } func setupRemoteVideo(uid: UInt, view: NSView?) { let videoCanvas = AgoraRtcVideoCanvas() videoCanvas.uid = uid videoCanvas.view = view // Assign view for joining, set to nil for leaving videoCanvas.renderMode = .hidden agoraKit.setupRemoteVideo(videoCanvas) } } // Extension for handling Agora SDK callbacks extension ViewController: AgoraRtcEngineDelegate { // Triggered when the local user successfully joins a channel func rtcEngine(_ engine: AgoraRtcEngineKit, didJoinChannel channel: String, withUid uid: UInt, elapsed: Int) { print("Successfully joined channel: \(channel) with UID: \(uid)") } // Triggered when a remote user joins the channel func rtcEngine(_ engine: AgoraRtcEngineKit, didJoinedOfUid uid: UInt, elapsed: Int) { // Assign the remote user’s video stream to the remote view setupRemoteVideo(uid: uid, view: remoteView) } // Triggered when a remote user leaves the channel func rtcEngine(_ engine: AgoraRtcEngineKit, didOfflineOfUid uid: UInt, reason: AgoraUserOfflineReason) { // Remove the remote video feed when the user disconnects setupRemoteVideo(uid: uid, view: nil) } } ``` ### Create a user interface [#create-a-user-interface-2] To connect the sample code to your existing UI, ensure that your `ViewController.swift` file includes the `UIView`s used to [Display the local video](#display-the-local-video) and [Display remote video](#display-remote-video). Alternatively, use the following sample code to generate a basic user interface. To use this interface, replace the contents of the `ViewController.swift` file with the following code: **Sample code to create the user interface** ```swift // ViewController.swift import Cocoa import AgoraRtcKit class ViewController: NSViewController { // UI view for displaying the local video stream var localView: NSView! // UI view for displaying the remote video stream var remoteView: NSView! override func viewDidLoad() { super.viewDidLoad() // Set up the user interface setupUI() } func setupUI() { guard let screenFrame = NSScreen.main?.frame else { return } // Ensure screen is available // Create the local video view covering the full screen localView = NSView(frame: screenFrame) // Create the remote video view positioned in the top-right corner remoteView = NSView(frame: CGRect(x: self.view.bounds.width - 135, y: 50, width: 250, height: 250)) // Add video views to the main view self.view.addSubview(localView) self.view.addSubview(remoteView) } } ``` ## Test the sample code [#test-the-sample-code-2] Take the following steps to test the sample code: 1. In your code update the `appId` and `token`, with the app ID and temporary token you obtained from Agora Console. Use the same `channelName` you filled in when generating the temporary token. 2. Click **Build** to run your project and wait a few seconds for the app installation to complete. 3. Allow the app to access the device's microphone and camera. 4. On a second macOS device, repeat the previous steps to install and launch the app. Alternatively, use the [Web demo](https://webdemo.agora.io/basicVideoCall/index.html) to join the same channel and test the following use-cases: * If users on both devices join the channel as hosts, they can see and hear each other. * If one user joins as host and the other as audience, the host can see themselves in the local video window; the audience can see the host in the remote video window and hear the host. ## Reference [#reference-2] This section contains content that completes the information on this page, or points you to documentation that explains other aspects to this product. * If a firewall is deployed in your network environment, refer to [Connect with Cloud Proxy](build/optimize-quality-and-connection/cloud-proxy.mdx) to use Agora services normally. ### Next steps [#next-steps-2] After implementing the quickstart sample, read the following documents to learn more: * To ensure communication security in a test or production environment, best practice is to obtain and use a token from an authentication server. For details, see [Secure authentication with tokens](build/authenticate-users/use-tokens.mdx). ### Sample project [#sample-project-2] Agora provides open source sample projects on [GitHub](https://github.com/AgoraIO/API-Examples) for your reference. Download or view the [JoinChannelVideo](https://github.com/AgoraIO/API-Examples/tree/main/macOS/APIExample/Examples/Basic/JoinChannelVideo) project for a more detailed example. ### API reference [#api-reference-2] * [`sharedEngine`](https://api-ref.agora.io/en/video-sdk/macos/4.x/documentation/agorartckit/agorartcenginekit/sharedengine\(withappid\:delegate:\)) * [`joinChannel`](https://api-ref.agora.io/en/video-sdk/macos/4.x/documentation/agorartckit/agorartcenginekit/joinchannel\(bytoken\:channelid\:info\:uid\:joinsuccess:\)) * [`enableVideo`](https://api-ref.agora.io/en/video-sdk/macos/4.x/documentation/agorartckit/agorartcenginekit/enablevideo\(\)) * [`startPreview`](https://api-ref.agora.io/en/video-sdk/macos/4.x/documentation/agorartckit/agorartcenginekit/startpreview\(\)) * [`leaveChannel`](https://api-ref.agora.io/en/video-sdk/macos/4.x/documentation/agorartckit/agorartcenginekit/leavechannel\(_:\)) * [`AgoraRtcEngineDelegate`](https://api-ref.agora.io/en/video-sdk/macos/4.x/documentation/agorartckit/agorartcenginedelegate) ### See also [#see-also-2] * [Error codes](reference/error-codes.md) * [Connection status management](build/optimize-quality-and-connection/connection-status-management.mdx) <_PlatformProcessedMarker close="true" /> <_PlatformPanel platform="web"> <_PlatformProcessedMarker groupMode="structured" canonicalPlatform="web" platform="web" /> Explore sample implementations to quickly integrate Conversational AI. * [RTC SDK API examples](https://github.com/AgoraIO/API-Examples-Web): Sample projects for the Agora RTC Web SDK 4.x, covering basic, advanced examples for Vue and React. This page provides a step-by-step guide on how to create a basic Broadcast Streaming app using the Agora Video SDK. ## Understand the tech [#understand-the-tech-3] To start a Broadcast Streaming session, implement the following steps in your app: * **Initialize the Agora Engine**: Before calling other APIs, create and initialize an Agora Engine instance. * **Join a channel**: Call methods to create and join a channel. * **Join as a host**: A live streaming event has one or more hosts. A host publishes audio and video to the channel. Hosts can also subscribe to streams from other hosts. * **Join as audience**: Audience members can only subscribe to streams published by hosts. * **Send and receive audio and video**: Hosts publish streams to the channel. Audience members subscribe to audio and video streams published by hosts. ![Streaming workflow](https://assets-docs.agora.io/images/video-sdk/get-started-ils-bs.svg) ## Prerequisites [#prerequisites-3] * A [supported browser](reference/supported-platforms.md). Agora strongly recommends using the latest stable version of Google Chrome. * [Node.js and npm](https://docs.npmjs.com/downloading-and-installing-node-js-and-npm) * A camera and a microphone * A valid Agora account and project. Please refer to [Agora account management](manage-agora-account.md) for details. ## Set up your project [#set-up-your-project-3] This section shows you how to set up your Web project and install the Agora Video SDK. **Create a new project** To initialize a new Vite project, take the following steps: 1. Open a terminal and run the following command: ```bash npm create vite@latest agora_web_quickstart -- --template vanilla ``` This creates a new folder named `agora_web_quickstart` and initializes a Vite project inside it using the `vanilla` JavaScript template. 2. Navigate to the newly created folder. Download and set up dependencies for your project: ```bash cd agora_web_quickstart npm install ``` 3. Create a user interface for your project. The UI consists of buttons to join and leave a channel. Refer to [Create a user interface](#create-a-user-interface) to get a bare bones html layout. **Add to an existing project** To add Broadcast Streaming to your existing project, take the following steps: 1. Create a JavaScript file in your project's `src` folder to add `AgoraRTCClient` code that implements specific application logic. 2. Add an HTML file to your project to create a user interface. The UI consists of buttons to join and leave a channel. Refer to [Create a user interface](#create-a-user-interface) to get a bare bones HTML layout. 3. Include the JavaScript file in your HTML file. ```html ``` ### Install the SDK [#install-the-sdk-3] Add the Video SDK to your project: ```bash npm install agora-rtc-sdk-ng ``` ## Implement Broadcast Streaming [#implement-broadcast-streaming-3] This section guides you through the implementation of basic real-time audio and video interaction in your app. ![image](https://assets-docs.agora.io/images/video-sdk/ils-quick-start-sequence-web.svg) This guide includes [complete sample code](#complete-sample-code) that demonstrates implementing basic real-time interaction. To understand the core API calls in the sample code, review the following implementation steps. ### Import the `AgoraRTC` SDK [#import-the-agorartc-sdk] ```js import AgoraRTC from "agora-rtc-sdk-ng"; ``` ### Initialize an instance of `AgoraRTCClient` [#initialize-an-instance-of-agorartcclient] Call [`createClient`](https://api-ref.agora.io/en/video-sdk/web/4.x/interfaces/iagorartc.html#createclient) to initialize an `AgoraRTCClient` object. Set the [Channel mode](#channel-modes) and [Video encoding format](#video-encoding-formats) based on your use case. For Broadcast Streaming Set `mode` to `live`. ```javascript // RTC client instance let client = null; // Initialize the AgoraRTC client function initializeClient() { client = AgoraRTC.createClient({ mode: "live", codec: "vp8", role: "host" }); setupEventListeners(); } ``` ### Join a channel [#join-a-channel-3] To join a channel, call `join` with the following parameters: * **App ID**: The [App ID](manage-agora-account.md) for your project. * **Channel name**: The name of the channel to join. Clients that pass the same channel name join the same channel. If a channel with the specified name does not exist, it is created when the first user joins. * **Authentication token**: A dynamic key that authenticates a user when the client joins a channel. In a production environment, you obtain a token from a [token server](build/authenticate-users/deploy-token-server.mdx) in your security infrastructure. For the purpose of this guide [Generate a temporary token](manage-agora-account.md). * **User ID**: A 32-bit signed integer that identifies a user in the channel. You can specify a unique user ID for each user yourself. If you set the user ID to `0` when joining a channel, the SDK generates a random number for the user ID. When you join as a host, you can publish local media streams in the channel; you can also subscribe to streams published by other hosts. As an audience member, you may only subscribe to media streams published by hosts. * Join as a host ```js // Join as a host async function joinChannel() { await client.join(appId, channel, token, uid); // A host can both publish tracks and subscribe to tracks client.setClientRole("host", clientRoleOptions); // Create and publish local tracks await createLocalMediaTracks(); await publishLocalTracks(); displayLocalVideo(); } ``` * Join as audience To join a channel as an audience member, first call `join`, then use `setClientRole` to set the role. Listen for the `user-published` callback to play the tracks published in the channel. ```js // Join as audience async function joinAsAudience() { await client.join(appId, channel, token, uid); // Low audience latency level for broadcast streaming let clientRoleOptions = { level: 1 }; // Audience can only subscribe to tracks client.setClientRole("audience", clientRoleOptions); } ``` ### Create local media tracks [#create-local-media-tracks] To set up the necessary local media tracks: * Call `createMicrophoneAudioTrack` to create a local audio track. * Call `createCameraVideoTrack` to create a local Video track. ```javascript // Declare variables for local tracks let localAudioTrack = null; let localVideoTrack = null; // Create local audio and video tracks async function createLocalMediaTracks() { localAudioTrack = await AgoraRTC.createMicrophoneAudioTrack(); localVideoTrack = await AgoraRTC.createCameraVideoTrack(); } ``` ### Publish local media tracks [#publish-local-media-tracks] To make the created audio and video tracks available for other users in the channel, use the `publish` method. ```javascript async function publishLocalTracks() { await client.publish([localAudioTrack, localVideoTrack]); } ``` See [Local audio and video tracks](#local--tracks) to learn more about local tracks. ### Set up event listeners [#set-up-event-listeners] Use the `on` method to register event listeners for SDK events. The SDK triggers the `user-published` event when a user publishes an audio track in the channel. Similarly, it triggers the `user-unpublished` event when a user leaves the channel, goes offline, or unpublishes a media track. ```javascript // Handle client events function setupEventListeners() { // Declare event handler for "user-published" client.on("user-published", async (user, mediaType) => { // Subscribe to media streams await client.subscribe(user, mediaType); if (mediaType === "video") { // Specify the ID of the DOM element or pass a DOM object. user.videoTrack.play(""); } if (mediaType === "audio") { user.audioTrack.play(); } }); // Handle the "user-unpublished" event to unsubscribe from the user's media tracks client.on("user-unpublished", async (user) => { const remotePlayerContainer = document.getElementById(user.uid); remotePlayerContainer && remotePlayerContainer.remove(); }); } ``` * After successfully unsubscribing, the SDK releases the corresponding `RemoteTrack` object. This automatically removes the video playback element and stops audio playback. * If a remote user actively stops publishing, the local user receives the `user-unpublished` callback. Upon receiving this callback, the SDK automatically releases the corresponding `RemoteTrack` object, so you do not need to call `unsubscribe` again. * The `unsubscribe` method is asynchronous and should be used with `Promise` or `async/await`. To ensure that you receive all Video SDK events, set up event listeners before joining a channel. For more information about other `AgoraRTCClient` events, refer to the [API reference](https://api-ref.agora.io/en/video-sdk/web/4.x/interfaces/iagorartcclient.html#event_channel_media_relay_event). ### Display the local video [#display-the-local-video-3] To play the local video, use the `play` method of the local video track. Pass an element ID or a DOM element from your UI where you want to render the video. ```javascript // Display local video function displayLocalVideo() { const localPlayerContainer = document.createElement("div"); localPlayerContainer.id = uid; localPlayerContainer.textContent = `Local user ${uid}`; localPlayerContainer.style.width = "640px"; localPlayerContainer.style.height = "480px"; document.body.append(localPlayerContainer); localVideoTrack.play(localPlayerContainer); } ``` ### Display remote video [#display-remote-video-3] To display the remote video, call the `play` method of the remote user's `videoTrack` and pass in either the element ID or a DOM element from your UI where you want to render the video. ```js // Display remote user's video function displayRemoteVideo(user) { const remotePlayerContainer = document.createElement("div"); remotePlayerContainer.id = user.uid.toString(); remotePlayerContainer.textContent = `Remote user ${user.uid}`; remotePlayerContainer.style.width = "640px"; remotePlayerContainer.style.height = "480px"; document.body.append(remotePlayerContainer); user.videoTrack.play(remotePlayerContainer); } ``` ### Leave the channel [#leave-the-channel] To exit the channel, close local audio and video tracks and call `leave`. ```javascript // Leave the channel and clean up async function leaveChannel() { // Stop the local media tracks to release the microphone and camera resources if (localAudioTrack) { localAudioTrack.close(); localAudioTrack = null; } if (localVideoTrack) { localVideoTrack.close(); localVideoTrack = null; } // Leave the channel await client.leave(); } ``` ### Complete sample code [#complete-sample-code-3] A complete code sample demonstrating the basic process of real-time interaction is provided here for your reference. To use the sample, copy the following code to the JavaScript file in the project's `src` folder. ```js import AgoraRTC from "agora-rtc-sdk-ng"; // RTC client instance let client = null; // Declare variables for local tracks let localAudioTrack = null; let localVideoTrack = null; // Connection parameters let appId = "<-- Insert app ID -->"; let channel = "<-- Insert channel name -->"; let token = "<-- Insert token -->"; let uid = 0; // User ID // Initialize the AgoraRTC client function initializeClient() { client = AgoraRTC.createClient({ mode: "live", codec: "vp8", role: "host" }); setupEventListeners(); } // Handle client events function setupEventListeners() { client.on("user-published", async (user, mediaType) => { await client.subscribe(user, mediaType); console.log("subscribe success"); if (mediaType === "video") { displayRemoteVideo(user); } if (mediaType === "audio") { user.audioTrack.play(); } }); client.on("user-unpublished", async (user) => { const remotePlayerContainer = document.getElementById(user.uid); remotePlayerContainer && remotePlayerContainer.remove(); }); } // Create and publish local tracks async function createLocalTracks() { localAudioTrack = await AgoraRTC.createMicrophoneAudioTrack(); localVideoTrack = await AgoraRTC.createCameraVideoTrack(); } // Display local video function displayLocalVideo() { const localPlayerContainer = document.createElement("div"); localPlayerContainer.id = uid; localPlayerContainer.textContent = `Local user ${uid}`; localPlayerContainer.style.width = "640px"; localPlayerContainer.style.height = "480px"; document.body.append(localPlayerContainer); localVideoTrack.play(localPlayerContainer); } // Join as a host async function joinAsHost() { await client.join(appId, channel, token, uid); // A host can both publish tracks and subscribe to tracks client.setClientRole("host"); // Create and publish local tracks await createLocalTracks(); await publishLocalTracks(); displayLocalVideo(); disableJoinButtons(); console.log("Host joined and published tracks."); } // Join as audience async function joinAsAudience() { await client.join(appId, channel, token, uid); // Set low latency level let clientRoleOptions = { level: 1 }; // Audience can only subscribe to tracks client.setClientRole("audience", clientRoleOptions); disableJoinButtons(); console.log("Audience joined."); } // Publish local tracks async function publishLocalTracks() { await client.publish([localAudioTrack, localVideoTrack]); } // Display remote user's video function displayRemoteVideo(user) { const remotePlayerContainer = document.createElement("div"); remotePlayerContainer.id = user.uid.toString(); remotePlayerContainer.textContent = `Remote user ${user.uid}`; remotePlayerContainer.style.width = "640px"; remotePlayerContainer.style.height = "480px"; document.body.append(remotePlayerContainer); user.videoTrack.play(remotePlayerContainer); } // Leave the channel async function leaveChannel() { if (localAudioTrack) { localAudioTrack.close(); localAudioTrack = null; } if (localVideoTrack) { localVideoTrack.close(); localVideoTrack = null; } const localPlayerContainer = document.getElementById(uid); localPlayerContainer && localPlayerContainer.remove(); client.remoteUsers.forEach((user) => { const playerContainer = document.getElementById(user.uid); playerContainer && playerContainer.remove(); }); await client.leave(); enableJoinButtons(); console.log("Left the channel."); } // Disable join buttons function disableJoinButtons() { document.getElementById("host-join").disabled = true; document.getElementById("audience-join").disabled = true; } // Enable join buttons function enableJoinButtons() { document.getElementById("host-join").disabled = false; document.getElementById("audience-join").disabled = false; } // Set up event listeners for buttons function setupButtonHandlers() { document.getElementById("host-join").onclick = joinAsHost; document.getElementById("audience-join").onclick = joinAsAudience; document.getElementById("leave").onclick = leaveChannel; } // Start live streaming function startBasicLiveStreaming() { initializeClient(); window.onload = setupButtonHandlers; } startBasicLiveStreaming(); ``` ### Create a user interface [#create-a-user-interface-3] Open `index.html` in the project's root folder and replace the contents with the following code to implement a basic client user interface: **Sample code to create the user interface** ```html Agora Web SDK Quickstart

Agora Web SDK Quickstart

``` ## Test the sample code [#test-the-sample-code-3] Take the following steps to run and test the sample code: 1. In your `.js` file, update the values for `appId`, and `token` with values from Agora Console. Use the same `channel` name you used to generate the token. 2. To start the development server, use the following command: ```bash npm run dev ``` 3. Open your browser and navigate to the URL displayed in your terminal, for example, `http://localhost:5173`. You see the following page: ![image](https://assets-docs.agora.io/images/video-sdk/quickstart-ui-web-ils.png) 4. On a second device, repeat the previous steps to install and launch the app. Alternatively, use the [Web demo](https://webdemo.agora.io/basicVideoCall/index.html) or clone the [sample project on Github](https://github.com/AgoraIO/API-Examples-Web) to join the same channel and test the following use-cases: * If users on both devices join the channel as hosts, they can see and hear each other. * If one user joins as host and the other as audience, the host can see themselves; the audience can see the host in the remote video view and hear the host. * Run the web app on a local server (localhost) for testing purposes only. When deploying to a production environment, use the HTTPS protocol. * Due to browser security policies that restrict HTTP addresses to 127.0.0.1, the Agora Web SDK only supports HTTPS protocol and `http://localhost` (`http://127.0.0.1`). Please do not use the HTTP protocol to access your project, except for `http://localhost` (`http://127.0.0.1`). ## Reference [#reference-3] This section contains content that completes the information on this page, or points you to documentation that explains other aspects to this product. * If a firewall is deployed in your network environment, refer to [Connect with Cloud Proxy](build/optimize-quality-and-connection/cloud-proxy.mdx) to use Agora services normally. ### Next steps [#next-steps-3] After implementing the quickstart sample, read the following documents to learn more: * To ensure communication security in a test or production environment, best practice is to obtain and use a token from an authentication server. For details, see [Secure authentication with tokens](build/authenticate-users/use-tokens.mdx). ### Sample project [#sample-project-3] * [Build a NextJS Video Call App](https://www.agora.io/en/blog/build-a-next-js-video-call-app/) * Agora provides open source sample projects on [GitHub](https://github.com/AgoraIO/API-Examples-Web) for your reference. Download or view the [basicVideoCall](https://github.com/AgoraIO/API-Examples-Web/tree/main/src/example/basic/basicVideoCall) project for a more detailed example. ### API reference [#api-reference-3] * [`AgoraRTC.createClient`](https://api-ref.agora.io/en/video-sdk/web/4.x/interfaces/iagorartc.html#createclient) * [`IAgoraRTCRemoteUser`](https://api-ref.agora.io/en/video-sdk/web/4.x/interfaces/iagorartcremoteuser.html) * [`play`](https://api-ref.agora.io/en/video-sdk/web/4.x/interfaces/iremoteaudiotrack.html#play) * [`join`](https://api-ref.agora.io/en/video-sdk/web/4.x/interfaces/iagorartcclient.html#join) * [`createMicrophoneAudioTrack`](https://api-ref.agora.io/en/video-sdk/web/4.x/interfaces/iagorartc.html#createmicrophoneaudiotrack) * [`createCameraVideoTrack`](https://api-ref.agora.io/en/video-sdk/web/4.x/interfaces/iagorartc.html#createcameravideotrack) * [`publish`](https://api-ref.agora.io/en/video-sdk/web/4.x/interfaces/iagorartcclient.html#publish) * [AgoraRTCClient events](https://api-ref.agora.io/en/video-sdk/web/4.x/interfaces/iagorartcclient.html#event_channel_media_relay_event) ### Channel modes [#channel-modes] The Video SDK supports the following channel modes: * `rtc`: Communication use-case * `live`: Live broadcast use-case #### Communication use-case [#communication-use-case] It is suitable for use-cases where all users in the channel need to communicate with each other and the total number of users is not too large, such as multi-person conferences and online chats. #### Live broadcast use-case [#live-broadcast-use-case] It is suitable for use-cases where there are few publishers but many subscribers. In this use-case, the SDK defines two user roles: audience (default) and host. Hosts can send and receive audio and video, but audience cannot send and may only receive audio and video. You can specify user roles by setting the parameters `createClient` of role, or you can call `setClientRole` to dynamically modify user roles. ### Video encoding formats [#video-encoding-formats] You can set the `codec` parameter in the `createClient` method to the following video encoding formats: * `vp8` (VP8) * `h264` (H.264) * `vp9` (VP9) This setting only affects the video encoding format of the host. For the audience, as long as their device and browser support the decoding of this format, the subscription can be completed normally. Support for these formats may vary across browsers and devices. The following table lists the `codec` formats supported by different browsers for reference: | Browser | VP8 | H.264 | VP9 | | :----------------- | --- | -------------------- | ----------------------- | | Desktop Chrome 58+ | ✔ | ✔ | ✔ | | Firefox 56+ | ✔ | ✔ | ✔(Requires Firefox 69+) | | Safari 12.1+ | ✔ | ✔ | ✔(Requires Safari 16+) | | Safari \< 12.1 | ✘ | ✔ | ✘ | | AndroidChrome 58+ | ✔ | No clear information | ✔(Requires Chrome 68+) | * H.264 support on Firefox depends on the `OpenH264` video codec plug-in from Cisco Systems, Inc. * For Chrome, support for H.264 on Android devices varies based on device hardware compatibility with hardware codecs mandated by Chrome. ### Local audio and video tracks [#local-audio-and-video-tracks] The SDK uses a hierarchy where all local track objects derive from the `LocalTrack` base class. This class defines the common behavior for all local tracks. Specific track types, such as `LocalAudioTrack` and `LocalVideoTrack` inherit from `LocalTrack` and extend its functionality. To publish a local track, you call the `publish` method of the client with the `LocalTrack` object as an input parameter. This approach makes publishing a track independent of how you create your local track. There are two main types of local tracks: `LocalAudioTrack` and `LocalVideoTrack` for publishing audio and video, respectively. Each type of `LocalTrack` comes with its own set of tools. For example, `LocalAudioTrack` lets you control the volume, while `LocalVideoTrack` has functions for customizing video. The SDK includes more specific classes based on `LocalAudioTrack` and `LocalVideoTrack`. For example, the `CameraVideoTrack`, is a type of `LocalVideoTrack` that you can use to publish video from your camera. It comes with extra features for controlling the camera and adjusting video quality. The following diagram shows the relationship between the `LocalTrack` classes: ![ILocalTrack](https://assets-docs.agora.io/images/video-sdk/ILocalTrack-web.png) ### Other integration methods [#other-integration-methods] When you use `npm` to install the Web SDK, you can enable tree shaking to reduce the size of the app after integration. For details, see [Using tree shaking](build/optimize-quality-and-connection/app-size-optimization.mdx). In addition to using `npm` to install the Web SDK, you can also use the following methods: * In the project HTML file, add the following tag to obtain the SDK from CDN: ```html ``` * Download the Video SDK package locally, save the `.js` files in the SDK package to the project directory, and then add the following tag to the project HTML file: ```html ``` Visit the [Download](/en/api-reference/sdks?product=video\&platform=web) page to obtain the link for the latest SDK version. ### Frequently asked questions [#frequently-asked-questions-2] **Why do I get a `digital envelope routines::unsupported` error when running the quickstart project locally?** This issue arises in projects configured with `webpack` for local execution due to changes in Node.js 16 and above. The modifications in Node.js, particularly its dependency on OpenSSL (detailed in the [node issue](https://github.com/nodejs/node/issues/29817)), impact the local development environment dependencies of the project. Refer to the [webpack issue](https://github.com/webpack/webpack/issues/14532) for details. Use one of the following solutions to resolve the issue: * Run the following command to set a temporary environment variable (Recommended): ```bash export NODE_OPTIONS=--openssl-legacy-provider ``` * Temporarily switch to a lower version of Node.js. ### See also [#see-also-3] * [Error codes](reference/error-codes.md) * [Connection status management](build/optimize-quality-and-connection/connection-status-management.mdx) <_PlatformProcessedMarker close="true" /> <_PlatformPanel platform="windows"> <_PlatformProcessedMarker groupMode="structured" canonicalPlatform="web" platform="windows" /> This page provides a step-by-step guide on how to create a basic Broadcast Streaming app using the Agora Video SDK. ## Understand the tech [#understand-the-tech-4] To start a Broadcast Streaming session, implement the following steps in your app: * **Initialize the Agora Engine**: Before calling other APIs, create and initialize an Agora Engine instance. * **Join a channel**: Call methods to create and join a channel. * **Join as a host**: A live streaming event has one or more hosts. A host publishes audio and video to the channel. Hosts can also subscribe to streams from other hosts. * **Join as audience**: Audience members can only subscribe to streams published by hosts. * **Send and receive audio and video**: Hosts publish streams to the channel. Audience members subscribe to audio and video streams published by hosts. ![Streaming workflow](https://assets-docs.agora.io/images/video-sdk/get-started-ils-bs.svg) ## Prerequisites [#prerequisites-4] * A device running Windows 7 or higher. * Microsoft Visual Studio 2017 or higher with [C++ desktop development](https://devblogs.microsoft.com/cppblog/windows-desktop-development-with-c-in-visual-studio/) support. C++ 11 or above. * If you use C# for development, you also need the [.NET framework](https://learn.microsoft.com/en-us/dotnet/framework/install/guide-for-developers). * A camera and a microphone * A valid Agora account and project. Please refer to [Agora account management](manage-agora-account.md) for details. ## Set up your project [#set-up-your-project-4] This section shows you how to set up your Windows project and install the Agora Video SDK. **Create a new project** The following steps outline the process to set up a new Visual Studio 2022 project on Windows 11 for implementing real-time audio and video interaction functions. 1. In Visual Studio, select **File > New > Project** to create a new project. In the pop-up window, select **MFC application** as the project template, click **Next**, update the project name to `AgoraQuickStart`, set the project storage location, and then click **Create**. 2. In the pop-up MFC application window, set the application type to **Dialog-based** , and set **Use MFC** to **Use MFC in a shared DLL**. Enter the generated class, set the generated class to **Dlg**, set the base class to **CDialog**, and click **Finish**. **Add to an existing project** To integrate real-time audio and video interaction into your project: 1. Launch Visual Studio 2022 and open your existing project by selecting **File > Open > Project/Solution**. 2. Navigate to your project directory and open the `.sln` file. 3. Create a user-interface for your app based on your application use-case. A basic UI consists of the following controls: * A Picture Control for displaying local video * A Picture Control for displaying remote video * An Edit Control for entering a channel name * Buttons to join and leave a channel Refer to [Create a user interface](#create-a-user-interface) to get a bare bones sample layout. ### Install the SDK [#install-the-sdk-4] Install the Agora Video SDK: 1. Download the latest Windows [SDK](/en/api-reference/sdks?product=video\&platform=windows). 2. Unzip and open the downloaded SDK. Copy all subfolders in `sdk/` to your solution folder. Make sure these subfolders are in the same directory as your `.sln` file. #### Configure the project [#configure-the-project] In the Solution Explorer window, right-click the project name and click **Properties** to configure the following: 1. Go to the **C/C++ > General > Additional Include Directory** menu, click **Edit**, and in the pop-up window enter `$(SolutionDir)sdk\high_level_api\include`. 2. Go to the **Linker > General > Additional Library Directory** menu, click **Edit**, and in the pop-up window: * for 64 bit Windows, enter `$(SolutionDir)sdk\x86_64`. * for x86 Windows, enter `$(SolutionDir)sdk\x86`. 3. Go to the **Linker > Input > Additional Dependencies** menu, click **Edit**, and in the pop-up window: * for 64 bit Windows, enter `$(SolutionDir)sdk\x86_64\agora_rtc_sdk.dll.lib`. * for x86 Windows, enter `$(SolutionDir)sdk\x86\agora_rtc_sdk.dll.lib`. 4. Enter the **Advanced** menu, and in **Advanced Properties**, set **Copy contents to OutDir** and **Copy C++ runtime to output directory** to `Yes`. 5. Go to the **Build Events > Post-Build Events > Command Line** menu and enter `copy $(SolutionDir)sdk\x86_64\*.dll $(SolutionDir)$(Platform)\$(Configuration)`. 6. Click **Apply** to save the configuration. ## Implement Broadcast Streaming [#implement-broadcast-streaming-4] This section guides you through the implementation of basic real-time audio and video interaction in your app. The following figure illustrates the essential steps: ![Quick start sequence](https://assets-docs.agora.io/images/video-sdk/quick-start-sequence.svg) This guide includes [complete sample code](#complete-sample-code) that demonstrates implementing basic real-time interaction. To understand the core API calls in the sample code, review the following implementation steps. ### Initialize the engine [#initialize-the-engine-3] For real-time communication, initialize an `IRtcEngine` instance and set up event handlers to manage user interactions within the channel. Use `RtcEngineContext` to specify the [App ID](manage-agora-account.md), and custom [event handler](#subscribe-to--events), then call `initialize` to initialize the engine, enabling further channel operations. Add the following code to your header and `.cpp` files: ```cpp // Declare the required variables IRtcEngine* m_rtcEngine = nullptr; // RTC engine instance CAgoraQuickStartRtcEngineEventHandler m_eventHandler; // IRtcEngineEventHandler ``` ```cpp void CAgoraQuickStartDlg::initializeAgoraEngine() { // Create IRtcEngine object m_rtcEngine = createAgoraRtcEngine(); // Create IRtcEngine context object RtcEngineContext context; // Input your App ID. You can obtain your project's App ID from the Agora Console context.appId = APP_ID; // Add event handler for callbacks and events context.eventHandler = &m_eventHandler; // Initialize int ret = m_rtcEngine->initialize(context); m_initialize = (ret == 0); if (m_initialize) { // Enable the video module m_rtcEngine->enableVideo(); } else { AfxMessageBox(_T("Failed to initialize Agora RTC engine")); } } ``` ### Join a channel [#join-a-channel-4] To join a channel, call `joinChannel`\[2/2] with the following parameters: * **Channel name**: The name of the channel to join. Clients that pass the same channel name join the same channel. If a channel with the specified name does not exist, it is created when the first user joins. * **Authentication token**: A dynamic key that authenticates a user when the client joins a channel. In a production environment, you obtain a token from a [token server](build/authenticate-users/deploy-token-server.mdx) in your security infrastructure. For the purpose of this guide [Generate a temporary token](manage-agora-account.md). * **User ID**: A 32-bit signed integer that identifies a user in the channel. You can specify a unique user ID for each user yourself. If you set the user ID to `0` when joining a channel, the SDK generates a random number for the user ID and returns the value in the `onJoinChannelSuccess` callback. * **Channel media options**: Configure `ChannelMediaOptions` to define publishing and subscription settings, optimize performance for your specific use-case, and set optional parameters. For Broadcast Streaming, set the `channelProfile` to `CHANNEL_PROFILE_LIVE_BROADCASTING`, the `clientRoleType` to `CLIENT_ROLE_BROADCASTER` (host) or `CLIENT_ROLE_AUDIENCE`, and the `audienceLatencyLevel` to `AUDIENCE_LATENCY_LEVEL_LOW_LATENCY`. ```cpp void CAgoraQuickStartDlg::joinChannel(const char* token, const char* channelName) { ChannelMediaOptions options; // Set channel profile to live broadcasting options.channelProfile = CHANNEL_PROFILE_LIVE_BROADCASTING; // Set user role to broadcaster; keep default value if setting user role to audience options.clientRoleType = CLIENT_ROLE_BROADCASTER; // Publish the audio stream captured by the microphone options.publishMicrophoneTrack = true; // Publish the camera track options.publishCameraTrack = true; // Automatically subscribe to all audio streams options.autoSubscribeAudio = true; // Automatically subscribe to all video streams options.autoSubscribeVideo = true; // Specify the audio latency level options.audienceLatencyLevel = AUDIENCE_LATENCY_LEVEL_LOW_LATENCY; // Join the channel using the token and channel name (both are const char*) m_rtcEngine->joinChannel(token, channelName, 0, options); } ``` ### Subscribe to Video SDK events [#subscribe-to-video-sdk-events-3] The Video SDK provides an interface for subscribing to channel events. To use it, create a custom event handler class by inheriting from `IRtcEngineEventHandler` and override its methods to handle real-time interaction events. Add callbacks to receive notification of users joining and leaving the channel. To ensure that you receive all Video SDK events, set the Agora Engine event handler before joining a channel. ```cpp // Define the CAgoraQuickStartRtcEngineEventHandler class to handle callback events such as users joining and leaving the channel class CAgoraQuickStartRtcEngineEventHandler : public IRtcEngineEventHandler { public: CAgoraQuickStartRtcEngineEventHandler() : m_hMsgHandler(nullptr) { } // Set the handle of the message receiving window void SetMsgReceiver(HWND hWnd) { m_hMsgHandler = hWnd; } // Register onJoinChannelSuccess callback // This callback is triggered when a local user successfully joins a channel virtual void onJoinChannelSuccess(const char* channel, uid_t uid, int elapsed) { if (m_hMsgHandler) { ::PostMessage(m_hMsgHandler, WM_MSGID(EID_JOIN_CHANNEL_SUCCESS), uid, 0); } } // Register onUserJoined callback // This callback is triggered when the remote host successfully joins the channel virtual void onUserJoined(uid_t uid, int elapsed) { if (m_hMsgHandler) { ::PostMessage(m_hMsgHandler, WM_MSGID(EID_USER_JOINED), uid, 0); } } // Register onUserOffline callback // This callback is triggered when the remote host leaves the channel or is offline virtual void onUserOffline(uid_t uid, USER_OFFLINE_REASON_TYPE reason) { if (m_hMsgHandler) { ::PostMessage(m_hMsgHandler, WM_MSGID(EID_USER_OFFLINE), uid, 0); } } private: HWND m_hMsgHandler; }; ``` Implement the callback functions. ```cpp LRESULT CAgoraQuickStartDlg::OnEIDJoinChannelSuccess(WPARAM wParam, LPARAM lParam) { // Join channel success callback uid_t localUid = wParam; return 0; } LRESULT CAgoraQuickStartDlg::OnEIDUserJoined(WPARAM wParam, LPARAM lParam) { // Remote user joined callback uid_t remoteUid = wParam; if (m_remoteRender) { return 0; } // Render remote view VideoCanvas canvas; canvas.renderMode = RENDER_MODE_TYPE::RENDER_MODE_HIDDEN; canvas.uid = remoteUid; canvas.view = m_staRemote.GetSafeHwnd(); m_rtcEngine->setupRemoteVideo(canvas); m_remoteRender = true; return 0; } LRESULT CAgoraQuickStartDlg::OnEIDUserOffline(WPARAM wParam, LPARAM lParam) { // Remote user left callback uid_t remoteUid = wParam; if (!m_remoteRender) { return 0; } // Clear remote view VideoCanvas canvas; canvas.uid = remoteUid; m_rtcEngine->setupRemoteVideo(canvas); m_remoteRender = false; return 0; } ``` ### Enable the video module [#enable-the-video-module-3] Call the `enableVideo` method to enable the video module. ```cpp // Enable the video module m_rtcEngine->enableVideo(); ``` ### Display the local video [#display-the-local-video-4] Follow these steps to set up and start the local video preview: 1. Create a `VideoCanvas` instance and configure its properties: 2. Set the video rendering mode. 3. Specify the user ID (`uid`). 4. Define the display window. 5. Call the `setupLocalVideo` method to apply the `VideoCanvas` configuration. 6. Call the `startPreview` method to start the local video preview. ```cpp void CAgoraQuickStartDlg::setupLocalVideo() { // Set local video display properties VideoCanvas canvas; // Set video to be scaled proportionally canvas.renderMode = RENDER_MODE_TYPE::RENDER_MODE_HIDDEN; // User ID canvas.uid = 0; // Video display window canvas.view = m_staLocal.GetSafeHwnd(); m_rtcEngine->setupLocalVideo(canvas); // Preview the local video m_rtcEngine->startPreview(); } ``` ### Display remote video [#display-remote-video-4] To display the remote user's video: 1. Define the video display properties using `VideoCanvas`. 2. Call `setupRemoteVideo` to render the video. ```cpp void CAgoraQuickStartDlg::setupRemoteVideo(uid_t remoteUid) { // Set remote video display properties VideoCanvas canvas; // Set video size to be proportionally scaled canvas.renderMode = RENDER_MODE_TYPE::RENDER_MODE_HIDDEN; // Remote user ID canvas.uid = remoteUid; // You can only choose to set either view or surfaceTexture. If both are set, only the settings in view take effect. canvas.view = m_staRemote.GetSafeHwnd(); m_rtcEngine->setupRemoteVideo(canvas); m_remoteRender = true; return 0; } ``` ### Leave the channel [#leave-the-channel-1] When a user ends a call, or closes the app call `leaveChannel` to exit the current channel. To stop the local video preview and clear the view: 1. Call `stopPreview` to stop playing the local video. 2. Call `setupLocalVideo`, passing an empty `VideoCanvas` to reset the view. ```cpp void CAgoraQuickStartDlg::LeaveChannel() { if (m_rtcEngine) { // Stop local video preview m_rtcEngine->stopPreview(); // Leave the channel m_rtcEngine->leaveChannel(); // Clear local view VideoCanvas canvas; canvas.uid = 0; m_rtcEngine->setupLocalVideo(canvas); m_remoteRender = false; } } ``` ### Release resources [#release-resources] To destroy the engine instance, call `release` : ```cpp // Release resources when the object is destroyed m_rtcEngine->release(true); m_rtcEngine = NULL; ``` After you call `Dispose`, you can no longer use any SDK methods or callbacks. To use real-time Broadcast Streaming again, you must create a new engine. For more information, see [Initialize the Engine](#initialize-the-engine). ### Complete sample code [#complete-sample-code-4] A complete code sample that implements the basic process of real-time interaction is presented here for your reference. Copy the sample code into your project to quickly implement the basic functions of real-time interaction. **AgoraQuickStartDlg.h** ```cpp #pragma once #include // Import related header files #include using namespace now; using namespace agora::rtc; using namespace agora::media; using namespace agora::media::base; // Define the message ID #define WM_MSGID(code) (WM_USER+0x200+code) #define EID_JOIN_CHANNEL_SUCCESS 0x00000002 #define EID_USER_JOINED 0x00000004 #define EID_USER_OFFLINE 0x00000004 // Define the CAgoraQuickStartRtcEngineEventHandler class to handle callback events such as users joining and leaving the channel class CAgoraQuickStartRtcEngineEventHandler : public IRtcEngineEventHandler { public: CAgoraQuickStartRtcEngineEventHandler() : m_hMsgHandler(nullptr) { } // Set the handle of the message receiving window void SetMsgReceiver(HWND hWnd) { m_hMsgHandler = hWnd; } // Register onJoinChannelSuccess callback // This callback is triggered when a local user successfully joins a channel virtual void onJoinChannelSuccess(const char* channel, uid_t uid, int elapsed) { if (m_hMsgHandler) { ::PostMessage(m_hMsgHandler, WM_MSGID(EID_JOIN_CHANNEL_SUCCESS), uid, 0); } } // Register onUserJoined callback // This callback is triggered when the remote host successfully joins the channel virtual void onUserJoined(uid_t uid, int elapsed) { if (m_hMsgHandler) { ::PostMessage(m_hMsgHandler, WM_MSGID(EID_USER_JOINED), uid, 0); } } // Register onUserOffline callback // This callback is triggered when the remote host leaves the channel or is offline virtual void onUserOffline(uid_t uid, USER_OFFLINE_REASON_TYPE reason) { if (m_hMsgHandler) { ::PostMessage(m_hMsgHandler, WM_MSGID(EID_USER_OFFLINE), uid, 0); } } private: HWND m_hMsgHandler; }; // CAgoraQuickStartDlg dialog box class CAgoraQuickStartDlg : public CDialog { // Construction public: CAgoraQuickStartDlg(CWnd* pParent = nullptr); // Standard constructor virtual ~CAgoraQuickStartDlg(); // Dialog box data #ifdef AFX_DESIGN_TIME enum { IDD = IDD_AGORAQUICKSTART_DIALOG }; #endif // Handle the join/leave button click event afx_msg void OnBnClickedBtnJoin(); afx_msg void OnBnClickedBtnLeave(); // Handle callback events such as user joining/user leaving afx_msg LRESULT OnEIDJoinChannelSuccess(WPARAM wParam, LPARAM lParam); afx_msg LRESULT OnEIDUserJoined(WPARAM wParam, LPARAM lParam); afx_msg LRESULT OnEIDUserOffline(WPARAM wParam, LPARAM lParam); protected: HICON m_hIcon; CEdit m_edtChannelName; // DDX/DDV support virtual void DoDataExchange(CDataExchange* pDX); virtual BOOL OnInitDialog(); afx_msg void OnSysCommand(UINT nID, LPARAM lParam); afx_msg void OnPaint(); afx_msg HCURSOR OnQueryDragIcon(); DECLARE_MESSAGE_MAP() std::string cs2utf8(CString str); private: IRtcEngine* m_rtcEngine = nullptr; CAgoraQuickStartRtcEngineEventHandler m_eventHandler; bool m_initialize = false; bool m_remoteRender = false; }; ``` **AgoraQuickStartDlg.cpp** ```cpp #include "pch.h" #include "framework.h" #include "AgoraQuickStart.h" #include "AgoraQuickStartDlg.h" #include "afxdialogex.h" #ifdef _DEBUG #define new DEBUG_NEW #endif // CAboutDlg dialog used for App "About" menu item class CAboutDlg : public CDialogEx { public: CAboutDlg(); // Dialog Data #ifdef AFX_DESIGN_TIME enum { IDD = IDD_ABOUTBOX }; #endif protected: virtual void DoDataExchange(CDataExchange* pDX); // DDX/DDV support // Implementation protected: DECLARE_MESSAGE_MAP() }; CAboutDlg::CAboutDlg() : CDialogEx(IDD_ABOUTBOX) {} void CAboutDlg::DoDataExchange(CDataExchange* pDX) { CDialogEx::DoDataExchange(pDX); } BEGIN_MESSAGE_MAP(CAboutDlg, CDialogEx) END_MESSAGE_MAP() // CAgoraQuickStartDlg dialog for handling main user interactions and callback events CAgoraQuickStartDlg::CAgoraQuickStartDlg(CWnd* pParent /*=nullptr*/) : CDialog(IDD_AGORAQUICKSTART_DIALOG, pParent) { m_hIcon = AfxGetApp()->LoadIcon(IDR_MAINFRAME); } CAgoraQuickStartDlg::~CAgoraQuickStartDlg() { if (m_rtcEngine) { m_rtcEngine->release(true); m_rtcEngine = NULL; } } void CAgoraQuickStartDlg::DoDataExchange(CDataExchange* pDX) { CDialog::DoDataExchange(pDX); DDX_Control(pDX, IDC_EDIT_CHANNEL, m_edtChannelName); DDX_Control(pDX, IDC_STATIC_REMOTE, m_staRemote); DDX_Control(pDX, IDC_STATIC_LOCAL, m_staLocal); } BEGIN_MESSAGE_MAP(CAgoraQuickStartDlg, CDialog) ON_WM_SYSCOMMAND() ON_WM_PAINT() ON_WM_QUERYDRAGICON() ON_BN_CLICKED(ID_BTN_JOIN, &CAgoraQuickStartDlg::OnBnClickedBtnJoin) ON_BN_CLICKED(ID_BTN_LEAVE, &CAgoraQuickStartDlg::OnBnClickedBtnLeave) ON_MESSAGE(WM_MSGID(EID_JOIN_CHANNEL_SUCCESS), CAgoraQuickStartDlg::OnEIDJoinChannelSuccess) ON_MESSAGE(WM_MSGID(EID_USER_JOINED), &CAgoraQuickStartDlg::OnEIDUserJoined) ON_MESSAGE(WM_MSGID(EID_USER_OFFLINE), &CAgoraQuickStartDlg::OnEIDUserOffline) END_MESSAGE_MAP() // Insert your project's App ID obtained from the Agora Console #define APP_ID "" // Insert the temporary token obtained from the Agora Console #define token "" void CAgoraQuickStartDlg::initializeAgoraEngine() { m_rtcEngine = createAgoraRtcEngine(); RtcEngineContext context; context.appId = APP_ID; context.eventHandler = &m_eventHandler; int ret = m_rtcEngine->initialize(context); m_initialize = (ret == 0); if (m_initialize) { // Enable the video module m_rtcEngine->enableVideo(); } else { AfxMessageBox(_T("Failed to initialize Agora RTC engine")); } } BOOL CAgoraQuickStartDlg::OnInitDialog() { CDialog::OnInitDialog(); // Add "About..." menu item to the system menu CMenu* pSysMenu = GetSystemMenu(FALSE); if (pSysMenu != nullptr) { CString strAboutMenu; strAboutMenu.LoadString(IDS_ABOUTBOX); if (!strAboutMenu.IsEmpty()) { pSysMenu->AppendMenu(MF_SEPARATOR); pSysMenu->AppendMenu(MF_STRING, IDM_ABOUTBOX, strAboutMenu); } } // Set the dialog icons SetIcon(m_hIcon, TRUE); // Set big icon SetIcon(m_hIcon, FALSE); // Set small icon // Set the message receiver m_eventHandler.SetMsgReceiver(m_hWnd); // Initialize Agora engine initializeAgoraEngine(); return TRUE; // Unless focus is set to a control, return TRUE } void CAgoraQuickStartDlg::OnSysCommand(UINT nID, LPARAM lParam) { if ((nID & 0xFFF0) == IDM_ABOUTBOX) { CAboutDlg dlgAbout; dlgAbout.DoModal(); } else { CDialog::OnSysCommand(nID, lParam); } } void CAgoraQuickStartDlg::OnPaint() { if (IsIconic()) { CPaintDC dc(this); SendMessage(WM_ICONERASEBKGND, reinterpret_cast(dc.GetSafeHdc()), 0); int cxIcon = GetSystemMetrics(SM_CXICON); int cyIcon = GetSystemMetrics(SM_CYICON); CRect rect; GetClientRect(&rect); int x = (rect.Width() - cxIcon + 1) / 2; int y = (rect.Height() - cyIcon + 1) / 2; dc.DrawIcon(x, y, m_hIcon); } else { CDialog::OnPaint(); } } HCURSOR CAgoraQuickStartDlg::OnQueryDragIcon() { return static_cast(m_hIcon); } std::string CAgoraQuickStartDlg::cs2utf8(CString str) { char szBuf[2 * MAX_PATH] = { 0 }; WideCharToMultiByte(CP_UTF8, 0, str.GetBuffer(0), str.GetLength(), szBuf, 2 * MAX_PATH, NULL, NULL); return szBuf; } void CAgoraQuickStartDlg::joinChannel(const char* token, const char* channelName) { ChannelMediaOptions options; options.channelProfile = CHANNEL_PROFILE_LIVE_BROADCASTING; options.clientRoleType = CLIENT_ROLE_BROADCASTER; options.publishMicrophoneTrack = true; options.publishCameraTrack = true; options.autoSubscribeAudio = true; options.autoSubscribeVideo = true; options.audienceLatencyLevel = AUDIENCE_LATENCY_LEVEL_LOW_LATENCY; m_rtcEngine->joinChannel(token, channelName, 0, options); } void CAgoraQuickStartDlg::setupRemoteVideo() { VideoCanvas canvas; canvas.renderMode = RENDER_MODE_TYPE::RENDER_MODE_HIDDEN; canvas.uid = remoteUid; canvas.view = m_staRemote.GetSafeHwnd(); m_rtcEngine->setupRemoteVideo(canvas); m_remoteRender = true; } void CAgoraQuickStartDlg::setupLocalVideo() { VideoCanvas canvas; canvas.renderMode = RENDER_MODE_TYPE::RENDER_MODE_HIDDEN; canvas.uid = 0; canvas.view = m_staLocal.GetSafeHwnd(); m_rtcEngine->setupLocalVideo(canvas); m_rtcEngine->startPreview(); } void CAgoraQuickStartDlg::OnBnClickedBtnJoin() { CString strChannelName; m_edtChannelName.GetWindowText(strChannelName); if (strChannelName.IsEmpty()) { AfxMessageBox(_T("Fill channel name first")); return; } joinChannel(token, CW2A(strChannelName)); setupLocalVideo(); } void CAgoraQuickStartDlg::OnBnClickedBtnLeave() { m_rtcEngine->leaveChannel(); VideoCanvas canvas; canvas.uid = 0; m_rtcEngine->setupLocalVideo(canvas); m_rtcEngine->startPreview(); m_remoteRender = false; } LRESULT CAgoraQuickStartDlg::OnEIDJoinChannelSuccess(WPARAM wParam, LPARAM lParam) { return 0; } LRESULT CAgoraQuickStartDlg::OnEIDUserJoined(WPARAM wParam, LPARAM lParam) { uid_t remoteUid = wParam; if (m_remoteRender) { return 0; } setupRemoteVideo(remoteUid); return 0; } LRESULT CAgoraQuickStartDlg::OnEIDUserOffline(WPARAM wParam, LPARAM lParam) { uid_t remoteUid = wParam; if (!m_remoteRender) { return 0; } VideoCanvas canvas; canvas.uid = remoteUid; m_rtcEngine->setupRemoteVideo(canvas); m_remoteRender = false; return 0; } ``` ### Create a user interface [#create-a-user-interface-4] To connect the sample code to your existing user interface, ensure that your UI includes the controls used to [Display the local video](#display-the-local-video) and [Display remote video](#display-remote-video). Alternatively, follow these steps to create a bare-bones UI for your project. **Steps to create a minimalistic UI** 1. Switch the project to resource view in the right menu bar, and then open the `.Dialog` file. 2. From **View > Toolbox**, select Add **Picture Control**, and in **Properties > Miscellaneous**, set the ID of the control to `IDC_STATIC_REMOTE`. 3. From **View > Toolbox** , select Add **Picture Control** , and in **Properties > Miscellaneous** , set the control's ID to `IDC_STATIC_LOCAL`. 4. To set up an input box for entering the channel name, from **View > Toolbox** , select add **Static Text** control, and change the description text to `Channel name` in the properties. Add an **Edit Control** as an input box, and in **Properties > Miscellaneous**, set the control's ID to `IDC_EDIT_CHANNEL`. 5. To add join and leave channel buttons, open **View > Toolbox** and add two **Button** controls. In **Properties > Miscellaneous** , set the IDs to `ID_BTN_JOIN` and `ID_BTN_LEAVE`, and set the description text to **Join** and **Leave** respectively. Your user interface looks similar to the following: ![image](https://assets-docs.agora.io/images/video-sdk/windows-quickstart-ui.png) Follow these steps to create a bare-bones UI for your project. ## Test the sample code [#test-the-sample-code-4] To test your app, follow these steps: 1. In Visual Studio, select local Windows debugger to start compiling the application. 2. Enter the name of the channel you want to join in the input box and click the **Join** button to join the channel. You see yourself in the local view. 3. Use the [Web demo](https://webdemo.agora.io/basicVideoCall/index.html) to join the same channel and test the following use-cases: * If users on both devices join the channel as hosts, they can see and hear each other. * If one user joins as host and the other as audience, the host can see themselves in the local video window; the audience can see the host in the remote video window and hear the host. ## Reference [#reference-4] This section contains content that completes the information on this page, or points you to documentation that explains other aspects to this product. * If a firewall is deployed in your network environment, refer to [Connect with Cloud Proxy](build/optimize-quality-and-connection/cloud-proxy.mdx) to use Agora services normally. ### Next steps [#next-steps-4] After implementing the quickstart sample, read the following documents to learn more: * To ensure communication security in a test or production environment, best practice is to obtain and use a token from an authentication server. For details, see [Secure authentication with tokens](build/authenticate-users/use-tokens.mdx). ### Sample project [#sample-project-4] Agora provides open source sample projects on [GitHub](https://github.com/AgoraIO/API-Examples/tree/main/windows/APIExample/) for your reference. * Download or view the [sample project](https://github.com/AgoraIO/API-Examples/tree/main/windows/APIExample/APIExample/Basic/LiveBroadcasting) for a more detailed example. * For a Windows C# implementation, see [this sample project](https://github.com/AgoraIO-Extensions/Agora-C_Sharp-SDK/tree/master/APIExample/src/Basic/JoinChannelVideo). ### API Reference [#api-reference-4] * [`createAgoraRtcEngine`](https://api-ref.agora.io/en/video-sdk/cpp/4.x/API/class_irtcengine.html#api_createagorartcengine) * [`joinChannel`](https://api-ref.agora.io/en/video-sdk/cpp/4.x/API/class_irtcengine.html#api_irtcengine_joinchannel2) * [`enableVideo`](https://api-ref.agora.io/en/video-sdk/cpp/4.x/API/class_irtcengine.html#api_irtcengine_enablevideo) * [`startPreview`](https://api-ref.agora.io/en/video-sdk/cpp/4.x/API/class_irtcengine.html#api_irtcengine_startpreview) * [`leaveChannel`](https://api-ref.agora.io/en/video-sdk/cpp/4.x/API/class_irtcengine.html#api_irtcengine_leavechannel) * [`release`](https://api-ref.agora.io/en/video-sdk/cpp/4.x/API/class_irtcengine.html#api_irtcengine_release) ### Frequently asked questions [#frequently-asked-questions-3] * [How can I fix black screen issues?](/en/api-reference/faq/quality/video_blank) * [Why can't I turn on the camera?](/en/api-reference/faq/quality/video_camera) * [How can I listen for audience joining or leaving a channel?](/en/api-reference/faq/integration/audience_event) * [How can I solve channel-related issues?](/en/api-reference/faq/integration/channel) * [How can I set the log file?](/en/api-reference/faq/integration/set_log_file) ### See also [#see-also-4] * [Error codes](reference/error-codes.md) * [Connection status management](build/optimize-quality-and-connection/connection-status-management.mdx) <_PlatformProcessedMarker close="true" /> <_PlatformPanel platform="electron"> <_PlatformProcessedMarker groupMode="structured" canonicalPlatform="web" platform="electron" /> This page provides a step-by-step guide on how to create a basic Broadcast Streaming app using the Agora Video SDK. ## Understand the tech [#understand-the-tech-5] To start a Broadcast Streaming session, implement the following steps in your app: * **Initialize the Agora Engine**: Before calling other APIs, create and initialize an Agora Engine instance. * **Join a channel**: Call methods to create and join a channel. * **Join as a host**: A live streaming event has one or more hosts. A host publishes audio and video to the channel. Hosts can also subscribe to streams from other hosts. * **Join as audience**: Audience members can only subscribe to streams published by hosts. * **Send and receive audio and video**: Hosts publish streams to the channel. Audience members subscribe to audio and video streams published by hosts. ![Streaming workflow](https://assets-docs.agora.io/images/video-sdk/get-started-ils-bs.svg) ## Prerequisites [#prerequisites-5] * [Node.js](https://nodejs.org/en/download/) 14 or higher. * A camera and a microphone * A valid Agora account and project. Please refer to [Agora account management](manage-agora-account.md) for details. ## Set up your project [#set-up-your-project-5] This section shows you how to set up your Electron project and install the Agora Video SDK. 1. Create a new directory for your Electron project in a local folder. Add the following files to the root of the project directory: * `package.json`: Manages project dependencies and scripts. * `index.html`: Defines the app's user interface. * `main.js`: The main process entry point. * `renderer.js`: The renderer process script, responsible for interacting with the Agora Electron SDK. 2. Create a user interface for your app based on your use-case. A basic user interface consists of an element for local video and an element for remote video. Refer to [Create a user interface](#create-a-user-interface) to get a bare bones sample layout. ### Install the SDK [#install-the-sdk-5] **npm integration** Add the Video SDK to your project using `npm`: 1. Configure `package.json` * macOS ```json { "name": "electron-demo-app", "version": "0.1.0", "author": "your name", "description": "My Electron app", "main": "main.js", "scripts": { "start": "electron ." }, "agora_electron": { "platform": "darwin", "prebuilt": true }, "dependencies": { "agora-electron-sdk": "latest" }, "devDependencies": { "electron": "latest" } } ``` * Windows ```json { "name": "electron-demo-app", "version": "0.1.0", "author": "your name", "description": "My Electron app", "main": "main.js", "scripts": { "start": "electron ." }, "agora_electron": { "platform": "win32", "prebuilt": true, "arch": "ia32" }, "dependencies": { "agora-electron-sdk": "latest" }, "devDependencies": {"electron": "latest" } } ``` **Configuration parameters** * `agora-electron-sdk`: Version number of the Agora Electron SDK. Set to latest for the most recent version, or specify a version number. See [`agora-electron-sdk`](https://www.npmjs.com/package/agora-electron-sdk) on npm for available versions. * `electron`: Electron version number. Versions 5.0.0 and above are supported. For macOS devices with M1 chips, use version 11.0.0 or higher. * `platform` (Optional): Target platform. Set to `darwin` for macOS or `win32` for Windows. * `prebuilt` (Optional): Set to `true` by default to prevent compatibility issues between Electron, Node.js, and the SDK. * `arch` (Optional): Target architecture. Defaults to your system architecture. Different versions of Electron have different environment requirements. If you see errors due to a mismatched environment when running the project, refer to the official [Electron documentation](https://releases.electronjs.org/releases/stable) to choose the appropriate Electron and Node.js versions. 2. To install the dependencies, open a terminal in your project root directory and run the following commands:: * macOS ```bash npm install ``` * Windows ```bash npm install -D --arch=ia32 electron npm install ``` * On Windows, you must first install the 32-bit version of Electron by running `npm install -D --arch=ia32 electron`, and then run `npm install`. Otherwise, you may encounter the error: “Not a valid Win32 application.” * If a `node_modules` folder exists in the project root directory, delete it before running `npm install` to avoid potential errors. **Manual Integration** Compile the latest SDK from the [Electron SDK repository](https://github.com/AgoraIO-Extensions/Electron-SDK/tree/main/) and integrate it into your application. ## Implement Broadcast Streaming [#implement-broadcast-streaming-5] This section guides you through the implementation of basic real-time audio and video interaction in your app. The following figure illustrates the essential steps: ![Quick start sequence](https://assets-docs.agora.io/images/video-sdk/quick-start-sequence.svg) This guide includes [complete sample code](#complete-sample-code) that demonstrates implementing basic real-time interaction. To understand the core API calls in the sample code, review the following implementation steps. ### Set up the main process [#set-up-the-main-process] The `main.js` file is the entry point for your Electron application. It defines the main process, which is responsible for creating and managing browser windows. This script initializes the app, loads the UI from `index.html`, and ensures proper behavior across platforms. ```js const { app, BrowserWindow } = require("electron"); const path = require("path"); // If using Electron 9.x or later, set allowRendererProcessReuse to false app.allowRendererProcessReuse = false; function createWindow() { // Create the browser window const mainWindow = new BrowserWindow({ width: 800, height: 600, webPreferences: { // preload: path.join(__dirname, "renderer.js"), // Enable Node integration and disable context isolation nodeIntegration: true, contextIsolation: false, }, }); // Load the contents of the index.html file mainWindow.loadFile("./index.html"); // Open the Developer Tools mainWindow.webContents.openDevTools(); } // Manage the browser window for the Electron app app.whenReady().then(() => { createWindow(); // On macOS, create a new window if none are open app.on("activate", function () { if (BrowserWindow.getAllWindows().length === 0) { createWindow(); } }); }); // Quit the Electron app when all windows are closed (except on macOS) app.on("window-all-closed", function () { if (process.platform !== "darwin") app.quit(); }); ``` ### Handle permissions [#handle-permissions-3] Perform this step only when the target platform is macOS. Starting with macOS v10.14, you must check and obtain permission before accessing the camera or microphone. Use Electron’s `askForMediaAccess` method to request these permissions from the user. Add the following code to the `main.js` file: ```js // Check and request device permissions async function checkAndApplyDeviceAccessPrivilege() { // Check and request camera permission const cameraPrivilege = systemPreferences.getMediaAccessStatus('camera'); console.log(`Camera privilege before applying: ${cameraPrivilege}`); if (cameraPrivilege !== 'granted') { await systemPreferences.askForMediaAccess('camera'); console.log('Requested camera access from user'); } // Check and request microphone permission const micPrivilege = systemPreferences.getMediaAccessStatus('microphone'); console.log(`Microphone privilege before applying: ${micPrivilege}`); if (micPrivilege !== 'granted') { await systemPreferences.askForMediaAccess('microphone'); console.log('Requested microphone access from user'); } } checkAndApplyDeviceAccessPrivilege(); ``` ### Import dependencies [#import-dependencies] Import the modules and functions required to build the app using Video SDK. Add the following code to `renderer.js`: ```js const { createAgoraRtcEngine, ChannelProfileType, ClientRoleType, VideoSourceType, VideoViewSetupMode, AudienceLatencyLevelType } = require("agora-electron-sdk"); ``` ### Specify connection parameters [#specify-connection-parameters] Provide the App ID, temporary token, and the channel name you used when generating the token. The engine uses these values to initialize and join the channel. ```json // Enter your App ID const APPID = "<-- Insert App Id -->"; // Enter your temporary token let token = "<-- Insert Token -->"; // Enter the channel name used when generating the token const channel = "<-- Insert Channel Name -->"; // Specify a unique user ID for this session let uid = 123; ``` ### Initialize the engine [#initialize-the-engine-4] For real-time communication, create an `IRtcEngine` object by calling `createAgoraRtcEngine` and then call `initialize` with `RtcEngineContext` to specify the [App ID](manage-agora-account.md). Call `registerEventHandler` to register a custom [event handler](#subscribe-to--events) for managing user interactions within the channel. ```js let rtcEngine; const os = require("os"); const path = require("path"); const sdkLogPath = path.resolve(os.homedir(), "./test.log"); // Create RtcEngine instance rtcEngine = createAgoraRtcEngine(); // Initialize RtcEngine instance rtcEngine.initialize({ appId: APPID, logConfig: { filePath: sdkLogPath } }); ``` ### Enable the video module [#enable-the-video-module-4] Call the `enableVideo` method to enable the video module, and then call `startPreview` to start the local video preview. ```js // Enable the video module rtcEngine.enableVideo(); // Start the local video preview rtcEngine.startPreview(); ``` ### Join a channel [#join-a-channel-5] To join a channel, call `joinChannel` with the following parameters: * **Channel name**: The name of the channel to join. Clients that pass the same channel name join the same channel. If a channel with the specified name does not exist, it is created when the first user joins. * **Authentication token**: A dynamic key that authenticates a user when the client joins a channel. In a production environment, you obtain a token from a [token server](build/authenticate-users/deploy-token-server.mdx) in your security infrastructure. For the purpose of this guide [Generate a temporary token](manage-agora-account.md). * **User ID**: A 32-bit signed integer that identifies a user in the channel. You can specify a unique user ID for each user yourself. If you set the user ID to `0` when joining a channel, the SDK generates a random number for the user ID and returns the value in the `onJoinChannelSuccess` callback. * **Channel media options**: Configure `ChannelMediaOptions` to define publishing and subscription settings, optimize performance for your specific use-case, and set optional parameters. For Broadcast Streaming, set the `channelProfile` to `ChannelProfileLiveBroadcasting`, the `clientRoleType` to `ClientRoleBroadcaster` (host) or `ClientRoleAudience`, and the `audienceLatencyLevelType` to `AudienceLatencyLevelLowLatency`. ```js // Join the channel using a temporary token rtcEngine.joinChannel(token, channel, uid, { // Set the channel profile to live broadcasting channelProfile: ChannelProfileType.ChannelProfileLiveBroadcasting, // Set the user role to broadcaster; keep the default value for audience role clientRoleType: ClientRoleType.ClientRoleBroadcaster, // Publish audio collected from the microphone publishMicrophoneTrack: true, // Publish video collected from the camera publishCameraTrack: true, // Automatically subscribe to all audio streams autoSubscribeAudio: true, // Automatically subscribe to all video streams autoSubscribeVideo: true, // For live streaming use-case (This setting takes effect only when the user role is set to ClientRoleAudience ) audienceLatencyLevelType: AudienceLatencyLevelType.AudienceLatencyLevelLowLatency, }); ``` ### Register event handlers [#register-event-handlers] Call the `registerEventHandler` method to register the following callback events: * `onJoinChannelSuccess`: Triggered when the local user successfully joins a channel. After joining, call `setupLocalVideo` to configure the local video window. * `onUserJoined`: Triggered when a remote user joins the current channel. After the remote user joins, call `setupRemoteVideo` to configure the remote video window. * `onUserOffline`: Triggered when a remote user leaves the current channel. After the remote user leaves, call `setupRemoteVideo` to close the remote video window. ```js const EventHandles = { // Listen for the local user joining the channel onJoinChannelSuccess: ({ channelId, localUid }, elapsed) => { console.log('Successfully joined channel: ' + channelId); // After the local user joins the channel, set up the local video view rtcEngine.setupLocalVideo({ sourceType: VideoSourceType.VideoSourceCameraPrimary, uid: uid, view: localVideoContainer, setupMode: VideoViewSetupMode.VideoViewSetupAdd, }); }, // Listen for remote users joining the channel onUserJoined: ({ channelId, localUid }, remoteUid, elapsed) => { console.log('Remote user ' + remoteUid + ' joined'); // After a remote user joins the channel, set up the remote video view rtcEngine.setupRemoteVideo( { sourceType: VideoSourceType.VideoSourceRemote, uid: remoteUid, view: remoteVideoContainer, setupMode: VideoViewSetupMode.VideoViewSetupAdd, }, { channelId }, ); }, // Listen for users leaving the channel onUserOffline: ( { channelId, localUid }, remoteUid, reason ) => { console.log('Remote user ' + remoteUid + ' left the channel'); // After a remote user leaves the channel, remove the remote video view rtcEngine.setupRemoteVideo( { sourceType: VideoSourceType.VideoSourceRemote, uid: remoteUid, view: remoteVideoContainer, setupMode: VideoViewSetupMode.VideoViewSetupRemove, }, ); }, }; // Register the event handler rtcEngine.registerEventHandler(EventHandles); ``` To ensure that you receive all Video SDK events, set the Agora Engine event handler before joining a channel. ### Display the local video [#display-the-local-video-5] To display the local video: ```js // Set up the local video view rtcEngine.setupLocalVideo({ sourceType: VideoSourceType.VideoSourceCameraPrimary, uid: uid, view: localVideoContainer, setupMode: VideoViewSetupMode.VideoViewSetupAdd, }); ``` ### Display remote video [#display-remote-video-5] To display the remote video, call `setupRemoteVideo` to configure the remote video feed. ```js // Set up the remote video view rtcEngine.setupRemoteVideo( { sourceType: VideoSourceType.VideoSourceRemote, uid: remoteUid, view: remoteVideoContainer, setupMode: VideoViewSetupMode.VideoViewSetupAdd, }, { channelId }, ); // Remove the remote video view rtcEngine.setupRemoteVideo( { sourceType: VideoSourceType.VideoSourceRemote, uid: remoteUid, view: remoteVideoContainer, setupMode: VideoViewSetupMode.VideoViewSetupRemove, }, ); ``` Call this method when the app receives the `onUserJoined` callback. ### Start and close the app [#start-and-close-the-app-3] When a user launches your app, start Broadcast Streaming. When a user closes the app, stop Broadcast Streaming and release resources. 1. To start Broadcast Streaming, [initialize the engine](#initialize-the-engine), [display the local video](#display-the-local-video) and [join a channel](#join-a-channel). 2. When Broadcast Streaming ends, leave the channel and clean up resources: ```javascript // Stop the preview rtcEngine.stopPreview(); // Leave the channel rtcEngine.leaveChannel(); ``` When you no longer need to interact, unregister the event handler and call `release` to release engine resources. ```javascript // Unregister the event handler rtcEngine.unregisterEventHandler(EventHandles); // Release the engine rtcEngine.release(); ``` After destroying the engine, you can no longer use SDK methods and callbacks. To use the real-time interaction functions again, create a new engine instance. See [Initialize the engine](#initialize-the-engine) for details. ### Complete sample code [#complete-sample-code-5] A complete code sample demonstrating the basic process of real-time interaction is provided for your reference. Replace the entire contents of the `main.js` and `renderer.js` with the following code samples: **`main.js`** ```js const { app, BrowserWindow, systemPreferences } = require("electron"); const path = require("path"); // If using Electron 9.x or later, set allowRendererProcessReuse to false app.allowRendererProcessReuse = false; // Check and request device permissions (macOS only) async function checkAndApplyDeviceAccessPrivilege() { if (process.platform === "darwin") { // Check and request camera permission const cameraPrivilege = systemPreferences.getMediaAccessStatus('camera'); console.log(`Camera privilege before applying: \${cameraPrivilege}`); if (cameraPrivilege !== 'granted') { await systemPreferences.askForMediaAccess('camera'); console.log('Requested camera access from user'); } // Check and request microphone permission const micPrivilege = systemPreferences.getMediaAccessStatus('microphone'); console.log(`Microphone privilege before applying: \${micPrivilege}`); if (micPrivilege !== 'granted') { await systemPreferences.askForMediaAccess('microphone'); console.log('Requested microphone access from user'); } } } function createWindow() { // Create a browser window const mainWindow = new BrowserWindow({ width: 800, height: 600, webPreferences: { // preload: path.join(__dirname, "renderer.js"), // Set nodeIntegration to true and contextIsolation to false nodeIntegration: true, contextIsolation: false, }, }); // Load the content of the index.html file mainWindow.loadFile("./index.html"); // Open Developer Tools mainWindow.webContents.openDevTools(); } // Manage the browser window for the Electron app app.whenReady().then(async () => { await checkAndApplyDeviceAccessPrivilege(); createWindow(); // Create a new window if none are open (macOS specific) app.on("activate", function () { if (BrowserWindow.getAllWindows().length === 0) { createWindow(); } }); }); // Quit the Electron app when all windows are closed (Windows specific) app.on("window-all-closed", function () { if (process.platform !== "darwin") app.quit(); }); ``` **`renderer.js`** ```js const { createAgoraRtcEngine, ChannelProfileType, ClientRoleType, VideoSourceType, VideoViewSetupMode, AudienceLatencyLevelType } = require("agora-electron-sdk"); let rtcEngine; let localVideoContainer; let remoteVideoContainer; // Enter your App ID const APPID = "<-- Insert App Id -->"; // Enter your temporary token let token = "<-- Insert Token -->"; // Enter the channel name used when generating the token const channel = "<-- Insert Channel Name -->"; // User ID, must be unique within the channel let uid = 123; const EventHandles = { // Listen for the local user joining the channel onJoinChannelSuccess: ({ channelId, localUid }, elapsed) => { console.log('Successfully joined channel: ' + channelId); // After the local user joins the channel, set up the local video window rtcEngine.setupLocalVideo({ sourceType: VideoSourceType.VideoSourceCameraPrimary, uid: uid, view: localVideoContainer, setupMode: VideoViewSetupMode.VideoViewSetupAdd, }); }, // Listen for remote users joining the channel onUserJoined: ({ channelId, localUid }, remoteUid, elapsed) => { console.log('Remote user ' + remoteUid + ' joined'); // After the remote user joins the channel, set up the remote video window rtcEngine.setupRemoteVideo( { sourceType: VideoSourceType.VideoSourceRemote, uid: remoteUid, view: remoteVideoContainer, setupMode: VideoViewSetupMode.VideoViewSetupAdd, }, { channelId }, ); }, // Listen for users leaving the channel onUserOffline: ( { channelId, localUid }, remoteUid, reason ) => { console.log('Remote user ' + remoteUid + ' left the channel'); // After the remote user leaves the channel, remove the remote video window rtcEngine.setupRemoteVideo( { sourceType: VideoSourceType.VideoSourceRemote, uid: remoteUid, view: remoteVideoContainer, setupMode: VideoViewSetupMode.VideoViewSetupRemove, }, ); }, }; window.onload = () => { const os = require("os"); const path = require("path"); localVideoContainer = document.getElementById("join-channel-local-video"); remoteVideoContainer = document.getElementById("join-channel-remote-video"); const sdkLogPath = path.resolve(os.homedir(), "./test.log"); // Create an instance of RtcEngine rtcEngine = createAgoraRtcEngine(); // Initialize the RtcEngine instance rtcEngine.initialize({ appId: APPID, logConfig: { filePath: sdkLogPath } }); // Register the event callbacks rtcEngine.registerEventHandler(EventHandles); // Enable the video module rtcEngine.enableVideo(); // Start local video preview rtcEngine.startPreview(); // Join the channel using a temporary token rtcEngine.joinChannel(token, channel, uid, { // Set the channel profile to live broadcasting channelProfile: ChannelProfileType.ChannelProfileLiveBroadcasting, // Set the user role to broadcaster; keep default for audience clientRoleType: ClientRoleType.ClientRoleBroadcaster, // Publish audio captured by the microphone publishMicrophoneTrack: true, // Publish video captured by the camera publishCameraTrack: true, // Automatically subscribe to all audio streams autoSubscribeAudio: true, // Automatically subscribe to all video streams autoSubscribeVideo: true, // For live streaming use-case (This setting takes effect only when the user role is set to ClientRoleAudience ) audienceLatencyLevelType: AudienceLatencyLevelType.AudienceLatencyLevelLowLatency, }); }; ``` * In `renderer.js`, replace the values for `APPID`, `token`, and `channel` with your App ID, the temporary token from Agora Console, and the channel name you used to generate the token. * If you're targeting macOS v10.14 or later, add the device permission code to `main.js`. For more details, see [Get device permissions](#get-device-permissions). To test the complete code, see [Test the sample code](#test-the-sample-code). ### Create a user interface [#create-a-user-interface-5] To connect the sample code to your existing UI, ensure that your `index.html` file includes the HTML elements used to [Display the local video](#display-the-local-video) and [Display remote video](#display-remote-video). Alternatively, use the following sample to create a minimal interface. Add the following code to `index.html`: **Sample code to create the user interface** ```html Electron Quickstart

Electron Quickstart

``` ## Test the sample code [#test-the-sample-code-5] Take the following steps to test the sample code: 1. Update the `APPID`, `token`, and `channel` parameter values in your code. 2. To run the app, execute the following command in the project root directory: ```bash npm start ``` You see a window pop up showing the local video. 3. Invite a friend to run the demo app on a second device. Alternatively, use the [Web demo](https://webdemo.agora.io/basicVideoCall/index.html) to join the same channel. * If users on both devices join the channel as hosts, they can see and hear each other. * If one user joins as host and the other as audience, the host can see themselves in the local video window; the audience can see the host in the remote video window and hear the host. ## Reference [#reference-5] This section contains content that completes the information on this page, or points you to documentation that explains other aspects to this product. * If a firewall is deployed in your network environment, refer to [Connect with Cloud Proxy](build/optimize-quality-and-connection/cloud-proxy.mdx) to use Agora services normally. ### Next steps [#next-steps-5] After implementing the quickstart sample, read the following documents to learn more: * To ensure communication security in a test or production environment, best practice is to obtain and use a token from an authentication server. For details, see [Secure authentication with tokens](build/authenticate-users/use-tokens.mdx). ### Sample project [#sample-project-5] Agora provides open source sample projects on [GitHub](https://github.com/AgoraIO-Extensions/Electron-SDK/tree/main/example/src/renderer/examples) for your reference. Download or view the [JoinChannelVideo](https://github.com/AgoraIO-Extensions/Electron-SDK/blob/main/example/src/renderer/examples/basic/JoinChannelVideo/JoinChannelVideo.tsx) project for a more detailed example. ### API Reference [#api-reference-5] * [`createAgoraRtcEngine`](https://api-ref.agora.io/en/video-sdk/electron/4.x/API/class_irtcengine.html#api_createagorartcengine) * [`joinChannel`](https://api-ref.agora.io/en/video-sdk/electron/4.x/API/class_irtcengine.html#api_irtcengine_joinchannel2) * [`enableVideo`](https://api-ref.agora.io/en/video-sdk/electron/4.x/API/class_irtcengine.html#api_irtcengine_enablevideo) * [`startPreview`](https://api-ref.agora.io/en/video-sdk/electron/4.x/API/class_irtcengine.html#api_irtcengine_startpreview) * [`leaveChannel`](https://api-ref.agora.io/en/video-sdk/electron/4.x/API/class_irtcengine.html#api_irtcengine_leavechannel2) * [`release`](https://api-ref.agora.io/en/video-sdk/electron/4.x/API/class_irtcengine.html#api_irtcengine_release) ### Frequently asked questions [#frequently-asked-questions-4] * [How can I resolve common development issues on Electron?](/en/api-reference/faq/integration/electron_faq) * [How can I fix black screen issues?](/en/api-reference/faq/quality/video_blank) * [Why can't I turn on the camera?](/en/api-reference/faq/quality/video_camera) * [How can I listen for audience joining or leaving a channel?](/en/api-reference/faq/integration/audience_event) * [How can I solve channel-related issues?](/en/api-reference/faq/integration/channel) * [How can I set the log file?](/en/api-reference/faq/integration/set_log_file) ### See also [#see-also-5] * [Error codes](reference/error-codes.md) * [Connection status management](build/optimize-quality-and-connection/connection-status-management.mdx) <_PlatformProcessedMarker close="true" /> <_PlatformPanel platform="flutter"> <_PlatformProcessedMarker groupMode="structured" canonicalPlatform="web" platform="flutter" /> This page provides a step-by-step guide on how to create a basic Broadcast Streaming app using the Agora Video SDK. ## Understand the tech [#understand-the-tech-6] To start a Broadcast Streaming session, implement the following steps in your app: * **Initialize the Agora Engine**: Before calling other APIs, create and initialize an Agora Engine instance. * **Join a channel**: Call methods to create and join a channel. * **Join as a host**: A live streaming event has one or more hosts. A host publishes audio and video to the channel. Hosts can also subscribe to streams from other hosts. * **Join as audience**: Audience members can only subscribe to streams published by hosts. * **Send and receive audio and video**: Hosts publish streams to the channel. Audience members subscribe to audio and video streams published by hosts. ![Streaming workflow](https://assets-docs.agora.io/images/video-sdk/get-started-ils-bs.svg) ## Prerequisites [#prerequisites-6] * [Flutter](https://docs.flutter.dev/get-started/install) 2.10.5 or higher with Dart 2.14.0 or higher. * [Android Studio](https://developer.android.com/studio), IntelliJ, VS Code, or any other IDE that supports Flutter. See [Set up an editor](https://docs.flutter.dev/get-started/editor). * Prepare your development and testing environment according to your [target platform](reference/supported-platforms.md). Run the `flutter doctor` command to confirm that your development environment is set up correctly for Flutter development. * A camera and a microphone * A valid Agora account and project. Please refer to [Agora account management](manage-agora-account.md) for details. ## Set up your project [#set-up-your-project-6] This section shows you how to set up your Flutter project and install the Agora Video SDK. **Create a new project** From the Terminal, run the following commands to create a new project named `agora_project`, or follow the [steps for your IDE](https://docs.flutter.dev/get-started/test-drive?tab=vscode#choose-your-ide): ```bash flutter create agora_project cd agora_project ``` **Add to an existing project** To add Broadcast Streaming to your existing project: 1. Open your Flutter project and navigate to the `lib` folder. 2. Add a new file to the `lib` folder and name it `agora_logic.dart`. ### Install the SDK [#install-the-sdk-6] Install the Agora Video SDK and other dependencies. 1. Add the [latest version](https://pub.dev/packages/agora_rtc_engine) of Agora Video SDK to your Flutter project: ```bash flutter pub add agora_rtc_engine ``` 2. Add the permission processing package: ```bash flutter pub add permission_handler ``` The `dependencies` in your `pubspec.yaml` file should look like the following: ```yaml dependencies: flutter: sdk: flutter agora_rtc_engine: ^6.5.0 # Agora Flutter SDK, please use the latest version permission_handler: ^11.3.1 # Package for managing runtime permissions cupertino_icons: ^1.0.8 ``` 3. Install the dependencies. Execute the following command in the project path: ```bash flutter pub get ``` ## Implement Broadcast Streaming [#implement-broadcast-streaming-6] This section guides you through the implementation of basic real-time audio and video interaction in your app. The following figure illustrates the essential steps: ![Quick start sequence](https://assets-docs.agora.io/images/video-sdk/quick-start-sequence.svg) This guide includes [complete sample code](#complete-sample-code) that demonstrates implementing basic real-time interaction. To understand the core API calls in the sample code, review the following implementation steps. ### Import packages [#import-packages] Import the following packages in your dart file. ```dart import 'dart:async'; import 'package:agora_rtc_engine/agora_rtc_engine.dart'; import 'package:flutter/material.dart'; import 'package:permission_handler/permission_handler.dart'; ``` ### Initialize the engine [#initialize-the-engine-5] For real-time communication, initialize an `RtcEngine` instance. Use `RtcEngineContext` to specify the [App ID](manage-agora-account.md), and other configuration parameters. In your dart file, add the following code: ```dart // Set up the Agora RTC engine instance Future _initializeAgoraVideoSDK() async { _engine = createAgoraRtcEngine(); await _engine.initialize(const RtcEngineContext( appId: appId, channelProfile: ChannelProfileType.channelProfileLiveBroadcasting, )); } ``` ### Join a channel [#join-a-channel-6] To join a channel, call `joinChannel` with the following parameters: * **Channel name**: The name of the channel to join. Clients that pass the same channel name join the same channel. If a channel with the specified name does not exist, it is created when the first user joins. * **Authentication token**: A dynamic key that authenticates a user when the client joins a channel. In a production environment, you obtain a token from a [token server](build/authenticate-users/deploy-token-server.mdx) in your security infrastructure. For the purpose of this guide [Generate a temporary token](manage-agora-account.md). * **User ID**: A 32-bit signed integer that identifies a user in the channel. You can specify a unique user ID for each user yourself. If you set the user ID to `0` when joining a channel, the SDK generates a random number for the user ID and returns the value in the `onJoinChannelSuccess` callback. * **Channel media options**: Configure `ChannelMediaOptions` to define publishing and subscription settings, optimize performance for your specific use-case, and set optional parameters. For Broadcast Streaming, set the `clientRoleType` to `clientRoleBroadcaster` (host) or `clientRoleAudience`, and the `audienceLatencyLevel` to `audienceLatencyLevelLowLatency`. ```dart // Join a channel Future _joinChannel() async { await _engine.joinChannel( token: token, channelId: channel, options: const ChannelMediaOptions( autoSubscribeVideo: true, // Automatically subscribe to all video streams autoSubscribeAudio: true, // Automatically subscribe to all audio streams publishCameraTrack: true, // Publish camera-captured video publishMicrophoneTrack: true, // Publish microphone-captured audio // Use clientRoleBroadcaster to act as a host or clientRoleAudience for audience clientRoleType: ClientRoleType.clientRoleBroadcaster, // Set the audience latency level audienceLatencyLevel: AudienceLatencyLevelType.audienceLatencyLevelLowLatency ), uid: 0, ); } ``` ### Subscribe to Video SDK events [#subscribe-to-video-sdk-events-4] The SDK provides the `RtcEngineEventHandler` for subscribing to channel events. To use it, pass an instance of `RtcEngineEventHandler` to `registerEventHandler` and implement the event methods you want to handle. Call `registerEventHandler` to bind the event handler to the SDK. ```dart // Register an event handler for Agora RTC void _setupEventHandlers() { _engine.registerEventHandler( RtcEngineEventHandler( onJoinChannelSuccess: (RtcConnection connection, int elapsed) { debugPrint("Local user ${connection.localUid} joined"); setState(() => _localUserJoined = true); }, onUserJoined: (RtcConnection connection, int remoteUid, int elapsed) { debugPrint("Remote user $remoteUid joined"); setState(() => _remoteUid = remoteUid); }, onUserOffline: (RtcConnection connection, int remoteUid, UserOfflineReasonType reason) { debugPrint("Remote user $remoteUid left"); setState(() => _remoteUid = null); }, ), ); } ``` When a remote user joins the channel, the `onUserJoined` callback is triggered. Use the remote user's `uid` returned in the callback, to create an `AgoraVideoView` control for displaying the video stream from the remote user. To ensure that you receive all Video SDK events, register the event handler before joining a channel. ### Display the local video [#display-the-local-video-6] To display the local video, enable the video module by calling `enableVideo`, then start the local video preview with `startPreview`. ```dart Future _setupLocalVideo() async { // The video module and preview are disabled by default. await _engine.enableVideo(); await _engine.startPreview(); } ``` To render the local video, add the following widget inside your UI’s widget tree, such as in the build method of your `StatefulWidget`: ```dart // Displays the local user's video view using the Agora engine. Widget _localVideo() { return AgoraVideoView( controller: VideoViewController( rtcEngine: _engine, // Uses the Agora engine instance canvas: const VideoCanvas( uid: 0, // Specifies the local user renderMode: RenderModeType.renderModeHidden, // Sets the video rendering mode ), ), ); } ``` ### Display remote video [#display-remote-video-6] To render a remote video, add the following widget inside your UI’s widget tree, such as in the build method of your `StatefulWidget`: ```dart // If a remote user has joined, render their video, else display a waiting message Widget _remoteVideo() { if (_remoteUid != null) { return AgoraVideoView( controller: VideoViewController.remote( rtcEngine: _engine, // Uses the Agora engine instance canvas: VideoCanvas(uid: _remoteUid), // Binds the remote user's video connection: const RtcConnection(channelId: channel), // Specifies the channel ), ); } else { return const Text( 'Waiting for remote user to join...', textAlign: TextAlign.center, ); } } ``` ### Handle permissions [#handle-permissions-4] Request microphone and camera permissions for Broadcast Streaming. ```dart Future _requestPermissions() async { await [Permission.microphone, Permission.camera].request(); } ``` If your target platform is iOS or macOS, add the microphone and camera permission declarations required for real-time interaction to [`Info.plist`](https://help.apple.com/xcode/mac/current/#/dev3f399a2a6). | **Device** | **Key** | **Value** | | :--------- | :------------------------------------- | :-------------- | | Microphone | Privacy - Microphone Usage Description | for audio calls | | Camera | Privacy - Camera Usage Description | for video calls | ### Start and close the app [#start-and-close-the-app-4] To start Broadcast Streaming, request microphone and camera permissions, initialize the Agora SDK instance, set up event handlers, join a channel, and display the local video. ```dart await _requestPermissions(); await _initializeAgoraVideoSDK(); await _setupLocalVideo(); _setupEventHandlers(); await _joinChannel(); ``` To stop Broadcast Streaming, leave the channel and release the engine instance. ```dart // Leaves the channel and releases resources Future _cleanupAgoraEngine() async { await _engine.leaveChannel(); await _engine.release(); } ``` After you call `release`, you no longer have access to the methods and callbacks of the SDK. To use Broadcast Streaming features again, create a new engine instance. ### Complete sample code [#complete-sample-code-6] A complete code sample demonstrating the basic process of real-time interaction is provided for your reference. To use the sample code, copy the following code to replace the entire contents of the `.dart` file in your project. ```dart import 'dart:async'; import 'package:agora_rtc_engine/agora_rtc_engine.dart'; import 'package:flutter/material.dart'; import 'package:permission_handler/permission_handler.dart'; void main() => runApp(const MyApp()); // Fill in the app ID obtained from Agora console const appId = "<-- Insert app Id -->"; // Fill in the temporary token generated from Agora Console const token = "<-- Insert token -->"; // Fill in the channel name you used to generate the token const channel = "<-- Insert channel name -->"; // Main App Widget class MyApp extends StatelessWidget { const MyApp({Key? key}) : super(key: key); @override Widget build(BuildContext context) { return const MaterialApp( home: _MainScreen(), ); } } // Video Call Screen Widget class _MainScreen extends StatefulWidget { const _MainScreen({Key? key}) : super(key: key); @override _MainScreenScreenState createState() => _MainScreenScreenState(); } class _MainScreenScreenState extends State<_MainScreen> { int? _remoteUid; // Stores remote user ID bool _localUserJoined = false; // Indicates if local user has joined the channel late RtcEngine _engine; // Stores Agora RTC Engine instance @override void initState() { super.initState(); _startBroadcastStreaming(); } // Initializes Agora SDK Future _startBroadcastStreaming() async { await _requestPermissions(); await _initializeAgoraVideoSDK(); await _setupLocalVideo(); _setupEventHandlers(); await _joinChannel(); } // Requests microphone and camera permissions Future _requestPermissions() async { await [Permission.microphone, Permission.camera].request(); } // Set up the Agora RTC engine instance Future _initializeAgoraVideoSDK() async { _engine = createAgoraRtcEngine(); await _engine.initialize(const RtcEngineContext( appId: appId, channelProfile: ChannelProfileType.channelProfileLiveBroadcasting, )); } // Enables and starts local video preview Future _setupLocalVideo() async { await _engine.enableVideo(); await _engine.startPreview(); } // Register an event handler for Agora RTC void _setupEventHandlers() { _engine.registerEventHandler( RtcEngineEventHandler( onJoinChannelSuccess: (RtcConnection connection, int elapsed) { debugPrint("Local user \${connection.localUid} joined"); setState(() => _localUserJoined = true); }, onUserJoined: (RtcConnection connection, int remoteUid, int elapsed) { debugPrint("Remote user \$remoteUid joined"); setState(() => _remoteUid = remoteUid); }, onUserOffline: (RtcConnection connection, int remoteUid, UserOfflineReasonType reason) { debugPrint("Remote user \$remoteUid left"); setState(() => _remoteUid = null); }, ), ); } // Join a channel Future _joinChannel() async { await _engine.joinChannel( token: token, channelId: channel, options: const ChannelMediaOptions( autoSubscribeVideo: true, autoSubscribeAudio: true, publishCameraTrack: true, publishMicrophoneTrack: true, clientRoleType: ClientRoleType.clientRoleBroadcaster, audienceLatencyLevel: AudienceLatencyLevelType.audienceLatencyLevelLowLatency ), uid: 0, ); } @override void dispose() { _cleanupAgoraEngine(); super.dispose(); } // Leaves the channel and releases resources Future _cleanupAgoraEngine() async { await _engine.leaveChannel(); await _engine.release(); } @override Widget build(BuildContext context) { return Scaffold( appBar: AppBar(title: const Text('Agora Broadcast Streaming')), body: Stack( children: [ Center(child: _remoteVideo()), Align( alignment: Alignment.topLeft, child: SizedBox( width: 100, height: 150, child: Center( child: _localUserJoined ? _localVideo() : const CircularProgressIndicator(), ), ), ), ], ), ); } // Displays remote video view Widget _localVideo() { return AgoraVideoView( controller: VideoViewController( rtcEngine: _engine, canvas: const VideoCanvas( uid: 0, renderMode: RenderModeType.renderModeHidden, ), ), ); } // Displays remote video view Widget _remoteVideo() { if (_remoteUid != null) { return AgoraVideoView( controller: VideoViewController.remote( rtcEngine: _engine, canvas: VideoCanvas(uid: _remoteUid), connection: const RtcConnection(channelId: channel), ), ); } else { return const Text( 'Waiting for remote user to join...', textAlign: TextAlign.center, ); } } } ``` In the `appId` and `token` fields, enter the corresponding values you obtained from Agora Console. Use the same `channel` name you filled in when generating the temporary token. ### Create a user interface [#create-a-user-interface-6] To connect the sample code to your existing UI, ensure that your widget tree includes the `_remoteVideo` and `_localVideo` widgets used to [Display the local video](#display-the-local-video) and [Display remote video](#display-remote-video). Alternatively, use the following sample code to generate a basic user interface: **Sample code to create the user interface** ```dart // Build UI to display local video and remote video @override Widget build(BuildContext context) { return Scaffold( appBar: AppBar(title: const Text('Agora Video Call')), body: Stack( children: [ Center(child: _remoteVideo()), Align( alignment: Alignment.topLeft, child: SizedBox( width: 100, height: 150, child: Center( child: _localUserJoined ? _localVideo() : const CircularProgressIndicator(), ), ), ), ], ), ); } ``` ## Test the sample code [#test-the-sample-code-6] Take the following steps to test the sample code: 1. In `main.dart` update the values for `appId`, and `token` with values from Agora Console. Fill in the same `channel` name you used to generate the token. 2. Connect a target device to your development computer. 3. Open Terminal and execute the following command in the project folder to run the sample project: ```bash flutter run ``` 4. Launch the App, grant microphone and camera permissions. If you set the user role to host, you see yourself in the local view. 5. On a second target device, repeat the previous steps to install and launch the app. Alternatively, use the [Web demo](https://webdemo.agora.io/basicVideoCall/index.html) to join the same channel and test the following use-cases: * If users on both devices join the channel as hosts, they can see and hear each other. * If one user joins as host and the other as audience, the host can see themselves in the local video window; the audience can see the host in the remote video window and hear the host. On an Android device, the app UI appears similar to the following: ![quickstart-ui-flutter.png](https://assets-docs.agora.io/images/video-sdk/quickstart-ui-flutter.png) ## Reference [#reference-6] This section contains content that completes the information on this page, or points you to documentation that explains other aspects to this product. * If a firewall is deployed in your network environment, refer to [Connect with Cloud Proxy](build/optimize-quality-and-connection/cloud-proxy.mdx) to use Agora services normally. ### Next steps [#next-steps-6] After implementing the quickstart sample, read the following documents to learn more: * To ensure communication security in a test or production environment, best practice is to obtain and use a token from an authentication server. For details, see [Secure authentication with tokens](build/authenticate-users/use-tokens.mdx). ### Sample project [#sample-project-6] Agora provides open source sample projects on [GitHub](https://github.com/AgoraIO-Extensions/Agora-Flutter-SDK/tree/main/example/lib/examples) for your reference. Download or view [join\_channel\_video.dart](https://github.com/AgoraIO-Extensions/Agora-Flutter-SDK/tree/main/example/lib/examples/basic/join_channel_video) for a more detailed example. ### API reference [#api-reference-6] * [`enableVideo`](https://api-ref.agora.io/en/video-sdk/flutter/6.x/API/class_irtcengine.html#api_irtcengine_enablevideo) * [`registerEventHandler`](https://api-ref.agora.io/en/video-sdk/flutter/6.x/API/class_irtcengine.html#api_irtcengine_addhandler) * [`setClientRole`](https://api-ref.agora.io/en/video-sdk/flutter/6.x/API/class_irtcengine.html#api_irtcengine_setclientrole2) * [`setChannelProfile`](https://api-ref.agora.io/en/video-sdk/flutter/6.x/API/class_irtcengine.html#api_irtcengine_setchannelprofile) * [`startPreview`](https://api-ref.agora.io/en/video-sdk/flutter/6.x/API/class_irtcengine.html#api_irtcengine_startpreview2) * [`AgoraVideoView`](https://api-ref.agora.io/en/video-sdk/flutter/6.x/API/class_agoravideoview.html) * [`joinChannel`](https://api-ref.agora.io/en/video-sdk/flutter/6.x/API/class_irtcengine.html#api_irtcengine_joinchannel2) * [`leaveChannel`](https://api-ref.agora.io/en/video-sdk/flutter/6.x/API/class_irtcengine.html#api_irtcengine_leavechannel2) ### Frequently asked questions [#frequently-asked-questions-5] * [How can I fix black screen issues?](/en/api-reference/faq/quality/video_blank) * [Why can't I turn on the camera?](/en/api-reference/faq/quality/video_camera) * [How can I listen for audience joining or leaving a channel?](/en/api-reference/faq/integration/audience_event) * [How can I solve channel-related issues?](/en/api-reference/faq/integration/channel) * [How can I set the log file?](/en/api-reference/faq/integration/set_log_file) ### See also [#see-also-6] * [Error codes](reference/error-codes.md) * [Connection status management](build/optimize-quality-and-connection/connection-status-management.mdx) <_PlatformProcessedMarker close="true" /> <_PlatformPanel platform="react-native"> <_PlatformProcessedMarker groupMode="structured" canonicalPlatform="web" platform="react-native" /> This page provides a step-by-step guide on how to create a basic Broadcast Streaming app using the Agora Video SDK. ## Understand the tech [#understand-the-tech-7] To start a Broadcast Streaming session, implement the following steps in your app: * **Initialize the Agora Engine**: Before calling other APIs, create and initialize an Agora Engine instance. * **Join a channel**: Call methods to create and join a channel. * **Join as a host**: A live streaming event has one or more hosts. A host publishes audio and video to the channel. Hosts can also subscribe to streams from other hosts. * **Join as audience**: Audience members can only subscribe to streams published by hosts. * **Send and receive audio and video**: Hosts publish streams to the channel. Audience members subscribe to audio and video streams published by hosts. ![Streaming workflow](https://assets-docs.agora.io/images/video-sdk/get-started-ils-bs.svg) ## Prerequisites [#prerequisites-7] * React Native 0.60 or later. For more information, see [Get Started with React Native](https://reactnative.dev/docs/environment-setup). * Node 16 or later For your target platform, prepare the following: Android iOS * A machine running macOS, Windows, or Linux * [Java Development Kit (JDK) 11](https://openjdk.org/projects/jdk/11/) or later * Latest version of Android Studio * A physical or virtual mobile device running Android 5.0 or later * A machine running macOS * Xcode 10 or later * CocoaPods * A physical or virtual mobile device running iOS 9.0 or later. If you use React Native 0.63 or later, ensure your iOS version is 10.0 or later. * A camera and a microphone * A valid Agora account and project. Please refer to [Agora account management](manage-agora-account.md) for details. ## Set up your project [#set-up-your-project-7] This section shows you how to set up your React Native project and install the Agora Video SDK. ### Create a project [#create-a-project] To create a new application using [React Native Community CLI](https://github.com/react-native-community/cli), see [Get Started Without a Framework](https://reactnative.dev/docs/getting-started-without-a-framework). ### Install the SDK [#install-the-sdk-7] Download and add the Agora Video SDK to your React Native project. Choose either of the following methods: **npm** In your project folder, execute the following command: ```bash npm i --save react-native-agora ``` **yarn** In your project folder, execute the following commands: ```bash # Install yarn npm install -g yarn # Use yarn to download Agora React Native SDK yarn add react-native-agora ``` React Native 0.60.0 and later support automatic linking of native modules. Manual linking is not recommended. See [Autolinking](https://github.com/react-native-community/cli/blob/master/docs/autolinking.md) for details ## Implement Broadcast Streaming [#implement-broadcast-streaming-7] This section guides you through the implementation of basic real-time audio and video interaction in your app. The following figure illustrates the essential steps: ![Quick start sequence](https://assets-docs.agora.io/images/video-sdk/quick-start-sequence.svg) This guide includes [complete sample code](#complete-sample-code) that demonstrates implementing basic real-time interaction. To understand the core API calls in the sample code, review the following implementation steps and use the code in your `App.tsx` file. ### Import dependencies [#import-dependencies-1] Add the required Agora dependencies to your app. ```tsx import { createAgoraRtcEngine, ChannelProfileType, ClientRoleType, AudienceLatencyLevelType, IRtcEngine, RtcSurfaceView, RtcConnection, IRtcEngineEventHandler, } from 'react-native-agora'; ``` ### Initialize the engine [#initialize-the-engine-6] For real-time communication, call `createAgoraRtcEngine` to create an `RtcEngine` instance, and then `initialize` the engine using `RtcEngineContext` to specify the [App ID](manage-agora-account.md). ```tsx const appId = '<-- Insert app ID -->'; const setupVideoSDKEngine = async () => { if (Platform.OS === 'android') { await getPermission(); } agoraEngineRef.current = createAgoraRtcEngine(); const agoraEngine = agoraEngineRef.current; await agoraEngine.initialize({ appId: appId }); }; ``` ### Join a channel [#join-a-channel-7] To join a channel, call `joinChannel` with the following parameters: * **Channel name**: The name of the channel to join. Clients that pass the same channel name join the same channel. If a channel with the specified name does not exist, it is created when the first user joins. * **Authentication token**: A dynamic key that authenticates a user when the client joins a channel. In a production environment, you obtain a token from a [token server](build/authenticate-users/deploy-token-server.mdx) in your security infrastructure. For the purpose of this guide [Generate a temporary token](manage-agora-account.md). * **User ID**: A 32-bit signed integer that identifies a user in the channel. You can specify a unique user ID for each user yourself. If you set the user ID to `0` when joining a channel, the SDK generates a random number for the user ID and returns the value in the `onJoinChannelSuccess` callback. * **Channel media options**: Configure `ChannelMediaOptions` to define publishing and subscription settings, optimize performance for your specific use-case, and set optional parameters. For Broadcast Streaming, set the `channelProfile` to `ChannelProfileLiveBroadcasting`, the `clientRoleType` to `ClientRoleBroadcaster` (host) or `ClientRoleAudience`, and the `audienceLatencyLevel` to `audienceLatencyLevelLowLatency`. ```tsx const token = '<-- Insert token -->'; const channelName = '<-- Insert channel name -->'; const localUid = 0; // Local user UID, no need to modify // Define the join method called after clicking the join channel button const join = async () => { if (isJoined) { return; } if (isHost) { // Join the channel as a broadcaster agoraEngineRef.current?.joinChannel(token, channelName, localUid, { // Set channel profile to live broadcast channelProfile: ChannelProfileType.ChannelProfileLiveBroadcasting, // Set user role to broadcaster clientRoleType: ClientRoleType.ClientRoleBroadcaster, // Publish audio collected by the microphone publishMicrophoneTrack: true, // Publish video collected by the camera publishCameraTrack: true, // Automatically subscribe to all audio streams autoSubscribeAudio: true, // Automatically subscribe to all video streams autoSubscribeVideo: true, }); } else { // Join the channel as an audience agoraEngineRef.current?.joinChannel(token, channelName, localUid, { // Set channel profile to live broadcast channelProfile: ChannelProfileType.ChannelProfileLiveBroadcasting, // Set user role to audience clientRoleType: ClientRoleType.ClientRoleAudience, // Do not publish audio collected by the microphone publishMicrophoneTrack: false, // Do not publish video collected by the camera publishCameraTrack: false, // Automatically subscribe to all audio streams autoSubscribeAudio: true, // Automatically subscribe to all video streams autoSubscribeVideo: true,', // Change the delay level of the audience to achieve ultra-fast live broadcast audienceLatencyLevel: AudienceLatencyLevelType.AudienceLatencyLevelLowLatency, }); } }; ``` Since the Agora Engine runs locally, reloading the app using the Metro bundler may cause the engine reference to be lost, preventing channel joining. To resolve this issue, restart the app. ### Subscribe to Video SDK events [#subscribe-to-video-sdk-events-5] The Video SDK provides an interface for subscribing to channel events. To use it, create an instance of `IRtcEngineEventHandler` and implement the event methods you want to handle. Call `registerEventHandler` to register your custom event handler. ```tsx const [remoteUid, setRsemoteUid] = useState(0); // Uid of the remote user const setupEventHandler = () => { eventHandler.current = { // Triggered when the local user successfully joins a channel onJoinChannelSuccess: () => { setMessage('Successfully joined channel: ' + channelName); setupLocalVideo(); setIsJoined(true); }, // Triggered when a remote user joins the channel onUserJoined: (_connection: RtcConnection, uid: number) => { setMessage('Remote user ' + uid + ' joined'); setRemoteUid(uid); }, // Triggered when a remote user leaves the channel onUserOffline: (_connection: RtcConnection, uid: number) => { setMessage('Remote user ' + uid + ' left the channel'); setRemoteUid(uid); }, }; // Register the event handler agoraEngineRef.current?.registerEventHandler(eventHandler.current); }; ``` To ensure that you receive all Video SDK events, register the event handler before joining a channel. ### Enable the video module [#enable-the-video-module-5] Call `enableVideo` to activate the video module, and use `startPreview` to enable the local video preview. ```tsx const setupLocalVideo = () => { agoraEngineRef.current?.enableVideo(); agoraEngineRef.current?.startPreview(); }; ``` ### Display the local video [#display-the-local-video-7] To display the local user's video stream, use the `RtcSurfaceView` component. Set the `uid` to `0` and `style` the video container. ```tsx ``` ### Display remote video [#display-remote-video-7] To display the remote user's video, use the `RtcSurfaceView` component. Pass the `remoteUid` of the remote user to the component and set the desired `style` for the video container. ```tsx ``` ### Handle permissions [#handle-permissions-5] To access the camera and microphone, ensure that the user grants the necessary permissions when the app starts. Android iOS On Android devices, pop up a prompt box to obtain permission to use the camera and microphone. ```tsx // Import components related to Android device permissions import {PermissionsAndroid, Platform} from 'react-native'; const getPermission = async () => { if (Platform.OS === 'android') { await PermissionsAndroid.requestMultiple([ PermissionsAndroid.PERMISSIONS.RECORD_AUDIO, PermissionsAndroid.PERMISSIONS.CAMERA, ]); } }; ``` In Xcode, open the `info.plist` file and add the following content to the list on the right to obtain the corresponding device permissions: | Key | Type | Value | | -------------------------------------- | ------ | ------------------------------------------------------------------------------------------- | | Privacy - Microphone Usage Description | String | The purpose of using the microphone, for example, for a call or live interactive streaming. | | Privacy - Camera Usage Description | String | The purpose of using the camera, for example, for a call or live interactive streaming. | ### Leave the channel [#leave-the-channel-2] To exit a Broadcast Streaming channel, call `leaveChannel`. ```tsx const leave = () => { // Leave the channel agoraEngineRef.current?.leaveChannel(); setIsJoined(false); }; ``` ### Clean up resources [#clean-up-resources] Before closing your app, unregister the event handler and call `release` to free up resources. ```tsx const cleanupAgoraEngine = () => { return () => { agoraEngineRef.current?.unregisterEventHandler(eventHandler.current!); agoraEngineRef.current?.release(); }; }; ``` After calling `release,` methods and callbacks of the SDK are no longer available. To use the SDK functions again, create a new engine instance. ### Complete sample code [#complete-sample-code-7] A complete code sample demonstrating the basic process of real-time interaction is provided for your reference. To use the sample code, copy the code into `ProjectName/App.tsx` to quickly implement the basic functionality. ```tsx // Import React Hooks import React, { useRef, useState, useEffect } from 'react'; // Import user interface elements import { SafeAreaView, ScrollView, StyleSheet, Text, View, Switch, } from 'react-native'; // Import components related to obtaining Android device permissions import { PermissionsAndroid, Platform } from 'react-native'; // Import Agora SDK import { createAgoraRtcEngine, ChannelProfileType, ClientRoleType, IRtcEngine, RtcSurfaceView, RtcConnection, IRtcEngineEventHandler, VideoSourceType, } from 'react-native-agora'; // Define basic information const appId = '<-- Insert App ID -->'; const token = '<-- Insert Token -->'; const channelName = '<-- Insert Channel Name -->'; const localUid = 0; // Local user Uid, no need to modify const App = () => { const agoraEngineRef = useRef(); // IRtcEngine instance const [isJoined, setIsJoined] = useState(false); // Whether the local user has joined the channel const [isHost, setIsHost] = useState(true); // User role const [remoteUid, setRemoteUid] = useState(0); // Uid of the remote user const [message, setMessage] = useState(''); // User prompt message const eventHandler = useRef(); // Implement callback functions useEffect(() => { const init = async () => { await setupVideoSDKEngine(); setupEventHandler(); }; init(); return () => { cleanupAgoraEngine(); // Ensure this is synchronous }; }, []); // Empty dependency array ensures it runs only once const setupEventHandler = () => { eventHandler.current = { onJoinChannelSuccess: () => { setMessage('Successfully joined channel: ' + channelName); setupLocalVideo(); setIsJoined(true); }, onUserJoined: (_connection: RtcConnection, uid: number) => { setMessage('Remote user ' + uid + ' joined'); setRemoteUid(uid); }, onUserOffline: (_connection: RtcConnection, uid: number) => { setMessage('Remote user ' + uid + ' left the channel'); setRemoteUid(uid); }, }; agoraEngineRef.current?.registerEventHandler(eventHandler.current); }; const setupVideoSDKEngine = async () => { try { if (Platform.OS === 'android') { await getPermission(); } agoraEngineRef.current = createAgoraRtcEngine(); const agoraEngine = agoraEngineRef.current; await agoraEngine.initialize({ appId: appId }); } catch (e) { console.error(e); } }; const setupLocalVideo = () => { agoraEngineRef.current?.enableVideo(); agoraEngineRef.current?.startPreview(); }; // Define the join method called after clicking the join channel button const join = async () => { if (isJoined) { return; } try { if (isHost) { // Join the channel as a broadcaster agoraEngineRef.current?.joinChannel(token, channelName, localUid, { // Set channel profile to live broadcast channelProfile: ChannelProfileType.ChannelProfileLiveBroadcasting, // Set user role to broadcaster clientRoleType: ClientRoleType.ClientRoleBroadcaster, // Publish audio collected by the microphone publishMicrophoneTrack: true, // Publish video collected by the camera publishCameraTrack: true, // Automatically subscribe to all audio streams autoSubscribeAudio: true, // Automatically subscribe to all video streams autoSubscribeVideo: true, }); } else { // Join the channel as an audience agoraEngineRef.current?.joinChannel(token, channelName, localUid, { // Set channel profile to live broadcast channelProfile: ChannelProfileType.ChannelProfileLiveBroadcasting, // Set user role to audience clientRoleType: ClientRoleType.ClientRoleAudience, // Do not publish audio collected by the microphone publishMicrophoneTrack: false, // Do not publish video collected by the camera publishCameraTrack: false, // Automatically subscribe to all audio streams autoSubscribeAudio: true, // Automatically subscribe to all video streams autoSubscribeVideo: true,', // Change the delay level of the audience to achieve ultra-fast live broadcast audienceLatencyLevel: AudienceLatencyLevelType.AudienceLatencyLevelLowLatency, }); } } catch (e) { console.log(e); } }; // Define the leave method called after clicking the leave channel button const leave = () => { try { // Call leaveChannel method to leave the channel agoraEngineRef.current?.leaveChannel(); setRemoteUid(0); setIsJoined(false); showMessage('Left the channel'); } catch (e) { console.log(e); } }; const cleanupAgoraEngine = () => { return () => { agoraEngineRef.current?.unregisterEventHandler(eventHandler.current!); agoraEngineRef.current?.release(); }; }; // Render user interface return ( Agora Video SDK Quickstart Join Channel Leave Channel Audience { setIsHost(switchValue); if (isJoined) { leave(); } }} value={isHost} /> Host {isJoined && isHost ? ( Local user uid: {localUid} ) : ( Join a channel )} {isJoined && remoteUid !== 0 ? ( Remote user uid: {remoteUid} ) : ( {isJoined && !isHost ? 'Waiting for remote user to join' : ''} )} {message} ); // Display information function showMessage(msg: string) { setMessage(msg); } }; // Define user interface styles const styles = StyleSheet.create({ button: { paddingHorizontal: 25, paddingVertical: 4, fontWeight: 'bold', color: '#ffffff', backgroundColor: '#0055cc', margin: 5, }, main: { flex: 1, alignItems: 'center' }, scroll: { flex: 1, backgroundColor: '#ddeeff', width: '100%' }, scrollContainer: { alignItems: 'center' }, videoView: { width: '90%', height: 200 }, btnContainer: { flexDirection: 'row', justifyContent: 'center' }, head: { fontSize: 20 }, info: { backgroundColor: '#ffffe0', paddingHorizontal: 8, color: '#0000ff' }, }); const getPermission = async () => { if (Platform.OS === 'android') { await PermissionsAndroid.requestMultiple([ PermissionsAndroid.PERMISSIONS.RECORD_AUDIO, PermissionsAndroid.PERMISSIONS.CAMERA, ]); } }; export default App; ``` ### Create a user interface [#create-a-user-interface-7] Design a user interface for your project based on your application use-case. A basic interface consists of two view frames to display local and remote videos. Add **Join channel** and **Leave channel** buttons to enable the user to join and leave a channel. To create such an interface, refer to the following code: **Sample code to create the user interface** ```tsx import React, { useState } from 'react'; // Import user interface elements import { SafeAreaView, ScrollView, StyleSheet, Text, View, Switch, } from 'react-native'; const localUid = 0; // Local user Uid, no need to modify const App = () => { const [isJoined, setIsJoined] = useState(false); // Whether the local user has joined the channel const [isHost, setIsHost] = useState(true); // User role const [remoteUid, setRemoteUid] = useState(0); // Uid of the remote user const [message, setMessage] = useState(''); // User prompt message // Render user interface return ( Agora Video SDK Quickstart Join Channel Leave Channel Audience { setIsHost(switchValue); if (isJoined) { leave(); } }} value={isHost} /> Host {isJoined && isHost ? ( Local user uid: {localUid} ) : ( Join a channel )} {isJoined && remoteUid !== 0 ? ( Remote user uid: {remoteUid} ) : ( {isJoined && !isHost ? 'Waiting for remote user to join' : ''} )} {message} ); }; // Define user interface styles const styles = StyleSheet.create({ button: { paddingHorizontal: 25, paddingVertical: 4, fontWeight: 'bold', color: '#ffffff', backgroundColor: '#0055cc', margin: 5, }, main: { flex: 1, alignItems: 'center' }, scroll: { flex: 1, backgroundColor: '#ddeeff', width: '100%' }, scrollContainer: { alignItems: 'center' }, videoView: { width: '90%', height: 200 }, btnContainer: { flexDirection: 'row', justifyContent: 'center' }, head: { fontSize: 20 }, info: { backgroundColor: '#ffffe0', paddingHorizontal: 8, color: '#0000ff' }, }); export default App; ``` ## Test the sample code [#test-the-sample-code-7] Take the following steps to test the sample code: 1. Update the values for `appId`, `token`, and `channelName` in your code. 2. Run the app. Some simulators may not support all the functions of this project. Best practice is to run the project on a physical device. Android iOS To run the app on an Android device: 1. Turn on the developer options of the Android device and connect the Android device to the computer through a USB cable. 2. In the project root directory, execute `npx react-native run-android`. To run on the app on an iOS device: 1. Open the `ProjectName/ios/ProjectName.xcworkspace` folder using Xcode. 2. Connect the iOS device to your computer through a USB cable. 3. In Xcode, click the **Build and Run** button. For detailed steps on running the app on a real Android or iOS device, please refer to [Running On Device](https://reactnative.dev/docs/running-on-device). 1. Launch the app and click the **Join channel** button to join a channel. 2. Invite a friend to install and run the app on a second device. Alternatively, use the [Web demo](https://webdemo.agora.io/basicVideoCall/index.html) to join the same channel. Once your friend joins the channel, you can see and hear each other. ## Reference [#reference-7] This section contains content that completes the information on this page, or points you to documentation that explains other aspects to this product. * If a firewall is deployed in your network environment, refer to [Connect with Cloud Proxy](build/optimize-quality-and-connection/cloud-proxy.mdx) to use Agora services normally. ### Next steps [#next-steps-7] After implementing the quickstart sample, read the following documents to learn more: * To ensure communication security in a test or production environment, best practice is to obtain and use a token from an authentication server. For details, see [Secure authentication with tokens](build/authenticate-users/use-tokens.mdx). ### Sample project [#sample-project-7] Agora provides open source sample projects on [GitHub](https://github.com/AgoraIO-Extensions/react-native-agora/blob/main/example/src/examples) for your reference. Download or view the [JoinChannelVideo](https://github.com/AgoraIO-Extensions/react-native-agora/blob/main/example/src/examples/basic/JoinChannelVideo) project for a more detailed example. ### API reference [#api-reference-7] * [`createAgoraRtcEngine`](https://api-ref.agora.io/en/video-sdk/react-native/4.x/API/class_irtcengine.html#api_createagorartcengine) * [`initialize`](https://api-ref.agora.io/en/video-sdk/react-native/4.x/API/class_irtcengine.html#api_irtcengine_initialize) * [`registerEventHandler`](https://api-ref.agora.io/en/video-sdk/react-native/4.x/API/class_irtcengine.html#api_irtcengine_addhandler) * [`setChannelProfile`](https://api-ref.agora.io/en/video-sdk/react-native/4.x/API/class_irtcengine.html#api_irtcengine_setchannelprofile) * [`joinChannel`](https://api-ref.agora.io/en/video-sdk/react-native/4.x/API/class_irtcengine.html#api_irtcengine_joinchannel2) * [`leaveChannel`](https://api-ref.agora.io/en/video-sdk/react-native/4.x/API/class_irtcengine.html#api_irtcengine_leavechannel2) ### Frequently asked questions [#frequently-asked-questions-6] * [How can I fix black screen issues?](/en/api-reference/faq/quality/video_blank) * [Why can't I turn on the camera?](/en/api-reference/faq/quality/video_camera) * [How can I listen for audience joining or leaving a channel?](/en/api-reference/faq/integration/audience_event) * [How can I solve channel-related issues?](/en/api-reference/faq/integration/channel) * [How can I set the log file?](/en/api-reference/faq/integration/set_log_file) ### See also [#see-also-7] * [Error codes](reference/error-codes.md) * [Connection status management](build/optimize-quality-and-connection/connection-status-management.mdx) <_PlatformProcessedMarker close="true" /> <_PlatformPanel platform="javascript"> <_PlatformProcessedMarker groupMode="structured" canonicalPlatform="web" platform="javascript" /> This page provides a step-by-step guide on how to create a basic Broadcast Streaming app using the Agora Video SDK. ## Understand the tech [#understand-the-tech-8] To start a Broadcast Streaming session, implement the following steps in your app: * **Initialize the Agora Engine**: Before calling other APIs, create and initialize an Agora Engine instance. * **Join a channel**: Call methods to create and join a channel. * **Join as a host**: A live streaming event has one or more hosts. A host publishes audio and video to the channel. Hosts can also subscribe to streams from other hosts. * **Join as audience**: Audience members can only subscribe to streams published by hosts. * **Send and receive audio and video**: Hosts publish streams to the channel. Audience members subscribe to audio and video streams published by hosts. ![Streaming workflow](https://assets-docs.agora.io/images/video-sdk/get-started-ils-bs.svg) The React SDK is built on Web SDK 4.x and includes its APIs. For example, `useLocalMicrophoneTrack` calls the Web SDK `createMicrophoneAudioTrack` method. Use the React SDK for basic real-time audio and video features. For advanced or complex scenarios, use the Web SDK. When a feature requires the Web SDK, the React SDK documentation links to the relevant Web SDK API pages. ## Prerequisites [#prerequisites-8] * A [supported browser](reference/supported-platforms.md). * A JavaScript package manager such as [npm](https://www.npmjs.com/package/npm). * A camera and a microphone * A valid Agora account and project. Please refer to [Agora account management](manage-agora-account.md) for details. ## Set up your project [#set-up-your-project-8] This section shows you how to set up your ReactJS project and install the Agora Video SDK. ### Create a project [#create-a-project-1] To create a new ReactJS project with the necessary dependencies: 1. Ensure that you have installed [Node.js LTS](https://nodejs.org/en) and `npm`. 2. Open a terminal and execute: ```bash npm create vite@latest agora-sdk-quickstart -- --template react-ts ``` This creates a new project folder named `agora-sdk-quickstart`. Open the folder in your preferred IDE. ### Install the SDK [#install-the-sdk-8] Use one of the following methods to install the project dependencies: **NPM** Navigate to the project folder and run the following command to install the Video SDK: ```bash npm i agora-rtc-react ``` **CDN** To integrate React JS dependencies and the Agora SDK using CDN, add the following code to your HTML file: ```html ``` ## Implement Broadcast Streaming [#implement-broadcast-streaming-8] This section guides you through the implementation of basic real-time audio and video interaction in your app. This guide includes [complete sample code](#complete-sample-code) that demonstrates implementing basic real-time interaction. To understand the core API calls in the sample code, review the following implementation steps. ### Import Agora hooks and components [#import-agora-hooks-and-components] To use Video SDK in your component, include the following imports: ```tsx import { LocalUser, // Plays the microphone audio track and the camera video track RemoteUser, // Plays the remote user audio and video tracks useIsConnected, // Returns whether the SDK is connected to Agora's server useJoin, // Automatically join and leave a channel on mount and unmount useLocalMicrophoneTrack, // Create a local microphone audio track useLocalCameraTrack, // Create a local camera video track usePublish, // Publish the local tracks useRemoteUsers, // Retrieve the list of remote users } from "agora-rtc-react"; import AgoraRTC, { AgoraRTCProvider } from "agora-rtc-react"; import { useState } from "react"; import AgoraRTC, { AgoraRTCProvider } from "agora-rtc-react"; ``` ### Initialize the client [#initialize-the-client] Use the `createClient` method provided by the SDK to create a client object for the `` component. Wrap the `` component with `` to enable state management and access operation-related hooks. For Broadcast Streaming, set the `mode` property of `ClientConfig` to `"live"`, the client `role` to `"host"` or `"audience"` and the audience latency `level` to `1`. ```tsx const client = AgoraRTC.createClient({ mode: "live", codec: "vp8" }); // Set the client role and latency level // Join as audience client.setClientRole("audience", { level: 1 }); // To join as a host, use: // client.setClientRole("host"); return( ); } ``` ### Join a channel [#join-a-channel-8] To join a channel, use the `useJoin` hook. You can specify the following `joinOptions`: * **App ID**: [`appid`](manage-agora-account.md) identifies the project you created in Agora Console. * **Channel name**: The name of the `channel` to join. Clients that pass the same channel name join the same channel. If a channel with the specified name does not exist, it is created when the first user joins. * **Authentication token**: A `token` is a dynamic key that authenticates a user when the client joins a channel. In a production environment, you obtain a token from a [token server](build/authenticate-users/deploy-token-server.mdx) in your security infrastructure. For the purpose of this guide [Generate a temporary token](manage-agora-account.md). * **User ID**: `uid` is a 32-bit signed integer that identifies a user in the channel. You can specify a unique user ID for each user yourself. If you do not set a user ID or set it to `0` when joining a channel, the SDK generates a random number for the user ID and returns the value. Add the following to `Basics`: ```tsx const [appId, setAppId] = useState("<-- Insert App ID -->"); const [channel, setChannel] = useState("<-- Insert Channel Name -->"); const [token, setToken] = useState("<-- Insert Token -->"); const [calling, setCalling] = useState(false); useJoin({appid: appId, channel: channel, token: token ? token : null}, calling); ``` ### Create local audio and video tracks [#create-local-audio-and-video-tracks] To create local audio and video tracks, use the `useLocalMicrophoneTrack` and `useLocalCameraTrack` hooks. Add the following to `Basics`: ```tsx const { localMicrophoneTrack } = useLocalMicrophoneTrack(micOn); const { localCameraTrack } = useLocalCameraTrack(cameraOn); ``` ### Publish tracks in the channel [#publish-tracks-in-the-channel] After joining a channel, publish the local audio and video tracks using the `usePublish` hook. Add the following to `Basics`: ```tsx const [micOn, setMic] = useState(true); const [cameraOn, setCamera] = useState(true); const { localMicrophoneTrack } = useLocalMicrophoneTrack(micOn); const { localCameraTrack } = useLocalCameraTrack(cameraOn); usePublish([localMicrophoneTrack, localCameraTrack]); ``` ### Display the local video [#display-the-local-video-8] To display the local video, use the `LocalUser` hook, which provides properties to manage local audio and video tracks. Assign the local video track to `videoTrack` and the audio track to `audioTrack`. Use the `micOn` and `cameraOn` parameters to toggle audio and video. Include the following code in the markup of your `Basics` component: ```tsx ``` ### Display remote video [#display-remote-video-8] To manage and display the list of remote users connected to a channel, use the `useRemoteUsers` hook. To show each user's video, pass the user object to the `RemoteUser` component along with the required properties. Follow these steps to implement the logic: 1. **Retrieve the list of remote users** Use the `useRemoteUsers` hook to get the current list of remote users: ```tsx const remoteUsers = useRemoteUsers(); ``` 2. **Render the remoteUser component** Loop through the `remoteUsers` list and nest the `RemoteUser` component where you want to display each user's video. Include the following code in the markup of your `Basics` component: ```tsx {remoteUsers.map((user) => (
{user.uid}
))} ``` ### Leave the channel [#leave-the-channel-3] To leave a channel, destroy the component, close the browser window, or refresh the browser tab. The `useJoin` hook automatically handles the destruction of the React component. ### Complete sample code [#complete-sample-code-8] A complete code sample demonstrating the basic process of real-time interaction is provided for your reference. To use the complete sample, add the following to your `src/App.tsx` file. ```tsx import { LocalUser, RemoteUser, useIsConnected, useJoin, useLocalMicrophoneTrack, useLocalCameraTrack, usePublish, useRemoteUsers, } from "agora-rtc-react"; import { useState } from "react"; import AgoraRTC, { AgoraRTCProvider } from "agora-rtc-react"; const client = AgoraRTC.createClient({ mode: "live", codec: "vp8" }); return( ); } const Basics = () => { const [calling, setCalling] = useState(false); const isConnected = useIsConnected(); // Store the user's connection status const [appId, setAppId] = useState("<-- Insert App ID -->"); const [channel, setChannel] = useState("<-- Insert Channel Name -->"); const [token, setToken] = useState("<-- Insert Token -->"); const [micOn, setMic] = useState(true); const [cameraOn, setCamera] = useState(true); const { localMicrophoneTrack } = useLocalMicrophoneTrack(micOn); const { localCameraTrack } = useLocalCameraTrack(cameraOn); useJoin({appid: appId, channel: channel, token: token ? token : null}, calling); usePublish([localMicrophoneTrack, localCameraTrack]); const remoteUsers = useRemoteUsers(); return ( <>
{isConnected ? (
You
{remoteUsers.map((user) => (
{user.uid}
))}
) : (
setAppId(e.target.value)} placeholder="" value={appId} /> setChannel(e.target.value)} placeholder="" value={channel} /> setToken(e.target.value)} placeholder="" value={token} />
)}
{isConnected && (
)} ); }; export default BroadcastStreaming; ```
## Test the sample code [#test-the-sample-code-8] Use [CodeSandbox](https://codesandbox.io/), to swiftly run the complete demo code and experience the React Video SDK functionality in just a few simple steps.