# Reference overview (/en/api-reference)
# Recipes (/en/api-reference/recipes) # SDKs (/en/api-reference/sdks) ## SDK catalog The catalog below lists the latest available SDK release for each supported platform. ### TypeScript #### Agora Agents SDK SDK for: Conversational AI — build and run server-side voice agents - Current release: Version 2.3.1 (Latest) - Install with npm: `npm i agora-agents@2.3.1` - Package manager: [Open package](https://www.npmjs.com/package/agora-agents/v/2.3.1) ### Python #### Agora Agents SDK SDK for: Conversational AI — build and run server-side voice agents - Current release: Version 2.3.0 (Latest) - Install with pip: `pip install agora-agents` - Package manager: [Open package](https://pypi.org/project/agora-agents/) ### Go #### Agora Agents SDK SDK for: Conversational AI — build and run server-side voice agents - Current release: Version 2.3.1 (Latest) - Install with Go: `go get github.com/AgoraIO/agora-agents-go/v2@v2.3.1` - Package manager: [Open package](https://pkg.go.dev/github.com/AgoraIO/agora-agents-go/v2@v2.3.1) ### Android #### RTC Voice SDK Voice only SDK for Calling, Interactive Live Streaming, and Broadcast Streaming - Current release: Version 4.6.4 (Latest) - Install with Gradle: `implementation 'io.agora.rtc:voice-sdk:4.6.4'` - Package manager: [Open package](https://central.sonatype.com/artifact/io.agora.rtc/voice-sdk/4.6.4/aar) - Download: [Download SDK](https://download.agora.io/sdk/release/Agora_Native_SDK_for_Android_v4.6.4_VOICE.zip) #### RTC SDK SDK for Video Calling, Interactive Live Streaming, and Broadcast Streaming - Current release: Version 4.6.4 (Latest) - Install with Gradle: `implementation 'io.agora.rtc:full-sdk:4.6.4'` - Package manager: [Open package](https://central.sonatype.com/artifact/io.agora.rtc/full-sdk/4.6.4/aar) - Download: [Download SDK](https://download.agora.io/sdk/release/Agora_Native_SDK_for_Android_v4.6.4_FULL.zip) - Current release: Version 4.6.4 Lite (Latest) - Install with Gradle: `implementation 'io.agora.rtc:lite-sdk:4.6.4'` - Package manager: [Open package](https://central.sonatype.com/artifact/io.agora.rtc/lite-sdk/4.6.4/aar) - Download: [Download SDK](https://download.agora.io/sdk/release/Agora_Native_SDK_for_Android_v4.6.4_LITE.zip) #### Signaling SDK SDK for Signaling - Current release: Version 2.3.0 (Latest) - Install with Gradle: `implementation 'io.agora.rtm:rtm-sdk:2.3.0'` - Package manager: [Open package](https://central.sonatype.com/artifact/io.agora.rtm/rtm-sdk/2.3.0) - Download: [Download SDK](https://download.agora.io/rtm2/release/Agora_RTM_JAVA_SDK_for_Android_v2.3.0.zip) #### Chat SDK SDK for Chat - Current release: Version 1.4.0 (Latest) - Install with Gradle: `implementation 'io.agora.rtc:chat-sdk:1.4.0'` - Package manager: [Open package](https://search.maven.org/artifact/io.agora.rtc/chat-sdk/1.4.0/aar) - Download: [Download SDK](https://download.agora.io/sdk/release/Agora_CHAT_SDK_for_Android_v1_4_0.zip) #### IoT SDK SDK for Embedded Devices - Current release: Version 1.8.0 (no longer updated) - Download: [Download SDK](https://download.agora.io/rtsasdk/release/Agora-RTSALite-LJAutRmAcAjCP-Android-v1.8.0-20230421_161341-262178.tgz) #### Interactive Whiteboard SDK SDK for Interactive Whiteboard - Current release: Version 2.16.100 (Latest) - Package manager: [Open package](https://github.com/netless-io/whiteboard-android/releases/tag/2.16.100) #### Interactive Whiteboard Fastboard SDK for: Interactive Whiteboard Fastboard - Current release: Version 1.7.2 (Latest) - Package manager: [Open package](https://github.com/netless-io/fastboard-android/releases/tag/1.7.2) ### iOS #### RTC Voice SDK SDK for Voice Calling, Interactive Live Streaming (voice only), and Broadcast Streaming (voice only) - Current release: Version 4.6.4 (Latest) - Install with Swift Package Manager: `https://github.com/AgoraIO/AgoraAudio_iOS` - Package manager: [Open package](https://swiftpackageindex.com/AgoraIO/AgoraAudio_iOS) - Download: [Download SDK](https://download.agora.io/sdk/release/Agora_Native_SDK_for_iOS_v4.6.4_VOICE.zip) #### RTC SDK SDK for Video Calling, Interactive Live Streaming, and Broadcast Streaming - Current release: Version 4.6.4 (Latest) - Install with Swift Package Manager: `https://github.com/AgoraIO/AgoraRtcEngine_iOS` - Package manager: [Open package](https://swiftpackageindex.com/AgoraIO/AgoraRtcEngine_iOS) - Download: [Download SDK](https://download.agora.io/sdk/release/Agora_Native_SDK_for_iOS_v4.6.4_FULL.zip) - Current release: Version 4.6.4 Lite (Latest) - Install with Swift Package Manager: `https://github.com/AgoraIO/AgoraLite_iOS` - Package manager: [Open package](https://swiftpackageindex.com/AgoraIO/AgoraLite_iOS) - Download: [Download SDK](https://download.agora.io/sdk/release/Agora_Native_SDK_for_iOS_v4.6.4_LITE.zip) #### Signaling SDK SDK for Signaling - Current release: Version 2.3.0 (Latest) - Package manager: [Open package](https://github.com/AgoraIO/AgoraRTM_iOS/tree/2.3.0) - Download: [Download SDK](https://download.agora.io/rtm2/release/Agora_RTM_OC_SDK_v2.3.0.zip) #### Chat SDK SDK for Chat - Current release: Version 1.4.0 (Latest) - Package manager: [Open package](https://github.com/AgoraIO/AgoraChat_iOS.git) - Download: [Download SDK](https://download.agora.io/sdk/release/AgoraChat1_4_0.zip) #### Interactive Whiteboard SDK SDK for Interactive Whiteboard - Current release: Version 2.16.112 (Latest) - Package manager: [Open package](https://github.com/netless-io/Whiteboard-iOS/releases/tag/2.16.112) #### Interactive Whiteboard Fastboard SDK for: Interactive Whiteboard Fastboard - Current release: Version 1.4.2 (Latest) - Package manager: [Open package](https://github.com/netless-io/fastboard-iOS/releases/tag/1.4.2) ### Web #### RTC SDK SDK for: Video Calling, Interactive Live Streaming, and Broadcast Streaming - Current release: Version 4.24.8 (Latest) - Install with npm: `npm i agora-rtc-sdk-ng@4.24.8` - Package manager: [Open package](https://www.npmjs.com/package/agora-rtc-sdk-ng/v/4.24.8) - Download: [Download SDK](https://download.agora.io/sdk/release/Agora_Web_SDK_v4_24_8_FULL.zip) #### Signaling SDK SDK for: Signaling - Current release: Version 2.3.0 (Latest) - Install with npm: `npm i agora-rtm-sdk@2.3.0` - Package manager: [Open package](https://www.npmjs.com/package/agora-rtm-sdk/v/2.3.0) - Download: [Download SDK](https://download.agora.io/rtm2/release/Agora_RTM_JavaScript_SDK_for_Web_v2.3.0.zip) #### Chat SDK SDK for: Chat - Current release: Version 1.4.0 (Latest) - Install with npm: `npm i agora-chat` - Package manager: [Open package](https://www.npmjs.com/package/agora-chat) - Download: [Download SDK](https://download.agora.io/sdk/release/agora-chat-1.4.0.zip) #### Interactive Whiteboard SDK SDK for: Interactive Whiteboard - Current release: Version 2.16.53 (Latest) - Install with npm: `npm i white-web-sdk@2.16.53` - Package manager: [Open package](https://www.npmjs.com/package/white-web-sdk/v/2.16.53) #### Interactive Whiteboard Fastboard SDK for: Interactive Whiteboard Fastboard - Current release: Version 1.1.0 (Latest) - Install with npm: `npm i @netless/fastboard@1.1.0` - Package manager: [Open package](https://www.npmjs.com/package/@netless/fastboard/v/1.1.0) ### macOS #### RTC SDK SDK for: Video Calling, Interactive Live Streaming, and Broadcast Streaming - Current release: Version 4.6.2 (Latest) - Install with Swift Package Manager: `https://github.com/AgoraIO/AgoraRtcEngine_macOS` - Package manager: [Open package](https://swiftpackageindex.com/AgoraIO/AgoraRtcEngine_macOS) - Download: [Download SDK](https://download.agora.io/sdk/release/Agora_Native_SDK_for_Mac_v4.6.2_FULL.zip) #### Signaling SDK SDK for: Signaling - Current release: Version 2.3.0 (Latest) - Package manager: [Open package](https://github.com/AgoraIO/AgoraRTM_macOS/tree/2.3.0) - Download: [Download SDK](https://download.agora.io/rtm2/release/Agora_RTM_OC_SDK_v2.3.0.zip) #### Interactive Whiteboard SDK SDK for: Interactive Whiteboard - Current release: Version 2.16.46 (Latest) - Package manager: [Open package](https://github.com/netless-io/whiteboard-ios) ### React Native #### RTC SDK SDK for: Video Calling, Interactive Live Streaming, and Broadcast Streaming - Current release: Version 4.6.4 (Latest) - Install with npm: `npm i react-native-agora@4.6.4` - Package manager: [Open package](https://www.npmjs.com/package/react-native-agora/v/4.6.4) #### Signaling SDK SDK for: Signaling - Current release: Version 2.2.4 (Latest) - Install with npm: `npm i agora-react-native-rtm` - Package manager: [Open package](https://www.npmjs.com/package/agora-react-native-rtm) #### Chat SDK SDK for: Chat - Current release: Version 1.4.0 (Latest) - Install with npm: `npm i react-native-agora-chat@1.4.0` - Package manager: [Open package](https://www.npmjs.com/package/react-native-agora-chat/v/1.4.0) ### ReactJS #### RTC SDK SDK for: Video Calling, Interactive Live Streaming, and Broadcast Streaming - Current release: Version 2.5.1 (Latest) - Install with npm: `npm i agora-rtc-react@2.5.1` - Package manager: [Open package](https://www.npmjs.com/package/agora-rtc-react/v/2.5.1) - Download: [Download SDK](https://download.agora.io/sdk/release/agora-rtc-react.2.5.1.js) ### Windows #### RTC SDK SDK for: Video Calling, Interactive Live Streaming, and Broadcast Streaming - Current release: Version 4.6.2 (Latest) - Download: [Download SDK](https://download.agora.io/sdk/release/Agora_Native_SDK_for_Windows_v4.6.2_FULL.zip) #### Signaling SDK SDK for: Signaling - Current release: Version 2.3.0 (Latest) - Download: [Download SDK](https://download.agora.io/rtm2/release/Agora_RTM_C%2B%2B_SDK_for_Windows_v2.3.0.zip) #### Chat SDK SDK for: Chat - Current release: Version 1.4.0 (Latest) - Download: [Download SDK](https://download.agora.io/sdk/release/agora_chat_sdk.1.4.0.nupkg) ### Flutter #### RTC SDK SDK for: Video Calling, Interactive Live Streaming, and Broadcast Streaming - Current release: Version 6.6.4 (Latest) - Install with Flutter: `flutter pub add agora_rtc_engine:6.6.4` - Package manager: [Open package](https://pub.dev/packages/agora_rtc_engine/versions/6.6.4) #### Chat SDK SDK for: Chat - Current release: Version 1.4.0 (Latest) - Install with Flutter: `flutter pub add agora_chat_sdk` - Package manager: [Open package](https://pub.dev/packages/agora_chat_sdk) #### Signaling SDK SDK for: Signaling - Current release: Version 2.2.6 (Latest) - Install with Flutter: `flutter pub add agora_rtm:2.2.6` - Package manager: [Open package](https://pub.dev/packages/agora_rtm/versions/2.2.6) ### Linux #### IoT SDK SDK for: Embedded Devices - Current release: x86_64-linux-gnu Version 1.9.7 (Latest) - Download: [Download SDK](https://download.agora.io/rtsasdk/release/Agora-RTSALite-RmRdRcAcAjCF-x86_64-linux-gnu-v1.9.7-20251127_103054-992914.tgz) - Current release: aarch64-linux-gnu Version 1.9.7 (Latest) - Download: [Download SDK](https://download.agora.io/rtsasdk/release/Agora-RTSALite-RmRdRcAcAjCF-aarch64-linux-gnu-v1.9.7-20251127_103054-992914.tgz) - Current release: aarch64-linux-musl Version 1.9.7 (Latest) - Download: [Download SDK](https://download.agora.io/rtsasdk/release/Agora-RTSALite-RmRdRcAcAjCF-aarch64-linux-musl-v1.9.7-20251127_103054-992914.tgz) - Current release: armv7a-linux-gnueabihf Version 1.9.7 (Latest) - Download: [Download SDK](https://download.agora.io/rtsasdk/release/Agora-RTSALite-RmRdRcAcAjCF-arm-linux-gnueabihf-v1.9.7-20251127_103054-992914.tgz) - Current release: armv7a-linux-gnueabi Version 1.9.7 (Latest) - Download: [Download SDK](https://download.agora.io/rtsasdk/release/Agora-RTSALite-RmRdRcAcAjCF-arm-linux-gnueabi-v1.9.7-20251127_103054-992914.tgz) - Current release: armv7a-linux-uclibceabihf Version 1.9.7 (Latest) - Download: [Download SDK](https://download.agora.io/rtsasdk/release/Agora-RTSALite-RmRdRcAcAjCF-arm-linux-uclibceabihf-v1.9.7-20251127_103054-992914.tgz) - Current release: armv7a-linux-uclibceabi Version 1.9.7 (Latest) - Download: [Download SDK](https://download.agora.io/rtsasdk/release/Agora-RTSALite-RmRdRcAcAjCF-arm-linux-uclibceabi-v1.9.7-20251127_103054-992914.tgz) - Current release: armv7a-linux-musleabihf Version 1.9.7 (Latest) - Download: [Download SDK](https://download.agora.io/rtsasdk/release/Agora-RTSALite-RmRdRcAcAjCF-arm-linux-musleabihf-v1.9.7-20251127_103054-992914.tgz) - Current release: armv7a-linux-musleabi Version 1.9.7 (Latest) - Download: [Download SDK](https://download.agora.io/rtsasdk/release/Agora-RTSALite-RmRdRcAcAjCF-arm-linux-musleabi-v1.9.7-20251127_103054-992914.tgz) - Current release: armv6-linux-gnueabihf Version 1.9.7 (Latest) - Download: [Download SDK](https://download.agora.io/rtsasdk/release/Agora-RTSALite-RmRdRcAcAjCF-armv6-linux-gnueabihf-v1.9.7-20251127_103054-992914.tgz) - Current release: armv6-linux-uclibceabi Version 1.9.7 (Latest) - Download: [Download SDK](https://download.agora.io/rtsasdk/release/Agora-RTSALite-RmRdRcAcAjCF-armv6-linux-uclibceabi-v1.9.7-20251127_103054-992914.tgz) - Current release: armv5-linux-gnueabi Version 1.9.7 (Latest) - Download: [Download SDK](https://download.agora.io/rtsasdk/release/Agora-RTSALite-RmRdRcAcAjCF-armv5-linux-gnueabi-v1.9.7-20251127_103054-992914.tgz) - Current release: armv5-linux-uclibceabi Version 1.9.7 (Latest) - Download: [Download SDK](https://download.agora.io/rtsasdk/release/Agora-RTSALite-RmRdRcAcAjCF-armv5-linux-uclibceabi-v1.9.7-20251127_103054-992914.tgz) - Current release: mipsel-linux-gnueabihf Version 1.9.7 (Latest) - Download: [Download SDK](https://download.agora.io/rtsasdk/release/Agora-RTSALite-RmRdRcAcAjCF-mipsel-linux-gnueabihf-v1.9.7-20251127_103054-992914.tgz) - Current release: mipsel-linux-uclibceabihf Version 1.9.7 (Latest) - Download: [Download SDK](https://download.agora.io/rtsasdk/release/Agora-RTSALite-RmRdRcAcAjCF-mipsel-linux-uclibceabihf-v1.9.7-20251127_103054-992914.tgz) - Current release: mipsel-linux-uclibceabi Version 1.9.7 (Latest) - Download: [Download SDK](https://download.agora.io/rtsasdk/release/Agora-RTSALite-RmRdRcAcAjCF-mipsel-linux-uclibceabi-v1.9.7-20251127_103054-992914.tgz) - Current release: arm-liteos-eabi Version 1.9.7 (Latest) - Download: [Download SDK](https://download.agora.io/rtsasdk/release/Agora-RTSALite-RmRdRcAcAjCF-arm-liteos-eabi-v1.9.7-20251127_103054-992914.tgz) - Current release: armv7a-harmony-eabi Version 1.9.7 (Latest) - Download: [Download SDK](https://download.agora.io/rtsasdk/release/Agora-RTSALite-RmRdRcAcAjCF-arm-harmony-eabi-v1.9.7-20251127_103054-992914.tgz) #### Agora On-Premise Recording SDK Compatible with: Voice Calling, Video Calling, or Interactive Live Streaming - Current release: x86_64 Version 4.4.151 for C++ (Latest) - Download: [Download SDK](https://download.agora.io/rtsasdk/release/Agora-RTC-x86_64-linux-gnu-v4.4.151-20250919_101833-891308.tgz) - Current release: arm64 Version 4.4.151 for C++ (Latest) - Download: [Download SDK](https://download.agora.io/rtsasdk/release/Agora-RTC-aarch64-linux-gnu-v4.4.151-20250919_102817-891319.tgz) - Current release: x86_64 Version 4.4.151 for Java (Latest) - Download: [Download SDK](https://download.agora.io/sdk/release/Agora-Linux-Recording-Java-SDK-v4.4.151.1-x86_64-891308-28c706d74a-20250919_142050.zip) - Current release: arm64 Version 4.4.151 for Java (Latest) - Download: [Download SDK](https://download.agora.io/sdk/release/Agora-Linux-Recording-Java-SDK-v4.4.151-aarch64-891319-952e64402b-20250919_140753.zip) #### Server Gateway SDK Compatible with: Voice Calling, Video Calling, or Interactive Live Streaming - Current release: Version 4.4.32 for Java (Latest) - Download: [Download SDK](https://download.agora.io/sdk/release/Agora-Linux-Java-SDK-v4.4.32-x86_64-675656-ccd9be501d-20250526_180235.zip) - Current release: Version 4.4.32 aarch64 for C++ (Latest) - Download: [Download SDK](https://download.agora.io/rtsasdk/release/Agora-RTC-aarch64-linux-gnu-v4.4.32-20250425_150503-675674.tgz) - Current release: Version 4.4.32 x86_64 for C++ (Latest) - Download: [Download SDK](https://download.agora.io/rtsasdk/release/Agora-RTC-x86_64-linux-gnu-v4.4.32-20250425_144419-675648.tgz) - Current release: Version 2.2.8 for Go (Latest) - Download: [Download SDK](https://github.com/AgoraIO-Extensions/Agora-Golang-Server-SDK/tree/release/2.2.8) - Current release: Version 2.2.4 for Python (Latest) - Install with pip: `pip install agora-python-server-sdk` - Package manager: [Open package](https://pypi.org/project/agora-python-server-sdk/) - Download: [Download SDK](https://github.com/AgoraIO-Extensions/Agora-Python-Server-SDK/tree/release/2.2.4) ### Linux C++ #### Signaling SDK SDK for: Real-Time Messaging - Current release: Version 2.3.0 (Latest) - Download: [Download SDK](https://download.agora.io/rtm2/release/Agora_RTM_C%2B%2B_SDK_for_Linux_v2.3.0.zip) ### Linux Java #### Signaling SDK SDK for: Real-Time Messaging - Current release: Version 2.3.0 for aarch64 (Latest) - Install with Gradle: `implementation 'io.agora:rtm-java-aarch64:2.3.0'` - Package manager: [Open package](https://central.sonatype.com/artifact/io.agora/rtm-java-aarch64/2.3.0) - Download: [Download SDK](https://download.agora.io/rtm2/release/Agora_RTM_JAVA_SDK_for_Linux_aarch64_v2.3.0.zip) ### Unity #### RTC SDK SDK for: Video Calling, Interactive Live Streaming, and Broadcast Streaming - Current release: Version 4.6.2 (Latest) - Download: [Download SDK](https://download.agora.io/sdk/release/Agora_Unity_RTC_SDK_FULL_20260212_633_4.6.2-build.1.zip) #### Signaling SDK SDK for: Signaling - Current release: Version 2.1.9 (Latest) - Download: [Download SDK](https://download.agora.io/sdk/release/Agora_Unity_RTM_SDK_v2.1.9.zip) #### Chat SDK SDK for: Chat - Current release: Version 1.4.0 (Latest) - Download: [Download SDK](https://download.agora.io/sdk/release/agora_chat_unity_sdk1.4.0.unitypackage) ### Electron #### RTC SDK SDK for: Video Calling, Interactive Live Streaming, and Broadcast Streaming - Current release: Version 4.6.2 (Latest) - Install with npm: `npm i agora-electron-sdk@4.6.2` - Package manager: [Open package](https://www.npmjs.com/package/agora-electron-sdk/v/4.6.2) ### Unreal Engine #### RTC SDK SDK for Video Calling, Interactive Live Streaming, and Broadcast Streaming - Current release: Version 4.5.1 (Latest) - Download: [Download SDK](https://download.agora.io/sdk/release/Agora_RTC_FULL_SDK_4.5.1_Unreal.zip) # Contribute an ASR Extension (/en/ai/create_asr_extension) This page has moved to [Contribute an ASR extension](./reference/ten-agent/create-asr-extension). # Contribute a TTS Extension (/en/ai/create_tts_extension) This page has moved to [Contribute a TTS extension](./reference/ten-agent/create-tts-extension). # Voice Agent overview (/en/ai) A Voice Agent is a real-time system that listens to a user, reasons over the conversation, and responds with speech. On Agora, that experience is built from a shared foundation: real-time transport, an agent runtime, AI models, and the endpoint where the conversation is delivered. Agora supports two Voice Agent paths: **in apps** and **on dedicated devices**. Choose **In apps** if your agent lives in a web, mobile, desktop, or backend-assisted app experience. Choose **On dedicated devices** if your endpoint is embedded or hardware-first, such as a toy, wearable, kiosk, or companion. Use the quickstart that matches your endpoint to jump directly into the right implementation path. ## Capabilities No matter where you ship it, a Voice Agent usually has the same core layers: * **Real-time transport**: moves audio, events, and conversation state between the user and the agent with low latency. * **Agent runtime**: manages the session lifecycle, turn taking, interruptions, memory, and tool orchestration. * **Models**: provide ASR, LLM, TTS, and optional multimodal capabilities. * **Endpoint**: presents the experience to the user, either through an app or a dedicated device. The first three layers stay conceptually similar across the docs. The main split in the sidebar happens at the **endpoint** layer, because app teams and device teams need different implementation workflows. ## How this section is organized The AI sidebar is structured in two paths: * **In apps**: for web, mobile, desktop, and backend-assisted app experiences. * **On dedicated devices**: for embedded or hardware-first endpoints such as toys, wearables, kiosks, and companions. Both paths share the same Voice Agent idea, but they differ in what teams need to build next. App teams usually care first about session orchestration, UI, model behavior, and runtime events. Device teams usually care first about bring-up, networking, firmware, controls, and hardware integration. # Release notes (/en/ai/release-notes) This document tracks notable changes and improvements to the Conversational AI Engine. ## Releases ### v2.12 Released on September 10, 2026. #### New features Included in this release: * **Call custom tools** You can now declare custom tools for the LLM to call during a conversation, directly through the RESTful API. Set `advanced_features.enable_tools` and declare a tool's name, description, parameters, and a synchronous `GET` or `POST` HTTPS endpoint using the `llm.tools` parameter. The LLM selects a tool and generates its parameters based on the conversation. The Conversational AI Engine then calls your endpoint and returns the result to the LLM to continue generating a response. For details, see [Call custom tools](build/custom-model-integration/custom-tools). * **Generated filler words** `filler_words.content` now supports a `generated` mode that produces a context-relevant filler phrase instead of a fixed phrase list, directly through the RESTful API. If a generated phrase isn't ready in time, the agent falls back to your configured static phrases. Generation is Agora-hosted, so no additional model, endpoint, or credentials are required. For details, see [Talking while waiting](build/shape-the-conversation/filler-words). * **Defer custom instructions with the `append` action** The `think` API's state actions now support `append`. Setting `on_listening_action`, `on_thinking_action`, or `on_speaking_action` to `append` doesn't interrupt the current interaction. Instead, the custom instruction is appended as a separate user message once the current turn's LLM output finishes, then starts a new turn. Available through the [**Send custom instructions**](/en/api-reference/api-ref/conversational-ai/think) endpoint. * **Client component support for direct speech and instruction injection** The [Android](/en/api-reference/api-ref/conversational-ai/client-toolkit/android), [iOS](/en/api-reference/api-ref/conversational-ai/client-toolkit/ios), and [Web](/en/api-reference/api-ref/conversational-ai/client-toolkit/web) client toolkits add `speak` and `think` methods. Use `speak` to bypass the LLM and make the agent speak text directly, or `think` to inject text as user input for the LLM to process. Both methods send messages over RTM and support message priority, interruption behavior, and per-state Think actions. * **MCP tool-use for MLLM agents** MLLM agents now support MCP tool-use, in addition to cascading LLM agents. Configure [`mllm.mcp_servers`](/en/api-reference/api-ref/conversational-ai/join#request-body-properties-mllm-mcp-servers) and set `advanced_features.enable_tools` to `true`, the same switch used for cascading LLM agents. Available directly through the RESTful API. For details, see [Use MCP tools](build/mcp-tools). * **New ASR provider** * [Gemini](models/asr/gemini) #### API changes This release introduces the following changes for custom tool calling, generated filler words, MCP tool-use for MLLM agents, and the `think` API's `append` action. * Changes to [**Start a conversational AI agent**](/en/api-reference/api-ref/conversational-ai/join) * **New parameters** * [`llm.tools`](/en/api-reference/api-ref/conversational-ai/join#request-body-properties-llm-tools): Custom tool definitions the LLM can call. Requires `advanced_features.enable_tools`. * [`filler_words.content.generated_config`](/en/api-reference/api-ref/conversational-ai/join#request-body-properties-filler-words-content-generated-config): Prompt and fallback strategy for generated filler words. Used when `filler_words.content.mode` is `generated`. * [`mllm.mcp_servers`](/en/api-reference/api-ref/conversational-ai/join#request-body-properties-mllm-mcp-servers): MCP server connections for MLLM agents. Requires `advanced_features.enable_tools`. * **Updated parameters** * [`advanced_features.enable_tools`](/en/api-reference/api-ref/conversational-ai/join#request-body-properties-advanced-features-enable-tools): Now also enables custom tools declared in `llm.tools` and MCP tools declared in `mllm.mcp_servers`, in addition to MCP tools configured in `llm.mcp_servers`. * [`filler_words.content.mode`](/en/api-reference/api-ref/conversational-ai/join#request-body-properties-filler-words-content-mode) value `generated`: Generate a filler phrase from the current context, with `filler_words.content.static_config` used as a fallback. * [`filler_words.content.static_config`](/en/api-reference/api-ref/conversational-ai/join#request-body-properties-filler-words-content-static-config): Now a named sibling property of `content` instead of the `"{mode}_config"` key. * Changes to [**Send custom instructions**](/en/api-reference/api-ref/conversational-ai/think) * **Updated parameters** * [`on_listening_action`](/en/api-reference/api-ref/conversational-ai/think#request-body-properties-on-listening-action), [`on_thinking_action`](/en/api-reference/api-ref/conversational-ai/think#request-body-properties-on-thinking-action), and [`on_speaking_action`](/en/api-reference/api-ref/conversational-ai/think#request-body-properties-on-speaking-action) value `append`: Don't interrupt the current interaction; append the instruction as a separate user message once the current turn's LLM output finishes. #### Toolkit API This release adds `speak` and `think` methods to the client toolkit, so you can bypass the LLM to play text directly or inject text as user input: Android iOS Web * [`speak`](/en/api-reference/api-ref/conversational-ai/client-toolkit/android#speak) * [`think`](/en/api-reference/api-ref/conversational-ai/client-toolkit/android#think) * `SpeakMessage`, `ThinkMessage`, `ThinkListeningAction`, `ThinkThinkingAction`, `ThinkSpeakingAction` * [`speak`](/en/api-reference/api-ref/conversational-ai/client-toolkit/ios#speak) * [`think`](/en/api-reference/api-ref/conversational-ai/client-toolkit/ios#think) * `SpeakMessage`, `ThinkMessage`, `ThinkListeningAction`, `ThinkThinkingAction`, `ThinkSpeakingAction` * [`speak`](/en/api-reference/api-ref/conversational-ai/client-toolkit/web#speak) * [`think`](/en/api-reference/api-ref/conversational-ai/client-toolkit/web#think) * `SpeakMessage`, `ThinkMessage`, `EThinkListeningAction`, `EThinkThinkingAction`, `EThinkSpeakingAction` #### Deprecation notice Important These changes may affect your existing integration. Review each change and update your integration as needed. * **`asr.params` becomes required for non-default ASR vendors in v2.13.** Starting in v2.13, a request that sets [`asr.vendor`](/en/api-reference/api-ref/conversational-ai/join#request-body-properties-asr-vendor) to a vendor other than `ares` and omits [`asr.params`](/en/api-reference/api-ref/conversational-ai/join#request-body-properties-asr-params) fails. Configure `asr.params` for your vendor, or omit `asr.vendor` to keep the default `ares` vendor. * **Legacy `turn_detection` fields and the `keywords`/`disabled` start-of-speech modes lose their support guarantee in a future release.** Migrate to [`turn_detection.config`](/en/api-reference/api-ref/conversational-ai/join#request-body-properties-turn-detection-config) and [`interruption`](/en/api-reference/api-ref/conversational-ai/join#request-body-properties-interruption), introduced in v2.6. * **`advanced_features.enable_mllm` loses its support guarantee in a future release.** The MLLM switch may stop working. Migrate to [`mllm.enable`](/en/api-reference/api-ref/conversational-ai/join#request-body-properties-mllm-enable), introduced in v2.6. ### v2.11 Released on August 11, 2026. #### New features Included in this release: * **ASR keywords** You can now configure keywords to improve ARES's recognition accuracy for specific terms, such as brand names, product names, or industry jargon. Configure keywords using the `AresSTT` vendor in the Agora Agents SDKs, or the `keywords` parameter within the `asr` block directly through the RESTful API. For details, see [Improve ASR accuracy with keywords](build/shape-the-conversation/asr-keywords). * **New TTS provider** * [Typecast](models/tts/typecast) * **New MLLM provider** * [Azure OpenAI Realtime API](models/mllm/azure) #### Improvements This release includes the following enhancement. * **Improved stability for long audio replies** This release improves resource protection and error-recovery logic for long streaming audio replies, reducing the likelihood of an abnormal disconnect or failed recovery. As a result, service stability improves for long replies and free-form conversation scenarios. #### API changes This release introduces the following changes for ASR keywords and the new TTS and MLLM providers. Python SDK TypeScript SDK Go SDK REST API * [`AresSTT`](/en/api-reference/api-ref/server-sdk/python#aresstt): New `keywords` parameter. * [`TypecastTTS`](/en/api-reference/api-ref/server-sdk/python#typecasttts): New TTS vendor class. * [`AzureOpenAIRealtime`](/en/api-reference/api-ref/server-sdk/python#azureopenairealtime): New MLLM vendor class. * [`AresSTT`](/en/api-reference/api-ref/server-sdk/typescript#other-stt-vendors): New `keywords` option. * [`TypecastTTS`](/en/api-reference/api-ref/server-sdk/typescript#other-tts-vendors): New TTS vendor class. * [`AzureOpenAIRealtime`](/en/api-reference/api-ref/server-sdk/typescript#azureopenairealtime): New MLLM vendor class. * [`NewAresSTT`](/en/api-reference/api-ref/server-sdk/go#newaresstt): New `Keywords` field. * [`NewTypecastTTS`](/en/api-reference/api-ref/server-sdk/go#newtypecasttts): New TTS vendor constructor. * [`NewAzureOpenAIRealtime`](/en/api-reference/api-ref/server-sdk/go#newazureopenairealtime): New MLLM vendor constructor. * Changes to [**Start a conversational AI agent**](/en/api-reference/api-ref/conversational-ai/join) * **New parameters** * [`asr.keywords`](/en/api-reference/api-ref/conversational-ai/join#request-body-properties-asr-keywords): ASR keywords used to improve recognition accuracy for specific terms. Only available when `asr.vendor` is not set or `ares`. * **Updated parameters** * [`tts.vendor`](/en/api-reference/api-ref/conversational-ai/join#request-body-properties-tts-vendor) value `typecast`: Use Typecast as the TTS provider. See [Typecast](/en/ai/models/tts/typecast). * [`mllm.vendor`](/en/api-reference/api-ref/conversational-ai/join#request-body-properties-mllm-vendor) value `azure`: Use Azure OpenAI Realtime API as the MLLM provider. See [Azure OpenAI Realtime API](/en/ai/models/mllm/azure). ### v2.10 Released on July 30, 2026. #### New features Included in this release: * **Connect custom TTS services using the OpenAI protocol** You can now use a self-built, privately deployed, or third-party TTS service that implements the OpenAI TTS protocol as your agent's speech synthesis module. Configure it using the `GenericTTS` vendor in the Agora Agents SDKs (Go and TypeScript v2.4.0 or later, Python v2.4.1 or later), or the `generic_http` TTS vendor directly through the RESTful API. For details, see [Connect your own TTS service](/en/ai/build/custom-model-integration/custom-tts) and [Generic TTS](/en/ai/models/tts/generic-http). * **Client toolkit support for manual turn control** The Android, iOS, and Web client toolkits now provide dedicated APIs for manual Start of Speech (SoS) and End of Speech (EoS) control, so you no longer need to construct raw Signaling (RTM) messages by hand. Call `manualSOS`/`manualEOS` to send requests, and register the new result callbacks to handle the server's response. This capability is suitable for scenarios that require explicit control of the user turn boundary from the client side, such as AI interviews, interactive quizzes, and walkie-talkie apps. For details, see [Manually control start and end of speech](/en/ai/best-practices/manual-turn-control). * **New TTS providers** * [Gradium](models/tts/gradium) * [Mistral](models/tts/mistral) * **New Avatar provider** * [LemonSlice](models/avatar/lemonslice) #### Toolkit API This release adds the following APIs to the client toolkit for manual turn control: Android iOS Web * [`manualSOS`](/en/api-reference/api-ref/conversational-ai/client-toolkit/android#manualsos) * [`manualEOS`](/en/api-reference/api-ref/conversational-ai/client-toolkit/android#manualeos) * [`onUserManualSosEvent`](/en/api-reference/api-ref/conversational-ai/client-toolkit/android#onusermanualsosevent) * [`onUserManualEosEvent`](/en/api-reference/api-ref/conversational-ai/client-toolkit/android#onusermanualeosevent) * [`onAgentManualEosEvent`](/en/api-reference/api-ref/conversational-ai/client-toolkit/android#onagentmanualeosevent) * `UserManualEventPayload`, `UserManualSosEvent`, `UserManualEosEvent`, `AgentManualEosPayload`, `AgentManualEosEvent` * [`manualSOS`](/en/api-reference/api-ref/conversational-ai/client-toolkit/ios#manualsos) * [`manualEOS`](/en/api-reference/api-ref/conversational-ai/client-toolkit/ios#manualeos) * [`onUserManualSosEvent`](/en/api-reference/api-ref/conversational-ai/client-toolkit/ios#onusermanualsosevent) * [`onUserManualEosEvent`](/en/api-reference/api-ref/conversational-ai/client-toolkit/ios#onusermanualeosevent) * [`onAgentManualEosEvent`](/en/api-reference/api-ref/conversational-ai/client-toolkit/ios#onagentmanualeosevent) * `UserManualEventPayload`, `UserManualSosEvent`, `UserManualEosEvent`, `AgentManualEosPayload`, `AgentManualEosEvent` * [`manualSOS`](/en/api-reference/api-ref/conversational-ai/client-toolkit/web#manualsos) * [`manualEOS`](/en/api-reference/api-ref/conversational-ai/client-toolkit/web#manualeos) * [`EConversationalAIAPIEvents`](/en/api-reference/api-ref/conversational-ai/client-toolkit/web#econversationalaiapievents) values `USER_MANUAL_SOS_RESULT`, `USER_MANUAL_EOS_RESULT`, and `AGENT_MANUAL_EOS_RESULT` * `TUserManualEventPayload`, `TUserManualSosEvent`, `TUserManualEosEvent`, `TAgentManualEosPayload`, `TAgentManualEosEvent` #### API changes This release introduces the following changes for connecting TTS services. Python SDK TypeScript SDK Go SDK REST API This release adds the following TTS vendor classes: * [`GenericTTS`](/en/api-reference/api-ref/server-sdk/python#generictts): Connect a custom TTS service using the OpenAI TTS protocol. * [`GradiumTTS`](/en/api-reference/api-ref/server-sdk/python#gradiumtts) * [`MistralTTS`](/en/api-reference/api-ref/server-sdk/python#mistraltts) This release adds the following TTS vendor classes: * [`GenericTTS`](/en/api-reference/api-ref/server-sdk/typescript#generictts): Connect a custom TTS service using the OpenAI TTS protocol. * [`GradiumTTS` and `MistralTTS`](/en/api-reference/api-ref/server-sdk/typescript#other-tts-vendors) This release adds the following TTS vendor constructors: * [`NewGenericTTS`](/en/api-reference/api-ref/server-sdk/go#newgenerictts): Connect a custom TTS service using the OpenAI TTS protocol. * [`NewGradiumTTS`](/en/api-reference/api-ref/server-sdk/go#newgradiumtts) * [`NewMistralTTS`](/en/api-reference/api-ref/server-sdk/go#newmistraltts) * Changes to [**Start a conversational AI agent**](/en/api-reference/api-ref/conversational-ai/join) * **New parameters** * [`tts.url`](/en/api-reference/api-ref/conversational-ai/join#request-body-properties-tts-url): The OpenAI TTS protocol compatible service endpoint. Required only when `tts.vendor` is `generic_http`. * [`tts.headers`](/en/api-reference/api-ref/conversational-ai/join#request-body-properties-tts-headers): Custom request headers to include in requests to the TTS service. Required only when `tts.vendor` is `generic_http`. * **Updated parameters** * [`tts.vendor`](/en/api-reference/api-ref/conversational-ai/join#request-body-properties-tts-vendor) value `generic_http`: Integrate a custom TTS service using the OpenAI TTS protocol. See [Generic TTS](/en/ai/models/tts/generic-http). * [`tts.vendor`](/en/api-reference/api-ref/conversational-ai/join#request-body-properties-tts-vendor) value `gradium`: Use Gradium as the TTS provider. See [Gradium](/en/ai/models/tts/gradium). * [`tts.vendor`](/en/api-reference/api-ref/conversational-ai/join#request-body-properties-tts-vendor) value `mistral`: Use Mistral as the TTS provider. See [Mistral](/en/ai/models/tts/mistral). * [`tts.params`](/en/api-reference/api-ref/conversational-ai/join#request-body-properties-tts-params): Now required when `tts.vendor` is `generic_http`. ### v2.9 Released on July 1, 2026. #### New features Included in this release: * **Manual turn control** You can now explicitly control when a user's speech starts and ends by sending manual Start of Speech (SoS) and End of Speech (EoS) requests over Signaling (RTM), instead of relying on automatic detection. This supports scenarios with strict turn-boundary requirements, such as AI interviews, interactive quizzes, or push-to-talk. Set [`start_of_speech.mode`](/en/api-reference/api-ref/conversational-ai/join#request-body-properties-turn-detection-config-start-of-speech-mode) and [`end_of_speech.mode`](/en/api-reference/api-ref/conversational-ai/join#request-body-properties-turn-detection-config-end-of-speech-mode) to `manual` when starting the agent. For details, see [Manually control start and end of speech](/en/ai/best-practices/manual-turn-control). * **Agent state notifications over RTM Message** You can now track agent state changes in real time using Signaling (RTM) Message events. The agent pushes `state.listening`, `state.thinking`, and `state.speaking` events as individual RTM messages, making it easier to record state change timestamps, process events sequentially, or consume state events alongside other RTM Message events. For details, see [Get agent state](/en/ai/best-practices/get-agent-state). * **Pre-recorded greeting audio** You can now play a pre-recorded audio file as the agent's greeting instead of synthesizing it with TTS. Set [`properties.llm.greeting_audio_url`](/en/api-reference/api-ref/conversational-ai/join#request-body-properties-llm-greeting-audio-url) to the URL of an `mp3`, `wav`, or `pcm` file when starting the agent. If the audio fails to download, decode, or is in an unsupported format, the agent automatically falls back to TTS synthesis using `greeting_message`. * **Managed mode** You can now use supported ASR, LLM, and TTS providers with Agora-managed credentials, without supplying your own API keys. Set [`credential_mode`](/en/api-reference/api-ref/conversational-ai/join#request-body-properties-asr-credential-mode) to `"managed"` within the `asr`, `llm`, or `tts` block when starting an agent. Managed mode replaces the deprecated `preset` parameter. For details, see [Use managed mode](/en/ai/build/custom-model-integration/managed-mode). #### Improvements This release includes the following enhancements. * **Speech captured during an uninterruptible greeting** When [`properties.llm.greeting_configs.interruptable`](/en/api-reference/api-ref/conversational-ai/join#request-body-properties-llm-greeting-configs-interruptable) is set to `false`, the agent now silently collects the user's ASR results while the greeting plays. After the greeting ends, the system concatenates the collected speech segments into a single user message and sends it to the LLM for a unified response, instead of discarding speech that occurs during playback. #### API changes This release introduces the following changes to the RESTful API. * Changes to [**Start a conversational AI agent**](/en/api-reference/api-ref/conversational-ai/join) * **New parameters** * [`properties.llm.greeting_audio_url`](/en/api-reference/api-ref/conversational-ai/join#request-body-properties-llm-greeting-audio-url): The URL of a pre-recorded audio file (`mp3`, `wav`, or `pcm`) to play as the agent's greeting instead of synthesizing it with TTS. * [`properties.llm.greeting_configs.audio_download_timeout_ms`](/en/api-reference/api-ref/conversational-ai/join#request-body-properties-llm-greeting-configs-audio-download-timeout-ms): The timeout in milliseconds for downloading the greeting audio file. Defaults to `1000`. * [`properties.llm.greeting_configs.audio_pcm_sample_rate`](/en/api-reference/api-ref/conversational-ai/join#request-body-properties-llm-greeting-configs-audio-pcm-sample-rate): The sample rate in Hz of the greeting audio file when the format is PCM. Defaults to `16000`. * [`properties.asr.credential_mode`](/en/api-reference/api-ref/conversational-ai/join#request-body-properties-asr-credential-mode), [`properties.llm.credential_mode`](/en/api-reference/api-ref/conversational-ai/join#request-body-properties-llm-credential-mode), and [`properties.tts.credential_mode`](/en/api-reference/api-ref/conversational-ai/join#request-body-properties-tts-credential-mode): Set to `"managed"` to use Agora-managed credentials for the provider without supplying your own API key. * **Updated parameters** * [`properties.turn_detection.config.start_of_speech.mode`](/en/api-reference/api-ref/conversational-ai/join#request-body-properties-turn-detection-config-start-of-speech-mode): Now accepts `manual`, which disables automatic start-of-speech detection so the client can explicitly signal the start of user speech over RTM. * [`properties.turn_detection.config.end_of_speech.mode`](/en/api-reference/api-ref/conversational-ai/join#request-body-properties-turn-detection-config-end-of-speech-mode): Now accepts `manual`, which disables automatic end-of-speech detection so the client can explicitly signal the end of user speech over RTM. * [`properties.llm.greeting_message`](/en/api-reference/api-ref/conversational-ai/join#request-body-properties-llm-greeting-message): In addition to serving as a standalone greeting, this field is now required as a fallback when `greeting_audio_url` is configured, and is used to maintain conversation context when an audio greeting is interrupted. * **Deprecated parameters** * [`preset`](/en/api-reference/api-ref/conversational-ai/join#preset): Deprecated in favor of setting `credential_mode` to `"managed"` within the `asr`, `llm`, or `tts` block. See [Use managed mode](/en/ai/build/custom-model-integration/managed-mode). * Changes to [**Notification event types**](/en/ai/reference/event-types) * **Updated events** * [`110 agent error`](/en/ai/reference/event-types#110-agent-error): Now reports greeting audio URL errors when the greeting audio file fails to download, times out, is in an unsupported format, or fails to decode. The agent automatically falls back to TTS synthesis using `greeting_message`. When `parameters.enable_error_message` is `true`, these errors are also delivered to the client via the `onMessageError` RTM callback. ### v2.8 Released on June 11, 2026. #### New features Included in this release: * **Session data retention control** By default, session interaction text and audio are temporarily retained for the minimum necessary period to support service operation, agent optimization, and troubleshooting. To disable data retention for a session, set [`properties.parameters.opt_out`](/en/api-reference/api-ref/conversational-ai/join#request-body-properties-parameters-opt-out) to `true` when starting an agent. * **RTC token expiry warning** A new [`104 agent expire`](reference/event-types#104-agent-expire) event is triggered when the agent's RTC token is about to expire. Upon receiving this event, call the [Update agent](/en/api-reference/api-ref/conversational-ai/update) API with a new token to refresh it. The agent does not exit the channel or interrupt the session when this event is triggered. * **New ASR provider** * [xAI](models/asr/xai) * **New LLM provider** * [xAI Grok](models/llm/xai) * **New TTS provider** * [xAI](models/tts/xai) #### Improvements This release includes the following enhancements. Important These changes may affect your existing integration. Review each change and update your integration as needed. * **Idle timeout behavior change** Setting `properties.idle_timeout` to `0` no longer means the agent runs indefinitely. From this release, `0` only disables the channel idle timeout. Regardless of the `idle_timeout` value, the maximum running time for a single session is 72 hours, after which the agent automatically exits. If your integration previously relied on `idle_timeout = 0` to keep an agent running until manually stopped, update your task scheduling and lifecycle management logic to account for the 72-hour limit. #### API changes This release introduces the following changes to the RESTful API. * Changes to [**Start a conversational AI agent**](/en/api-reference/api-ref/conversational-ai/join) * **New parameters** * [`properties.parameters.opt_out`](/en/api-reference/api-ref/conversational-ai/join#request-body-properties-parameters-opt-out): Set to `true` to disable data retention for the current session. * **Updated parameters** * [`properties.idle_timeout`](/en/api-reference/api-ref/conversational-ai/join#request-body-properties-idle-timeout): Valid range is now `0` to `259200` seconds (72 hours). Regardless of the value set, the maximum running time for a single session is 72 hours. * Changes to [**Retrieve agent history**](/en/api-reference/api-ref/conversational-ai/history) * **New response fields** * [`contents[].speech_start_ms`](/en/api-reference/api-ref/conversational-ai/history#contents-speech-start-ms): Unix timestamp in milliseconds indicating when the user started speaking or the agent started TTS playback. Only returned when `llm.vendor` is `custom`. * [`contents[].speech_end_ms`](/en/api-reference/api-ref/conversational-ai/history#contents-speech-end-ms): Unix timestamp in milliseconds indicating when the user stopped speaking, or when TTS playback completed or was interrupted. Only returned when `llm.vendor` is `custom`. * [`contents[].speech_algorithmic_delay`](/en/api-reference/api-ref/conversational-ai/history#contents-speech-algorithmic-delay): The total delay in milliseconds introduced by audio processing algorithms after audio is captured from the user's microphone. Only returned when `llm.vendor` is `custom`, `contents[].role` is `user`, and actual voice input is present. * Changes to [**Notification event types**](reference/event-types) * **New events** * [`104 agent expire`](reference/event-types#104-agent-expire): Triggered when the agent's RTC token is about to expire. Call the Update agent API with a new token upon receiving this event. * **Updated events** * [`102 agent left`](reference/event-types#102-agent-left): Added `Task lifetime limit exceeded` as a new exit reason when the 72-hour session limit is reached. * [`103 agent history`](reference/event-types#103-agent-history): Added `contents[].speech_start_ms`, `contents[].speech_end_ms`, and `contents[].speech_algorithmic_delay`. Only returned when `llm.vendor` is `custom`. * Changes to [**Retrieve a list of agents**](/en/api-reference/api-ref/conversational-ai/list) * **Updated parameters** * `state`: Now accepts a comma-separated list of values to filter by multiple agent states in a single request. For example, `state=0,1,2`. ### v2.7 Released on May 20, 2026. #### New features Included in this release: * **New MLLM provider** * [xAI Grok](models/mllm/xai) * **New Avatar providers** * [Generic avatar (Beta)](models/avatar/generic) * **Greeting interruption control** The `properties.llm.greeting_configs` object in the [Start a conversational AI agent](/en/api-reference/api-ref/conversational-ai/join) API now supports an `interruptable` parameter, which controls whether user speech can interrupt a greeting during playback. #### Improvements This release includes the following enhancements: Important These changes may affect your existing integration. Review each change and update your integration as needed. * **Response status codes updated** This release adjusts the HTTP response status codes and `reason` values returned by the RESTful API. New status codes `401`, `429`, and `500` are added. New `reason` values `ServiceNotEnabled`, `AccountSuspended`, and `ResourceAllocationFailed` are introduced. The existing `InvalidRequest` reason is replaced by `InvalidRequestBody`, `MissingRequiredField`, and `InvalidFieldValue`. For details, see [Status codes and error messages](/en/api-reference/api-ref/conversational-ai/status-codes). If your integration relies on `reason` values for error handling, retry logic, or alerting, update your logic to handle the new values accordingly. * **Query conversation turns API now returns paginated results** The [Query conversation turn information](/en/api-reference/api-ref/conversational-ai/turns) API no longer returns all turns in a single response. Results are now paginated, defaulting to page 1 with up to 50 turns per page. This affects sessions with more than 50 conversation turns. Update your integration to handle the new `total_turn_count` and `pagination` fields in the response, and retrieve subsequent pages as needed. This release also adds the [112 turns finished](reference/event-types#112-turns-finished) notification event, which delivers conversation turn data in batches after a session ends as an alternative way to retrieve turn data. * **`on_listening_action` default behavior change** The default value of `on_listening_action` in the [Send a custom instruction](/en/api-reference/api-ref/conversational-ai/think) API has changed from `inject` to `interrupt`. When set to `interrupt`, the API immediately interrupts the current flow and initiates a new round of dialogue. If you rely on the previous default behavior, explicitly set `on_listening_action` to `inject` in your requests. #### API changes This release introduces the following changes to the RESTful API. * Changes to [**Start a conversational AI agent**](/en/api-reference/api-ref/conversational-ai/join) * **New parameters** * [`properties.llm.greeting_configs.interruptable`](/en/api-reference/api-ref/conversational-ai/join#request-body-properties-llm-greeting-configs-interruptable): Controls whether user speech can interrupt a greeting during playback. * Changes to [**Query conversation turn information**](/en/api-reference/api-ref/conversational-ai/turns) * **New query parameters** Adds `page_index` and `page_size` fields to [query parameters](/en/api-reference/api-ref/conversational-ai/turns#query-parameters). * **New response parameters** The [response](/en/api-reference/api-ref/conversational-ai/turns#response) now includes additional top-level fields: * `agent_id`: The unique identifier of the agent. * `name`: The name of the agent. * `channel`: The name of the RTC channel the agent joined. * `total_turn_count`: The total number of dialogue turns in the session. * `pagination`: Pagination details for the response, including `page_index`, `total_pages`, and `is_last_page`. * Changes to [**Send a custom instruction**](/en/api-reference/api-ref/conversational-ai/think) * **Behavior changes** * [`on_listening_action`](/en/api-reference/api-ref/conversational-ai/think#on-listening-action): The default value has changed from `inject` to `interrupt`. When set to `interrupt`, the API immediately interrupts the current flow and initiates a new round of dialogue. * Changes to [**Notification event types**](reference/event-types) * **New events** * `112 turns finished`: Triggered after a session ends, delivering conversation turn data in batches for post-session analysis and processing. ### v2.6 Released on April 22, 2026. #### New features Included in this release: * **New TTS provider** * [Deepgram](models/tts/deepgram) * **New endpoint: Send a custom instruction** Adds a new endpoint to inject a custom text instruction into the agent's current conversation pipeline. The instruction is processed as user input, enabling scenarios such as implicit instruction injection, client-side event triggering, and voice and text collaboration. For details, see [Send a custom instruction](/en/api-reference/api-ref/conversational-ai/think). * **MLLM turn detection refactoring** Introduces a new [`mllm.turn_detection`](/en/api-reference/api-ref/conversational-ai/join#request-body-properties-mllm-turn-detection) object for configuring turn detection when using MLLM. When defined, this overrides the top-level `turn_detection` object. Supported modes vary by vendor: * OpenAI Realtime API: `agora_vad`, `server_vad`, `semantic_vad` * Google Gemini Live: `agora_vad`, `server_vad` * **Interruption control** Adds a new top-level [`interruption`](/en/api-reference/api-ref/conversational-ai/join#request-body-properties-interruption) object for unified management of agent interruption behavior. Supports `start_of_speech` and `keywords` trigger modes, and configurable handling strategies when interruption is disabled. #### Improvements This release includes the following enhancements: * **New agent state callbacks** Adds three callbacks to the [Android](/en/api-reference/api-ref/conversational-ai/client-toolkit/android#onagentlisteningchanged), [iOS](/en/api-reference/api-ref/conversational-ai/client-toolkit/ios#onagentlisteningchanged), and [web](/en/api-reference/api-ref/conversational-ai/client-toolkit/web#econversationalaiapievents) Toolkit event handler interface: * `onAgentListeningChanged`: Listen for changes in the agent's listening state to monitor when the agent starts or stops listening to user input. * `onAgentThinkingChanged`: Listen for changes in the agent's thinking state to monitor when the agent starts or stops processing a request. * `onAgentSpeakingChanged`: Listen for changes in the agent's speaking state to monitor when the agent starts or stops playing back speech. * **Added `name` field to all [NCS event](reference/event-types) payloads** Returns the agent name provided when starting the agent. #### API changes This release introduces the following changes to the RESTful API. * **New endpoint** * [Send a custom instruction](/en/api-reference/api-ref/conversational-ai/think): `POST /v2/projects/{appid}/agents/{agentId}/think` * Changes to [**Start a conversational AI agent**](/en/api-reference/api-ref/conversational-ai/join) * **New parameters** * [`interruption`](/en/api-reference/api-ref/conversational-ai/join#request-body-properties-interruption): Unified interruption control configuration. Supports `start_of_speech` and `keywords` trigger modes, and configurable handling strategies when interruption is disabled. * [`greeting_configs.delay_ms`](/en/api-reference/api-ref/conversational-ai/join#request-body-properties-llm-greeting-configs-delay-ms): Specifies the delay in milliseconds before the agent plays the greeting message after the user joins the channel. * [`llm.headers`](/en/api-reference/api-ref/conversational-ai/join#request-body-properties-llm-headers): Lets you pass custom headers in LLM requests, such as business-specific fields or tenant identifiers. * [`mllm.enable`](/en/api-reference/api-ref/conversational-ai/join#request-body-properties-mllm-enable): Enables the MLLM module. Replaces the deprecated `advanced_features.enable_mllm`. * [`mllm.turn_detection`](/en/api-reference/api-ref/conversational-ai/join#request-body-properties-mllm-turn-detection): Turn detection configuration for the MLLM module. When defined, overrides the top-level `turn_detection` object. * [`mllm.turn_detection.server_vad_config.idle_timeout_ms`](/en/api-reference/api-ref/conversational-ai/join#request-body-properties-mllm-turn-detection-server-vad-config-idle-timeout-ms): Idle timeout in milliseconds for server VAD mode. Applicable to OpenAI Realtime API only. * [`mllm.vendor`](/en/api-reference/api-ref/conversational-ai/join#request-body-properties-mllm-vendor) value `gemini`: Enables Google Gemini Live integration using the Gemini Developer API. * **Behavior changes** * `turn_detection` now handles SoS and EoS detection only. Interruption handling strategies have moved to the new top-level `interruption` field. * `turn_detection.config.start_of_speech.mode` values `keywords` and `disabled` are deprecated. Use the new `interruption` field instead. * When `turn_detection.end_of_speech.mode` is set to `semantic`, EOS detection supports English and Chinese only. For unsupported languages, the engine automatically falls back to VAD. * **Deprecated fields** * `advanced_features.enable_mllm`: Use `mllm.enable` instead. * `turn_detection.config.start_of_speech.mode` value `keywords`: Use `interruption.mode = "keywords"` instead. * `turn_detection.config.start_of_speech.mode` value `disabled`: Use `interruption.enable = false` with `interruption.disabled_config.strategy` instead. * **Removed fields** * `mllm.style` * `mllm.create_response` and `mllm.interrupt_response` for OpenAI Realtime API. * Changes to [**Notification event types**](reference/event-types) * **New fields** * Added `name` to all [NCS event payloads](reference/event-types). Returns the agent name provided when starting the agent. ### v2.5 Released on March 31, 2026. #### New features Included in this release: * **Preset model configurations** Adds a [`preset`](/en/api-reference/api-ref/conversational-ai/join#preset) parameter to the Join API. Pass a comma-separated list of preset names to apply predefined configurations for ASR, LLM, and TTS providers. When you use a preset, you do not need to provide the endpoint URL, API key, or model for the preset provider. Use the `asr`, `llm`, and `tts` fields to configure additional settings. When you use a preset, the corresponding ASR, LLM, or TTS service is provided through Agora-managed accounts. For details, see the [pricing](reference/pricing) page. Available presets: * ASR: `deepgram_nova_2`, `deepgram_nova_3` * LLM: `openai_gpt_4o_mini`, `openai_gpt_4_1_mini`, `openai_gpt_5_nano`, `openai_gpt_5_mini` * TTS: `minimax_speech_2_6_turbo`, `minimax_speech_2_8_turbo`, `openai_tts_1` * **Pause state detection** When enabled, the agent uses semantic understanding to detect when a user intends to pause the conversation. For example, if the user says "hold on" or "just a moment", the agent waits for further input instead of forwarding the utterance to the LLM. Controlled by the new [`pause_state_enabled`](/en/api-reference/api-ref/conversational-ai/join#request-body-properties-turn-detection-config-end-of-speech--mode-config-pause-state-enabled) parameter. * **Conversation turn insights** You can now query per-turn start, end, and latency metrics for completed agent sessions. See [Query conversation turn information](/en/api-reference/api-ref/conversational-ai/turns). * **Real-time TTS parameter updates with custom LLM** In scenarios with a custom LLM integrated, this release adds support for real-time TTS parameter updates during conversations, enabling a more natural conversational experience. This feature applies to the following use cases: * The custom LLM detects a user request for the agent to change its voice or speaking style. * The custom LLM identifies a user's positive emotion and adjusts TTS volume, pitch, speech rate, and other parameters in real time to better match the user's mood. For implementation details, refer to [Real-time TTS parameter updates](build/custom-model-integration/custom-llm#update-tts-parameters-in-real-time). * **New TTS provider** * [Murf (Beta)](models/tts/murf) * **New Avatar provider** * [Anam (Beta)](models/avatar/anam) #### Improvements This release includes the following enhancements: * **Migration from HeyGen API to LiveAvatar (Beta) API** Updates the HeyGen avatar model configuration to use the [LiveAvatar (Beta)](models/avatar/heygen) API. #### API changes This release introduces the following changes to the RESTful API. * **New endpoint: Query conversation turns** Adds a new GET endpoint to query conversation turn information for an agent session. After a conversation ends, use this endpoint to retrieve the start event, end event, and latency metrics for each turn within the session. For details, see [Query conversation turn information](/en/api-reference/api-ref/conversational-ai/turns). * Changes to [**Start a conversational AI agent**](/en/api-reference/api-ref/conversational-ai/join) * **New parameters added** * [`preset`](/en/api-reference/api-ref/conversational-ai/join#preset) * [`pause_state_enabled`](/en/api-reference/api-ref/conversational-ai/join#request-body-properties-turn-detection-config-end-of-speech--mode-config-pause-state-enabled) * **Deprecated fields deleted** The following deprecated fields are deleted: * `properties.silence_timeout` * `properties.advanced_features.enable_aivad` * `properties.llm.silence_message` ### v2.4 Released on February 2, 2026. #### New features Included in this release: * **MCP integration** Adds a [`llm.mcp_servers`](/en/api-reference/api-ref/conversational-ai/join#request-body-properties-llm-mcp-servers) field to the [Start a conversational AI agent](/en/api-reference/api-ref/conversational-ai/join) API to connect the LLM to an MCP (Model Context Protocol) server. Set [`advanced_features.enable_tools`](/en/api-reference/api-ref/conversational-ai/join#request-body-properties-advanced-features-enable-tools) to `true` to enable tool calls. This allows the agent to call tools provided by external services to extend functionality. For details, see [Use MCP tools](build/mcp-tools). * **Filler phrases** Adds a [`properties.filler_words`](/en/api-reference/api-ref/conversational-ai/join#request-body-properties-filler-words) field to insert pre-set or LLM-generated filler phrases during conversations to fill periods of silence while waiting for LLM output. This feature smooths dialogue flow, reduces user anxiety, and makes agent speech sound more natural. #### Improvements This release includes the following enhancements: * **Turn detection configuration optimization** The `turn_detection` parameter structure has been updated to provide more flexible conversation turn detection. The new structure uses a `mode` + `config` pattern with separate Start of Speech (SoS) and End of Speech (EoS) detection configurations. **Basic configuration** * `mode`: Conversation turn detection mode (currently supports `default`) * `config.speech_threshold`: Voice activity detection sensitivity **Start of Speech (SoS) detection** Detects when the user begins speaking. When SoS is detected while the agent is speaking, an interruption occurs. The [`config.start_of_speech`](/en/api-reference/api-ref/conversational-ai/join#request-body-properties-turn-detection-config-start-of-speech) parameter supports three detection modes: * **VAD mode** (`vad`): Triggered by voice activity detection. Supports configuration of interruption threshold, prefix padding, and ignore word list. * **Keyword mode (Beta)** (`keywords`): Based on keyword triggering. The agent begins conversation after detecting a specified keyword. * **Disabled mode** (`disabled`): Disables interruption. Supports append or ignore strategies. **End of Speech (EoS) detection** Detects when the user finishes speaking. When EoS is detected, the agent generates a response. The [`config.end_of_speech`](/en/api-reference/api-ref/conversational-ai/join#request-body-properties-turn-detection-config-end-of-speech) parameter supports two detection modes: * **VAD mode** (`vad`): Determines conversation end based on silence duration. * **Semantic mode** (`semantic`): Determines conversation end based on semantic understanding. Supports configurable maximum wait time. #### API changes This release introduces the following changes to the RESTful API. * Changes to [**Start a conversational AI agent**](/en/api-reference/api-ref/conversational-ai/join) * **[`turn_detection`](/en/api-reference/api-ref/conversational-ai/join#request-body-properties-turn-detection) has a revamped structure** All previous fields under `turn_detection` have been deprecated and replaced with a new data structure. * Deprecated * `turn_detection.interrupt_mode` * `turn_detection.interrupt_keywords` * `turn_detection.interrupt_duration_ms` * `turn_detection.prefix_padding_ms` * `turn_detection.silence_duration_ms` * `turn_detection.threshold` * **AIVAD activation** The `advanced_features.enable_aivad` field is now deprecated; to configure AIVAD activation use [`turn_detection.config.end_of_speech.mode = "semantic"`](/en/api-reference/api-ref/conversational-ai/join#request-body-properties-turn-detection-config-end-of-speech-mode). * **New parameters added** * [`advanced_features.enable_tools`](/en/api-reference/api-ref/conversational-ai/join#request-body-properties-advanced-features-enable-tools) * [`llm.mcp_servers`](/en/api-reference/api-ref/conversational-ai/join#request-body-properties-llm-mcp-servers) * [`properties.filler_words`](/en/api-reference/api-ref/conversational-ai/join#request-body-properties-filler-words) * [`turn_detection.mode`](/en/api-reference/api-ref/conversational-ai/join#request-body-properties-turn-detection-mode) * [`turn_detection.config`](/en/api-reference/api-ref/conversational-ai/join#request-body-properties-turn-detection-config) ### v2.3 Released on January 7, 2026. #### New features Included in this release: * **Control LLM response interruption in custom LLM scenarios** Adds a `metadata.interruptable` field in the first chunk of the `chat.completion.custom_metadata` object to control whether user speech can interrupt the agent's TTS output when using custom LLMs with streaming response (SSE). Use this to prevent interruptions when delivering critical information such as regulations, policies, or pricing. For more information, see [Configure LLM response interruption](build/custom-model-integration/custom-llm#configure-llm-response-interruption). * **RTC media content encryption** Adds an `rtc` parameter to the [Start a conversational AI agent](/en/api-reference/api-ref/conversational-ai/join) API for configuring RTC encryption. * **New ASR provider** * [Sarvam (Beta)](models/asr/sarvam) #### Improvements This release includes the following enhancements: * **ElevenLabs TTS compatibility optimization** Supports SSML parsing and forwarding to ensure ElevenLabs TTS correctly renders intended speech effects such as pauses, emphasis, and pronunciation. * **Latency optimization best practices** Adds a guide to [Optimize conversation latency](best-practices/optimize-latency). * **Cloud Recording best practices** Adds a guide to [Record conversations with Cloud Recording](best-practices/record-agent-conversation). #### API changes This release introduces the following changes to the RESTful API. * **Stop agent API is now asynchronous** The [Stop a conversational AI agent](/en/api-reference/api-ref/conversational-ai/leave) API now responds immediately after request parameters are validated. The request is processed asynchronously after the API returns. * Changes to [**Start a conversational AI agent**](/en/api-reference/api-ref/conversational-ai/join) * **New parameters added** * `properties.rtc` ### v2.2 Released on December 15, 2025. #### New features Included in this release: * **Geofencing** Use `geofence` configuration when you [Start a conversational AI agent](/en/api-reference/api-ref/conversational-ai/join) to limit which Agora servers the Conversational AI Engine can access based on geographic regions. See [Restrict agent zones](best-practices/regional-restrictions) for details. * **Agent greeting mode** Adds a field `llm.greeting_configs.mode` to the [Start a conversational AI agent](/en/api-reference/api-ref/conversational-ai/join) request to set the agent's greeting broadcast mode. The following modes are supported: * `single_every` (Default): The agent broadcasts a greeting every time a user joins a channel where there are no other users. * `single_first`: The agent broadcasts a greeting only when the first user joins a channel. * **New TTS providers** * [Sarvam (Beta)](models/tts/sarvam) #### Improvements This release includes the following enhancements: * **TTS parameter update** Adds `base_url` field to the TTS parameters for [OpenAI](models/tts/openai) and [ElevenLabs](models/tts/elevenlabs). This field specifies the endpoint URL for the TTS service. #### API changes This release introduces the following changes to the RESTful API. * Changes to [**Start a conversational AI agent**](/en/api-reference/api-ref/conversational-ai/join) * **New parameters added** * `properties.geofence` * `llm.greeting_configs.mode` ### v2.1 Released on December 5, 2025. #### New features Included in this release: * **Template variables** This version adds the `llm.template_variables` field to the [Start a conversational AI agent](/en/api-reference/api-ref/conversational-ai/join) API, used to insert variables into the agent's `system_messages`, `greeting_message`, `failure_message`, and `parameters.silence_config.content` text. By configuring these variables, the Conversational AI engine automatically replaces them with the corresponding values defined in `llm.template_variables`. Template variables, combined with prompt customization and SIP outbound calling functionality, enable you to dynamically inject content to automate processes such as automatic hang-up, voicemail recognition, automatic message leaving, and call transfer. * **Custom labels** This version adds a `labels` field to the [Start a conversational AI agent](/en/api-reference/api-ref/conversational-ai/join) API, enabling agents to carry custom labels. These labels are bound to the agent and returned in the `payload` field of all message notification callbacks from the Conversational AI engine, allowing you to implement custom business logic, such as tagging activity IDs, customer groups, and business scenarios. In SIP outbound call scenarios, you can pass a custom label in the `properties.labels` field when calling the outbound call interface to mark the call. #### API changes This release introduces the following changes to the RESTful API. * Changes to [**Start a conversational AI agent**](/en/api-reference/api-ref/conversational-ai/join) * **New parameters added** * `properties.labels` * `llm.template_variables` ### v2.0 Released on November 15, 2025. #### New features Included in this release: * **Telephony (Beta)** This version adds an outbound calling feature that enables the conversational AI agent to initiate an outbound call to a specified number through a POST API. After the call is answered, the agent can engage in real-time dialogue with the callee. This feature supports a wide range of outbound AI call scenarios. A set of phone number management APIs is also added for handling numbers connected to Conversational AI Engine. Refer to the API changes section for details. Telephony pricing The telephony feature is currently in **Beta** and is provided free of charge. Pricing terms may change upon official release. * **Selective attention locking (Beta)** This version adds the selective attention lock feature. Register voiceprints to enable the agent to identify specific speakers and suppress background voices and environmental noise, ensuring clearer, more focused conversations. * **Graceful exit** This version adds a new `farewell_config` field to [Start a conversational AI agent](/en/api-reference/api-ref/conversational-ai/join) API to configure the graceful exit feature. When enabled, calling the [Stop a Conversational Agent](/en/api-reference/api-ref/conversational-ai/leave) API causes the agent to enter an `IDLE` state before leaving the channel. * **Keyword interruption mode** This release adds a new option to the `turn_detection.interrupt_mode` field in the [Start a conversational AI agent](/en/api-reference/api-ref/conversational-ai/join) API. Set this field to `"keyword"` to enable keyword interruption mode. When this mode is enabled, the agent stops its current behavior after detecting any of the keywords specified in the `turn_detection.interrupt_keywords` field. * **Adaptive interruption mode** This release adds a new option to the `turn_detection.interrupt_mode` field in the [Start a conversational AI agent](/en/api-reference/api-ref/conversational-ai/join) API. Set this field to `"adaptive"` to enable adaptive interruption mode. When this mode is enabled, the agent dynamically increases the voice continuity threshold while speaking to reduce accidental interruptions. * **New ASR, LLM, MLLM, and TTS providers** * ASR * [OpenAI (Beta)](models/asr/openai) * [Speechmatics](models/asr/speechmatics) * [Google (Beta)](models/asr/google) * Amazon Transcribe (Beta) * [AssemblyAI (Beta)](models/asr/assembly-ai) * LLM * [Groq](models/llm/groq) * [Amazon Bedrock](models/llm/amazon) * MLLM * [Google Gemini Live](models/mllm/gemini) * TTS * [Rime (Beta)](models/tts/rime) * [Fish Audio (Beta)](models/tts/fish-audio) * [Google (Beta)](models/tts/google) * [Amazon Polly (Beta)](models/tts/amazon) * **New webhook notification events** This release adds three new webhook notification event types to support metrics reporting and call-state monitoring: * `111`: [agent metrics](reference/event-types#111-agent-metrics) Notifies real-time performance metrics for each dialogue turn, including ASR, LLM, and TTS latency measurements. * `201`: [inbound call state](reference/event-types#201-inbound-call-state) Reports state changes for incoming calls, such as when a call starts, is answered, transferred, or hung up. * `202`: [outbound call state](reference/event-types#202-outbound-call-state) Reports state changes for outbound calls initiated by the agent, including call start, dialing, ringing, answer, and hang-up events. #### Improvements This release includes the following enhancements: * **Support for avatars with MLLMs** Added support for using avatars with MLLMs. #### API changes This release introduces the following changes to the RESTful API. * Changes to [**Start a conversational AI agent**](/en/api-reference/api-ref/conversational-ai/join) * **New parameters added** * `properties.parameters.farewell_config` * `properties.advanced_features.enable_sal` * `properties.sal` * `properties.sal.sal_mode` * `properties.sal.sample_urls` * `properties.turn_detection.interrupt_mode` (supports `adaptive` and `keyword` values) * `properties.turn_detection.interrupt_keywords` * `properties.turn_detection.interrupt_duration_ms` (migrated from `vad.interrupt_duration_ms`) * `properties.turn_detection.prefix_padding_ms` (migrated from `vad.prefix_padding_ms`) * `properties.turn_detection.silence_duration_ms` (migrated from `vad.silence_duration_ms`) * `properties.turn_detection.threshold` (migrated from `vad.threshold`) * **Deprecated** * The `vad` interface is deprecated. All configuration items have been moved to the `turn_detection` field. * **New APIs** * Telephony (Beta) * Initiate an outbound call * Retrieve call records * Hang-up a call * Retrieve call status * Phone number management (Beta) * Retrieve list of numbers * Import number * Retrieve number information * Update number configuration * Delete number #### Toolkit API This release renames all APIs and parameters containing the word `transcription` in the client-side subtitle API to use `transcript`, as shown below: #### Android * `onTranscriptionUpdated` renamed to `onTranscriptUpdated` * `TranscriptionRenderMode` renamed to `TranscriptRenderMode` * `TranscriptionType` renamed to `TranscriptType` * `TranscriptionStatus` renamed to `TranscriptStatus` * `Transcription` renamed to `Transcript` #### iOS * `onTranscriptionUpdated` renamed to `onTranscriptUpdated` * `TranscriptionRenderMode` renamed to `TranscriptRenderMode` * `TranscriptionType` renamed to `TranscriptType` * `TranscriptionStatus` renamed to `TranscriptStatus` * `Transcription` renamed to `Transcript` #### Web * `TRANSCRIPTION_UPDATED` renamed to `TRANSCRIPT_UPDATED` ### v1.7 Released on July 31, 2025. #### New features * **AI avatars** Create visual avatar representations for your conversational agents using third-party avatar providers. AI avatars provide a visual presence during voice interactions, making conversations feel more natural and engaging. Enable AI avatars by setting `avatar.enable` to `true` and configuring the `avatar.vendor` and `avatar.params` fields when calling [Start a conversational AI agent](/en/api-reference/api-ref/conversational-ai/join) to create your agent. Info AI avatars require video streaming and incur additional charges. See [video calling pricing](/en/realtime-media/rtc/reference/pricing) for details. * **Selective attention locking (Beta)** This version introduces the selective attention locking feature, which uses voiceprint recognition technology to identify and filter out the speaker while suppressing background noise. This enhances the efficiency of conversational AI, particularly improving speech recognition accuracy. To experience this feature, contact [technical support](mailto\:support@agora.io). * **Send picture messages (Beta)** The toolkit now includes an API for [sending picture messages](build/send-multimodal-messages). You can send image URLs to the main model, which automatically references the image in future interactions to generate more relevant responses. A new callback is available to receive image message receipt details after successful transmission. Info * The picture messaging feature is currently in Beta and free for a limited time. * Image processing depends on the capabilities of the integrated LLM. Ensure the LLM you connect to the Conversational AI Engine supports image input. #### API changes This release introduces the following modifications to the RESTful API. * [Start a conversational AI agent](/en/api-reference/api-ref/conversational-ai/join) * **New parameters added** * `avatar.enable` * `avatar.vendor` * `avatar.params` ##### Toolkit API * Android: * `chat` * `ImageMessage` * `onMessageReceiptUpdated` * `MessageReceipt` * iOS: * `chat` * `ChatMessage` * `ChatMessageType` * `ImageMessage` * `onMessageReceiptUpdated` * `MessageReceipt` * Web: * `chat` * `TMessageReceipt` * `EChatMessagePriority` * `EChatMessageType` * `IChatMessageBase` * `IChatMessageImage` ### v1.6 Released on July 15, 2025. #### New features * **Support for OpenAI Realtime API** Integrate Multimodal Large Language Models (MLLMs) with Conversational AI Engine to enable end-to-end real-time audio and text interactions. See [OpenAI Realtime API](models/mllm/openai) for integration details. * **Support for more TTS vendors** Conversational AI Engine now supports the following additional TTS vendors: * [Cartesia](models/tts/cartesia) * [OpenAI](models/tts/openai) * **Custom ASR provider support** To improve flexibility in configuring conversational agents, this release allows you to select a custom automatic speech recognition (ASR) provider. The [Start a conversational AI agent](/en/api-reference/api-ref/conversational-ai/join) API now includes the following new parameters: * `asr.vendor`: Specify the ASR provider * `asr.params`: Configure ASR parameters The following ASR providers are supported: * **ARES** (default) * **Microsoft Azure** * **Deepgram** **Billing update:** In earlier versions, the service fee included the cost of the Ares ASR provider. Starting in v1.6, the pricing is restructured as follows: * If you use **ARES ASR**, the total price remains unchanged: ***Total cost = Conversational AI Engine Audio Basic Task + ARES ASR Task*** * If you use **a different ASR provider**, you are charged **only** the new **Conversational AI Engine Audio Basic Task** fee. For further details, see [Pricing](reference/pricing). * **Multi-platform toolkit** Agora now offers a toolkit to help you quickly build conversational agent apps. The toolkit is available for [**iOS**](/en/api-reference/api-ref/conversational-ai/client-toolkit/ios), [**Android**](/en/api-reference/api-ref/conversational-ai/client-toolkit/android), and [**Web**](/en/api-reference/api-ref/conversational-ai/client-toolkit/web), and includes APIs for common scenarios. Call these APIs to combine the capabilities of the Agora Voice SDK and Signaling SDK to achieve the following functions: * [**Display live transcript**](build/transcripts) Display real-time text output of user-agent conversations. The transcript component is now more robust, with better error handling, session management, and extensibility. * [**Interrupt the agent**](build/shape-the-conversation/interrupt-agent) Stop the agent from speaking or thinking mid-conversation. * [**Client-side events**](build/handle-runtime-events/event-notifications) Track changes in conversation state, performance metrics, and error events. * [**Optimize audio settings**](best-practices/audio-setup) Quickly apply best-practice audio configurations to improve agent responsiveness and clarity. #### API changes ##### REST API This release introduces several important modifications to the RESTful API. * [Start a conversational AI agent](/en/api-reference/api-ref/conversational-ai/join) * **New parameters added** * `asr.vendor` * `asr.params` * `advanced_features.enable_mllm` * `properties.mllm` * `turn_detection.type` * `turn_detection.interrupt_duration_ms` * `turn_detection.prefix_padding_ms` * `turn_detection.silence_duration_ms` * `turn_detection.threshold` * `turn_detection.create_response` * `turn_detection.interrupt_response` * `turn_detection.eagerness` * `parameters.enable_metrics` * `parameters.data_channel` * `parameters.enable_error_message` ##### Toolkit APIs * [Android SDK](/en/api-reference/api-ref/conversational-ai/client-toolkit/android) * [iOS SDK](/en/api-reference/api-ref/conversational-ai/client-toolkit/ios) * [Web SDK](/en/api-reference/api-ref/conversational-ai/client-toolkit/web) ### v1.5 Released on June 9, 2025. #### New features * **Voice interruption mode** This release adds the `turn_detection.interrupt_mode` parameter to the [Start a conversational AI agent](/en/api-reference/api-ref/conversational-ai/join) API, allowing you to control how the agent handles human voice interruptions. The following modes are supported: * **`interrupt`**: (Default) The human voice immediately interrupts the agent. The agent terminates the current interaction and processes the new human voice input. * **`append`**: The human voice does not interrupt the agent. The agent processes the newly received human voice request after the current interaction ends. * **`ignore`**: The agent ignores human voice requests received during speaking or thinking. These requests are discarded and not stored in the context. * **TTS filtering** This release adds the `tts.skip_patterns` parameter to the [Start a conversational AI agent](/en/api-reference/api-ref/conversational-ai/join) API. This parameter controls whether the TTS module skips bracketed content when reading LLM response text. This prevents the agent from vocalizing structural prompt information like tone indicators, action descriptions, and system prompts, creating a more natural and immersive listening experience. #### API changes This release introduces several important modifications to the RESTful API. * [Start a conversational AI agent](/en/api-reference/api-ref/conversational-ai/join) * **New parameters added** * `turn_detection.interrupt_mode` * `parameters.silence_config` * `tts.skip_patterns` ### v1.4 Released on May 29, 2025. #### New features * **Metadata support for LLM requests** This release adds the `llm.vendor` parameter to the [Start a conversational AI agent](/en/api-reference/api-ref/conversational-ai/join) API. When set to `"custom"`, the agent includes additional metadata when calling the LLM, such as `turn_id` and `timestamp`. * **Support for Anthropic** Conversational AI Engine now supports `anthropic` as a request style for chat completion. Refer to the `llm.style` parameter in [Start a conversational AI agent](/en/api-reference/api-ref/conversational-ai/join). #### Improvements This release includes the following enhancements: * **Advanced LLM configuration**: The [Update agent configuration](/en/api-reference/api-ref/conversational-ai/update) API now supports: * `llm.system_messages` for updating system prompts * `llm.params` for modifying configuration parameters used when calling the large language model #### API changes This release introduces several important modifications to the RESTful API. * [Start a conversational AI agent](/en/api-reference/api-ref/conversational-ai/join) * **New parameters added** * `llm.vendor` * **Removed parameters:** * `agent_rtm_uid` * [Update agent configuration](/en/api-reference/api-ref/conversational-ai/update) * **New parameters added** * `llm.system_messages` * `llm.params` ### v1.3 Released on April 16, 2025. #### New features * **Agent conversation history**: This version adds two methods to retrieve an agent's history. The history includes messages exchanged between the user and the agent and timestamps of agent creation and exit. * Call the RESTful API `history` endpoint to [Retrieve agent history](/en/api-reference/api-ref/conversational-ai/history). * Subscribe to the [agent history event](reference/event-types#103-agent-history) through the [Agora message notification service](build/handle-runtime-events/webhooks). When the agent stops, Agora automatically sends the agent's history to your business server through a Webhook callback. #### Improvements * **Customize the priority of broadcast information**: This version upgrades the [Broadcast a message using TTS](/en/api-reference/api-ref/conversational-ai/speak) interface and adds two new configuration parameters related to broadcast interruption logic: * `priority`: Sets the priority of the message broadcast. Supports setting the following priorities: * `INTERRUPT` High priority * `APPEND`: Medium priority * `IGNORE`: Low priority * `interruptable`: Configure whether to allow human voice to interrupt the agent's broadcast. #### API changes * Adds the [Retrieve agent history](/en/api-reference/api-ref/conversational-ai/history) method. * Adds `priority` and `interruptable` parameters to the [Broadcast a message using TTS](/en/api-reference/api-ref/conversational-ai/speak) method. ### v1.2 Released on April 10, 2025. #### New features * **Broadcast a message using TTS**: A new message broadcast interface enables a specified agent to deliver a custom message. When interacting with an agent, calling this interface interrupts the agent's speech and thinking process, allowing the TTS module to immediately broadcast the custom message. * **Interrupt the agent**: The interrupt agent endpoint lets you stop the specified agent's speech and thinking process. #### API changes This version adds the following APIs: * [Broadcast a message using TTS](/en/api-reference/api-ref/conversational-ai/speak) * [Interrupt the agent](/en/api-reference/api-ref/conversational-ai/interrupt) ### v1.1 Released on March 27, 2025. #### New features The [Start a conversational agent](/en/api-reference/api-ref/conversational-ai/join) API adds the `enable_rtm` and `agent_rtm_uid` parameters to enable Signaling integration with the conversational AI agent. When this feature is enabled, the agent can leverage the Signaling SDK to obtain a user's custom context information such as speaking status, selected text, signature, and score, and pass this data to the agent to generate more relevant content. For details, see [Transmit custom information](build/shape-the-conversation/custom-information). #### Improvements To help you quickly integrate a custom large language model (LLM), this version adds documentation for [Custom LLMs](build/custom-model-integration/custom-llm). Refer to the sample code in the documentation to integrate your custom model into the Conversational AI Engine and enable advanced capabilities such as Retrieval-Augmented Generation (RAG), multi-modal processing, and tool invocation. #### API changes The `POST` method to [Start a conversational AI agent](/en/api-reference/api-ref/conversational-ai/join) now includes the `enable_rtm` and `agent_rtm_uid` parameters. ### v1.0 (Public Beta) This version, released on March 4, 2025, adds pricing information for the Agora Conversational AI Engine. For more information, see [Pricing](reference/pricing). #### Integration guide To achieve the best conversation experience, use Agora Conversational AI Engine with the following Agora SDKs: * Agora RTC Native SDK, v4.5.1 or later. * Agora RTC Web SDK, version 4.23.2 or later. #### New features * **Live transcripts**: Supports real-time text output of conversations between users and the AI agent for transcript display in your app's UI. Agora provides an open-source transcript processing module. Integrate the module and call its API to implement live transcript. For details, see [Display live transcripts](build/transcripts). * **Message Notification Service**: Introduces a new Conversational AI Engine message notification service. Configure it in the Agora console and subscribe to agent creation, stop, and error events. When a subscribed event occurs, Agora sends the details to your specified callback address. See [Receive event notifications](build/handle-runtime-events/webhooks). * **Keywords**: Enhances recognition accuracy of Conversational AI Engine for proprietary words by adding keywords. This feature is currently in Beta stage. For details, [contact technical support](https://agoraio.zendesk.com/hc/en-us). ### v1.0 (Private Beta) Released on February 18, 2025. The first beta release of the Conversational AI Engine brings natural, smooth, low-latency, and highly reliable real-time voice conversations with AI agents to Agora channels. It enables you to efficiently build intelligent and immersive interactive experiences. See [Product overview](/en/ai) for details. #### Core features * **Real-time voice conversation** Supports natural and smooth real-time voice conversations with AI. It delivers a low-latency, ultra-responsive interactive experience as if the user is communicating with a real person. * **Intelligent noise suppression** Intelligently identifies and suppresses background noise, ensuring clear sound transmission even in noisy environments to provide users with a high-quality audio experience. * **Background human voice suppression** Suppresses background voices and noise while accurately preserving the primary speaker's voice. This ensures a clear and focused interactive experience in multi-speaker environments. * **Intelligent interruption handling** Allows users to interrupt AI at any time to ensure quick and natural responses. This feature enables smooth transitions and avoids mechanical interactions. * **Intelligent transmission** An AI-optimized transmission algorithm ensures stable voice data delivery even in weak network conditions where packet loss reaches 80%. This guarantees conversation continuity and reliability across diverse network environments. * **Flexible arrangement** Supports multiple Large Language Model (LLM) and Text-to-Speech (TTS) providers, enabling flexible orchestration to meet diverse business needs and deliver highly customizable AI dialogue solutions. * **Multi-platform support** Compatible with iOS, Android, Web, and various embedded hardware platforms, providing a seamless and consistent cross-platform experience. #### Integration guide * For the best conversational experience, Agora recommends using Conversational AI Engine with specific Agora Video/Voice SDK versions. For details, [contact technical support](mailto\:support@agora.io). * The number of Peak Concurrent Users (PCU) allowed to call the server API under a single App ID is limited to 20. If you need to increase this limit, [contact technical support](mailto\:support@agora.io). # About Agora (/en/introduction/about-agora) ## What Agora is Agora is a real-time platform for products where people, devices, services, and AI systems need to stay connected inside the same live session. Instead of treating voice, video, messaging, AI orchestration, and media processing as separate stacks, Agora gives you a shared real-time foundation you can build on. You can start with one immediate use case, such as voice calling or an AI agent, and then extend the same session architecture into messaging, cloud processing, analytics, and production operations. Think in live sessions, not isolated features Agora is easiest to understand when you treat it as a shared real-time layer for interaction, coordination, AI participation, and media workflows. ## What problems it solves Teams usually come to Agora when they need to solve more than one live-product problem at the same time: * Keep voice and video interactions usable under real-world network conditions. * Coordinate room state, presence, events, and messaging alongside live media. * Add voice AI into a live session instead of bolting AI onto a disconnected backend flow. * Record, transcribe, translate, transform, or distribute media after the session starts. * Operate production systems with the credentials, security, monitoring, and governance live products require. In practice, this means Agora is most useful when your product is not just sending a message or making a single API call, but managing an ongoing real-time experience. ## Product families Agora's docs are easier to navigate when you group the platform into four product families instead of reading it as a list of unrelated SKUs. Build voice agents, assistants, tutors, support flows, and multimodal AI experiences that participate in a live real-time session. Build calling, meetings, classrooms, livestream interaction, smart-device media flows, and other low-latency communication products. Keep users, devices, and room state synchronized with chat, presence, metadata, signaling, and collaboration surfaces. Extend live sessions with recording, speech-to-text, translation, push and pull workflows, transcoding, and distribution tooling. ## How these products fit together The most useful mental model is to think in layers of responsibility: 1. **Projects and credentials** define access, billing scope, and service boundaries. 2. **Real-time sessions** connect users, devices, and agents inside a shared live context. 3. **Voice, video, and messaging** carry interaction, room state, and events through that context. 4. **AI and server-side control** add orchestration, automation, callbacks, and business logic. 5. **Cloud media services** capture, transform, translate, distribute, and observe the live session. You do not need every layer on day one. The value of the platform is that these layers can be added over time without forcing you to replace the underlying session model. ## Common solution patterns Most implementations start from a small number of repeatable patterns: Start with live voice or video when the main requirement is low-latency communication between participants. Add room messaging, presence, and coordination when participants also need shared state, chat, or event-driven flows. Add a voice agent when an AI system needs to join the session as a live participant instead of a separate backend service. Extend a live session with archive, compliance, captions, transcripts, translation, or post-session processing. Build smart-device or embedded experiences where microphones, speakers, sensors, and agents operate through the same real-time path. These patterns are helpful because they let you choose a narrow first milestone while keeping a clear path to the rest of the platform. ## Where to start Choose the next page based on the first problem you need to solve: * Start with [Start with AI](/en/introduction/start-with-ai) if you want an AI-assisted path using Skills, MCP, and agent-friendly docs. * Go to [Conversational AI](/en/introduction/conversational-ai) if your main product is a voice agent, assistant, tutor, or spoken AI experience. * Go to [Real-Time Voice & Video](/en/introduction/realtime-audio-video) if your first milestone is calling, meetings, classrooms, livestream interaction, or device media transport. * Go to [Messaging & Presence](/en/introduction/messaging-presence) if chat, presence, signaling, or shared room state is your main coordination problem. * Go to [Cloud Media Services](/en/introduction/cloud-media-services) if you need recording, transcription, translation, media routing, or stream transformation around a live session. * Read [Core Concepts](/en/introduction/core-concepts) if you want the platform primitives before choosing a build path. When you move from prototype to production, continue with [Projects](/en/introduction/projects), [Console setup](/en/introduction/console-setup), [Billing](/en/introduction/billing), and [Security and privacy](/en/introduction/security-privacy). # Account (/en/introduction/account) This page shows you how to sign up for an Agora account, create a new project, and get the App ID and App Certificate to generate a temporary token. ## Get started with Agora Use [Agora Console](https://console.agora.io) to create and configure projects, and manage your Agora account. Each project provides the credentials your app needs to integrate Agora products. ### Sign up for an Agora account To use Agora products and services, you need an Agora account. Go to the [Agora sign-up page](https://sso2.agora.io/en/signup) and sign up with an email address, or with a Google or GitHub account. After you sign up, you are logged in automatically. For later visits, log in to [Agora Console](https://console.agora.io) with your email address or linked third-party account. ### Your first Agora project When you create a new account, Agora creates a default project and opens it for you when you log in. To create a new project: 1. In [Agora Console](https://console.agora.io), expand the **Projects** dropdown at the top of the page. 2. Select **Create project**. ![Create a new project button](https://assets-docs.agora.io/images/console/create-new-project-button.png) 3. Enter a **Project Name**. Agora creates the project with a default setup: App ID and token authentication enabled, and Conversational AI with Signaling (RTM) enabled. ![Create project dialog](https://assets-docs.agora.io/images/console/create-new-project.png) 4. Click **Create Project**. You see the new project selected in the **Project** dropdown. ### Get the App ID and App Certificate Agora automatically assigns a unique identifier to each project, called an App ID. To generate authentication tokens on your app server, you also need the project's App Certificate. To get the App ID and App Certificate for a project: 1. On the **Home** page, select **Manage credentials** on the project card. 2. Use the copy buttons next to **App ID** and **Primary certificate** to get your App ID and App Certificate, respectively. ![Project credentials panel](https://assets-docs.agora.io/images/console/project-credentials.png) For full App Certificate management details, see [Projects](/en/introduction/projects#manage-app-certificates). ### Generate temporary tokens To ensure communication security, best practice is to use tokens to authenticate users who log in from your client app. To generate a temporary RTC + RTM token for your project: 1. On the **Home** page, select **Manage credentials** on the project card. 2. Select **Generate Temp Token**. 3. Enter a **Channel** name and **RTC UID** in the pop-up box and select **Generate token**. ![Generate Temp Token dialog](https://assets-docs.agora.io/images/console/generate-temp-token.png) 4. Copy the generated token. For more information on managing other aspects of your Agora account, see [Console setup](/en/introduction/console-setup). # Agora CLI (/en/introduction/agora-cli) Install the Agora CLI when you want the shortest path to a working Agora starter. Use it to log in, initialize an official starter, write environment values, switch projects, and diagnose setup problems from the terminal. If you want to get a demo running before you fine-tune prompts or architecture, start here. After the starter works, add [Agora MCP](/en/introduction/agora-mcp) or [Agora Skills](/en/introduction/agora-skills) if you want your assistant to stay grounded in current docs and Agora-specific workflows. ## Install the Agora CLI macOS and Linux Windows (PowerShell) ```bash curl -fsSL https://dl.agora.io/cli/install.sh | sh agora --help ``` ```powershell irm https://dl.agora.io/cli/install.ps1 | iex agora --help ``` ## Quick start Follow this path if you want to get an official starter running as quickly as possible. TypeScript Python Go ```bash agora login agora init my-nextjs-demo --template nextjs cd my-nextjs-demo pnpm install pnpm dev ``` ```bash agora login agora init my-python-demo --template python cd my-python-demo bun install bun run dev ``` ```bash agora login agora init my-go-demo --template go cd my-go-demo make setup make dev ``` ### What happens in this flow 1. `agora login` authenticates your machine. 2. `agora init` clones an official starter, binds it to your Agora project, and writes the environment you need locally. 3. The runtime-specific install and dev commands start the application. If the starter runs, move on to application-specific work. If not, troubleshoot in the same order you experienced the setup below. ## If something fails ### The `agora` command is not available If `agora --help` does not work after installation, rerun the installer first. On macOS and Linux, make sure you used the install command with `--add-to-path`. If PowerShell blocks the Windows install script, download `install.ps1` and run: ```text powershell -ExecutionPolicy Bypass -File .\install.ps1 ``` ### Login or starter setup fails Run the setup path again in order: ```bash agora login agora init my-demo --template nextjs ``` This covers the two most common first-run failures: the machine is not authenticated, or the starter was not initialized correctly. ### The starter runs incorrectly Use the CLI to check readiness, credentials, and feature enablement: ```bash agora project doctor ``` ### Project settings or credentials look wrong Inspect the current environment values that the CLI exported for your active project: ```bash agora project env --shell ``` This is the right check when the local app starts but the agent does not join, transcripts do not appear, or the active project looks wrong. ## Build from scratch instead If you are not using an official starter, create or switch projects first, then export credentials into the local environment. TypeScript Python Go ```bash agora login agora project create conv-ai-tutorial agora project use conv-ai-tutorial agora project env --shell ``` ```bash agora login agora project create conv-ai-tutorial agora project use conv-ai-tutorial agora project env --shell ``` ```bash agora login agora project create conv-ai-tutorial agora project use conv-ai-tutorial agora project env --shell ``` You can also switch the active project later or create a new one explicitly: ```bash agora project create my-project agora project use my-project ``` The CLI gives your assistant reproducible local steps it can execute and verify, especially during starter bootstrapping, project binding, environment generation, and first-line diagnosis. # Agora MCP (/en/introduction/agora-mcp) Install Agora MCP when you want your assistant to check the latest Agora docs before it writes or changes code. Use it to confirm current SDK methods, parameters, quickstarts, and product or platform differences while you build or debug. This is most useful when documentation freshness is the risky part of the task. If you also want help choosing the right Agora workflow or starter, add [Agora Skills](/en/introduction/agora-skills). ## Install the MCP server `Agora MCP` is included when you install [Agora Skills](/en/introduction/agora-skills), but you can also install the MCP server on its own: ```text https://mcp.agora.io ``` Agora MCP is available only in coding tools where you install or configure it. A regular browser-based chat session does not connect to this server automatically. Cursor Claude Code Codex Gemini CLI Manual installation Add the following JSON to your MCP configuration: ```json { "mcpServers": { "agora-docs": { "url": "https://mcp.agora.io" } } } ``` ```bash claude mcp add --transport http agora-docs https://mcp.agora.io ``` ```bash codex mcp add --url https://mcp.agora.io agora-docs ``` ```bash gemini mcp add --transport http agora-docs https://mcp.agora.io ``` Add `https://mcp.agora.io` to your MCP client and use `http` or `Streamable HTTP` transport if the client asks for a transport type. ## Start with a prompt Install the server in your coding tool, then ask your assistant to search Agora docs before it writes code. Include the Agora product, the target platform, and the task type in the same prompt. ```text Use Agora MCP to find the current web quickstart for Conversational AI and explain the minimum setup for a browser voice demo. ``` ## Make docs lookup part of the workflow If your repository includes `AGENTS.md` or similar instructions, tell your assistant to prefer Agora MCP for product-specific lookups. That turns docs checks into a normal part of the workflow instead of an optional extra step. If your MCP client is not available, these fallback resources still help: * Use `https://docs.agora.io/llms.txt` as the lightweight documentation index. Its entries link directly to Markdown pages. * Append `.md` to a documentation URL to read that page as Markdown. For example, use `https://docs.agora.io/en/ai/get-started/quickstart.md`. * Use `https://docs.agora.io/llms-full.txt` only for offline indexing or bulk processing. It is too large to use as the default context for an interactive assistant. # Agora Skills (/en/introduction/agora-skills) Install Agora Skills when you want your assistant to choose the right Agora workflow before it starts coding. Skills helps you pick the right product path, starter, and setup sequence, so your assistant can follow official Agora patterns instead of guessing. Use Skills when docs lookup alone is not enough and the harder part is deciding what to build first or which official path to follow. If you also want fresh documentation while your assistant works, pair Skills with [Agora MCP](/en/introduction/agora-mcp). ## Install Agora Skills Skills CLI Manual installation ```bash npx skills add AgoraIO/skills ``` Skills activate automatically when the agent detects a relevant Agora task. Clone the repository and point your coding assistant to the Agora skill files: ```bash git clone https://github.com/AgoraIO/skills.git ~/agora-skills ``` Use `skills/agora/` as the directory entry point, or load `SKILL.md` directly if your tool expects a single root file. If you install Skills manually, place the files where your assistant already looks for local instructions: Claude Code Cursor Windsurf GitHub Copilot Other tools Run the following commands inside Claude Code: * **User-level** (available across all your projects) ```bash /plugin marketplace add AgoraIO/skills /plugin install agora ``` * **Project-level** (shared with your team via version control) ```bash /plugin marketplace add AgoraIO/skills /plugin install agora --scope project ``` Copy or symlink `skills/agora/` into `.cursor/rules/`. Add `skills/agora/` to Cascade context. Reference the files with `@workspace` or add them to Copilot instructions. Point the tool at `skills/agora/` or the top-level `SKILL.md`. ## Start with a prompt Install Skills, then ask your assistant to choose the correct Agora path before it starts scaffolding or editing code. ```text Use Agora Skills and Agora MCP to help me build a web-based conversational AI demo. Check the official docs first, choose the right starter, and use the Agora CLI for setup. ``` From there, let your assistant choose the starter and setup path before it edits code. # Cloud Media Services (/en/introduction/cloud-media-services) Cloud Media Services is the capability layer for products that need to do more with live media than simply transport it between participants. This category covers the cloud-side workflows that capture, transform, route, translate, and extend live sessions. Agora's product map for this layer includes: * **Recording** in cloud or local-server form * **Media Push / Pull** and Media Gateway workflows * **Speech-to-Text / Translation** * **Transcoding / Compositing** * **Extensions / Analytics** and adjacent cloud-side operational services ## What this category is good for This category is the right entry point when you need: * recording for archive, replay, compliance, or review * pushing or pulling media between Agora and external streaming systems * speech transcription or live translation as a cloud-side service * stream composition, transcoding, and output shaping * analytics, extensions, and post-session media workflows The shared requirement is that the media session must continue to create value after, around, or beyond the primary live interaction. ## Capability map Capture live sessions into replayable and reviewable assets through cloud-managed or self-managed recording paths. Move media between Agora and external live-streaming or online media systems through push, pull, and gateway workflows. Turn live speech into text and translated text for accessibility, AI preprocessing, records, and multilingual products. Reshape streams into layouts, output formats, and distribution-friendly media workflows for playback and broadcast. Extend product behavior and add production observation, diagnosis, and ecosystem integrations around the media path. ## Start here ## Key characteristics Handle what happens to live media before, during, and after the user-facing session. Connect live interaction with storage, broadcast delivery, analytics, moderation, and external streaming workflows. Work with recordings, transformed streams, transcripts, translations, and playback-oriented outputs as one service layer. Add capabilities that teams usually need once a live experience moves beyond prototype stage. Reuse the same cloud media services whether the upstream source is RTC, AI interaction, or host-audience live streaming. ## What to read next * [Media processing and distribution](/en/realtime-media/media-processing-and-distribution) * [Cloud Recording](/en/realtime-media/cloud-recording) * [Media Gateway](/en/realtime-media/rtmp-gateway) * [Transcoding](/en/realtime-media/transcoding) * [Real-time Transcription and Translation](/en/realtime-media/speech-to-text) # Community Resources (/en/introduction/community-resources) Agora Console is the unified portal for configuring, purchasing, and managing Agora products and services. ## Dashboard overview After logging in to Console, the overview dashboard provides quick access to: * projects * billing * usage * members and roles * documentation * code samples * support * Agora Analytics * Extensions Marketplace ## Additional resources * [Support](/en/introduction/support) * [Release Notes](/en/ai/release-notes) * [Usage Analytics](/en/introduction/usage-analytics) * [AI best practices](/en/ai/best-practices) * [AI product map](/en/ai/domain-overview) ## Status and operations Use the Agora Status Page and the Support Center to monitor service health, track incidents, and follow maintenance events. # Console overview (/en/introduction/console-setup) Use Agora Console to manage your projects and account settings. Console settings are organized into two groups: * **Project-level settings**: Apply to a specific project. * **Account settings**: Apply across your entire Agora account. ## Project-level view When you log in to Agora Console, you land on the **Home** page for the selected project. Use the **Project** dropdown at the top of the page to switch between projects. Agora creates a default project for you when you create a new account. ![Home page](https://assets-docs.agora.io/images/console/project-overview.png) Use the sidebar to switch between project-level pages. ### Home The **Home** page is the landing page for a project in Agora Console. It shows quick links to start building an AI agent, RTC services, your project's App ID and certificate, and a summary of your usage and billing across RTC, Signaling, and Conversational AI. ### Credentials Select **Manage credentials** on the project card, on the **Home** page, to open the **Project credentials** panel. This panel shows the basic identity, security configuration, and credentials for the selected project. * **Project Name**: An editable, human-readable name for the project. Select **Edit** to rename the project. * **Project ID**: An internal ID for your project. This is **not** the App ID. * **Status**: The status of your Agora project. Select **Disable project** to disable it. * **App ID**: A unique, non-editable identifier for the project. Select the copy icon to copy the ID to your clipboard. * **Primary certificate**: The project's primary server-side secret, used to generate authentication tokens. Enabling the primary certificate (default) secures the project and requires clients to authenticate with a token. Treat the primary certificate like a database password. * **Secondary certificate**: An optional fallback certificate used during certificate rotation. Enable it using the **Enable Secondary** button. You can't delete a secondary certificate while it is enabled; disable it first before deleting. You can copy your **App ID** and **Primary certificate** individually, generate a temporary token, or download your Agora `.env` file for use in your projects. ![Project credentials panel](https://assets-docs.agora.io/images/console/project-credentials.png) **Rotate certificates** If you suspect your primary certificate has been compromised, rotate it using the secondary certificate. Replacing the primary certificate may impact your live business, so only rotate certificates when necessary. 1. Select **Add a Certificate** to enable the secondary certificate. 2. Select **Make Primary** against the secondary certificate to swap the primary and secondary certificates. 3. Select **Disable Secondary** to disable the secondary certificate (the original primary certificate). 4. Once your users have switched to the new certificate, select **Delete Secondary** to delete it. Once deleted, a certificate can't be restored. Before deleting it, confirm that users have switched to the new certificate. Deleting a certificate immediately invalidates all tokens generated with it, and users already connected can't renew their tokens using it. ### RTC Services Select **RTC Services** in the sidebar to view and configure the Agora products and services available for the current project. Services are grouped into the following categories: * **Intelligence**: Conversational AI Engine, Real-Time Speech-to-Text. * **Media**: Cloud Recording, Media Push, Video Screenshot Upload, Media Gateway, Media Pull. * **Messaging and Signaling**: Signaling, Chat. * **Collaboration**: Whiteboard, Flexible Classroom. * **Infrastructure**: Cloud Proxy, Co-Host Authentication. Use the toggle on each product panel to enable or disable it for the project. Select the settings icon to open its configuration panel. For an overview of the available configuration options, refer to the specific product documentation. ![RTC Services page](https://assets-docs.agora.io/images/console/realtime-services.png) ### Webhooks A webhook is a callback over HTTPS that allows your app or back-end system to receive notifications. Agora calls your webhook endpoint from its servers to send notifications when selected events occur. Select **Webhooks** in the sidebar to view and manage webhooks for the current project. ![Webhooks page](https://assets-docs.agora.io/images/console/webhooks-list.png) To configure webhooks: 1. In [Agora Console](https://console.agora.io), select **Webhooks** in the sidebar. 2. Select **New Webhook** to open the webhook configuration panel. ![Webhook configuration in Agora Console](https://assets-docs.agora.io/images/console/webhook-rtc.png) 3. From the **Product** dropdown, select the product for which you want to set up Webhook notifications. 4. Copy the **Signing secret**. Use this secret to add signature verification. 5. Fill in the following: * **Region**: Select the region where your notifications receiving server is located. Agora connects to the nearest Agora node server based on your selection. * **Receiving URL**: The `HTTPS` public address of your server that receives the notifications, for example `https://1111-123-456-789-99.ap.ngrok.io/ncsNotify`. Only `HTTPS` endpoints are supported. * **Subscribe to events**: Select all the events for which you want to receive notifications. If the selected events generate a high number of queries per second (QPS), ensure that your server has sufficient processing capacity. * **IP whitelist**: If your server is behind a firewall, enable this option, then call the [IP address query API](/en/api-reference/api-ref/rtc/query-ip-address) to get the IP addresses of the Agora Notifications server and add them to the firewall's allowed IP list. To reduce the delay in notification delivery, best practice is to activate `HTTP` persistent connection (also called `HTTP` keep-alive) on your server with the following settings: * `MaxKeepAliveRequests`: 100 or more * `KeepAliveTimeout`: 10 seconds or more 6. Select **Run health check**. Agora performs a health test for your configuration as follows: 1. The health test generates test events that correspond to your subscribed events, and sends callbacks to your server. In test event callbacks, the channelName is `test_webhook`, and the uid is `12121212`. 2. After receiving each test callback, your server must respond within 10 seconds with a status code of `200`. The response body must be in JSON format. 7. When the health check succeeds, select **Save**. If the health check fails, follow the prompt in Agora Console to troubleshoot the error. Common errors include the following: | Status code | Error | Description | | ----------- | ----------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `590` | Request timeout | Your server does not return the status code `200` within 10 seconds. Check whether your server responds to the request properly. If your server responds to the request properly, contact Agora Technical Support to check if the network connection between the Agora Notifications server and your server is working. | | `591` | Domain name unreachable | The domain name is invalid and cannot be resolved to the target IP address. Check whether your server is properly deployed. | | `592` | Certificate error | The Agora Notifications server fails to verify the SSL certificates returned by your server. Check if the SSL certificates of your server are valid. If your server is behind a firewall, check whether you have added all IP addresses of the Agora Notifications server to the firewall's allowed IP list. | | Other | Other response errors | Your server returns a response with a status code other than `200`. See the prompt in Agora Console for the specific status code and error messages. | ### Agora analytics Select **Agora Analytics** in the sidebar to open analytics in a new tab. Use it to track and analyze usage and quality data for RTC and Chat. For details, see [Agora Analytics overview](/en/realtime-media/agora-analytics/product-overview). ## Account settings Account settings apply across your entire Agora account, not to a single project. **Usage**, **Projects**, and **Billing** are available directly in the sidebar. Access other options by clicking the username at the bottom of the sidebar. ![Account popup menu](https://assets-docs.agora.io/images/console/account-menu.png) The popup menu shows your current workspace and gives access to **Profile**, **Subscriptions**, **Extensions**, **RESTful API Keys**, and **Settings**. It also gives you access to developer and support resources, including documentation, support tickets, and community links (**Get Support**), as well as account-wide actions like switching the theme (**Theme**), reverting to the legacy Console (**Back to classic console**), and signing out (**Sign out**). ### Profile Use the profile page to update your personal and security details such as your name, email, and phone number. You can also reset your password or connect a Google or GitHub account to enable alternate sign-in methods. ### Usage Select **Usage** in the sidebar. The **Usage** page shows your account's usage for the current billing cycle. * **Total Usage**: A summary of RTC minutes, agent minutes, and Signaling messages used in the current billing cycle. * **Detailed Usage**: A breakdown of usage by product, filterable by project and date range. Select **View Breakdown** to chart usage over time, and export the data using the download icon. ![Detailed Usage chart with filters](https://assets-docs.agora.io/images/console/usage-detailed.png) For deeper usage insights, use [Agora Analytics](/en/realtime-media/agora-analytics/product-overview). ### Projects Select **Projects** in the sidebar to view and manage all projects in your account. * Search projects by name or App ID. * Select **Create project** to create a new project. * Click the **Actions** menu next to a project to open its **Credentials** panel, or perform other project actions. ![Projects page](https://assets-docs.agora.io/images/console/projects-list.png) ### Subscriptions The **Subscriptions** page shows available monthly subscription plans and top-up packs for Agora products. You can view your current subscription, compare included features between various plans, and upgrade or downgrade your subscription. You can also view your purchase history on this page. ![Subscription plans](https://assets-docs.agora.io/images/console/subscription-plans.png) For detailed pricing for a specific product, see the product's pricing page. ### Billing Select **Billing** in the sidebar. The **Billing** page shows your available and reserved balance, invoices, transactions, payment methods, and billing information. ![Billing page showing balance, invoices, and transactions](https://assets-docs.agora.io/images/console/billing.png) * **Account Balance**: Your available balance, with options to top up your wallet or withdraw funds once a payment method is on file. * **Reserved Balance**: Funds reserved against estimated usage. Select **View Estimate** to see the breakdown. * **Your Invoices**: Monthly and pre-paid invoices, filterable by date range. * **Transactions**: A log of balance movements, filterable by type, plus a separate tab for withdraw requests. * **Payment Methods**: Saved cards and bank transfer details used for billing. * **Cards**: Select **Add New Card** to save a credit card for billing and invoice settlement. Provide the card number, name on card, expiration date, and CVC. Optionally set the card as the default for future payments. ![Add New Card dialog](https://assets-docs.agora.io/images/console/add-new-card.png) * **Bank Transfer**: Switch to the **Bank Transfer** tab for wire transfer instructions, including the beneficiary's name, account number, bank, SWIFT code, and bank address. Include the provided note (your Company ID) in the transfer's notes field. Transfers made without the provided note can't be matched to your account and will be disregarded. * **Billing Information**: The name and address associated with your billing card. * **Billing FAQ**: Information about billing, such as schedule, account suspension policy, payment orders, and refunds. ### Extensions Use the **Extensions** page to browse and manage extensions that add features to your app on top of Agora SDKs. ![Extensions marketplace](https://assets-docs.agora.io/images/console/extensions.png) Select an extension from the list, then click **View details** to open the configuration page. ### RESTful API keys This page lists your RESTful API keys, used to authenticate with Agora's RESTful API. Each key consists of a Customer ID and Customer Secret. Select **Create API Key** to generate a new key. You can download the Customer Secret only once, so save it somewhere secure. Add a note to a key to help identify it. Select the trash icon to delete a key. You can have a maximum of 10 keys per account. ![RESTful API keys](https://assets-docs.agora.io/images/console/restful-api-keys.png) ### Settings The **Settings** page lets you manage your account information, team members, SSO integration, and notification settings. #### Account Use the **Account** tab to see your organization name, obtain your company ID, or change your account email. Toggle **Two-factor authentication** to enable or disable enhanced authentication for sensitive account actions. You can also use this tab to allow Agora customer support to operate your Console for the next 24 hours. #### Teams and members The **Teams and Members** tab lets you manage who has access to your account. * **Members**: Lists each member, their role, status, and any remarks. Select **Invite Member** to add a new member. * **Roles**: Lists each role, its permissions, and the projects it applies to. Permissions cover Usage, Finance, Teams & Members, Projects, XLA, and Agora Analytics access. Agora provides five built-in roles: **Finance**, **Engineer**, **CS & Maintenance**, **Product & Operation**, and **Admin**. Built-in roles are locked and can't be edited. ![Roles table showing permissions by module](https://assets-docs.agora.io/images/console/members-roles.png) Select **Create Role** to define a custom role. When creating a role, set: * **Role name**: A name for the role. * **Permissions**: Access level (**None**, **View**, or **Edit**, depending on the module) for Usage, Finance, Teams & Members, Projects, and XLA. * **Agora Analytics**: Access to Call Inspector, Data Insights, and Live Data. * **Data**: The projects the role applies to. Select **All Projects**, or choose specific projects from the list. ![Create Role dialog with permissions form](https://assets-docs.agora.io/images/console/create-role.png) #### SSO management The **SSO Management** tab lets you integrate Agora Console with Okta as your identity provider, so your team can log in using their Okta SSO credentials. This integration manages access using SAML and SCIM, and requires an active Premium or Enterprise Support Plan. Select **Request** to request the integration. #### Notification preferences Use the **Notification Preferences** tab to configure whether you want to receive notifications from Agora by email or text messages. You can set fine-grained preferences for each type of alert. ![Notification preferences](https://assets-docs.agora.io/images/console/notification-preferences.png) To view your notifications press the bell icon next to your username to open the **Notifications** page. The page lists notifications related to your Agora account. Filter by category using **All**, **Unread**, **Account**, **Billing**, **Product**, or **System**. Select a notification to read its full content. ### Get support The **Get support** sub-menu provides useful links to Agora documentation, sales, support tickets, and the community platform. # Conversational AI (/en/introduction/conversational-ai) Conversational AI is the capability layer for products where the primary interface is a live conversation with an AI system. In this category, voice is not just an input channel. It is the product experience itself. Agora groups several related surfaces under this layer: * **Conversational AI Engine** for production voice-agent workflows * **AI developer toolkits** for client and server integration * **OpenAI Realtime integration** for teams that want to combine Agora transport with OpenAI Realtime-style model interaction ## What this category is good for This category is the right entry point when you need: * voice-first AI assistants and copilots * customer service and sales agents * tutors, role-play systems, and spoken learning experiences * AI-powered smart devices and hardware endpoints * multimodal products where an agent participates in a live session The common requirement is natural turn-taking, low response latency, interruption handling, and a live connection between the user and the AI system. ## Products and paths in this category The main product path for building real-time voice agents with managed speech, reasoning, interruption handling, and agent lifecycle control. Client and server integration surfaces for web, mobile, and backend teams that need to build custom AI experiences on top of Agora. A guided path for teams that want to connect Agora real-time transport with OpenAI Realtime-style voice interaction flows. ## Start here ## Key characteristics Keep spoken interaction responsive enough for natural back-and-forth conversation rather than turn-based chatbot behavior. Combine real-time audio transport with ASR, LLM, TTS, interruption handling, and agent lifecycle control in one interaction model. Start from Agora-managed paths or integrate custom LLM, ASR, and TTS combinations depending on control and quality requirements. Build across web, mobile, backend, and device surfaces with a shared mental model for sessions, events, and agent behavior. Add custom data, event subscriptions, server-side orchestration, and downstream analytics without redesigning the whole interaction loop. Reuse Agora's voice path and session model so AI can participate as part of a live system, not only as a detached API call. ## What to read next * [Conversational AI](/en/ai) * [AI quickstart](/en/ai/get-started/quickstart) * [OpenAI Realtime](/en/ai/reference/openai-realtime-integration) * [API Reference](/en/api-reference/conversational-ai) # Core concepts (/en/introduction/core-concepts) Voice Agents are real-time AI experiences where users speak naturally with an AI system. On Agora, voice agents listen, reason over the conversation, and respond with low-latency speech inside a live session. RTC (Real-Time Communication) technology allows almost instant exchange of audio, video, and other media data between the sender and the receiver. Agora Signaling, formerly known as RTM, provides real-time delivery of messages, presence, metadata, transcripts, state updates, and control events. In many voice agent experiences, RTC carries the live audio path, while Signaling carries the events and data that coordinate the experience. Agora SDKs provide real-time audio, video, and messaging interaction services, with multi-platform and multi-device support. This includes high-definition video calls, voice-only calls, interactive live streaming, real-time messaging, as well as one-on-one and multi-group chats. This article introduces the key processes and concepts you need to know to use Agora SDKs. ## Before you start Most quickstarts and guides ask for the same values: | Value | What it identifies | | ----------------------------------- | ------------------------------------------------------------------- | | [App ID](#app-id) | The Agora project that your app belongs to. | | [App Certificate](#app-certificate) | The secret used to enable token authentication. | | [Token](#token) | A dynamic key that grants a user permission to access a channel. | | [Channel](#channel) | The live space, identified by a channel name, where users interact. | | [UID](#user-id-uid) | The unique user identity of one participant in a channel. | ## Set up your Agora project To use Agora SDKs, first create a project in Agora Console. See [Agora account management](/en/introduction/account) for details. ![Create project in Agora Console](https://assets-docs.agora.io/images/common/create-project.svg) [Agora Console](https://console.agora.io/) is the main dashboard where you manage your Agora projects and services. After registering an [Agora account](https://console.agora.io/), use Agora Console to: * Manage the account * Create and configure Agora projects and services * Get an App ID * Manage members and roles * Check call quality and usage * Check bills and make payments * Access product resources Agora also provides REST APIs for tasks such as creating projects and retrieving usage data programmatically. ## General concepts Agora uses the following basic concepts across its SDKs and cloud services: ### App ID The App ID is a string generated in [Agora Console](https://console.agora.io) when you create a project. Each project has a unique App ID. Pass it when you initialize an Agora SDK, and use it when generating the authentication tokens that control access to a channel. You can [retrieve your App ID](https://console.agora.io/projects) in Agora Console. Agora uses the App ID to identify your project and associate usage and billing data with it. For production environments, use the **App ID + token** authentication mechanism. Without a token, anyone with access to your App ID can join your channels. ### App Certificate An App Certificate is a secret generated by Agora Console to enable token authentication. It is required when generating RTC or Signaling tokens. To use your App Certificate to set up a token server, see [Deploy a token server](/en/realtime-media/rtc/build/authenticate-users/deploy-token-server). ### Token A token is a dynamic key that is used by the Agora authentication server to check user permissions. A token is usually scoped to a channel, a UID, a product capability, and an expiration time. You use Agora Console to generate a temporary token for testing purposes during the development process. In a production environment, you implement a token server in your security infrastructure to control access to your channels. After obtaining the App ID, App Certificate, and token in Agora Console, you can start implementing basic audio and video communication in your app. ### Channel Agora uses the *channel name* to identify a channel. Users who specify the same *channel name* join a common channel and interact with each other. A channel is created when the first user joins. It ceases to exist when the last user leaves. You create a channel by calling the methods for transmitting real-time data. Agora uses different channels to transmit different types of data. The RTC SDK channel transmits audio or video data, while the Signaling channel transmits messaging or signaling data. The RTC SDK and Signaling channels are independent of each other. Additional components provided by Agora, such as On-Premise Recording and Cloud Recording, join the RTC SDK channel and provide real-time recording, transmission acceleration, media playback, and content moderation. ### Channel profile The SDK applies different optimization methods according to the selected channel profile. Agora supports the following channel profiles: | Channel profile | Description | | ------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `COMMUNICATION` | This profile is suitable for one-on-one or group calls, where all users in the channel talk freely. | | `LIVE_BROADCASTING` | In a live streaming channel, users have two client roles: *host* and *audience*. The *host* sends and receives audio or video, while the *audience* only receives audio or video with the sending function disabled. | ### Stream A stream is a sequence of digitally-encoded coherent signals that contains audio or video data. Users in a channel [publish](#publish) local streams and [subscribe](#subscribe) to remote streams from other users. ### Publish Publishing is the act of sending a user's audio or video data to the channel. Usually, the published stream is created by the audio data sampled from a microphone or the video data captured by a camera. You can also publish media streams from other sources, such as an online music file or the user's screen. After successfully publishing a stream, the SDK continues sending media data to other users in the channel. By publishing the local stream and subscribing to remote streams, users communicate with each other in real time. ### Subscribe Subscribing is the act of receiving media streams published by remote users in the channel. A user receives audio and video data from other users by subscribing to one or more of their streams. You either directly play the subscribed streams or process incoming data for other purposes such as recording or capturing screenshots. ### User ID (UID) A User ID (UID) uniquely identifies a user within an Agora service. Both RTC and Signaling (formerly RTM) use UIDs, but the UID format, identity scope, and session behavior differ between the two services. In RTC, the UID is an integer that uniquely identifies a user within a channel. When joining a channel, you can assign a specific UID or let Agora generate one automatically by passing `0` or, where supported, `null`. Each user in the same channel must have a unique UID. If multiple users join the same channel with the same UID, unexpected behavior may occur. Agora services and components use the UID to identify and manage the user within that channel. In Signaling, the UID is a unique string supplied with the App ID when initializing the SDK. It identifies the user when logging in to Signaling and throughout the session, including for billing and online status notifications. Because the UID is associated with the user during initialization, the user only needs to provide the channel name when joining a channel. The same UID cannot be logged in to Signaling from multiple devices at the same time. If another client logs in with an active UID, the previously connected client is disconnected and receives a notification. ### User role A user role is used to define whether users in the channel have permission to publish streams. There are two user roles: * Host: A user who can publish streams in a channel. * Audience: A user who cannot publish streams in a channel. Users with this role can only subscribe to remote audio and video streams. ### Connection (RtcConnection) A connection is the link between the SDK and the channel. When you need to publish or receive multiple streams in multiple channels, use a connection to specify the target channel. ### Agora SDRTN® Agora's core engagement services are powered by its Software-Defined Real-Time Network (SDRTN®) that is accessible and available anytime, anywhere around the world. The software-defined network isn't confined by device, phone numbers, or a telecommunication provider's coverage area like traditional networks. Agora SDRTN® has data centers globally that cover over 200+ countries and regions. The network delivers sub-second latency and high availability of real-time video and audio anywhere on the globe. With Agora SDRTN®, Agora can deliver live user engagement experiences in the form of real-time communication (RTC) with the following advantages: * Unmatched quality of service * High availability and accessibility * True scalability * Low Cost # Firewall requirements (/en/introduction/firewall) To allow you to use Agora products in environments with restricted network access, Agora provides the following solutions: the firewall whitelist and the Agora cloud proxy. The following table lists the support of Agora products for the two solutions: | Agora Products | Firewall Whitelist | Agora Cloud Proxy | | ---------------------------------------- | :----------------: | :---------------: | | RTC SDK (Native, third-party frameworks) | ✘ | ✔ | | RTC SDK (Web) | ✔ | ✔ | | Signaling SDK (Native) | ✔ | ✘ | | Signaling SDK (Web) | ✔ | ✔ | | On-Premise Recording SDK | ✘ | ✔ | * When using the firewall whitelist, add the domains and ports to the firewall whitelist, and do not set restrictions on IP addresses. * When using Agora cloud proxy, refer to [Connect to Agora through a restricted network](/en/realtime-media/rtc/build/manage-connection-and-quality/cloud-proxy) ### RTC SDK (Web) Add the following destination domains and the corresponding ports to your firewall whitelist. #### Domains ```text .agora.io .edge.agora.io .sd-rtn.com .edge.sd-rtn.com .ap.sd-rtn.com .statscollector.sd-rtn.com .webrtc-cloud-proxy.sd-rtn.com .rtnsvc.com .edge.rtnsvc.com .rtesvc.com .edge.rtesvc.com ``` To improve connectivity over public networks, we recommend adding the domains listed above to your firewall allowlist. #### Ports | Destination ports | Port type | Operation | | -------------------------------------------------------------------------------------------------------- | --------- | --------- | | 80; 443; 3433; 3478; 4700 - 5000; 5668; 5669; 6080; 6443; 8667; 9667; 30011 - 30013 (for RTMP converter) | TCP | Allow | | 3478; 4700 - 5000 (2.9.0 or later); 10000 - 65535 (before 2.9.0) | UDP | Allow | ### Signaling SDK (Web) #### Message channel For a message channel, you need to add the following content to the firewall whitelist: * **Domains**: ```text .edge.agora.io .edge.sd-rtn.com web-1.ap.sd-rtn.com web-2.ap.sd-rtn.com web-3.ap.sd-rtn.com web-4.ap.sd-rtn.com ap-web-1.agora.io ap-web-2.agora.io ap-web-3.agora.io ap-web-4.agora.io webcollector-rtm.agora.io logservice-rtm.agora.io rtm.statscollector.sd-rtn.com rtm.logservice.sd-rtn.com ``` * **Ports**: | **Destination port** | **Protocol** | **Operate** | | ---------------------- | ------------ | ----------- | | 443; 9591; 9593; 27387 | TCP | Allow | Info If you are using Signaling 1.x, also add port 9601. #### Stream channel For a stream channel, you need to add the following to your firewall whitelist: * **Domains**: ```text .agora.io .edge.agora.io .sd-rtn.com .edge.sd-rtn.com ``` * **Ports**: | **Destination port** | **Protocol** | **Operate** | | --------------------------------------------------------- | ------------ | ----------- | | 80; 3433; 4700 - 5000; 5668; 5669; 6080; 6443; 8667; 9667 | TCP | Allow | | 3478; 4700 - 5000 | UDP | Allow | ### Signaling SDK (Native) #### Message channel For a message channel, you need to add the following content to the firewall whitelist: * **Domains**: ```text .agora.io ``` * **Ports**: | **Destination port** | **Protocol** | **Operate** | | ----------------------------------------------------- | ------------ | ----------- | | 443; 7384; 8443; 9130; 9131; 9136; 9137; 9140; 9141 | TCP | Allow | | 1080; 3000; 8000; 8130; 8443; 9120; 9121; 9700; 25000 | UDP | Allow | #### Stream channel For a stream channel, you need to add the following to your firewall whitelist: * **Ports**: | **Destination port** | **Protocol** | **Operate** | | -------------------- | ------------ | ----------- | | 4001 - 4150 | UDP | Allow | Info The target ports listed in this section may be adjusted according to actual conditions. If you encounter any issues, contact [rtm@agora.io](mailto\:rtm@agora.io). # Glossary (/en/introduction/glossary) ## A ### Agora Analytics Agora Analytics is a site for developers to track and analyze the usage and quality of calls. ### Agora Cloud Backup Agora Cloud Backup is a backup cloud storage service used in cloud recording. If the recording service cannot upload the recorded files to the specified third-party cloud storage, then the service automatically and temporarily stores them in the backup cloud. ### Agora Console Agora Console is a site for developers to manage Agora projects and services. ### App ID An app ID is a randomly generated string provided by Agora and is the unique identifier of an app. ### App certificate An app certificate is a randomly generated string provided by Agora for enabling token authentication. It is one of the required arguments for generating a token. ### Audience Audience are users who do not have streaming permissions in a channel. An audience user can subscribe to remote audio and video streams, but cannot publish audio and video streams. For more information, see [user role](#user-role). ### Audience (becoming) Becoming an audience describes a use-case within an Interactive Live Streaming channel (the channel profile is Live-Broadcast) when a host switches the user role and becomes an audience. ### Audio mixing Audio mixing means combining multiple audio streams into one. ### Audio profile An audio profile includes the sample rate, encoding scheme, number of channels, and bitrate for encoded audio data. ### Audio route The audio route is the pathway audio data takes through audio hardware components during playback. ## C ### Callee A callee is a Signaling user who receives a [call invitation](#call-invitation). ### Caller A caller is a Signaling user who sends a [call invitation](#call-invitation). ### Call invitation Call invitation is a communication protocol based on the peer-to-peer messaging functionality of the Agora Signaling SDK. Call invitation supports starting, ending, accepting, and refusing calls. ### Channel In Agora's platform, a channel is a way of grouping users together and is identified by a unique channel name. Users who connect to the same channel can communicate with each other. A channel is created when the first user joins and ceases to exist when the last user leaves. ### Channel attribute Channel attributes are tags added to Signaling channels, including the property name, property value, the ID of the last Signaling user who updated the attribute, and the time of the last update. ### Channel message A channel message is a message that a Signaling user sends to all Signaling users in a channel. ### Channel profile The channel profile is a configuration that Agora uses to apply optimized algorithms for different real-time use-cases. ### Cloud proxy Cloud proxy is a proxy service that enables users to connect to Agora services through a firewall by using fixed IP addresses. ### CREC CREC is a component provided by Agora for recording and saving voice and video calls and interactive streaming on a third-party cloud storage through RESTful APIs. ### Co-hosting Co-hosting describes a use-case with more than one host. ### Composite recording mode Composite recording mode generates a single mixed audio and video file for all UIDs in a channel. ### Custom rendering Custom rendering is the process where developers collect raw data from the SDK and process it according to specific needs. ### Custom source Custom source is the process where an app captures raw data by itself. ## D ### Delay In real-time audio and video communication, delay refers to the time elapsed from when the data is sent to when it is received. ### Dual-stream mode In the dual-stream mode, the RTC SDK simultaneously transmits a higher-resolution video stream along with an additional low-resolution, low bitrate video stream. ## F ### Freeze Freeze refers to choppy audio or video playback caused by a poor network connection or limited device performance during real-time audio and video communication. ## H ### High-quality video stream In dual-stream mode, the SDK transmits two video streams of differing quality at the same time. See [dual stream mode](#dual-stream-mode) for details. ### Host The host refers to a user who has streaming permissions in a channel. A host can publish audio and video. A host may also subscribe to audio and video published by other hosts. ### Host (becoming) Becoming a host describes a use-case within an Interactive Live Streaming channel (the channel profile is Live-Broadcast) when an audience switches the user role and becomes a host. ## I ### Individual recording mode Individual recording mode records audio and video of each UID as separate files. ### Inject online media stream Inject online media stream refers to injecting an online media stream in an Interactive Live Streaming channel to share the stream with all users in the channel. The Agora RTC SDK provides a method for developers to inject an online mixed audio and video stream or an audio only stream to a channel. ### Interactive Live Streaming Enabled by either Agora’s RTC SDK or Voice SDK, Interactive Live Streaming gives you full control over the streaming experience from a standard one-to-many stream to a highly-interactive live event. ## J ### Jitter In real-time audio and video communication, jitter is the variation in the delay of data packets transmitted continuously on the network. ## L ### Last mile The last mile refers to the network between the Agora edge server and the end user's device. ### Loopback test A loopback test sends a signal from a communication device and is then returned (looped back) to it. It is often used to determine whether a device is working properly. ### Low-quality video stream In dual-stream mode, the SDK transmits two video streams of differing quality at the same time. The low-quality video stream has a lower resolution and bitrate than the high-quality video stream. See [dual stream mode](#dual-stream-mode) for details. ## M ### MediaPlayer kit The mediaplayer kit is a plug-in of the RTC SDK to play local and online media resources and publish the media streams to other users in an Interactive Live Streaming channel. ### Media stream A media stream is an object that contains media data. ### MPUSH MPUSH enables you to upload audio and video streams from Agora channels and upload them to a Content Delivery Network (CDN) to reach a larger audience. ### Mirror Mirroring is an effect that a video image renders. ## O ### Offline Offline describes the status of a Signaling user who has successfully logged out of Signaling. ### Offline message An offline message is a peer-to-peer message that an online Signaling user sends to an offline Signaling user. ### Online Online describes the status of a user who has successfully logged in to the Agora Signaling system or stays disconnected from the Agora Signaling system for more than 30 seconds. ### OPREC OPREC is a component provided by Agora for recording and saving voice and video calls and interactive streaming on a Linux server. ## P ### Packet loss Packet loss refers to the data packets transmitted on the network failing to arrive at their intended destination. ### Peer-to-peer message A peer-to-peer message is a message that an online Signaling user sends to an online or offline user. ### Publish Publishing is the action of sending the user's audio and/or video data to the channel. ## R ### Raw data Raw data, including raw audio data and raw video data, is the unprocessed data which developers can collect during real-time communication. ### Render the first video frame Rendering the first video frame is the action of rendering the first video frame on the local device. ## S ### Agora SDRTN® Software-Defined Real-Time Network (SDRTN®) is a real-time transmission network built by Agora and is the only network infrastructure specifically designed for real-time communications in the world. ### Signaling SDK You use the Signaling SDK to implement real-time messaging use-cases that require low latency and high concurrency for a global audience. ### Slice Slicing means cutting recorded audio or video into separate files according to specific rules. During an Agora CREC, the recording service cuts the streams and generates multiple slice files (TS or WebM files) and M3U8 files that serve as a playlist of the slice files. ### Sound localization Sound localization means determining the distance to and direction of a sound through hearing the difference of volume, time, and timbre between users' ears. ### Stream fallback In use-cases where multiple users engage in real-time audio and video communication, user experience can be impaired if the network condition is too poor to guarantee both audio and video at the same time. ### Stream mixing Stream mixing means combining multiple media streams into one. It may include the mixing of video streams (video mixing) and audio streams (audio mixing). ### Subscribe In the Agora RTC SDK, subscribing is the action of receiving media streams published to the channel. In the Agora Signaling SDK, subscribing is the action of monitoring the online status of one or multiple Signaling users. ## T ### TCP TCP (Transmission Control Protocol) is a connection-oriented and reliable transport layer communication protocol. ### Token A token, also known as a dynamic key, is used for authentication when an app user joins a channel or logs onto the Agora Signaling. ### Transcoding Transcoding is the process of decoding audio and video data and then re-encoding them into the target conversion output or format. ## U ### UDP UDP (User Datagram Protocol) is a connectionless-oriented and unreliable transport layer communication protocol. ### User attribute User attributes are tags added to Signaling users, including property names and property values. ### User ID (uid) In the Agora RTC SDK, a user ID identifies a user in the channel. The user ID is a 32-bit signed integer, with a value range from -231 to 231-1, that you can specify yourself. If you specify `0` for the user ID when joining a channel, the SDK generates a random number and returns the value in the join channel success callback. In the Agora Signaling SDK, a user ID identifies a user in Signaling. The user ID in the Agora RTC SDK and the Agora Signaling SDK are independent of each other. ### User role The type of user role determines whether the user in the channel has streaming permissions. ## V ### Video layout Video layout arranges the display of users when multiple users are mixed into one stream, such as in Media Push or a composite recording. ### Video mixing Video mixing means combining multiple video streams into one. ### Video profile The video profile refers to a set of video attributes, such as resolution, bitrate, and frame rate. ### RTC SDK An SDK developed by Agora to enable developers to add real-time audio interaction to their projects. ### Voice SDK Agora provides the Voice SDK to enable real-time audio communication. ## W ### Web page recording mode In web page recording mode, the content and audio of a specified web page are recorded in a single file. # Agora Documentation (/en/introduction) ## Build a Voice Agent ## Build Real-Time Applications ## Start with AI Tools ## Real-time use cases ## SDKs and APIs
## Get Help # Members & Roles (/en/introduction/members-roles) ## Manage members and teams Once you add members to your Agora account, you are assigned to the **Admin** team and can add these members to different teams with specified permissions. ### Add a member 1. Log in to Agora Console, click your account name in the top-right corner, and click **Settings**. 2. In the left navigation panel, click **Teams and members**, then select the **Members** tab. 3. Click **Add New Member**, fill in the email address of the new member, and choose a team from the dropdown. 4. Click **OK**. 5. Agora sends a confirmation email to this address. The new member must follow the instructions in the email to finish joining the project. ### Manage members On the **Teams and members** page, **Members** tab, you can do the following: * Change the team that a member belongs to. * Delete a member. Only the main account can delete a member account. Member accounts assigned to the **Admin** team cannot delete a member account. ### Manage teams and permissions On the **Teams and members** page, **Teams** tab, you can view the permissions assigned to teams. Agora predefines the following teams: * **Admin**: full access to all projects; can view usage data and finance information, manage members and teams, manage projects, and view Analytics reports. * **Finance**: can view finance information for all projects. * **Product/Operation**: can view usage data for all projects. * **CS/Maintenance**: can view Analytics reports for all projects. * **Engineer**: can manage projects and view Analytics reports for all projects. ### Add a custom team 1. On the **Teams and Members** page, **Teams** tab, click **Add New Team**. 2. Fill in the team name. 3. Select the permissions of this team in the **Usage**, **Finance**, **Teams and Members**, **Project**, **Analytics**, and **Data** columns. 4. Click **OK**. # Messaging & Presence (/en/introduction/messaging-presence) ![Signaling overview](https://assets-docs.agora.io/images/signaling/signaling-overview.png) Messaging & Presence is the capability layer for products that need more than media. This category covers human messaging, participant presence, room state, and shared collaboration surfaces that keep live systems coordinated. Agora's product map for this layer includes: * **IM / Chat** * **RTM / Signaling** * **Interactive Whiteboard** ## What this category is good for This category is the right entry point when you need: * in-room chat and social messaging * user presence and online state * metadata synchronization and coordination signals * collaborative session control and shared state * whiteboard-backed visual collaboration inside a live product The shared requirement is that participants must stay aligned with each other and with the state of the session in real time. ## Capability map IM-oriented communication for user conversation, room interaction, and product workflows that depend on message exchange. RTM-oriented channels, topics, presence, metadata, and coordination events that support real-time systems. Shared visual state for drawing, annotation, presentation, and collaborative interaction around content. ## Start here ## Key characteristics Support both user-facing communication and the metadata flows that keep live products synchronized. Track who is online, what state they are in, and how session behavior should react as users and devices change status. Extend from text and events into shared whiteboard or content collaboration when the product needs a richer interaction layer. Combine easily with RTC sessions and AI agents when messages and events need to travel alongside audio and video. Choose between IM-style user communication and RTM-style signaling depending on whether the main problem is conversation, coordination, or both. Keep users, agents, and session tools operating against the same live room state instead of fragmenting the product experience. ## What to read next * [Real-time Messaging RTM](/en/realtime-media/rtm) * [Instant Messaging IM](/en/realtime-media/im) * [Interactive Whiteboard](/en/realtime-media/whiteboard) # Projects (/en/introduction/projects) ## Why this page matters In Agora, a project is the operational boundary for a product environment. It holds the App ID, certificate configuration, enabled services, and billing context that the rest of your integration depends on. If your project setup is wrong, the rest of the docs may still be correct but your implementation will fail later through missing credentials, disabled services, or the wrong security mode. ## What you do here Use the **Projects** page in Agora Console to: * create a new project for a product or environment * choose the authentication mode for that project * copy the App ID used by clients and services * manage App Certificates for token-based access * check whether the project is configured the way your integration expects ## Recommended setup For most teams: * create separate projects for development, testing, and production * use **App ID + Token** instead of App ID only * keep certificate management on the server side * confirm the correct project is being used before debugging client or backend code ## Create a project 1. Open the **Projects** page in Agora Console. 2. Click **Create New**. 3. Enter a project name and use case. 4. Select **App ID + Token (Recommended)** as the authentication mechanism. 5. Click **Submit**. ## What to check in project details On the project details page, confirm that you can: * copy the App ID used by your application * view whether token-based security is enabled * access the App Certificate when your backend needs token generation * verify that the project is the one your current environment should use ## Service access To access most Agora products you need: * An App ID. * An App Certificate if your backend generates tokens. * A project with the required service enabled. For production, use token-based authentication instead of App ID only. ## What to verify before launch Before moving beyond a prototype, confirm: * the required service is enabled for the correct project * the authentication mode matches your intended production design * the billing unit makes sense for your usage pattern * the right people on your team can access billing and financial records ## Manage App Certificates Agora provides two certificate roles: * **Primary Certificate** for normal token generation * **Secondary Certificate** for production rotation and controlled cutovers If you suspect a certificate is compromised: 1. Enable the Secondary Certificate. 2. Swap the certificates. 3. Disable the old secondary certificate. 4. Delete the original primary certificate after clients and services have moved over. Deleting a certificate invalidates all tokens generated with that certificate. ## Restrictions * If your account has multiple members, only those assigned to the **Admin**, **Engineer**, or an authorized custom team can access the **Projects** page. * Each Agora account can create up to 20 projects. If you need to create more projects, submit a support ticket. ## Where to go next * Open [Console setup](/en/introduction/console-setup) to understand the first-time console workflow around credentials and service activation. * Read [Billing](/en/introduction/billing) to understand free allowances, settlement, and package options before launch. * Read [Security and privacy](/en/introduction/security-privacy) before production rollout. # Real-Time Voice & Video (/en/introduction/realtime-audio-video) Real-Time Audio & Video is the capability layer for products where live media is the primary user experience. This category covers two-way voice and video interaction, host-audience live experiences, large-scale broadcast delivery, and device-oriented real-time connectivity. Agora's product map for this layer includes: * **Voice Calling** * **Video Calling** * **Interactive Live Streaming** * **Fusion CDN broadcast delivery** * **IoT / RTSA edge-device connectivity** ## What this category is good for This category is the right entry point when you need: * one-to-one or group voice and video calling * meetings, classrooms, and telehealth sessions * creator-led or host-audience live interaction * broadcast-style delivery to larger audiences * smart-device, camera, or edge-terminal media connectivity The shared requirement is stable, low-latency media transport with clear session behavior across networks, devices, and participant roles. ## Capability map Low-latency voice interaction for calling, social audio, customer communication, and lightweight live sessions. Real-time face-to-face interaction for meetings, classrooms, telehealth, and collaboration products. Host-audience interaction for creator events, live communities, and interactive broadcast-style products. Fusion CDN-based distribution for larger viewer populations and playback-oriented live delivery. RTSA-based device connectivity for smart cameras, displays, embedded endpoints, and IoT-oriented real-time media. ## Start here ## Key characteristics Use the same real-time transport model across calling, live interaction, broadcast expansion, and device connectivity. Support peer-to-peer, group, host-audience, and device-oriented topologies without switching to a different conceptual model. Start from interactive sessions, then extend into larger audience delivery or downstream cloud media services. Reach web, mobile, desktop, and edge-device products with consistent session and media semantics. Combine media transport with messaging, AI, transcription, recording, and analytics as product requirements grow. Keep the live experience usable under real-world packet loss, geography, and device constraints. ## What to read next * [RTC](/en/realtime-media/rtc) * [Fusion CDN](/en/realtime-media/fusion-cdn) * [RTSA](/en/realtime-media/rtsa) # Security & Privacy (/en/introduction/security-privacy) ## Information security policy This section provides information security policy for Agora services. Agora services provide built-in encryption and customized encryption. You can use either of them to implement encryption. The following diagram describes the encrypted data transmission process: ![Encrypted data transmission process](https://assets-docs.agora.io/images/common/encrypted-data-transmission-process.svg) #### Purpose Agora is committed to safeguarding the confidentiality, integrity, and availability of all users' physical and electronic information assets. * Confidentiality against unauthorized access and eavesdropping * Integrity against tampering and forgery * Availability of data transmission through the Agora SDRTN® #### Scope This article describes how Agora protects customer data with security controls. ### Data classification All customer data, in all formats or media types, is classified according to the following categories and protected accordingly. | Category | Description | | --------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Customer Account Data | Information related to the customers' Agora accounts, including customer ID, customer IP, network type, operating system, email address, telephone, product interest, programming platform, UTM information, identity, company name, company URL, billing information, trade information, purchased package information, project status, ticket, member information and console access records. | | End User Data | Information collected by customers related to end users' personal devices and network, including microphone and camera information, CPU status, memory status, battery status, system version, phone model, phone signal level, received signal strength indicator (RSSI), network type, user attributes and channel attributes. | | Call Content | Audio and video data of the user during a call. | | Log | Media server logs generated by the Agora servers when accessing the Agora SDRTN®. Media server logs do not contain text messages or personal information. | ### Data security The communication between the user and the Agora server is protected by transmission protocols, such as the Agora private transmission protocol, Transport Layer Security (TLS) and WebSocket Secure (WSS). You can also use Advanced Encryption Standard (AES) or a customized encryption algorithm for the encryption of audio and video data. During data transmission, the Agora SDRTN® does not transmit any encryption key information. Call content information can only be decrypted on the terminal device (such as the client app and the customer's on-premise recording server) through the client authorization key. ### Data availability * Large and distributed data centers: Agora has multiple data centers providing services globally, and any attack on one data center cannot affect others. * Rapid recovery: When a data center is subjected to malicious attacks that are difficult to prevent, such as a distributed denial-of-service (DDoS) attack, Agora will automatically isolate the data center and avoid affecting users' services. * DDoS attack prevention: Agora has deployed anti-DDoS firewalls in each core cloud data center. Agora has more than two hundred distributed data centers around the world, which guarantees sufficient capabilities and resources to control the risk of DDoS attacks. ### Data storage Agora provides customers with the Agora On-Premise Recording SDK and Agora Cloud Recording, enabling customers to record part or all of the call contents. When using the recording services, all recorded video or audio files are stored on the storage server provided by the customer. ### Access authorization End users can access the Agora SDRTN® using a dynamic key. For details, see [Secure authentication with tokens](/en/realtime-media/rtc/build/authenticate-users/authentication-workflow). ### Electronic access controls Agora strictly controls the data access on all internal systems. All users have independent internal accounts and authorization systems, and must pass two-step verification. All access records are recorded. All servers involving user data are strictly audited and protected. Agora employees will only access the production server when necessary, and only by obtaining temporary authorization. Operation records are kept for the whole process. ### Physical access controls All operating servers are hosted in computer rooms that meet ISO 27001 or above information security management certification standards, and all computer rooms protect information security in accordance with the escrow agreement or the Data Protection Association (DPA). Both third-party hosting providers and Agora employees must be approved before they can access servers and equipment. ### Roles and responsibilities Agora sets the responsibilities for its own staff and customers. See the following sections for more information. ### Roles and responsibilities within Agora Security roles and responsibilities within Agora are categorized as follows: | Role | Responsibilities | | ---------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Information Security Sub-Committee | The Information Security Sub-committee (ISSC) is responsible for the development and implementation of policies and procedures. The ISSC monitors company adherence, and conducts regular technical and non-technical evaluations of Agora security policies. The ISSC also designates which employees are authorized employees. | | Authorized Employee | An authorized employee has access to the production machines required for the support and maintenance of the Agora service as determined by the ISSC. | | All Other Staff | All other staff are required to maintain confidentiality as required by their terms of employment and are required to immediately report any security breach. | ### Shared responsibilities When evaluating the Agora SDRTN®, it is important for customers to understand and distinguish the following security measures: * The security measures that Agora implements and operates. * The security measures that customers implement and operate to ensure the security of call content information and applications when using the Agora SDRTN® services. Customers retain control of the security measures they choose to protect their personal information, platform, applications, systems, and networks. Agora customers should be responsible for all information they collect from users and themselves, such as application logins, identities, passwords, payment information, names, and addresses. Agora recommends adding obvious prompts before accessing the users' personal information. For example, add an Enable/Disable button with which the user can agree or disagree to share personal information. ### Security training Agora conducts information protection, security, and compliance training for all new employees. Agora also ensures that all employees receive information confidentiality training at least once a year. All authorized employees (employees who have access to the production machines) will receive additional training. ### Non-compliance Employees must comply with confidentiality agreements and internal security systems. If a non-compliant situation occurs, Agora will take the corresponding measures depending on the severity of the situation, including but not limited to having conversations with the at-fault employees, strengthening training and education, dissolving labor agreements, and pursuing other legal liabilities. ### Reporting potential security risks Please report any potential risks of Agora services that you may notice to [security@agora.io](mailto\:security@agora.io). Please use our PGP public key (Key ID: 2F4553BE) to encrypt your report, as potential security risks are usually sensitive. We assure you that our security experts will handle the issue immediately once they receive your report. To facilitate troubleshooting and verification, please include the following information in your email: * Your contact information. * The version of the impacted SDK or solution. * Description of the potential risks. * Miscellaneous technical details, such as your system configurations, methods or steps for reproducing the issue. To better protect our system and customers, Agora also invites security researchers to report any bugs or vulnerabilities they discover to us. Click [Bug Bounty](#agora-bug-bounty-program) to learn more. ### Security FAQ To ensure the security of transmitted data, Agora services provide encryption for audio and video data. Customers can use AES-128/AES-256 or other algorithms preset by Agora, or use customized encryption algorithms. For details, see [Channel Encryption](/en/realtime-media/rtc/build/secure-and-protect-channels/media-stream-encryption). The encryption key is completely generated and distributed by the customer. Agora recommends that customers use separate keys for each channel to achieve the highest level of data security. For network protocol encryption, Agora SDKs use AES-128 mode to encrypt network payloads. For channel encryption, Agora SDKs support AES-128/AES-256 mode to encrypt audio and video. For those who have higher requirements for content security, Agora recommends using our customized encryption. Note that the On-Premise Recording SDK and Cloud Recording do not support customized encryption. For details, please send an email to [support@agora.io](mailto\:support@agora.io). Yes. Agora regularly scans core network nodes to check and clear possible security holes. Agora also configures anti-DDoS firewalls in each core cloud data center to protect them from attacks. Agora has more than two hundred distributed data centers around the world, which provides sufficient capabilities and resources to control the risk of DDoS attacks. When using Agora services, the audio and video data are transmitted through the Agora servers. Agora servers cache the audio and video data for 10 seconds during the transmission and release all audio and video data immediately after the call. ## Whitepaper People engage longer when they see, hear, and interact with each other. The future of meaningful human connections is made possible now with Agora’s Real-Time Engagement Platform. People rely on Agora’s Real-Time Engagement Platform to exchange millions of calls and messages, with vivid voice and video embedded in any application, on any device, anywhere. Agora, Inc. is proud to offer a professional Real-Time Engagement Platform as a Service (RTE PaaS) with Compliance, Safety, Security, and Trust. Our commitment to compliance, data and information security, and privacy protection is part of the core values of our company. From our Software-Defined Real-Time Network (SDRTN®) architecture to our day-to-day business operations, Agora continually invests in innovations and business processes that build trust with our customers, investors, and developer community. Agora works to high standards to follow the best security practices and comply with strict privacy regulations and standards as we respect the privacy of all our customers. The information contained in this document is intended to provide transparency in relation to Agora’s security stance and processes. If you think you may have found a security vulnerability within any of Agora’s services, please contact our security team directly at [security@agora.io](mailto\:security@agora.io). Agora strives to incorporate security into all our products and services and integrates the best security practices into everyday business operations. To meet these primary goals and improve the overall information security posture in an efficient and effective manner, Agora has built its security framework against the ISO/IEC 27001 Information Security Management Standard. As threats to information security continue to evolve, having dedicated security resources is essential. Agora’s Executive Security Committee meets regularly to address security concerns and coordinate company-wide security initiatives. The Agora Strategic Security Program Roadmap has been developed and approved by the committee to guide the implementation of our security programs. Our dedicated security team, led by the Chief Information Security Officer, has the responsibility for building and enforcing information security programs. Agora is always looking to better protect its systems and customers. Therefore, we continuously monitor and improve our information security programs by implementing the following: * Agora conducts an internal audit of its information security management system at least once a year to ensure effectiveness; * Agora follows industry best practice software development lifecycle management processes and conducts internal security reviews and testing before deployment into production systems; * Agora engages third-party security experts to carry out regular penetration tests; * Through our Bug Bounty Program, Agora works with security researchers to keep our customers' data more secure by identifying and reporting vulnerabilities in our products and services. Agora adheres to regional and international information security standards as well as industry requirements and is committed to using international best practices. We engage with independent third parties to verify the compliance of Agora. Certified by various reputable agencies across the globe, we are recognized by industry and security organizations for excellence. **ISO/IEC 27001 Information Security Management Standard** Agora is certified to ISO/IEC 27001:2022 by DNV GL, demonstrating our information security maturity level. Our security team implements the Information Security Management System in partnership with Ernst & Young. Security is a top priority at Agora, and this achievement demonstrates our commitment and continuous efforts to improve the efficiency of information security controls. Download the certificate: [ISO/IEC 27001:2022](/files/Agora_ISO_27001.pdf) **ISO/IEC 27018 Information Technology – Security techniques – Code of practice for protection of personally identifiable information (PII) in public clouds acting as PII processors** Agora is certified to ISO/IEC 27018 by DNV GL. This standard is a Code of Practice for protecting personal data in the cloud environment. Agora continuously strives to protect our customers' sensitive data. Download the certificate: [ISO/IEC 27018:2019](/files/Agora_ISO_27018.pdf) **SOC 2 Report** Agora is confident in our security practices, and we continue to engage independent third parties to perform strict SOC 2 audits on our internal processes, security controls, and the design of Agora products. We meet the audit requirements set by the American Institute of Certified Public Accountants (AICPA) standards for security, availability, and confidentiality, and have obtained a SOC 2 report. **PCI DSS** Agora prioritizes the security and privacy of its customers. Since we are neither a merchant nor a service provider as defined by the PCI Security Standards Council, we do not need to conduct an annual PCI assessment. However, Agora ensures that all transactions and customer data are handled with the highest level of security, in alignment with industry best practices. Should you have any questions or require further information regarding our security measures, please feel free to contact us at [security@agora.io](mailto\:security@agora.io). **General Data Protection Regulation - GDPR** Agora is aligned with GDPR and we are committed to providing GDPR-compliant products and services to our customers in the EU region or with our customers who conduct business within the EU. **Health Insurance Portability and Accountability Act - HIPAA** Agora is aware of the sensitivity of transmitting and processing health information and we have invested in both the creation and ongoing maintenance of a HIPAA compliance program. **California Consumer Privacy Act - CCPA** The CCPA is the first comprehensive privacy law in the United States that aims to provide a variety of privacy rights to California consumers. As a service provider, Agora is aligned with CCPA through the implementation of our security programs. **Children’s Online Privacy Protection Act - COPPA** The COPPA regulates the privacy protection requirements for children under the age of thirteen. Agora has engaged privacy experts in meeting the requirements of COPPA. Agora strongly believes in the principles of Secure by Design and Defense in Depth. Therefore, Agora adopts industry-recognized security standards and best security practices at every layer, from infrastructure to application, to perfect our products and environment and to secure the organization and our customers. Securing access to your environment starts with identity and access controls. Agora provides you with a solution to ensure that only authorized people can access your services and resources. The Agora Console is a role-based access control tool that you use to restrict access based on the "need-to-know" principle. The console is an interactive interface where you can easily create accounts, remove members, and assign roles and permissions. This tool can help you enforce your security policies. Furthermore, Agora provides static key, dynamic key, and hybrid authentication methods to secure the communication channels in different use cases. At Agora, you choose how your content is secured. We offer you various options for your content in transit and provide you with full control of your own encryption keys. These features include: * Agora Software Development Kits (SDKs) provide built-in encryption algorithms, including AES-128 and AES-256, to protect all data transmitted between the end users and Agora services. * We also provide you with the choice to use your own encryption algorithm to protect users’ media streams during real-time engagement. The encryption key is completely under your control. * The communication between the end users and Agora network (that is, Agora SDRTN®) is protected by encrypted transmission protocols such as the Agora Private Transmission Protocol, Transport Layer Security (TLS) and WebSocket Secure (WSS). Agora offers our customers the ability to record real-time communication with Agora On-Premise Recording SDK and Agora Cloud Recording SDK. The recording files can be stored on users’ local devices or in a designated cloud storage service chosen by our customers. The local recordings or cloud recordings can be further encrypted through any encryption form of your choosing. Agora does not store any streaming data or user data except for caching for transmission purposes. The cached streaming data of users will be immediately released after the service. Data centers hosting Agora services are maintained by certified and industry-leading cloud service providers, offering state-of-the-art physical protection for the servers and infrastructure that comprise the Agora environment. The production environment, where all our customers' data and functional servers reside, is completely separated from our internal organization network, including the development and testing environments. This guarantees that all our customers' data will stay in the production environment and never be used for development or testing purposes. Access to the Agora production and non-production network is minimized to the greatest extent. Agora also implements network segmentation in the production network based on various factors, such as the type of business, the criticality of data, and potential risks, to secure sensitive customer data. **DDoS prevention** Agora regularly scans our core network nodes in the production environment to check and clear potential security vulnerabilities. Anti-DDoS firewalls are configured in each core cloud data center for protection. With more than two hundred distributed data centers around the world, Agora can provide sufficient capabilities and resources to minimize the impact of DDoS attacks and ensure high availability of real-time video and audio anywhere around the globe. **Monitoring, logging and analysis** Agora continuously monitors and analyzes log events to gain a comprehensive view of the security state of our production environment. The logging covers both successful and unsuccessful security events, with an emphasis on the event data of critical infrastructure. To provide customers with better visibility and security insights, Agora Analytics is made available to consumers as a tracking and analysis tool. This tool enables customers to efficiently locate quality issues and identify root causes for a better end user experience. The tool contains the Real-Time Alarm function, which enables you to monitor call quality and informs you in real time when the user’s communication experience is below expectation. With Real-Time Alarm, you can undertake the following actions in real time: * Monitor users who are having a poor communication experience. * Identify abnormalities, analyze quality factors, and locate the source of an abnormal issue. **Network geo-fencing** Agora has embedded Network Geo-Fencing in Real-Time Voice, Video and Messaging SDKs to ensure your data is protected against rising concerns about network security and privacy breaches. Agora Network Geo-Fencing establishes a virtual boundary within Agora SDRTN®, and you have the choice to restrict your network traffic to one or more designated regions. **Network redundancy** Agora has more than 200 data center POPs (Points of Presence) across the world, covering the United States, Europe, China, Japan, India, the Asia-Pacific region, and other areas. The POPs in the SDRTN® network adopt the full mesh topology with superior routing capabilities. This ensures that the network services are not interrupted due to a single point of failure. The POPs build fault tolerance and disaster recovery capabilities for Agora services across regions. POPs also measure the performance of every possible path through the global network to find the “optimized” paths to ensure high data packet delivery success rate within the smallest time window. Agora provides expert guidance to our customers on how to leverage our security features and embed best practices into every layer of your application. Agora is continuously monitoring, auditing, and improving the design and operating effectiveness of our security controls. These activities are regularly performed by both third-party credentialed assessors and Agora’s internal risk and compliance team. Audit results are shared with senior management and all findings are tracked to resolution in a timely manner. In addition to third-party security compliance audits, Agora engages Trustwave SpiderLabs to conduct network penetration tests at least annually. Results of the penetration testing are shared with senior management and are triaged, prioritized, and remediated in a timely manner. Agora customers may receive executive summaries of these activities by requesting them from their account managers. Data security and user privacy are the top priorities of Agora. Agora is committed to building a professional RTE PaaS with Compliance, Safety, Security, and Trust. It is a critical responsibility for Agora to help ensure the confidentiality, integrity, and availability of systems and data, and Agora continues to work hard to maintain that trust. If you have any questions or concerns, please contact our security team or account managers. ## ISO certifications Security and compliance are basic requirements for real-time interaction. Agora complies with the compliance requirements of different countries and industries to create safe and reliable cloud services. The ISO certification of Agora has passed the supervision and accreditation of three different certification organizations: IFA CNAS (China National Accreditation Service for Conformity Assessment), UKAS (United Kingdom Accreditation Service), and DNV (Det Norske Veritas). Agora builds cloud service technology worldwide in accordance with internationally recognized standards to ensure the information security of global customers. #### ISO/IEC 27001:2022 ISO/IEC 27001:2022 is the most authoritative, widely accepted, and applied system certification standard in the field of information security. This system covers information security management during the global operation and maintenance of Agora's cloud platform for real-time engagement. Download the certificate: [ISO/IEC 27001:2022](/files/Agora_ISO_27001.pdf) #### ISO/IEC 27017:2015 ISO/IEC 27017:2015 is a supplementary standard to ISO 27001, which provides information security implementation specifications for cloud service providers. Download the certificate: [ISO/IEC 27017:2015](/files/Agora_ISO_27017.pdf) #### ISO/IEC 27018:2019 ISO/IEC 27018:2019 is a supplementary standard to ISO 27001, which provides implementation specifications for the protection of personal information for cloud service providers. Download the certificate: [ISO/IEC 27018:2019](/files/Agora_ISO_27018.pdf) #### ISO/IEC 27701:2019 ISO/IEC 27701:2019 is a privacy extension to ISO/IEC 27001 information security management and ISO/IEC 27002 security controls. It is an international management system standard that provides guidance on the protection of personal privacy, including how organizations should manage personal information. Download the certificate: [ISO/IEC 27701:2019](/files/Agora_ISO_27701.pdf) #### ISO/IEC 42001:2023 ISO/IEC 42001:2023 is the first international management system standard for artificial intelligence. It provides requirements for establishing, implementing, maintaining, and continually improving an AI management system within organizations that develop, provide, or use AI-based products or services. Agora's certification covers the full life cycle of the Agora Conversational AI Platform and AI interactive systems, spanning the organizational roles of AI producer, provider, and user, in accordance with Statement of Applicability version 1.0. Download the certificate: [ISO/IEC 42001:2023](/files/Agora_ISO_42001.pdf) ## Security best practices Security and compliance are essential for real-time engagements through technology. In order to provide safe and reliable cloud services, Agora adheres to the compliance requirements of different countries, regions, and industries, in addition to being certified to ISO/IEC 27001. For more information, see [ISO certifications](#iso-certifications). Agora products and services are designed and built with multiple protection measures against attacks commonly seen in the real-time engagement industry. This article describes some of the security best practices that Agora has adopted, as well as security tools it provides for developers, as follows: | Protection measures | Applied by default | Recommended use cases | | -------------------------- | ------------------ | ---------------------------------------------------------------------------------------------------------- | | Channel separation | Yes | All real-time use cases. | | Token-based authentication | No | All real-time apps in a production environment should use Token-based end-user authentication. | | Encryption | No | Real-time use cases that require confidentiality. | | Network geofencing | No | Real-time use cases where customers wish to restrict access to Agora servers to within a specified region. | ### Level 1 - Channel separation The channel architecture is the first built-in layer of protection. Agora creates an independent and isolated channel for each audio, video, or messaging data transmission. All channels are logically separated, and only authenticated users from the same App ID can join the same channel. ### Level 2 - Token-based authentication The second layer of protection is the authentication layer. This is implemented through dynamic Token-based authentication. The Token is a short-lived access key that is generated by the app backend and allows users to access the Agora platform after the user has been properly validated by the app. A Token is generated with important information such as App ID, user ID (`uid`), channel name, and expiration date. ![Authentication workflow](https://assets-docs.agora.io/images/chat/chat-authentication.svg) The app developer can enable Token-based authentication (App Certificate) on [Agora Console](https://console.agora.io/). When enabled, all users' requests to join a channel must use a valid Token. ![img](https://assets-docs.agora.io/images/certificate-enable.png) As a security best practice, set the Token expiration time (24 hours by default). A Token has three expiration timestamps: * Token expiration: How long a Token is valid for, that is, how long a user can stay in a channel. * Privilege expiration: * Join a channel: How long a Token can be used to join a channel. * Streaming privilege: Whether a user can send audio, video, or messaging to the channel, or join as an audience member. This privilege is not enabled by default. ### Level 3 - Encryption The next layer of security in the Agora platform is encryption. Agora supports transmission encryption and data encryption. To guarantee data confidentiality during transmission, Agora uses the AUT (Agora Universal Transport) encryption protocol, Agora's proprietary secured transport layer. Data encryption encrypts all the audio and video streams with a symmetric key and encryption controlled by the app developer. At this level, the app provides a symmetric encryption key to the local SDK libraries. The SDK encrypts all the captured media using the key and the configured AES-128/256 encryption. The data is sent encrypted to Agora SDRTN® and from there to the other endpoints in the channel. The receiving endpoint uses the key provided by the app layer to decrypt the media streams and send them to the renderers. With this method, only the application knows the keys. In the Native SDK (iOS, Android, macOS, Windows), the keys are not sent to Agora servers. ![Token encryption](https://assets-docs.agora.io/images/common/token-encryption.svg) When using other Agora services like Web SDK, Cloud Recording, Content Moderation, and Transcoding, encryption is done a bit differently and it is not end-to-end. In this case, media is still encrypted, but the Agora service needs to be aware of the key to be able to connect to the channel and provide the service. For example, in Web SDK, user/browser protection is provided through web server protection (HTTPS) as well as WebRTC standard security practices, such as encryption and key management. More information on WebRTC security can be found [here](https://webrtc-security.github.io/). Media encryption in Web SDK is based on the WebRTC standard, but interoperability with Agora is handled using Agora’s encryption engine, and the encryption key is passed securely to the Web SDK servers through APIs. The key is required because the Agora edge server converts from WebRTC protocols to Agora’s protocol and allows interoperability with Native SDK. A similar situation occurs with Cloud Recording, Content Moderation, and other services, where RESTful APIs are used to securely pass the key for the channel. ![img](https://assets-docs.agora.io/images/common/media-encryption.svg) ### Level 4 - Network geofencing To conform to the laws and regulations of different countries and regions, the Agora RTC SDK and the Signaling SDK support network geofencing, which limits the transmission of data to within a specified region. These SDKs support network geofencing in the following regions: global (default), North America, Europe, Asia (excluding Mainland China), Japan, India, and Mainland China. Once a customer specifies a region using geofencing, no audio, video, or message can access Agora servers outside that region. ### Security best practice checklist Use this list to quickly check what measures you have or have not taken to best protect the security of your app and users: 1. [Enable Token-based authentication](/en/introduction/projects#manage-app-certificates) on [Agora Console](https://console.agora.io/). 2. Disable *No certificate* in your project management page. Once this is done, your app authenticates users with Tokens only. 3. [Deploy a Token server](/en/realtime-media/rtc/build/authenticate-users/deploy-token-server) in your backend services. 4. Protect the Token server and only allow the app backend server to connect to the Token server. 5. Set the Token expiration time to a reasonable value. See [Deploy a Token server](/en/realtime-media/rtc/build/authenticate-users/deploy-token-server). 6. If needed, contact [support@agora.io](mailto\:support@agora.io) to enable Token privilege and set Token per role. 7. For additional security, work with the Agora SA team if needed to customize the Token server and modify the Token join privilege to a short time, such as 5 minutes. This is more advanced and recommended after app development is completed. 8. Channel encryption key management: Generate a random key (secret) per channel at the time of creating the channel; do not reuse encryption keys. 9. Pass the channel encryption key (secret) securely outside the Agora engine to authenticated endpoints that are allowed to join the channel. 10. For additional security, you can also do the following: 1. Set the “Agora channel name” to a one-time random string managed by your app. 2. Set the Agora `uid` to a one-time numeric ID that is mapped to the user on the app level. Do not use users' real IDs in your app as the Agora `uid`. ## Agora bug bounty program Agora is always looking to better protect our system and customers. Therefore, we invite security researchers to report any bugs or vulnerabilities they discover to us. Once we receive notice of a bug or vulnerability, Agora customer service and security teams will respond quickly to address the issue. In addition to our gratitude, those who report a vulnerability may be eligible for a monetary “bounty” based on the risk associated with the vulnerability and the importance of the affected system. ### System importance classifications The following table shows system importance classifications (in descending order), along with some example Agora assets: | Classification | Asset Examples | | -------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Core system | Agora SDRTN®, the mail system, and official websites such as [www.agora.io](http://www.agora.io/), [sso2.agora.io](http://sso2.agora.io/), and `api.agora.io` | | General system | Forums and the Developer Portal ([docs.agora.io](https://docs.agora.io/)) | | Fringe system | Test sites | ### Vulnerability severity classifications The following list outlines detailed examples of how various vulnerabilities would be classified: **Critical** 1. Unauthorized system privileges Examples include command injection (execution), code injection (execution), web shell execution, SQL injection, and buffer overflow that gains system privileges on the core system. 2. Direct denial of service Examples include actions that make service unavailable, reduce service quality, and so on. 3. Sensitive information leakage Examples include SQL injection of the core database (identity, order), unauthorized disclosure of sensitive information relating to a user, product order, or payment method, and so on. 4. Serious logic design flaws and process defects Examples include the ability to send batches of fraudulent messages, account consumption through a business interface, and large-scale modification of account passwords, and so on. **High** 1. Sensitive information leakage Examples include unauthorized disclosure of sensitive information relating to source code, hardcoded passwords, and so on. 2. Unauthorized access to sensitive information Examples include bypassing authentication or backend password checks, leading to unauthorized access to sensitive intranet information. 3. Unauthorized sensitive operations Examples include manipulating important information without authorization, such as orders, major business configurations, and so on. **Medium** 1. Vulnerabilities that require interaction and affect users Examples include stored XSS (cross-site scripting). 2. General unauthorized operations Examples include incorrect direct object references, unauthorized access to orders, unauthorized access to user information, and so on. 3. General information leakage Examples include client-side stored plaintext passwords, system path traversal, and so on. 4. General logic design flaws and process defects **Low** 1. Local denial of service vulnerabilities, CSRF (cross-site request forgery), reflected XSS, and so on. 2. Minor information leakage, such as path information, SVN information, exception information, the local SQL injection of a client-side application (limited to database name, field name, log print), and so on. 3. Vulnerabilities that are difficult to exploit but still have security implications, such as plaintext transmission of passwords. **Info** 1. Exposure of software banners, internal IP addresses, some public email addresses or phone numbers, and so on. 2. Using outdated versions of a system, supporting outdated versions of encryption protocols, such as SSL (Secure Sockets Layer) or TLS (Transport Layer Security) 1.0, supporting low-strength encryption algorithms, and so on. ### Bug Bounties Please report any potential risks of Agora services that you may notice to [security@agora.io](mailto\:security@agora.io). Any bounties awarded will conform roughly to the following ranges (based on the severity and system location of the bug or vulnerability, payable in USD): * Critical: $300 to $5000 * High: $200 to $2000 * Medium: $150 * Low: $50 * Info: $0 Note: Agora reserves the sole right to determine any reward amount given. # Start with AI (/en/introduction/start-with-ai) Use this page when you want your coding assistant to help you get started without guessing. The steps are simple: install the CLI, add the Agora skill, then give the assistant a prompt with the product, platform, language, and goal. If you also want live docs lookup while the assistant works, pair [Agora Skills](/en/introduction/agora-skills) with [Agora MCP](/en/introduction/agora-mcp). Skills handles workflow guidance, and MCP handles current documentation. ## Quick start ### 1. Install CLI macOS and Linux Windows (PowerShell) ```bash curl -fsSL https://dl.agora.io/cli/install.sh | sh agora --help ``` ```powershell irm https://dl.agora.io/cli/install.ps1 | iex agora --help ``` If your execution policy blocks inline scripts, download `install.ps1` and run: ```powershell powershell -ExecutionPolicy Bypass -File .\install.ps1 ``` After installation, confirm `agora --help` works in the same shell. ### 2. Add skill Install [Agora Skills](/en/introduction/agora-skills) so your assistant can choose the right Agora workflow, starter, and setup sequence before it edits code. Skills CLI Manual installation ```bash npx skills add AgoraIO/skills ``` Clone the repository and point your coding assistant to the Agora skill files: ```bash git clone https://github.com/AgoraIO/skills.git ~/agora-skills ``` Use `skills/agora/` as the directory entry point, or load `SKILL.md` directly if your tool expects a single root file. If your tool prefers local instruction files, point it at `skills/agora/` or the top-level `SKILL.md`. If you only want docs lookup and not workflow guidance, install [Agora MCP](/en/introduction/agora-mcp) instead. Many teams use both. ### 3. Prompt Now tell the assistant what you want it to build and how to approach the setup. A strong prompt includes the Agora product, the target platform, the language or starter preference, and the end goal. ```text Use Agora Skills to help me build a web-based conversational AI demo in Python. ``` From there, let the assistant choose the starter, run `agora login`, initialize the project, and explain any environment values it writes locally. ## What success looks like You are on the right path when the assistant can: * choose an official starter instead of inventing one * explain why that starter matches your product and platform * use the Agora CLI for setup instead of ad hoc manual steps * stay grounded in current Agora docs when platform details matter # Support (/en/introduction/support) ## Submit a ticket To ask Agora Support a question: 1. Log in to Agora Console. 2. Click **Get Support** > **Create Support Ticket**. 3. Type your question or keywords to see whether the issue has already been answered. 4. If you cannot find an answer, select a category and submit a ticket to customer support. You can track ticket status under **Tickets**. ## Status page Agora Status Page provides up-to-date information about product and service status, including: * historical service stability for the past 90 days * real-time quality data for the past 24 hours * maintenance events and disruptions * RSS event subscriptions You can access the status page directly at [status.agora.io](https://status.agora.io/). The status dashboard currently provides status information for: * Real-Time Communication * Cloud Recording * Agora Chat * Interactive Whiteboard # Usage Analytics (/en/introduction/usage-analytics) ![Analytics overview](https://assets-docs.agora.io/images/analytics/analytics-overview.png) Agora Analytics tracks and analyzes the usage, quality, and performance of real-time voice, video, live streaming, chat, and other Agora products. ## Features * **Call Inspector**: identify, analyze, and respond to performance quality issues with detailed call and user metrics. * **Data Insights**: periodic call usage and quality statistics. * **Real Time Monitoring**: visualized data for multiple call metrics in real time. * **Alert notifications**: alerts when abnormal metrics or events are detected. * **RESTful APIs**: retrieve call statistics and quality metrics for your own application or DataOps workflow. * **Embedding**: embed Analytics pages in internal web portals using a low-code approach. ## Console usage page The Console **Usage** page lets you check usage for the past 12 months. You can: * Select a time frame and data granularity. * Filter by all projects or a specific project. * Select a product. * Turn on **View Breakdown**. Only **Admin**, **Product/Operation**, and authorized custom teams can access the **Usage** page. # Foundational real-time capabilities (/en/realtime-media/foundation-realtime)
## RTC Owns audio, video, and channel sessions as the real-time base for calling, live streaming, meetings, and classrooms.
## RTM Provides low-latency, reliable messaging plus state synchronization for channel messages, presence, and workflow coordination.
## Instant Messaging IM Fits broader cloud-chat workloads where a full messaging service is needed beyond lightweight real-time coordination.
## Real-time Transcription & Translation Provides ultra-low-latency, high-accuracy speech transcription and translation for captions, multilingual communication, and content understanding.
## RTSA Provides real-time media-stream and signaling transport for device-oriented and terminal-style workloads. # Overview (/en/realtime-media) Realtime & Media is the capability domain for products that need something to happen live. Use this section when you are deciding how people, devices, backend services, and media streams should connect, stay in sync, and keep working while a session is in progress. Instead of organizing docs around a single product story, this tab is organized around the functional layers most teams assemble when building a real-time experience: live interaction, session intelligence, media routing, large-scale delivery, and device or server participation. ## What you can build from this section ### Create live interaction surfaces Use these docs when you are defining how users join a session, exchange media, coordinate actions, or collaborate in the same live space. * [Voice & Video](/en/realtime-media/rtc): calls, meetings, co-hosting, interactive live rooms, and media quality control * [Signaling](/en/realtime-media/rtm): channel messaging, presence, state sync, and real-time coordination * [Chat](/en/realtime-media/im): persistent and full-featured messaging systems beyond lightweight room coordination * [Whiteboard](/en/realtime-media/whiteboard): shared visual collaboration inside live sessions ### Understand, capture, and transform session content Use these docs when media inside the session needs to become text, records, composites, or downstream assets. * [Transcription & Translation](/en/realtime-media/speech-to-text): captions, speech understanding, and multilingual live experiences * [Cloud Recording](/en/realtime-media/cloud-recording): archive, replay, compliance, QA, and post-session review workflows * [Transcoding](/en/realtime-media/transcoding): mixing, layout composition, and output transformation ### Bridge real-time sessions with external media systems Use these docs when your product must ingest outside streams, publish session media to other systems, or interoperate with existing streaming infrastructure. * [Media Push](/en/realtime-media/media-push): send RTC channel media to CDN pipelines or downstream media systems * [Media Pull](/en/realtime-media/media-pull): bring online media streams into an interactive real-time session * [Media Gateway](/en/realtime-media/rtmp-gateway): connect RTMP-based devices and systems to Agora ### Deliver playback to larger audiences Use these docs when interactive participation and large-scale viewing are separate concerns in your architecture. * [Fusion CDN](/en/realtime-media/fusion-cdn): multi-CDN distribution for live playback at audience scale ### Extend the session to devices and backend services Use these docs when participants are not just mobile or web clients, but also embedded devices, operator consoles, or server-side workers. * [IoT & Edge](/en/realtime-media/rtsa): device and edge connectivity for cameras, terminals, and embedded endpoints * [Server Gateway](/en/realtime-media/rtc-server-sdk): backend participation in media send, receive, subscribe, and control flows ## How to navigate this tab * Start with [Voice & Video](/en/realtime-media/rtc), [Signaling](/en/realtime-media/rtm), [Chat](/en/realtime-media/im), or [Whiteboard](/en/realtime-media/whiteboard) when your main question is how users interact inside a live session. * Start with [Transcription & Translation](/en/realtime-media/speech-to-text), [Cloud Recording](/en/realtime-media/cloud-recording), or [Transcoding](/en/realtime-media/transcoding) when your main question is what should happen to the media during or after the session. * Start with [Media Push](/en/realtime-media/media-push), [Media Pull](/en/realtime-media/media-pull), [Media Gateway](/en/realtime-media/rtmp-gateway), or [Fusion CDN](/en/realtime-media/fusion-cdn) when your main question is how to move media across systems or out to larger audiences. * Start with [IoT & Edge](/en/realtime-media/rtsa) or [Server Gateway](/en/realtime-media/rtc-server-sdk) when your main question is how devices or backend services participate in the real-time workflow. ## Common reading paths * Interactive live streaming with large audience playback: [Voice & Video](/en/realtime-media/rtc) -> [Media Push](/en/realtime-media/media-push) -> [Fusion CDN](/en/realtime-media/fusion-cdn) * Meeting archive and searchable records: [Voice & Video](/en/realtime-media/rtc) -> [Cloud Recording](/en/realtime-media/cloud-recording) -> [Transcription & Translation](/en/realtime-media/speech-to-text) * External stream into an interactive session: [Media Pull](/en/realtime-media/media-pull) or [Media Gateway](/en/realtime-media/rtmp-gateway) -> [Voice & Video](/en/realtime-media/rtc) * Smart device connectivity with backend media control: [IoT & Edge](/en/realtime-media/rtsa) -> [Server Gateway](/en/realtime-media/rtc-server-sdk) # Media processing and distribution (/en/realtime-media/media-processing-and-distribution)
## Recording Recording captures live interaction content into your storage flow for archive, replay, review, and compliance-oriented workflows. ### Cloud Recording Fits teams that want a managed recording path from Agora.
### Local Server Recording Fits teams that need self-managed recording servers and finer-grained control over the recording workflow.
## Media Push Pushes interactive channel content into CDN or broadcast-style distribution paths.
## Media Pull Brings external online media streams into an existing real-time scenario.
## Transcoding Transcoding handles media transformation and composition for live streaming and multi-layout output scenarios. ### Cloud Transcoding Fits teams that need hosted stream composition and output transformation.
## Media Gateway Bridges standard live protocol flows with real-time interaction systems.
## Fusion CDN Live Streaming Optimizes end-to-end playback through multi-CDN scheduling and distribution.
Additional note: the broader Agora content system also includes PPT Conversion. For this IA pass, it can stay as an extension of transcoding rather than a first-class left-nav item. # RTC overview (/en/realtime-media/overview) ## Build live interaction ## Extend live interaction ## Process session content ## Connect external media ## Deliver to audiences # Server-side and extensions (/en/realtime-media/server-and-extensions)
## Server Gateway Lets the backend participate directly in RTC media send, receive, and control flows.
## SDK extension plugins Support capability enhancement and customized audio-video effects on top of the existing SDK stack.
## Marketplace Provides a faster path to selecting, purchasing, and connecting external real-time modules into your product. # Set up service and credentials (/en/realtime-media/setup-service-and-credentials) ## Why this always comes first No matter whether you end up building AI, RTC, RTM, recording, or transcoding, the first step is rarely "write code." It is usually confirming Console access, project state, App ID, credentials, and authentication. ## What you should prepare * project and App ID * customer ID and customer secret * token generation strategy * service enablement status * the boundary between test and production environments ## Recommended practice Prepare the control plane and credential flow first, then move into product-specific docs. That prevents expensive rework when permissions, quotas, or auth assumptions diverge later. # FAQ (/en/api-reference/faq) # API reference (/en/api-reference/api-ref) # Channel Management REST API (/en/api-reference/api-ref/iot-channel-management-rest-api) In addition to the SDK that you integrate into the app client, Agora provides server-side RESTful APIs to manage real-time channels. The IoT Channel Management REST API uses the shared RTC REST API reference. Use these APIs to query channel information, manage user privileges, and query Agora Notifications service IP addresses. ## REST API reference * [RTC REST API overview](/en/api-reference/api-ref/rtc) * [How to call RESTful APIs](/en/api-reference/api-ref/rtc/how-to-call-api) * [RESTful authentication](/en/api-reference/api-ref/rtc/authentication) * [Response status codes](/en/api-reference/api-ref/rtc/response-status-codes) ## Endpoints **Channel information** * [Query the channel list](/en/api-reference/api-ref/rtc/query-channel-list) * [Query the user list](/en/api-reference/api-ref/rtc/query-user-list) * [Query the host list](/en/api-reference/api-ref/rtc/query-host-list) * [Query the user status](/en/api-reference/api-ref/rtc/query-user-status) **User privilege banning** * [Create a banning rule](/en/api-reference/api-ref/rtc/create-ban-rule) * [Delete a banning rule](/en/api-reference/api-ref/rtc/delete-ban-rule) * [Get the banning rule list](/en/api-reference/api-ref/rtc/get-ban-rule-list) * [Update the banning rule expiration](/en/api-reference/api-ref/rtc/update-ban-expiration) **Message notification service** * [Query the IP address](/en/api-reference/api-ref/rtc/query-ip-address) ## Supporting topics * [Ban user privileges best practices](/en/api-reference/api-ref/rtc/ban-user-privileges-best-practices) * [Ensure service reliability](/en/api-reference/api-ref/rtc/ensure-service-reliability) * [Channel event types](/en/api-reference/api-ref/rtc/channel-event-types) # Fastboard API (/en/api-reference/api-ref/uikit-sdk) This page provides the API reference for the Fastboard SDK. ## Reference ## FastboardView class ### getFastboard ```java public Fastboard getFastboard() ``` Get the `Fastboard` object. The Fastboard SDK does not support initializing a `Fastboard` instance directly. To get the `Fastboard` object, you need to add the `FastboardView` object to the app's layout, and then call the `getFastboard` method. **Note** Call this method to obtain the `Fastboard` object before calling other APIs. **Returns** The `Fastboard` object is returned when the method call is successful. ## Fastboard class The Fastboard class provides methods for creating `FastRoom` objects. ### createFastRoom ```java public FastRoom createFastRoom(FastRoomOptions roomOptions) ``` Create a `FastRoom` object. **Note** Call this method after obtaining the `Fastboard` object. **Parameters** * `roomOptions`: Configuration options for the whiteboard room. See [FastRoomOptions](#roomoptions) for details. **Returns** A `FastRoom` object is returned when the method call is successful. ### preloadWhiteboardView ```java public static void preloadWhiteboardView() { WhiteboardViewManager.get().preload(); } ``` Preload the whiteboard view. This method only takes effect when `autoPreload` in [`FastboardConfig`](#fastboardconfig) is set to `false`. Successfully calling this method will consume memory to perform one whiteboard view preload, improving the speed of joining rooms next time. ### destroy ```java public void destroy() ``` Destroy the room and release resources. ### setConfig ```java public static void setConfig(FastboardConfig config) ``` Modifies the auxiliary configuration of the `Fastboard` object. **Parameters** * `config`: The auxiliary configuration of the `Fastboard` object. See [FastboardConfig](#fastboardconfig).
### FastRoomOptions Whiteboard room configuration options. ```java public class FastRoomOptions { private final String appId; private final String uuid; private final String token; private final String uid; private final boolean writable; private final FastRegion fastRegion; private Float containerSizeRatio; private FastUserPayload userPayload; public FastRoomOptions(String appId, String uuid, String token, String uid, FastRegion fastRegion) { this(appId, uuid, token, uid, fastRegion, true); } public FastRoomOptions(String appId, String uuid, String token, String uid, FastRegion fastRegion, boolean writable) { this.appId = appId; this.uuid = uuid; this.token = token; this.uid = uid; this.fastRegion = fastRegion; this.writable = writable; } } ``` The `FastRoomOptions` class contains the following properties: * `appId`: String. App Identifier for the interactive whiteboard project. For details, see [Get security credentials for your whiteboard project](/en/realtime-media/whiteboard/build/set-up-and-build-your-first-app/enable-whiteboard#get-security-credentials-for-your-whiteboard-project). * `uuid`: String. The UUID of the room, which is the unique identifier of the room. For details, see [Create a room (POST)](/en/api-reference/api-ref/whiteboard/room-management#create-a-room-post) for the value of the `uuid` parameter in the response package body after the request is successful. * `token`: String. Room Token of the room, used for user authentication when joining the room. It can be obtained in the following ways: * Call the [Generate a room token (POST)](/en/realtime-media/whiteboard/build/authenticate-users/generate-token-rest#generate-a-room-token-post) RESTful API. * Build a token generator at your app server. See [Generate a token at app server](/en/realtime-media/whiteboard/build/authenticate-users/generate-token-app-server). * `uid`: String. The unique identifier of a user in string format. The maximum length is 1,024 bytes. Ensure that the `uid` of each user in the same room is unique. * `writable`: boolean. Whether the user joins the whiteboard room in interactive mode: * `true`: Join the whiteboard room in interactive mode, that is, with read and write permissions. * `false`: Join the whiteboard room in subscription mode, that is, with read-only permission. * `fastRegion`: The data center, which must be the same as the data center you chose when [creating the whiteboard room](/en/api-reference/api-ref/whiteboard/room-management#create-a-room-post). See [FastRegion](#fastregion). * `containerSizeRatio`: Float. In the local display window, the aspect ratio of the content is `0.56` by default, which is `9:16`. * `userPayload`: User information displayed by the user's cursor, including the user's nickname and avatar. See [`FastUserPayload`](#fastuserpayload) for details. ### FastRegion Data center, containing the following enumeration values: * `CN_HZ`: Hangzhou, China, the service area covers East Asia, Southeast Asia, and other areas not covered by the data center. * `US_SV`: Silicon Valley, USA, the service area covers North America and South America. * `SG`: Singapore, the service area covers Singapore, East Asia, and Southeast Asia. * `IN_MUM`: Mumbai, India, the service area covers India. * `EU`: London, UK, the service area covers Europe. ### FastUserPayload ```java public class FastUserPayload { private final String nickName; private final String avatar; public FastUserPayload(String nickName) { this(nickName, null); } public FastUserPayload(String nickName, String avatar) { this.nickName = nickName; this.avatar = avatar; } } ``` `FastUserPayload` object, used to store the user information displayed on the cursor, contains the following member variables: * `nickName`: String. The user's nickname displayed on the user's cursor. * `avatar`: String. (Optional) The user avatar displayed on the user cursor, the URL address corresponding to the avatar should be passed in. ### FastboardConfig ```java public class FastboardConfig { private final boolean enablePreload; private final int preloadCount; private final boolean autoPreload; } ``` The auxiliary configuration class of the `Fastboard` object, including the following attributes: * `enablePreload`: Boolean. Whether to enable whiteboard preloading: * `true`: Enable whiteboard preloading. * `false`: (Default) Disable whiteboard preloading. * `preloadCount`: Integer. The number of WebViews to preload, defaults to `0`. The actual effect of this attribute is limited by the memory capacity. You need to adjust the settings appropriately according to the actual situation. * `autoPreload`: Boolean. This property only takes effect when `enablePreload` is `true`. Whether to enable automatic preloading: * `true`: (Default) Enable automatic preloading. After enabling, whiteboard views will be automatically preloaded to improve loading speed, but will occupy additional memory. * `false`: Disable automatic preloading. After disabling, you can call [`preloadWhiteboardView`](#preloadwhiteboardview) to manually perform preloading. ## FastRoom class The `FastRoom` class provides methods for managing interactive whiteboard real-time rooms. ### join \[1/2] ```java public void join() ``` Join a whiteboard room. **Note** This method needs to be called after the `FastRoom` object is successfully created. ### join \[2/2] ```java public void join(@Nullable OnRoomReadyCallback onRoomReadyCallback) ``` Join a whiteboard room. **Note** This method needs to be called after the `FastRoom` object is successfully created. **Parameters** * `onRoomReadyCallback`: [OnRoomReadyCallback](#roomready) interface instance. Passing in `null` means not to register the interface. ### OnRoomReadyCallback The `OnRoomReadyCallback` interface is used to send room event notifications to the app and includes the following member methods: ```java void onRoomReady(FastRoom fastRoom); ``` Room ready callback. **Parameters** * `fastRoom`: `FastRoom` object. ### isReady ```java public boolean isReady() ``` Get whether the room is ready. After calling the `join` method, you need to call this method to get whether the room is ready. After the room is ready, other methods in the `FastRoom` class can be called to operate the whiteboard. **Returns** Is the room ready? * `true`: ready. * `false`: Not ready yet. ### isWritable ```java public boolean isWritable() ``` Get whether the local user's current interactive whiteboard real-time room is in interactive mode. **Returns** Get whether the local user is in interactive mode: * `true`: interactive mode, that is, with read and write permissions. * `false`: Subscription mode, that is, with read-only permissions. ### redo ```java public void redo() ``` Redo, that is, roll back the undo operation. ### setWritable \[1/2] ```java public void setWritable(boolean writable) ``` Set whether the user is in interactive mode in the room. **Note** This method needs to be called after the room is ready. **Parameters** * `writable`: whether the user is in interactive mode in the room: * `true`: interactive mode, that is, with read and write permissions. * `false`: Subscription mode, that is, with read-only permissions. ### setWritable \[2/2] ```java public void setWritable(boolean writable, FastResult result) ``` Set whether the user is in interactive mode in the room. **Note** This method needs to be called after the room is ready. **Parameters** * `writable`: whether the user is in interactive mode in the room: * `true`: interactive mode, that is, with read and write permissions. * `false`: Subscription mode, that is, with read-only permissions. * `result`: The result of the `setWritable` method call. See [FastResult](#result) for details. Passing in the `FastResult` instance, the SDK will trigger the callback implemented in the `FastResult` interface and report whether the `setWritable` method call is successful; passing in `null` means not to listen to the callback. ### registerApp Registers a third-party window application to the whiteboard room. ```java public void registerApp(FastRegisterAppParams params, FastResult result) ``` This method allows you to register external applications that can run as independent windows within the whiteboard. Registration supports two methods: 1. **Local script registration**: Uses local JavaScript code (recommended for its high reliability) 2. **Remote package registration**: Uses a remote URL (may fail due to network issues) Caution * You must complete the registration before joining the room to ensure the application loads correctly. * Ensure that all clients use the same version of the application to avoid runtime issues. * Register the application before joining the room. * Registered applications can be added to the room using `getRoom().addApp()`. **Parameters** * `params`: Registration parameters, including app configurations. See [FastRegisterAppParams](#fastregisterappparams) for details. * `result`: The result of the `registerApp` method call. See [FastResult](#result) for details. If you pass a `FastResult` instance, the SDK triggers the callback implemented in the `FastResult` interface to report whether the `registerApp` method call was successful. Passing `null` means the callback is not listened to. ### FastRegisterAppParams A configuration class for registering third-party window app parameters. ```java public class FastRegisterAppParams { private String javascriptString; private String kind; private String url; private Map appOptions; private String variable; public FastRegisterAppParams(String javascriptString, String kind, String variable, Map appOptions) { this.javascriptString = javascriptString; this.kind = kind; this.appOptions = appOptions; this.variable = variable; } public FastRegisterAppParams(String url, String kind, Map appOptions) { this.url = url; this.kind = kind; this.appOptions = appOptions; } public String getJavascriptString() { return javascriptString; } public String getKind() { return kind; } public String getUrl() { return url; } public Map getAppOptions() { return appOptions; } public String getVariable() { return variable; } } ``` The `FastRegisterAppParams` class provides two registration methods: 1. **Local script registration**: Registers using a local JavaScript string (recommended for its high reliability) * Use the `FastRegisterAppParams(String javascriptString, String kind, String variable, Map appOptions)` constructor 2. **Remote package registration**: Registers using a published package's URL (may fail due to network issues) * Use the `FastRegisterAppParams(String url, String kind, Map appOptions)` constructor The `FastRegisterAppParams` class includes the following properties: * `javascriptString`: String. Local JavaScript script code for local script registration. * `kind`: String. The name of the registered app, representing the type name of the app. * `url`: String. The URL address of the remote registration package for remote package registration. * `appOptions`: Map\. Parameters passed during the initialization of the app instance. This configuration is not synchronized to other clients and is considered a local setting, often used to toggle debug mode. * `variable`: String. The variable name mounted on the window, appearing as `window.variable` after mounting. ### undo ```java public void undo() ``` Undo the previous operation. ### setStrokeColor ```java public void setStrokeColor(@ColorInt int color) ``` Set the line color. **Parameters** * `color`: Integer. Line color, RGB format, for example `0x0000FF` means blue ### setAppliance ```java public void setAppliance(FastAppliance fastAppliance) ``` Set the currently used whiteboard tool. **Parameters** * `fastAppliance`: Whiteboard tool. See [FastAppliance](#appliance) for details. ### FastAppliance Whiteboard tool, containing the following enumeration values: * `CLICKER(Appliance.CLICKER)`: Click tool. Currently mainly used for clicking content on HTML5 files. * `SELECTOR(Appliance.SELECTOR)`: Select tool. * `HAND(Appliance.HAND)`: Grab tool, used to move the view. * `PENCIL(Appliance.PENCIL)`: Pencil. * `RECTANGLE(Appliance.RECTANGLE)`: Rectangle tool. * `ELLIPSE(Appliance.ELLIPSE)`: Ellipse tool. * `TEXT(Appliance.TEXT)`: text tool. * `ERASER(Appliance.ERASER)`: Eraser tool. * `PENCIL_ERASER(Appliance.PENCIL_ERASER)`: Pencil eraser tool, used to erase local pencil strokes. * `LASER_POINTER(Appliance.LASER_POINTER)`: Laser pointer. * `ARROW(Appliance.ARROW)`: Arrow. * `STRAIGHT(Appliance.STRAIGHT)`: Straight line. * `PENTAGRAM(Appliance.SHAPE, ShapeType.Pentagram)`: Pentagram. * `RHOMBUS(Appliance.SHAPE, ShapeType.Rhombus)`: rhombus. * `TRIANGLE(Appliance.SHAPE, ShapeType.Triangle)`: triangle. * `BUBBLE(Appliance.SHAPE, ShapeType.SpeechBalloon)`: Speech bubble. * `OTHER_CLEAR()`: Clear the whiteboard content. ### setStrokeWidth ```java public void setStrokeWidth(int width) ``` Set the width of the line. **Parameters** * `width`: Integer. Line width (px). ### cleanScene ```java public void cleanScene() ``` Clear the whiteboard content. ### setWritable ```java public void setWritable(boolean writable) ``` Set whether the user is in interactive mode in the room. **Parameters** * `writable`: boolean. Whether the user is in interactive mode: * `true`: interactive mode, that is, with read and write permissions. * `false`: Subscription mode, that is, with read-only permissions. ### insertImage ```java public void insertImage(String url, int width, int height) ``` Insert picture. This method can insert and display the specified network image onto the current whiteboard page. **Parameters** * `url`: String. The URL address of the image. Please ensure that the app client can access the URL, otherwise the image will not be displayed properly. * `width`: Integer. The width of the image (px). * `height`: Integer. The height of the image (px). ### insertVideo ```java public void insertVideo(String url, String title) ``` Insert and play audio and video in the whiteboard sub-window. **Parameters** * `url`: URL address of audio and video files. Please ensure that the app client can access the URL, otherwise the audio and video files cannot be loaded normally. * `title`: window title. ### insertStaticDoc ```java public void insertStaticDoc(DocPage[] pages, String title, FastResult result) ``` Inserts a static document into a whiteboard subwindow. **Parameters** * `pages`: DocPage array. The page settings for the inserted document. See [DocPage](#docpage). * `title`: String. The title of the subwindow. * `result`: An object that implements the `FastResult` interface, used to handle the result of the method call. The SDK will use this object to report the result of the operation. If you do not need to listen to the callback, please pass in `null`. ### DocPage ```java public class DocPage { private String src; private String preview; private Double width; private Double height; } ``` A class for configuring the page settings for inserting documents. Includes the following attributes * `src`: String. The address of the image or dynamic PPT page. * Image: URL address, which can be generated by yourself or by the document conversion function. You can get the address from the image field returned by the query conversion task progress. Make sure the App client can access the URL, otherwise the image cannot be loaded normally. * Dynamic PPT page: URI address generated by the document conversion function. * `preview`: String. (Optional) URL address of the image or dynamic PPT preview image. The URL address of the dynamic PPT preview image can be obtained from the preview field returned by the query document conversion progress. After passing in, the preview image will be displayed on the left side of the App. Make sure the App client can access the URL, otherwise the preview image cannot be loaded normally. * `width`: Integer. The width of the inserted document in the whiteboard (px). * `height`: Integer. The height of the inserted document in the whiteboard (px). ### insertPptx ```java public void insertPptx(String taskUuid, String prefixUrl, String title, FastResult result) ``` Insert a dynamic document in a whiteboard sub-window. This method can insert dynamic HTML web pages converted by the document conversion function. **Parameters** * `taskUUID`: String. The Task UUID of the document conversion task, which is the value of the `uuid` field in the response body when the initiate document conversion task request is successful. * `prefixUrl`: String. The prefix path of the converted result file address, which is the value of the `prefix` field in the response body when the query document conversion progress request is successful. * `title`: String. The title of the sub-window. * `result`: An object that implements the `FastResult` interface, used to handle the result of the method call. The SDK will use this object to report the result of the operation. If you do not need to listen to the callback, please pass in `null`. ### insertDocs ```java public void insertDocs(FastInsertDocParams params, FastResult result) ``` **Note** This method is deprecated since v1.6.0. Use [insertStaticDoc](#insertstaticdoc) or [insertPptx](#insertpptx) instead. Insert and display documents in the whiteboard subwindow. After successfully initiating the [document conversion task](/en/api-reference/api-ref/whiteboard/file-conversion), you can call this method and pass in the relevant parameters of the converted document. The SDK will automatically create a sub-window, insert and display the converted document in pages. **Parameters** * `params`: Insert the parameter settings of the document. See [FastInsertDocParams](#docparams) for details. * `result`: `insertDocs` method call result. See [FastResult](#result) for details. Passing in the `FastResult` instance, the SDK will trigger the callback implemented in the `FastResult` interface and report whether the `insertDocs` method call is successful; passing in `null` means not to listen to the callback. ### FastInsertDocParams Document parameter settings. ```java public class FastInsertDocParams { private String taskUUID; private String taskToken; private FastRegion region; private ConverterType converterType; private String fileType; private Boolean dynamicDoc; private String title; } ``` The `FastInsertDocParams` class contains the following properties: * `taskUUID`: String. Task UUID of the document conversion task, that is, [Initiate document conversion task API](/en/api-reference/api-ref/whiteboard/file-conversion). When the request is successful, the uuid parameter in the response package body value. * `taskToken`: String. The Task Token of the document conversion task must be consistent with the Task Token passed in when [Initiate document conversion task](/en/api-reference/api-ref/whiteboard/file-conversion). * `converterType`: enumeration. The version of the document conversion service, the values are as follows: * `Projector`: new version. For details, see [New version of document conversion service](/en/realtime-media/whiteboard/build/display-files-and-manage-scenes/file-conversion-overview). * `WhiteboardConverter`: legacy (default). For details, see [Old version of the document conversion service](/en/realtime-media/whiteboard/reference/file-conversion-overview-deprecated). * `fileType`: String. Document type: * `pdf`: static document. * `pptx`: dynamic document. * `dynamicDoc`: boolean. Whether the document conversion task type is a dynamic conversion task. * `title`: String. Window title. ### setFastStyle ```java public void setFastStyle(FastStyle style) ``` Style the whiteboard user interface. **Parameters** * `style`: The style of the whiteboard user interface. See [FastStyle](#faststyle) for details. ### FastStyle Whiteboard user interface style. ```java public class FastStyle { private int mainColor; private boolean darkMode; public FastStyle() { } public int getMainColor() { return mainColor; } public void setMainColor(@ColorInt int color) { this.mainColor = color; } public boolean isDarkMode() { return darkMode; } public void setDarkMode(boolean darkMode) { this.darkMode = darkMode; } public FastStyle copy() { FastStyle style = new FastStyle(); style.mainColor = mainColor; style.darkMode = darkMode; return style; } } ``` The `FastStyle` class contains the following member methods: **getMainColor** Gets the theme color of the whiteboard user interface. **Returns** The theme color for the whiteboard user interface. **setMainColor** Set the theme color of the whiteboard user interface. This method can set the color of some button borders and prompt text when the whiteboard is loaded. **Parameters** * `color`: The theme color of the whiteboard user interface, RGB format, for example `0x0000FF` means blue. **isDarkMode** Gets whether the whiteboard user interface is in dark mode. **Returns** * `true`: Whiteboard user interface is in dark mode. * `false`: The whiteboard user interface is in light mode. **setDarkMode** Set the whiteboard user interface to dark mode. **Parameters** * `darkMode`: whether to use dark mode: * `true`: dark mode. * `false`: light mode. ### setResource ```java public void setResource(FastResource fastResource) ``` Set resources related to whiteboard color. **Parameters** * `fastResource`: Whiteboard color-related resources. See [FastResource](#fastresource) for details. ### FastResource Resources related to whiteboard colors. ```java public class FastResource { @ColorInt public int getBackgroundColor(boolean darkMode) { return color(darkMode ? R.color.fast_dark_mode_bg : R.color.fast_light_mode_bg ); } @ColorInt public int getBoardBackgroundColor(boolean darkMode) { return getBackgroundColor(darkMode); } } ``` Contains the following member methods, all of which can be overridden to customize colors: **getBackgroundColor** Get the background color of the whiteboard control. **Parameters** * `darkMode`: boolean. Whether the background color of the whiteboard control is dark mode: * `true`: dark mode. * `false`: light mode. **Returns** Hexadecimal color value. **getBoardBackgroundColor** Get the whiteboard background color. **Note** If you do not override this method, `getBackgroundColor` will be called by default. **Parameters** * `darkMode`: boolean. Whether the whiteboard background color is dark mode: * `true`: dark mode. * `false`: light mode. **Returns** Hexadecimal color value. ## FastUiSettings class The `FastUiSettings` class provides methods for setting up the whiteboard user interface. ### showRoomController ```java public void showRoomController(ControllerId... ids) ``` Demonstrates controls on a whiteboard user interface. **Parameters** * `ids`: The identifier of the control. See [ControllerId](#controllerId) for details. ### hideRoomController ```java public void hideRoomController(ControllerId... ids) ``` Hides controls on the whiteboard user interface. **Parameters** * `ids`: The identifier of the control. See [ControllerId](#controllerId) for details. ### ControllerId Controls on the whiteboard user interface, including the following enumeration values: * `RedoUndo`: redo and undo buttons. * `ToolBox`: Toolbar. * `PageIndicator`: Page indicator. ### setToolsExpandAppliances ```java public static void setToolsExpandAppliances(List> toolsExpandAppliances) ``` Sets the tool set included in the toolbar in expanded mode. If the default toolbar provided by Fastboard SDK cannot meet your needs, you can call this method to customize the tools contained in the toolbar and set the toolbar to expand mode. You can pass in a secondary tool list in this method. The elements in the primary list will be expanded and displayed on the toolbar, and the elements in the secondary list will be collapsed. After successfully calling the `setToolsExpandAppliances` method, if you need to switch the toolbar to collapsed mode, you can call [setToolboxExpand](#settoolboxexpand). **Note** This method needs to be called before joining the whiteboard room. **Parameters** * `toolsExpandAppliances`: Tools included in the toolbar in expanded mode. See [FastAppliance](#appliance) for details. **Example** ```java ArrayList> config = new ArrayList<>(); config.add(Arrays.asList( FastAppliance.CLICKER, FastAppliance.PENCIL, FastAppliance.TEXT, FastAppliance.SELECTOR, FastAppliance.ERASER )); config.add(Arrays.asList(FastAppliance.SELECTOR)); config.add(Arrays.asList(FastAppliance.PENCIL)); config.add(Arrays.asList(FastAppliance.TEXT)); config.add(Arrays.asList(FastAppliance.ERASER)); config.add(Arrays.asList( FastAppliance.STRAIGHT, FastAppliance.ARROW, FastAppliance.RECTANGLE, FastAppliance.ELLIPSE, FastAppliance.PENTAGRAM, FastAppliance.RHOMBUS, FastAppliance.BUBBLE, FastAppliance.TRIANGLE )); config.add(Arrays.asList(FastAppliance.OTHER_CLEAR)); FastUiSettings.setToolsExpandAppliances(config); ``` ### setToolsCollapseAppliances ```java public static void setToolsCollapseAppliances(List toolsCollapseAppliances) ``` Sets the tool set included in the toolbar in collapsed mode. If the default toolbar provided by Fastboard SDK cannot meet your needs, you can call this method to customize the tools included in the toolbar and set the toolbar to fold mode. You can pass in a first-level tool list in this method, and the elements in the list will be collapsed on the toolbar by default. After successfully calling the `setToolsCollapseAppliances` method, if you need to switch the toolbar to expanded mode, you can call [setToolboxExpand](#settoolboxexpand). **Note** This method needs to be called before joining the whiteboard room. **Parameters** * `toolsCollapseAppliances`: Tools included in the whiteboard toolbar. See [FastAppliance](#appliance) for details. **Example** ```java ArrayList collapseAppliances = new ArrayList<>(); collapseAppliances.add(FastAppliance.PENCIL); collapseAppliances.add(FastAppliance.ERASER); collapseAppliances.add(FastAppliance.ARROW); collapseAppliances.add(FastAppliance.SELECTOR); collapseAppliances.add(FastAppliance.TEXT); collapseAppliances.add(FastAppliance.OTHER_CLEAR); FastUiSettings.setToolsCollapseAppliances(collapseAppliances); ``` ### setToolsColors ```java public static void setToolsColors(List toolsColors) ``` Set the color used by the whiteboard tool. This method sets the color of graphics, lines, or text drawn using the whiteboard tool. **Note** This method needs to be called before joining the whiteboard room. **Parameters** * `toolsColors`: Color of whiteboard tools, RGB format, for example `0x0000FF` means blue. ### setStrokeRange ```java public void setStrokeRange(int min, int max) ``` Sets the range for the line width slider in the toolbox. This method configures the minimum and maximum values of the line width slider in the toolbox extension panel. Users can adjust the line width within this range when using drawing tools like pencils, rectangles, circles, etc. The default range is \[1, 24]. Caution * The line width slider will be restricted to the specified range. * If the current line width exceeds the new range, it will be constrained within the new range. * This setting affects all drawing tools that support line width adjustments. **Parameters** * `min`: int. The minimum value for the line width (inclusive). * `max`: int. The maximum value for the line width (inclusive). ### setToolboxEdgeMargin ```java public void setToolboxEdgeMargin(int margin) ``` Set the margins between the whiteboard toolbar and the sides. The definition of the margin is determined by the toolbar position set in `setToolboxGravity`: * The toolbar is located on the left side of the whiteboard: the margin refers to the distance between the left side of the toolbar and the left side of the whiteboard. * The toolbar is located on the right side of the whiteboard: the margin refers to the distance between the right side of the toolbar and the right side of the whiteboard. **Parameters** * `margin`: Integer. The distance between the toolbar and the side of the whiteboard, in `px`. ### setToolboxGravity ```java public void setToolboxGravity(int gravity) ``` Set the position of the toolbar on the whiteboard. **Parameters** * `gravity`: The position of the toolbar on the whiteboard: * `Gravity.LEFT`: left. * `Gravity.RIGHT`: Right. ### setToolboxExpand ```java public void setToolboxExpand(boolean expand) ``` Set whether to expand the toolbar. The default display state of the toolbar is related to the device. It is expanded by default on tablets and collapsed by default on mobile phones. You can call this method to modify the display state of the toolbar. **Parameters** * `expand`: Whether to expand the toolbar: * `true`: expand. * `false`: fold. ## FastLogger Class The `FastLogger` class provides a flexible whiteboard runtime log recording tool that can easily record four different levels of log information: `debug`, `info`, `warn`, and `error`. The `FastLogger` class uses a `Logger` interface and provides a default implementation `DefaultLogger` to use the system's `Log` class. ### setLogger ```java public static void setLogger(Logger logger) ``` Set custom logger. **Parameters** * `logger`: The logger instance. If you need to customize the logger, you need to pass in an implementation class of the `Logger` interface, see [`Logger`](#logger); passing in `null` will result in using the system default logger. ### Logger Logger specification, including recording methods for all log levels. ### debug ```java static void debug(String msg) ``` Records a debug log. **Parameters** * `msg`: String. Log message. ### info ```java static void info(String msg) ``` Records an info log. **Parameters** * `msg`: String. Log message. ### warn ```java static void warn(String msg) ``` Records a warn log. **Parameters** * `msg`: String. Log message. ### error ```java static void error(String msg) ``` Records an error. **Parameters** * `msg`: String. Log message. ## FastResult The result of the method call. ```java public interface FastResult { void onSuccess(T value); void onError(Exception exception); } ``` The `FastResult` interface reports the results of method calls and includes the following callback methods: * `onSuccess`: callback when the method call is successful. * `onError`: callback when an error occurs. ## Platform-specific versions - [Android](/en/api-reference/api-ref/uikit-sdk/android.md) - [iOS](/en/api-reference/api-ref/uikit-sdk/ios.md) - [Web](/en/api-reference/api-ref/uikit-sdk/web.md) # Optimize audio quality (/en/ai/best-practices/audio-setup) In real-time audio interactions, the rhythm, continuity, and intonation of conversations between humans and AI often differ from those between humans. To improve the AI–human conversation experience, it's important to optimize audio settings. When using the Android, iOS, or Web RTC SDK with the Conversational AI Engine, follow the best practices in this guide to improve conversation fluency and reliability, especially in complex network environments. ## Server configuration When calling the server API to create a conversational AI agent, use the default values for audio-related parameters to ensure the best audio experience. ## Client configuration To configure the client app, implement the following: ### Integrate the required dynamic libraries For the best Conversational AI Engine audio experience, integrate and load the following dynamic libraries in your project: Android iOS Web * AI noise suppression plugin: `libagora_ai_noise_suppression_extension.so` * AI echo cancellation plug-in: `libagora_ai_echo_cancellation_extension.so` For integration details, refer to [App size optimization](/en/realtime-media/rtc/build/optimize-and-operate/app-size-optimization/android). * AI noise suppression plugin: `AgoraAiNoiseSuppressionExtension.xcframework` * AI echo cancellation plug-in: `AgoraAiEchoCancellationExtension.xcframework` For integration details, refer to [App size optimization](/en/realtime-media/rtc/build/optimize-and-operate/app-size-optimization/ios). * Integrate the `agora-extension-ai-denoiser` extension. Refer to [AI Noise Suppression](/en/realtime-media/rtc/build/enhance-the-audio-experience/ai-noise-suppression/web). Info Optimizing audio uses AI Noise Suppression, which is a paid feature. ## Optimize audio for optimal performance You can optimize audio settings in the following ways: * **(Recommended) Use the Toolkit APIs** Supported in Video/Voice SDK version 4.5.1 and above. * **Use the Video/Voice SDK APIs directly** Supported in SDK version 4.3.1 and above. ### Use the toolkit APIs In this solution, you use the toolkit APIs to optimize audio settings. Android iOS Web 1. **Integrate the toolkit** Add the toolkit to your project using Maven or source code. See [Install the Android toolkit](/en/api-reference/api-ref/conversational-ai/client-toolkit/android#installation). 2. **Create a toolkit instance** ```kotlin val config = ConversationalAIAPIConfig( rtcEngine = rtcEngine, rtmClient = rtmClient, renderMode = TranscriptRenderMode.Word, enableLog = true, enableRenderModeFallback = true ) val api = ConversationalAIAPIImpl(config) ``` 3. **Set optimal audio settings** Before joining the RTC channel, call `loadAudioSettings()` to apply the optimal audio parameters. ```kotlin api.loadAudioSettings() rtcEngine.joinChannel(token, channelName, null, userId) ``` 4. **Release resources** ```kotlin api.destroy() ``` 1. **Integrate the toolkit** Add the toolkit to your project using CocoaPods, Swift Package Manager, or source code. See [Install the iOS toolkit](/en/api-reference/api-ref/conversational-ai/client-toolkit/ios#installation). 2. **Create a toolkit instance** ```swift let config = ConversationalAIAPIConfig( rtcEngine: rtcEngine, rtmEngine: rtmEngine, renderMode: .words, enableLog: true, enableRenderModeFallback: true ) convoAIAPI = ConversationalAIAPIImpl(config: config) ``` 3. **Set optimal audio settings** Before joining the RTC channel, call `loadAudioSettings()` to apply the optimal audio parameters. ```swift convoAIAPI.loadAudioSettings() rtcEngine.joinChannel(rtcToken: token, channelName: channelName, uid: uid, isIndependent: independent) ``` 4. **Release resources** ```swift convoAIAPI.destroy() ``` * **Integrate the transcript processing toolkit** Add the toolkit to your project using a package manager or source code. See [Install the Web toolkit](/en/api-reference/api-ref/conversational-ai/client-toolkit/web#installation). Refer to [Folder structure](#folder-structure) to understand the role of each file. ### Use the SDK APIs In this solution, you use the RTC SDK to optimize audio settings. #### Set audio parameters The settings in this section apply to Video/Voice SDK versions 4.3.1 and above. If you are using an earlier version, upgrade to version 4.5.1 or above or [Contact Technical Support](mailto\:support@agora.io). Android iOS Web For the best conversational AI audio experience, apply the following settings: 1. **Set the audio scenario**: When initializing the engine, set the audio scenario to the AI client scenario. You can also set the scenario before joining a channel by calling the `setAudioScenario` method. 2. **Configure audio parameters**: Call `setParameters` before joining a channel and whenever the `onAudioRouteChanged` callback is triggered. This configuration sets audio 3A plug-ins (acoustic echo cancellation, noise suppression, and automatic gain control), the audio sampling rate, the audio processing mode, and other settings. For recommended parameter values, refer to the sample code. Info Since Video/Voice SDK versions 4.3.1 to 4.5.0 do not support the AI client audio scenario, set the scenario to `AUDIO_SCENARIO_CHORUS` to improve the audio experience. However, the audio experience cannot be aligned with versions 4.5.1 and above. To get the best audio experience, upgrade the SDK to version 4.5.1 or higher. The following sample code defines a `setAudioConfigParameters` function to configure audio parameters. Call this function before joining a channel and whenever the audio route changes. ```kotlin private var rtcEngine: RtcEngineEx? = null private var mAudioRouting = Constants.AUDIO_ROUTE_DEFAULT // highlight-start // Set audio configuration parameters private fun setAudioConfigParameters(routing: Int) { mAudioRouting = routing rtcEngine?.apply { setParameters("{\"che.audio.aec.split_srate_for_48k\":16000}") setParameters("{\"che.audio.sf.enabled\":true}") setParameters("{\"che.audio.sf.stftType\":6}") setParameters("{\"che.audio.sf.ainlpLowLatencyFlag\":1}") setParameters("{\"che.audio.sf.ainsLowLatencyFlag\":1}") setParameters("{\"che.audio.sf.procChainMode\":1}") setParameters("{\"che.audio.sf.nlpDynamicMode\":1}") if (routing == Constants.AUDIO_ROUTE_HEADSET // 0 || routing == Constants.AUDIO_ROUTE_EARPIECE // 1 || routing == Constants.AUDIO_ROUTE_HEADSETNOMIC // 2 || routing == Constants.AUDIO_ROUTE_BLUETOOTH_DEVICE_HFP // 5 || routing == Constants.AUDIO_ROUTE_BLUETOOTH_DEVICE_A2DP) { // 10 setParameters("{\"che.audio.sf.nlpAlgRoute\":0}") } else { setParameters("{\"che.audio.sf.nlpAlgRoute\":1}") } setParameters("{\"che.audio.sf.ainlpModelPref\":10}") setParameters("{\"che.audio.sf.nsngAlgRoute\":12}") setParameters("{\"che.audio.sf.ainsModelPref\":10}") setParameters("{\"che.audio.sf.nsngPredefAgg\":11}") setParameters("{\"che.audio.agc.enable\":false}") } } // highlight-end // Create and initialize the RTC engine fun createRtcEngine(rtcCallback: IRtcEngineEventHandler): RtcEngineEx { val config = RtcEngineConfig() config.mContext = AgentApp.instance() config.mAppId = ServerConfig.rtcAppId config.mChannelProfile = Constants.CHANNEL_PROFILE_LIVE_BROADCASTING // highlight-start // Set the audio scene to AI dialogue scene (supported by 4.5.1 and above) // Version 4.3.1 ~ 4.5.0 is set to chorus scene AUDIO_SCENARIO_CHORUS config.mAudioScenario = Constants.AUDIO_SCENARIO_AI_CLIENT // Register audio route change callback config.mEventHandler = object : IRtcEngineEventHandler() { override fun onAudioRouteChanged(routing: Int) { super.onAudioRouteChanged(routing) // Set audio related parameters setAudioConfigParameters(routing) } } // highlight-end try { rtcEngine = (RtcEngine.create(config) as RtcEngineEx).apply { // highlight-start // Load the audio plugin loadExtensionProvider("ai_echo_cancellation_extension") loadExtensionProvider("ai_noise_suppression_extension") // highlight-end } } catch (e: Exception) { Log.e("CovAgoraManager", "createRtcEngine error: $e") } return rtcEngine!! } // Join the channel fun joinChannel(rtcToken: String, channelName: String, uid: Int, isIndependent: Boolean = false) { // highlight-start // Initialize audio configuration parameters setAudioConfigParameters(mAudioRouting) // highlight-end // Configure channel options and join the channel val options = ChannelMediaOptions() options.clientRoleType = CLIENT_ROLE_BROADCASTER options.publishMicrophoneTrack = true options.publishCameraTrack = false options.autoSubscribeAudio = true options.autoSubscribeVideo = false val ret = rtcEngine?.joinChannel(rtcToken, channelName, uid, options) } ``` For the best conversational AI audio experience, apply the following settings: 1. **Set the audio scenario**: When initializing the engine, set the audio scenario to the AI client scenario. You can also set the scenario before joining a channel by calling the `setAudioScenario` method. 2. **Configure audio parameters**: Call `setParameters` before joining a channel and whenever the `rtcEngine:didAudioRouteChanged:` callback is triggered. This configuration sets audio 3A plug-ins (acoustic echo cancellation, noise suppression, and automatic gain control), the audio sampling rate, the audio processing mode, and other settings. For recommended parameter values, refer to the sample code. Info Since Video/Voice SDK versions 4.3.1 to 4.5.0 do not support the AI client audio scenario, set the scenario to `AgoraAudioScenarioChorus` to improve the audio experience. However, the audio experience cannot be aligned with versions 4.5.1 and above. To get the best audio experience, upgrade the SDK to version 4.5.1 or higher. The following sample code defines a `setAudioConfigParameters` function to configure audio parameters. Call this function before joining a channel and whenever the audio route changes. ```swift class RTCManager: NSObject { private var rtcEngine: AgoraRtcEngineKit! private var audioDumpEnabled: Bool = false private var audioRouting = AgoraAudioOutputRouting.default // highlight-start // Set audio related parameters private func setAudioConfigParameters(routing: AgoraAudioOutputRouting) { audioRouting = routing rtcEngine.setParameters("{\"che.audio.aec.split_srate_for_48k\":16000}") rtcEngine.setParameters("{\"che.audio.sf.enabled\":true}") rtcEngine.setParameters("{\"che.audio.sf.stftType\":6}") rtcEngine.setParameters("{\"che.audio.sf.ainlpLowLatencyFlag\":1}") rtcEngine.setParameters("{\"che.audio.sf.ainsLowLatencyFlag\":1}") rtcEngine.setParameters("{\"che.audio.sf.procChainMode\":1}") rtcEngine.setParameters("{\"che.audio.sf.nlpDynamicMode\":1}") if routing == .headset || routing == .earpiece || routing == .headsetNoMic || routing == .bluetoothDeviceHfp || routing == .bluetoothDeviceA2dp { rtcEngine.setParameters("{\"che.audio.sf.nlpAlgRoute\":0}") } else { rtcEngine.setParameters("{\"che.audio.sf.nlpAlgRoute\":1}") } rtcEngine.setParameters("{\"che.audio.sf.ainlpModelPref\":10}") rtcEngine.setParameters("{\"che.audio.sf.nsngAlgRoute\":12}") rtcEngine.setParameters("{\"che.audio.sf.ainsModelPref\":10}") rtcEngine.setParameters("{\"che.audio.sf.nsngPredefAgg\":11}") rtcEngine.setParameters("{\"che.audio.agc.enable\":false}") } // highlight-end } extension RTCManager: RTCManagerProtocol { func createRtcEngine(delegate: AgoraRtcEngineDelegate) -> AgoraRtcEngineKit { let config = AgoraRtcEngineConfig() config.appId = AppContext.shared.appId config.channelProfile = .liveBroadcasting // highlight-start // Set the audio scene to AI dialogue scene (supported by 4.5.1 and above) // Versions 4.3.1 ~ 4.5.0 support chorus scenes .chorus config.audioScenario = .aiClient rtcEngine = AgoraRtcEngineKit.sharedEngine(with: config, delegate: delegate) // Register audio route change callback rtcEngine.addDelegate(self) // highlight-end return rtcEngine } func joinChannel(rtcToken: String, channelName: String, uid: String) { // highlight-start // Initialize audio configuration parameters setAudioConfigParameters(routing: audioRouting) // highlight-end // Configure channel options and join the channel let options = AgoraRtcChannelMediaOptions() options.clientRoleType = .broadcaster options.publishMicrophoneTrack = true options.publishCameraTrack = false options.autoSubscribeAudio = true options.autoSubscribeVideo = false let ret = rtcEngine.joinChannel(byToken: rtcToken, channelId: channelName, uid: UInt(uid) ?? 0, mediaOptions: options) } } // highlight-start // Implement the AgoraRtcEngineDelegate interface to handle audio route change callbacks extension RTCManager: AgoraRtcEngineDelegate { public func rtcEngine(_ engine: AgoraRtcEngineKit, didAudioRouteChanged routing: AgoraAudioOutputRouting) { setAudioConfigParameters(routing: routing) } } // highlight-end ``` Integrate the `agora-extension-ai-denoiser` extension. Refer to [AI Noise Suppression](/en/realtime-media/rtc/build/enhance-the-audio-experience/ai-noise-suppression/web). Info Use Web SDK version 4.15.0 or later. ## Reference This section contains content that completes the information on this page, or points you to documentation that explains other aspects to this product. ### Sample project Refer to the following open-source sample code to set audio-related parameters. Android iOS * [`CovRtcManager.kt`](https://github.com/AgoraIO-Community/Conversational-AI-Demo/blob/main/Android/scenes/convoai/src/main/java/io/agora/scene/convoai/rtc/CovRtcManager.kt) * [`RTCManager.swift`](https://github.com/AgoraIO-Community/Conversational-AI-Demo/blob/main/iOS/Scenes/ConvoAI/ConvoAI/ConvoAI/Classes/Manager/RTCManager.swift) ### Folder structure Android iOS Web * `IConversationalAIAPI.kt`: API interface and related data structures and enumerations * `ConversationalAIAPIImpl.kt`: ConversationalAI API main implementation logic * `ConversationalAIUtils.kt`: Tool functions and event callback management * `subRender/` * `v3/`: Transcript module * `TranscriptionController.kt`: Transcript Controller * `MessageParser.kt`: Message Parser * `ConversationalAIAPI.swift`: API interface and related data structures and enumerations * `ConversationalAIAPIImpl.swift`: ConversationalAI API main implementation logic * `Transcription/` * `TranscriptionController.swift`: Transcript Controller * `index.ts`: API Class * `type.ts`: API interface and related data structures and enumerations * `utils/` * `index.ts`: API utility functions * `events.ts`: Event management class, which can be extended to easily implement event monitoring and broadcasting * `sub-render.ts`: Transcript module ### API reference Android iOS * SDK * `setAudioScenario` * `setParameters` * `onAudioRouteChanged` * Toolkit * [`loadAudioSettings`](/en/api-reference/api-ref/conversational-ai/client-toolkit/android#loadaudiosettings) * [`destroy`](/en/api-reference/api-ref/conversational-ai/client-toolkit/android#destroy) * SDK * `setAudioScenario` * `setParameters` * `rtcEngine:didAudioRouteChanged:` * Toolkit * [`loadAudioSettings`](/en/api-reference/api-ref/conversational-ai/client-toolkit/ios#loadaudiosettings12) * [`destroy`](/en/api-reference/api-ref/conversational-ai/client-toolkit/ios#destroy) * Toolkit * [`IConversationalAIAPIEventHandlers interface`](/en/api-reference/api-ref/conversational-ai/client-toolkit/web#iconversationalaiapieventhandlers-interface) * [`EConversationalAIAPIEvents`](/en/api-reference/api-ref/conversational-ai/client-toolkit/web#econversationalaiapievents) * [`subscribeMessage`](/en/api-reference/api-ref/conversational-ai/client-toolkit/web#subscribemessage) * [`unsubscribeMessage`](/en/api-reference/api-ref/conversational-ai/client-toolkit/web#unsubscribe) * [`destroy`](/en/api-reference/api-ref/conversational-ai/client-toolkit/web#destroy) # Get agent state (/en/ai/best-practices/get-agent-state) If you need to track agent state changes in your client app, you can use either Signaling (RTM) Message or RTM Presence. This guide explains how to get agent state using RTM Message and describes [how the two approaches differ](#rtm-message-vs-rtm-presence). Info * To monitor state changes using client components, see [Listen to agent events](/en/ai/build/handle-runtime-events/webhooks). * To get agent state through RTM Presence, continue using your existing Presence implementation. ## Understand the tech The Conversational AI Engine pushes agent state change events to the client as individual RTM messages through the `onMessageEvent` callback. Unlike RTM Presence, which synchronizes state as key-value pairs, RTM Message wraps each state change in a separate event message. This makes it easier to record state change timestamps, process events sequentially, or consume state events together with other RTM Message events. The inner `message` field of each RTM message is a JSON string that requires a second parse to access the state event data. Use the `event_type` field in the inner message to identify the event type. The following agent states are pushed through RTM Message: * `state.listening` * `state.thinking` * `state.speaking` The flow works as follows: ```text Agent state change ├─ listening -> true / false ├─ thinking -> true / false └─ speaking -> true / false ↓ RTM Message ↓ Client parses inner payload and identifies event by event_type ``` ### State event types The Conversational AI Engine sends an RTM Message each time the agent state changes. Each message corresponds to a single state change event. | `event_type` | Description | | :---------------- | :-------------------------------------------------- | | `state.listening` | The agent starts or stops listening for user input. | | `state.thinking` | The agent starts or stops processing user input. | | `state.speaking` | The agent starts or stops speaking. | ### Message structure The inner `message` field is a JSON string. After deserialization, the structure is as follows: ```json { "event_id": "xxxx", "event_type": "state.listening", "event_ms": 1611566412672, "payload": { "value": true, "timestamp": 1611566412600 } } ``` | Field | Type | Description | | :------------------ | :------ | :-------------------------------------------------------------------------------------- | | `event_id` | string | Unique event ID. | | `event_type` | string | State event type. One of `state.listening`, `state.thinking`, or `state.speaking`. | | `event_ms` | integer | Timestamp when the event was sent, in milliseconds. | | `payload.value` | boolean | Current state value. `true` means the agent entered the state; `false` means it exited. | | `payload.timestamp` | integer | Timestamp when the state change actually occurred, in milliseconds. | ## Prerequisites Before you begin, ensure that you have: * Completed the basic agent integration. See [Quickstart](/en/ai/get-started/quickstart). * Integrated the RTM SDK and implemented message listener logic. See [RTM quickstart](https://docs.agora.io/en/signaling/get-started/sdk-quickstart). * Enabled the RTM service and set `data_channel` to `rtm` when starting the agent. ## Implementation ### Enable RTM when starting the agent When calling [Start a conversational AI agent](/en/api-reference/api-ref/conversational-ai/join), include the following configuration: ```json { "properties": { "advanced_features": { "enable_rtm": true }, "parameters": { "data_channel": "rtm" } } } ``` ### Listen for and parse state messages After logging in to RTM and subscribing to the channel with the same name as the agent, parse state messages in the RTM message callback. The handling logic consists of the following steps: 1. Deserialize the RTM `message` field into a state event object. 2. Check that the deserialized result contains `event_type` and `payload`. 3. Dispatch handling based on `event_type`, and use `payload.value` to update your app state. In the following examples: * The inner `message` field is a JSON string and must be deserialized again to obtain `event_type` and `payload`. * `event_type` identifies the specific state type. * `payload.value` indicates whether the agent is entering (`true`) or exiting (`false`) the state. * `payload.timestamp` can be used for precise ordering or logging. Android iOS Web ```kotlin val rtmConfig = RtmConfig() rtmConfig.eventListener = object : RtmEventListener { override fun onMessageEvent(event: MessageEvent) { // 1. Deserialize the inner message (JSON string) val payload = JSONObject(event.message.data) if (!payload.has("event_type") || !payload.has("payload")) { return } val eventType = payload.getString("event_type") val statePayload = payload.getJSONObject("payload") val value = statePayload.getBoolean("value") val timestamp = statePayload.getLong("timestamp") // 2. Dispatch based on event_type when (eventType) { "state.listening" -> handleListeningChanged(value, timestamp) "state.thinking" -> handleThinkingChanged(value, timestamp) "state.speaking" -> handleSpeakingChanged(value, timestamp) } } } mRtmClient = RtmClient.create(rtmConfig) ``` ```swift func rtmKit(_ rtmKit: AgoraRtmClientKit, didReceiveMessageEvent event: AgoraRtmMessageEvent) { // 1. Deserialize the inner message (JSON string) guard let jsonString = event.message.stringData, let data = jsonString.data(using: .utf8), let payload = try? JSONSerialization.jsonObject(with: data) as? [String: Any], let eventType = payload["event_type"] as? String, let statePayload = payload["payload"] as? [String: Any] else { return } let value = statePayload["value"] as? Bool ?? false let timestamp = statePayload["timestamp"] as? Int ?? 0 // 2. Dispatch based on event_type switch eventType { case "state.listening": handleListeningChanged(value, timestamp) case "state.thinking": handleThinkingChanged(value, timestamp) case "state.speaking": handleSpeakingChanged(value, timestamp) default: break } } ``` ```typescript rtm.addEventListener("message", (event) => { // 1. Deserialize the inner message (JSON string) const payload = JSON.parse(event.message as string); if (!payload.event_type || !payload.payload) { return; } const { event_type, payload: statePayload } = payload; // 2. Dispatch based on event_type switch (event_type) { case "state.listening": handleListeningChanged(statePayload.value, statePayload.timestamp); break; case "state.thinking": handleThinkingChanged(statePayload.value, statePayload.timestamp); break; case "state.speaking": handleSpeakingChanged(statePayload.value, statePayload.timestamp); break; } }); ``` ### Update app state Use the received events to update your UI or internal state machine. For example: * When `state.listening` with `payload.value = true` is received, display "Listening". * When `state.thinking` with `payload.value = true` is received, display "Thinking". * When `state.speaking` with `payload.value = true` is received, display "Speaking". ## RTM Message vs RTM Presence The Conversational AI Engine supports getting agent state through both RTM Message and RTM Presence. The key differences are: | | RTM Presence | RTM Message | | :---------- | :------------------------------- | :------------------------------------------------------- | | Return form | State key-value pairs | State event messages | | Granularity | Better for reading current state | Better for handling each state change | | Timing info | Relies on message arrival time | Includes explicit `event_ms` and `payload.timestamp` | | Use case | Simple UI state display | Event-driven logic, logging, unified message consumption | Choose based on your needs: * If you only need to update the UI based on whether the agent is currently `listening`, `thinking`, or `speaking`, RTM Presence is simpler to integrate. * If you want to process state changes together with captions, interruptions, or performance metrics in a unified RTM Message handler, use RTM Message. * If you need to record the timestamp of each state change or analyze state transitions in sequence, use RTM Message. ## Considerations Keep the following in mind when implementing this feature: * **Double deserialization**: The inner `message` field is a JSON string and must be deserialized a second time to access `event_type` and `payload`. * **Avoid duplicate UI updates**: If you are already driving UI state through RTM Presence, avoid having both implementations update the same UI state simultaneously, as this can cause redundant updates or state flickering. * **`state.speaking` end timing**: A `state.speaking` value of `false` does not guarantee that the client has finished playing audio. The server estimates the speaking end time based on TTS audio length, so there may be a delay between the state switching to `false` and the client completing playback. * **Scope of state messages**: State messages cover only `listening`, `thinking`, and `speaking` agent state events. Manual SoS and EoS signals and their callbacks are not included. # Manually control start and end of speech (/en/ai/best-practices/manual-turn-control) When your business scenario isn't a good fit for letting the server automatically determine when a user starts or stops speaking, use manual Start of Speech (SoS) and End of Speech (EoS) to have the client explicitly declare the user's turn boundaries. This is useful for AI interviews, interactive quizzes, walkie-talkie style push-to-talk, and other "press to start, tap to submit" interactions. This guide explains how to implement manual control over user speech boundaries by configuring the agent and using the client toolkit. ## Understand the tech Manual turn control consists of two independent capabilities: * **Manual SoS**: The client explicitly declares that the user has started speaking. * **Manual EoS**: The client explicitly declares that the user has finished speaking. `start_of_speech.mode` and `end_of_speech.mode` are independent settings. Choose which boundary needs manual control based on your use case: | Business scenario | `start_of_speech.mode` | `end_of_speech.mode` | Client call | | --------------------------------------------------------------------------------------------- | ---------------------- | -------------------- | ---------------------------------------- | | The user can start speaking freely, but must tap "submit answer" | `vad` | `manual` | Call only `manualEOS` | | The user taps or holds a button to start speaking, and the server determines when speech ends | `manual` | `vad` | Call only `manualSOS` | | Press-and-hold to talk, release to send (walkie-talkie mode) | `manual` | `manual` | Call `manualSOS` first, then `manualEOS` | * If only `end_of_speech.mode` is `manual`, the server still uses VAD to automatically detect when the user starts speaking. Your app only needs to call `manualEOS` when the user taps submit. * If only `start_of_speech.mode` is `manual`, the server starts accepting audio for the current turn only after your app calls `manualSOS`. End of speech is still detected automatically by VAD, so your app does not need to call `manualEOS`. * If both `start_of_speech.mode` and `end_of_speech.mode` are `manual`, the server starts accepting audio for the current turn only after your app calls `manualSOS`, and submits the turn only after your app calls `manualEOS`. ## Prerequisites Before you begin, ensure that you have: * Completed the basic steps for interacting with an agent. See [Quickstart](/en/ai/get-started/quickstart). * Integrated Video SDK v4.5.1 or later and followed the [Quickstart](../../realtime-media/video/quickstart) guide to implement basic real-time audio and video features. * Enabled Signaling for your project in the Agora Console and followed the [Signaling Quickstart](/en/realtime-media/rtm/quickstart) to implement real-time messaging. * Made sure the RTC engine instance is initialized and the app is logged in to Signaling. The client toolkit does not handle initialization, lifecycle management, authentication, or login for Video SDK or Signaling. ## Implement manual control Manual turn control requires configuring both the agent and the client toolkit: 1. Configure manual mode when you start the agent. 2. Initialize the client toolkit and subscribe to channel messages. 3. Register handlers for the manual SoS/EoS result events. 4. Call `manualSOS`/`manualEOS` when the user taps start or submit. ### Enable manual turn control when starting the agent When calling [Start a conversational AI agent](/en/api-reference/api-ref/conversational-ai/join), use the following parameter settings: * `advanced_features.enable_rtm: true`: Start Signaling (Required) * `parameters.data_channel: "rtm"`: Enable the RTM data transmission channel (Required) * `parameters.enable_metrics: true`: Receive agent performance data (Enable on demand) * `parameters.enable_error_message: true`: Receive agent error events (Enable on demand) Then set the boundary or boundaries you want to control manually to `"manual"`. The following `curl` examples show three common configurations: Manual EoS only Manual SoS only Manual SoS + EoS If you only need to manually control end of speech, keep automatic SoS and set only `end_of_speech.mode` to `manual`. ```shell curl --request POST \ --url https://api.agora.io/api/conversational-ai-agent/v2/projects//join \ --header 'Authorization: agora token="007abcxxxxxxx123"' \ --data ' { "name": "manual-eos-agent", "properties": { "channel": "channel_name", "token": "token", "agent_rtc_uid": "0", "remote_rtc_uids": [ "123" ], "advanced_features": { "enable_rtm": true }, "asr": { "language": "en-US" }, "llm": { "url": "https://api.xxxx/v1/xxxx", "api_key": "xxx", "system_messages": [ { "role": "system", "content": "You are a helpful chatbot." } ], "greeting_message": "Hello, how can I help you?", "failure_message": "Sorry, I am unable to answer that question.", "max_history": 10, "params": { "model": "xxxx" } }, "tts": { "vendor": "minimax", "params": { "key": "your-minimax-key", "model": "speech-01-turbo", "voice_setting": { "voice_id": "female-shaonv", "speed": 1, "vol": 1, "pitch": 0, "emotion": "happy" }, "audio_setting": { "sample_rate": 16000 } } }, "turn_detection": { "mode": "default", "config": { "start_of_speech": { "mode": "vad", "vad_config": { "interrupt_duration_ms": 160, "speaking_interrupt_duration_ms": 320, "prefix_padding_ms": 800 } }, "end_of_speech": { "mode": "manual" } } }, "parameters": { "data_channel": "rtm" } } } ' ``` If you only need to manually control start of speech, set `start_of_speech.mode` to `manual` and keep automatic EoS. ```shell curl --request POST \ --url https://api.agora.io/api/conversational-ai-agent/v2/projects//join \ --header 'Authorization: agora token="007abcxxxxxxx123"' \ --data ' { "name": "manual-sos-agent", "properties": { "channel": "channel_name", "token": "token", "agent_rtc_uid": "0", "remote_rtc_uids": [ "123" ], "advanced_features": { "enable_rtm": true }, "asr": { "language": "en-US" }, "llm": { "url": "https://api.xxxx/v1/xxxx", "api_key": "xxx", "system_messages": [ { "role": "system", "content": "You are a helpful chatbot." } ], "greeting_message": "Hello, how can I help you?", "failure_message": "Sorry, I am unable to answer that question.", "max_history": 10, "params": { "model": "xxxx" } }, "tts": { "vendor": "minimax", "params": { "key": "your-minimax-key", "model": "speech-01-turbo", "voice_setting": { "voice_id": "female-shaonv", "speed": 1, "vol": 1, "pitch": 0, "emotion": "happy" }, "audio_setting": { "sample_rate": 16000 } } }, "turn_detection": { "mode": "default", "config": { "start_of_speech": { "mode": "manual" }, "end_of_speech": { "mode": "vad", "vad_config": { "silence_duration_ms": 480 } } } }, "parameters": { "data_channel": "rtm" } } } ' ``` If you need a complete push-to-talk experience, set both `start_of_speech.mode` and `end_of_speech.mode` to `manual`. ```shell curl --request POST \ --url https://api.agora.io/api/conversational-ai-agent/v2/projects//join \ --header 'Authorization: agora token="007abcxxxxxxx123"' \ --data ' { "name": "manual-sos-eos-agent", "properties": { "channel": "channel_name", "token": "token", "agent_rtc_uid": "0", "remote_rtc_uids": [ "123" ], "advanced_features": { "enable_rtm": true }, "asr": { "language": "en-US" }, "llm": { "url": "https://api.xxxx/v1/xxxx", "api_key": "xxx", "system_messages": [ { "role": "system", "content": "You are a helpful chatbot." } ], "greeting_message": "Hello, how can I help you?", "failure_message": "Sorry, I am unable to answer that question.", "max_history": 10, "params": { "model": "xxxx" } }, "tts": { "vendor": "minimax", "params": { "key": "your-minimax-key", "model": "speech-01-turbo", "voice_setting": { "voice_id": "female-shaonv", "speed": 1, "vol": 1, "pitch": 0, "emotion": "happy" }, "audio_setting": { "sample_rate": 16000 } } }, "turn_detection": { "mode": "default", "config": { "start_of_speech": { "mode": "manual" }, "end_of_speech": { "mode": "manual" } } }, "parameters": { "data_channel": "rtm" } } } ' ``` After a successful response, the agent joins the specified RTC channel and is ready to interact with the user. ### Integrate and initialize the client toolkit Android iOS Web 1. Add the toolkit to your project using Maven or source code. See [Install the Android toolkit](/en/api-reference/api-ref/conversational-ai/client-toolkit/android#installation). Refer to the [component structure](#component-structure) to understand the role of each file. 2. Create a configuration object for the RTC engine and Signaling client instances, then use it to initialize the component instance. ```kotlin val config = ConversationalAIAPIConfig( rtcEngine = rtcEngine, rtmClient = rtmClient, renderMode = TranscriptRenderMode.Word, enableLog = true, enableRenderModeFallback = true ) val api = ConversationalAIAPIImpl(config) ``` 3. Register a handler for the manual SoS/EoS result events, then subscribe to the agent's channel. ```kotlin api.addHandler(object : IConversationalAIAPIEventHandler { override fun onUserManualSosEvent(agentUserId: String, event: UserManualSosEvent) { // Handle the manual SoS result. event.payload.success indicates whether the server accepted it. } override fun onUserManualEosEvent(agentUserId: String, event: UserManualEosEvent) { // Handle the manual EoS result. } override fun onAgentManualEosEvent(agentUserId: String, event: AgentManualEosEvent) { // Handle the server-triggered automatic EoS notification. } }) api.subscribeMessage("channelName") { error -> if (error != null) { // Handle subscription failure. } } ``` 1. Add the toolkit to your project using CocoaPods, Swift Package Manager, or source code. See [Install the iOS toolkit](/en/api-reference/api-ref/conversational-ai/client-toolkit/ios#installation). Refer to the [component structure](#component-structure) to understand the role of each file. 2. Create a configuration object for the RTC engine and Signaling client instances, then use it to initialize the component instance. ```swift let config = ConversationalAIAPIConfig( rtcEngine: rtcEngine, rtmEngine: rtmEngine, renderMode: .words, enableLog: true, enableRenderModeFallback: true ) convoAIAPI = ConversationalAIAPIImpl(config: config) ``` 3. Register a handler for the manual SoS/EoS result events, then subscribe to the agent's channel. ```swift convoAIAPI.addHandler(handler: self) func onUserManualSosEvent(agentUserId: String, event: UserManualSosEvent) { // Handle the manual SoS result. event.payload.success indicates whether the server accepted it. } func onUserManualEosEvent(agentUserId: String, event: UserManualEosEvent) { // Handle the manual EoS result. } func onAgentManualEosEvent(agentUserId: String, event: AgentManualEosEvent) { // Handle the server-triggered automatic EoS notification. } convoAIAPI.subscribeMessage(channelName: channelName) { error in if let error = error { // Handle subscription failure. } } ``` 1. Add the toolkit to your project using a package manager or source code. See [Install the Web toolkit](/en/api-reference/api-ref/conversational-ai/client-toolkit/web#installation). Refer to the [component structure](#component-structure) to understand the role of each file. 2. Create a configuration object for the RTC engine and Signaling client instances, then use it to initialize the component instance. ```ts const config: IConversationalAIAPIConfig = { rtcEngine, rtmEngine, renderMode: ETranscriptHelperMode.WORD, enableLog: true, enableRenderModeFallback: true, } const conversationalAIAPI = await ConversationalAIAPI.init(config) ``` 3. Register a handler for the manual SoS/EoS result events, then subscribe to the agent's channel. ```ts conversationalAIAPI.on(EConversationalAIAPIEvents.USER_MANUAL_SOS_RESULT, (agentUserId, event) => { // Handle the manual SoS result. event.payload.success indicates whether the server accepted it. }) conversationalAIAPI.on(EConversationalAIAPIEvents.USER_MANUAL_EOS_RESULT, (agentUserId, event) => { // Handle the manual EoS result. }) conversationalAIAPI.on(EConversationalAIAPIEvents.AGENT_MANUAL_EOS_RESULT, (agentUserId, event) => { // Handle the server-triggered automatic EoS notification. }) conversationalAIAPI.subscribeMessage(channelName) ``` Info These handlers receive the server's processing result for your manual SoS/EoS requests. Use the `event`, `requestId`, and error information to implement your own business logic. See the [client toolkit API reference](#toolkit) for details. ### Send manual SoS when the user starts speaking Call `manualSOS` only when `start_of_speech.mode` is `"manual"`. If you use automatic SoS through VAD, skip this step. Android iOS Web ```kotlin api.manualSOS(agentUserId = "agentUserId") { requestId, error -> if (error != null) { Log.e("ManualSOS", "Send failed: ${error.errorMessage}, requestId=$requestId") } else { Log.i("ManualSOS", "SoS sent, requestId=$requestId") } } ``` ```swift convoAIAPI.manualSOS(agentUserId: agentUid) { requestId, error in if let error = error { print("Send failed: \(error.message), requestId=\(requestId)") } else { print("SoS sent, requestId=\(requestId)") } } ``` ```ts const requestId = await conversationalAIAPI.manualSOS(agentUserId) ``` Once manual SoS takes effect, the server starts counting subsequent audio toward the current user turn. Audio that arrives before this point is not counted. ### Send manual EoS when the user finishes speaking Call `manualEOS` only when `end_of_speech.mode` is `"manual"`. If you use automatic EoS through VAD, skip this step. Android iOS Web ```kotlin api.manualEOS(agentUserId = "agentUserId") { requestId, error -> if (error != null) { Log.e("ManualEOS", "Send failed: ${error.errorMessage}, requestId=$requestId") } else { Log.i("ManualEOS", "EoS sent, requestId=$requestId") } } ``` ```swift convoAIAPI.manualEOS(agentUserId: agentUid) { requestId, error in if let error = error { print("Send failed: \(error.message), requestId=\(requestId)") } else { print("EoS sent, requestId=\(requestId)") } } ``` ```ts const requestId = await conversationalAIAPI.manualEOS(agentUserId) ``` Once manual EoS takes effect, the agent submits the turn for ASR and LLM inference as usual. This signal only marks the end of the user's current speech segment — it does not immediately end the entire turn. ### Destroy the component instance When the agent interaction ends, destroy the component instance to release all resources. Android iOS Web ```kotlin api.destroy() ``` ```swift convoAIAPI.destroy() ``` ```ts conversationalAIAPI.destroy() ``` ## Reference ### API reference #### RESTful API * [Start a conversational AI agent](/en/api-reference/api-ref/conversational-ai/join) #### Toolkit Android iOS Web * [`addHandler`](/en/api-reference/api-ref/conversational-ai/client-toolkit/android#addhandler) * [`subscribeMessage`](/en/api-reference/api-ref/conversational-ai/client-toolkit/android#subscribemessage) * [`manualSOS`](/en/api-reference/api-ref/conversational-ai/client-toolkit/android#manualsos) * [`manualEOS`](/en/api-reference/api-ref/conversational-ai/client-toolkit/android#manualeos) * [`destroy`](/en/api-reference/api-ref/conversational-ai/client-toolkit/android#destroy) * [`onUserManualSosEvent`](/en/api-reference/api-ref/conversational-ai/client-toolkit/android#onusermanualsosevent) * [`onUserManualEosEvent`](/en/api-reference/api-ref/conversational-ai/client-toolkit/android#onusermanualeosevent) * [`onAgentManualEosEvent`](/en/api-reference/api-ref/conversational-ai/client-toolkit/android#onagentmanualeosevent) * [`addHandler`](/en/api-reference/api-ref/conversational-ai/client-toolkit/ios#addhandler) * [`subscribeMessage`](/en/api-reference/api-ref/conversational-ai/client-toolkit/ios#subscribemessage) * [`manualSOS`](/en/api-reference/api-ref/conversational-ai/client-toolkit/ios#manualsos) * [`manualEOS`](/en/api-reference/api-ref/conversational-ai/client-toolkit/ios#manualeos) * [`destroy`](/en/api-reference/api-ref/conversational-ai/client-toolkit/ios#destroy) * [`onUserManualSosEvent`](/en/api-reference/api-ref/conversational-ai/client-toolkit/ios#onusermanualsosevent) * [`onUserManualEosEvent`](/en/api-reference/api-ref/conversational-ai/client-toolkit/ios#onusermanualeosevent) * [`onAgentManualEosEvent`](/en/api-reference/api-ref/conversational-ai/client-toolkit/ios#onagentmanualeosevent) * [`subscribeMessage`](/en/api-reference/api-ref/conversational-ai/client-toolkit/web#subscribemessage) * [`manualSOS`](/en/api-reference/api-ref/conversational-ai/client-toolkit/web#manualsos) * [`manualEOS`](/en/api-reference/api-ref/conversational-ai/client-toolkit/web#manualeos) * [`destroy`](/en/api-reference/api-ref/conversational-ai/client-toolkit/web#destroy) * [`EConversationalAIAPIEvents`](/en/api-reference/api-ref/conversational-ai/client-toolkit/web#econversationalaiapievents) # Optimize latency (/en/ai/best-practices/optimize-latency) Latency is a key factor affecting user experience in conversational AI agent scenarios. This guide helps you understand and optimize conversation latency to improve user experience. ## How latency works Understanding latency composition helps you identify optimization opportunities. ### Cascaded architecture latency When using the cascaded architecture with ASR, LLM, and TTS components, end-to-end latency consists of the following: ```text End-to-end latency = RTC latency + Algorithm preprocessing latency + ASR latency + LLM latency + TTS latency (+ Avatar latency) ``` ### Latency for each component The following table shows typical latency ranges for each component based on actual test data: | Component | Latency metric | Description | Typical latency range (ms) | | ----------------------- | ----------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------- | | RTC | Audio and video latency | Includes audio capture, encoding, network transmission, decoding, and playback | 150-300 | | Algorithm preprocessing | Preprocessing latency | Includes VAD (Voice Activity Detection), intelligent interruption handling (AIVAD), and other algorithm processing time | 720-940\* | | ASR | `asr_ttlw` | Time To Last Word. The latency from when the user stops speaking to when ASR outputs the last word. | 400-700 | | LLM | `llm_ttfb` / `llm_ttfs` | TTFB: Time To First Byte, the first byte latency.
TTFS: Time To First Sentence, the first sentence latency. | 250-1000 | | TTS | `tts_ttfb` | Time To First Byte. The response latency from when the TTS request starts to when the first byte is received. | 100-350 | | Avatar rendering | Rendering latency | The latency from when the avatar module receives the first frame of TTS audio to when it generates and synchronizes the first frame of audio and video (if enabled) | 50-200 | \* Algorithm preprocessing latency is based on `silence_duration_ms` set to 640 ms. Adjusting this parameter affects preprocessing latency. Info Test data shows that LLM typically contributes the most to overall latency. Optimizing LLM selection and configuration is key to reducing end-to-end latency. ### Real-world latency example The following example shows latency data from three conversation turns in an actual conversation. This data comes from the [`111 agent metrics`](../reference/event-types#111-agent-metrics) event in Agora's [message notification service](../build/handle-runtime-events/webhooks): ```json { "metrics": [ { "turn_id": 1, "tts_ttfb": 61 }, { "turn_id": 2, "asr_ttlw": 141, "llm_ttfb": 270, "llm_ttfs": 482, "tts_ttfb": 90 }, { "turn_id": 3, "asr_ttlw": 103, "llm_ttfb": 306, "llm_ttfs": 948, "tts_ttfb": 106 } ] } ``` {/* ### MLLM architecture latency When using MLLM (Multimodal Large Language Model), the model processes audio input directly and generates audio output without separate ASR, LLM, and TTS components. This results in simpler latency composition: ```text End-to-end latency = RTC latency + Algorithm preprocessing latency + MLLM latency ``` You can obtain the `mllm_ttfb` metric by listening to the `111 agent metrics` event. This metric represents the response latency from when the MLLM request starts to when the first byte is received. :::info[Info] * MLLM architecture typically achieves lower end-to-end latency by reducing data conversion and transmission between components. * When MLLM is enabled, the graceful interruption (AIVAD) feature is disabled, which reduces algorithm preprocessing latency. ::: */} ## Monitor latency metrics The Conversational AI engine provides two ways to monitor latency metrics for each conversation turn: ### Use client components If you use client components (Android, iOS, or Web), you can listen to agent performance metrics in real time by registering the `onAgentMetrics` callback. Android iOS Web ```kotlin api.addHandler(object : IConversationalAIAPIEventHandler { override fun onAgentMetrics(agentUserId: String, metric: Metric) { when (metric.type) { ModuleType.ASR -> { Log.d("Metrics", "ASR TTLW: \${metric.value}ms") } ModuleType.LLM -> { // metric.name can be "ttfb" or "ttfs" Log.d("Metrics", "LLM \${metric.name}: \${metric.value}ms") } ModuleType.TTS -> { Log.d("Metrics", "TTS TTFB: \${metric.value}ms") } ModuleType.TOTAL -> { Log.d("Metrics", "Total Delay: \${metric.value}ms") } else -> { Log.d("Metrics", "\${metric.type}: \${metric.name} = \${metric.value}ms") } } } }) ``` ```swift func onAgentMetrics(agentUserId: String, metrics: Metric) { switch metrics.type { case .asr: print("ASR TTLW: \(metrics.value)ms") case .llm: print("LLM \(metrics.name): \(metrics.value)ms") case .tts: print("TTS TTFB: \(metrics.value)ms") case .total: print("Total Delay: \(metrics.value)ms") case .unknown: print("Unknown metric: \(metrics.name) = \(metrics.value)ms") } } ``` ```ts conversationalAIAPI.on( EConversationalAIAPIEvents.AGENT_METRICS, (agentUserId: string, metrics: Metric) => { console.log(`[\${metrics.type}] \${metrics.name}: \${metrics.value}ms`); if (metrics.type === 'TOTAL') { console.log(`Total delay for turn: \${metrics.value}ms`); } } ); ``` For detailed integration steps and API reference, see [Webhooks](../build/handle-runtime-events/webhooks). ### Use Message Notification Service If you have enabled Agora's Message Notification Service, you can obtain agent performance metrics by receiving the `agent metrics` event where `eventType` is `111`. **Event callback example** ```json { "noticeId": "2000001428:4330:107", "productId": 17, "eventType": 111, "notifyMs": 1611566412672, "payload": { "agent_id": "A42AC47Hxxxxxxxx4PK27ND25E", "start_ts": 1000, "stop_ts": 1672531200, "channel": "test-channel", "metrics": [ { "turn_id": 1, "tts_ttfb": 61 }, { "turn_id": 2, "asr_ttlw": 141, "llm_ttfb": 270, "llm_ttfs": 482, "tts_ttfb": 90, } ] } } ``` For detailed event field descriptions, see [Notification event 111 agent metrics](../reference/event-types#111-agent-metrics). ## Optimize cascaded architecture latency To reduce latency in cascaded architecture, focus on optimizing individual components, geographic deployment, and RTC settings. ### Optimize LLM, ASR, and TTS components LLM is typically the component that contributes the most to latency. Optimizing LLM can significantly reduce overall latency. **Choose low-latency vendors** LLM, ASR, and TTS vendors and models vary significantly in response speed. Refer to the [Conversational AI Performance Lab](https://www.agora.io/en/conversational-ai-performance-lab) to compare performance metrics across vendors. **Optimize parameter configuration** When creating an agent, read the vendor documentation for ASR, LLM, and TTS to understand available parameters and tune them for your use case. The following are general optimization approaches: * **LLM** * **Choose smaller models**: Models such as `gpt-4o-mini` and `gemini-2.5-flash` typically respond faster than larger models. * **Limit `max_tokens`**: Reducing the maximum number of tokens generated can lower TTFS (first sentence latency). * **Enable streaming response**: Ensure `stream: true` so the agent can start speaking as soon as possible. * **ASR** * **Use vendor-recommended sampling rate**: Use the sampling rate recommended by the vendor, such as 16 kHz, to avoid unnecessary resampling. * **Limit language model**: Use the `phrases` or `context` parameter to provide domain-specific vocabulary and improve recognition speed. * **Disable non-essential features**: Some ASR vendors provide advanced parameters such as punctuation and tone output. You can disable these based on your use case to improve response speed. * **TTS** * **Choose faster modes**: Some TTS providers offer modes such as turbo or low-latency, which typically respond faster than default mode. * **Choose simpler voices**: Some TTS providers offer voices with varying complexity. Choosing less complex voices can reduce generation time. * **Disable non-essential features**: Some TTS vendors provide advanced parameters such as profanity\_filter, punctuation\_filter, and diarization. You can disable these based on your use case to improve response speed. The following example shows how to optimize LLM parameters: ```json { "properties": { "llm": { "url": "https://api.openai.com/v1/chat/completions", "api_key": "your_api_key", "params": { "model": "gpt-4o-mini", // Select the faster-responding model "temperature": 0.7, "max_tokens": 150, // Limit the generation length to reduce latency "stream": true // Enable streaming response } } } } ``` {/* ### Optimize geographic deployment **Understand the default location selection strategy** When regional access restrictions are not enabled or when using the `GLOBAL` region, the Conversational AI engine automatically selects the nearest server to deploy the agent based on the IP address of the configured LLM URL. The system includes the following capabilities: 1. **Smart deployment**: The system deploys the Conversational AI engine service in the region that corresponds to the IP address of the LLM URL. 2. **Proximity selection**: If no servers are available in the corresponding region, the system selects the nearest available region. 3. **Failover**: When the service in a region becomes unavailable, the system switches to another available region. **Optimize deployment region** To minimize network latency, Agora recommends: 1. **Deploy ASR, LLM, and TTS in the same region**: Choose vendors that support the same region to reduce cross-region network latency. 2. **Use regional access restrictions**: If your users are concentrated in a specific region, use the `geofence` parameter to restrict the Conversational AI engine to servers in that region: ```json { "properties": { "geofence": { "area": "NORTH_AMERICA" // Or EUROPE, ASIA, etc. }, "llm": { "url": "https://api.openai.com/v1/chat/completions" // Use LLM endpoint in North America region }, "tts": { "url": "wss://api.elevenlabs.io/v1/text-to-speech/{voice_id}/stream-input" // Use TTS endpoint in North America region } } } ``` For detailed regional access restriction configuration, see [Restrict agent zones](regional-restrictions). */} ### Optimize RTC latency RTC latency includes audio capture, encoding, network transmission, decoding, and playback. To optimize RTC latency, configure audio settings on the client side. **Use AI conversation scenario** Agora RTC SDK 4.5.1 and later supports the AI conversation scenario (`AUDIO_SCENARIO_AI_CLIENT`), which is specifically optimized for AI conversations and includes: * Optimized audio 3A algorithms (echo cancellation, noise reduction, and gain control) * Lower audio capture and playback latency * Audio processing tailored to AI voice characteristics Android iOS **Use the client-side component API (recommended)** ```kotlin val config = ConversationalAIAPIConfig( rtcEngine = rtcEngine, rtmClient = rtmClient, renderMode = TranscriptRenderMode.Word, enableLog = true, enableRenderModeFallback = true ) val api = ConversationalAIAPIImpl(config) // Load optimal audio settings api.loadAudioSettings() ``` **Configure the RTC SDK directly** ```kotlin val config = RtcEngineConfig() config.mAudioScenario = Constants.AUDIO_SCENARIO_AI_CLIENT rtcEngine = RtcEngine.create(config) ``` **Use the client-side component API (recommended)** ```swift let config = ConversationalAIAPIConfig( rtcEngine: rtcEngine, rtmEngine: rtmEngine, renderMode: .words, enableLog: true, enableRenderModeFallback: true ) convoAIAPI = ConversationalAIAPIImpl(config: config) // Load optimal audio settings convoAIAPI.loadAudioSettings() ``` **Configure the RTC SDK directly** ```swift let config = AgoraRtcEngineConfig() config.audioScenario = .aiClient rtcEngine = AgoraRtcEngineKit.sharedEngine(with: config, delegate: delegate) ``` For detailed audio setting optimization, see [Optimize audio](audio-setup). {/* ## Optimize MLLM architecture latency If you use an MLLM (Multimodal Large Language Model) architecture, optimizing latency primarily relies on choosing low-latency MLLM vendors. MLLM vendors vary in real-time performance. The Conversational AI engine currently supports: * OpenAI Realtime API: Provides low-latency real-time audio processing capabilities * Google Gemini Live: Supports multimodal real-time interaction For detailed MLLM integration guides, see the MLLM provider pages under `AI > Models > MLLM`. */} ## Latency optimization checklist Use the following checklist to systematically optimize your conversational AI agent latency: ### Server-side optimization * **Choose low-latency LLM models**: Refer to [Conversational AI Performance Lab](https://www.agora.io/en/conversational-ai-performance-lab) to select models with excellent TTFT and throughput performance. * **Enable streaming response**: Ensure `stream: true`. * **Choose low-latency ASR and TTS vendors**: Enable low-latency or turbo modes when available. Refer to [Conversational AI Performance Lab](https://www.agora.io/en/conversational-ai-performance-lab) to select models with low latency and high throughput performance. * **Optimize geographic deployment**: Deploy ASR, LLM, and TTS in the same region. ### Client-side optimization * **Use AI conversation scenario**: Set `AUDIO_SCENARIO_AI_CLIENT` (RTC SDK 4.5.1 and later). * **Load optimal audio settings**: Call the `loadAudioSettings()` method of the client component. * **Integrate required audio plugins**: Ensure integration of AI noise reduction and AI echo cancellation plugins. * **Optimize network conditions**: Ensure stable user network connection and consider using SD-RTN™ to optimize network transmission. ### Monitoring and analysis * **Monitor latency metrics in real time**: Obtain latency data for each conversation turn through client components or NCS. * **Identify latency bottlenecks**: Analyze which component contributes the most to latency. * **Continuously optimize**: Adjust configuration based on actual data and conduct A/B testing. ## Balance latency and quality When optimizing latency, find a balance between response speed and conversation quality: | Optimization strategy | Latency impact | Quality impact | Recommended scenario | | ---------------------------- | ----------------------- | -------------------------- | ---------------------------------------------------------------- | | Use smaller LLM models | ✅ Significantly reduces | ⚠️ May reduce | Latency-sensitive scenarios with relatively simple conversations | | Limit `max_tokens` | ✅ Moderately reduces | ⚠️ May affect completeness | Scenarios requiring short responses | | Regional access restrictions | ✅ Moderately reduces | No impact | Users concentrated in a specific region | | Optimize RTC settings | ✅ Moderately reduces | No impact | All scenarios | Tip Monitor latency metrics to identify bottlenecks, then optimize the highest-contributing component rather than pursuing minimal latency across all components. ## References Refer to the following resources for further details. * [Webhooks](../build/handle-runtime-events/webhooks) * [Optimize audio](audio-setup) * [Notification event types](../reference/event-types) * [Conversational AI Performance Lab](https://www.agora.io/en/conversational-ai-performance-lab) {/* * MLLM provider pages under `AI > Models > MLLM` */} # Record agent conversations (/en/ai/best-practices/record-agent-conversation) This guide explains how to configure Agora [Cloud Recording](/en/api-reference/api-ref/cloud-recording) for Voice Agent conversations and avoid audio-scene conflicts when recording starts. ## Why this needs special handling When you run a Voice Agent and Cloud Recording in the same channel, recording startup can briefly interrupt audio. The issue is usually caused by the agent audio scene falling back to a default mode when Cloud Recording joins the channel. To keep audio stable, set the agent `audio_scenario` to `"chorus"` when you start the agent. ## Configure the agent Set the agent's RTC audio scenario to `chorus` when you configure the agent: Python SDK TypeScript SDK Go SDK REST API ```python from agora_agent import Agent # client is your configured Agora client agent = ( Agent(client) .with_stt(...) # configure your STT vendor .with_llm(...) # configure your LLM vendor .with_tts(...) # configure your TTS vendor .with_audio_scenario('chorus') ) ``` ```typescript import { Agent } from 'agora-agents'; // client is your configured Agora client const agent = new Agent({ client }) .withStt(/* configure your STT vendor */) .withLlm(/* configure your LLM vendor */) .withTts(/* configure your TTS vendor */) .withAudioScenario('chorus'); ``` ```go import "github.com/AgoraIO/agora-agents-go/v2/agentkit" // client is your configured *agentkit.AgoraClient agent := agentkit.NewAgent(client). WithStt(/* configure your STT vendor */). WithLlm(/* configure your LLM vendor */). WithTts(/* configure your TTS vendor */). WithAudioScenario(agentkit.AudioScenarioChorus) ``` Add or modify the `audio_scenario` field in the agent startup parameters: ```json { "properties": { "parameters": { "audio_scenario": "chorus" } } } ``` ## Complete example The following example shows an agent launch request with the recommended recording-safe configuration: ```json { "name": "TestConvoAgent", "properties": { "channel": "{{AccessChannel}}", "token": "{{token}}", "agent_rtc_uid": "0", "remote_rtc_uids": [ "*" ], "idle_timeout": 30, "advanced_features": { "enable_bhvs": true }, "llm": { "url": "https://api.openai.com/v1/chat/completions", "api_key": "", "system_messages": [ { "role": "system", "content": "You are a helpful chatbot." } ], "greeting_message": "Hello, how can I help you?", "failure_message": "Sorry, I don't know how to answer this question.", "max_history": 10, "params": { "model": "gpt-4o-mini" } }, "tts": { "vendor": "microsoft", "params": { "key": "", "region": "eastus", "voice_name": "en-US-AndrewMultilingualNeural" } }, "turn_detection": { "silence_duration_ms": 640 }, "parameters": { "enable_dump": true, "enable_error_message": true, "audio_scenario": "chorus", "enable_delay": true } } } ``` ## Start recording After the agent is running, call the [Start Cloud Recording API](/en/api-reference/api-ref/cloud-recording/start) to begin recording. Example Cloud Recording request body: ```json { "cname": "{{AccessChannel}}", "uid": "{{RecordingUID}}", "clientRequest": { "token": "{{token}}", "recordingConfig": { "maxIdleTime": 120, "streamTypes": 0, "audioProfile": 1, "channelType": 1 }, "recordingFileConfig": { "avFileType": [ "hls", "mp4" ] }, "storageConfig": { "vendor": "{{Vendor}}", "region": "{{Region}}", "bucket": "{{Bucket}}", "accessKey": "{{AccessKey}}", "secretKey": "{{SecretKey}}" } } } ``` ## Related pages * [Start and stop an agent](../build/start-stop-agent) * [Cloud Recording API reference](/en/api-reference/api-ref/cloud-recording) # Restrict where agent can run (/en/ai/best-practices/regional-restrictions) import { Parameter, ParameterList } from '@/components/mdx'; To comply with laws and regulations in different countries and regions, the Conversational AI Engine supports regional access restrictions. When you enable regional access restrictions, the Conversational AI Engine only accesses Agora servers in the designated region, regardless of the user's location. For example, if you specify North America as the access region, two users who initiate calls from North America and Singapore have different experiences: | Specified access region | User location | Actual access region | User experience | |:-------|:-------|:-------|:-------| | North America | North America | North America | Normal quality. | | North America | Singapore | North America | Quality may be significantly impacted. Cross-regional public internet connections between the designated region and the user's location can result in poor network quality. If all servers in the specified region are unavailable, the service returns an error. | ## Understand the tech When regional access restrictions are disabled, the Conversational AI Engine automatically selects the nearest server to deploy the agent based on the IP address of the LLM URL and supports failover: 1. **Intelligent deployment**: The system automatically deploys the Conversational AI Engine service in the region that corresponds to the IP address of the configured LLM URL. 1. **Nearest available region**: If no server is available in the corresponding region, the system automatically selects the nearest available region. 1. **Failover**: When a service becomes unavailable in a region, the system automatically switches to another available region. :::caution[Caution] When you enable regional access restrictions, the system strictly limits access to the specified region and does not perform cross-regional failover. ::: ## Implementation ### Configure regional access To configure region access restrictions, set `properties.geofence` when you [Start a conversational AI agent](/en/api-reference/api-ref/conversational-ai/join). The allowed region for server access. The excluded region. Only available when `area` is set to `GLOBAL`. ### Configuration examples Refer to the following examples to understand `geofence` configuration. #### Restrict access to North America only Python SDK TypeScript SDK Go SDK REST API ```python from agora_agent import Agent from agora_agent.agentkit import GeofenceConfig # client is your configured Agora client agent = ( Agent(client) .with_llm(...) # configure your LLM vendor .with_tts(...) # configure your TTS vendor .with_geofence(GeofenceConfig( area='NORTH_AMERICA', )) ) ``` ```typescript import { Agent } from 'agora-agents'; // client is your configured Agora client const agent = new Agent({ client }) .withLlm(/* configure your LLM vendor */) .withTts(/* configure your TTS vendor */) .withGeofence({ area: 'NORTH_AMERICA', }); ``` ```go import "github.com/AgoraIO/agora-agents-go/v2/agentkit" // client is your configured *agentkit.AgoraClient agent := agentkit.NewAgent(client). WithLlm(/* configure your LLM vendor */). WithTts(/* configure your TTS vendor */). WithGeofence(&agentkit.GeofenceConfig{ Area: agentkit.GeofenceAreaNorthAmerica, }) ``` Use the following request body when you [Start a conversational AI agent](/en/api-reference/api-ref/conversational-ai/join): ```json { "name": "customer_service", "properties": { "channel": "test_channel", "token": "your_rtc_token", "agent_rtc_uid": "123", "remote_rtc_uids": ["456"], "geofence": { "area": "NORTH_AMERICA" }, "llm": { // ... LLM configuration }, "tts": { // ... TTS configuration } } } ``` #### Allow global access but exclude India Python SDK TypeScript SDK Go SDK REST API ```python from agora_agent import Agent from agora_agent.agentkit import GeofenceConfig # client is your configured Agora client agent = ( Agent(client) .with_llm(...) # configure your LLM vendor .with_tts(...) # configure your TTS vendor .with_geofence(GeofenceConfig( area='GLOBAL', exclude_area='INDIA', )) ) ``` ```typescript import { Agent } from 'agora-agents'; // client is your configured Agora client const agent = new Agent({ client }) .withLlm(/* configure your LLM vendor */) .withTts(/* configure your TTS vendor */) .withGeofence({ area: 'GLOBAL', exclude_area: 'INDIA', }); ``` ```go import "github.com/AgoraIO/agora-agents-go/v2/agentkit" // client is your configured *agentkit.AgoraClient agent := agentkit.NewAgent(client). WithLlm(/* configure your LLM vendor */). WithTts(/* configure your TTS vendor */). WithGeofence(&agentkit.GeofenceConfig{ Area: agentkit.GeofenceAreaGlobal, ExcludeArea: agentkit.GeofenceExcludeAreaIndia.Ptr(), }) ``` Use the following request body when you [Start a conversational AI agent](/en/api-reference/api-ref/conversational-ai/join): ```json { "name": "customer_service", "properties": { "channel": "test_channel", "token": "your_rtc_token", "agent_rtc_uid": "123", "remote_rtc_uids": ["456"], "geofence": { "area": "GLOBAL", "exclude_area": "INDIA" }, "llm": { // ... LLM configuration }, "tts": { // ... TTS configuration } } } ``` ### Configure RTC regional access The Conversational AI Engine and Agora RTC services are independent, and their regional access restriction configurations are separate. To implement complete end-to-end regional access restrictions and ensure that the entire call chain with the agent is confined to a specified region, you must also configure RTC regional access restrictions. For more information, see [Restrict media zones](../../realtime-media/rtc/build/manage-connection-and-quality/geofencing). ## Data residency In addition to configuring regional access restrictions for the Conversational AI Engine, some LLM, TTS, and ASR vendors offer data residency services in different regions. You can ensure that data is not transferred across regions by configuring the URLs used by the LLM, TTS, and ASR modules. For example, ElevenLabs supports selecting data processing regions through different URL endpoints. For more information, see the [ElevenLabs documentation](https://elevenlabs.io/docs/overview/administration/data-residency). The following example shows how to configure the ElevenLabs TTS service using a URL endpoint for the European region: ```json { "properties": { "tts": { "vendor": { "name": "elevenlabs" }, "url": "wss://api.eu.elevenlabs.io/v1/text-to-speech/{voice_id}/stream-input" } } } ``` ## Best practices To ensure that data remains entirely within the designated region, follow these recommendations: 1. **Configure regional restrictions for the Conversational AI Engine**: Use the `geofence` object to restrict server access regions. 2. **Configure RTC regional restrictions**: Configure the corresponding regional access restrictions in the RTC SDK. 3. **Select regional AI services**: When configuring LLM, TTS, and ASR services, select a provider that supports regional data residency and use the corresponding regional URL endpoint. 4. **Verify the configuration**: Before deploying to production, thoroughly test that regional access restrictions work correctly and that data is not transferred across regions. ## Considerations Keep the following in mind when implementing regional access restrictions: - **Performance impact**: Regional access restrictions may affect user experience. If a user's location is far from the designated region, network latency may increase. - **Service availability**: When regional access restrictions are enabled and the server in the specified region is unavailable or lacks sufficient resources, the service returns an error and does not automatically switch to another region. - **Configuration consistency**: Ensure that the regional configurations of the Conversational AI Engine, RTC SDK, and LLM, TTS, and ASR services are consistent to prevent data transfer across regions. - **Compliance requirements**: Before configuring regional access restrictions, understand the legal and regulatory requirements of the target region to ensure that the configuration complies with local data protection and privacy regulations. # Architecture (/en/ai/build/architecture) The TEN framework uses a graph structure of interconnected nodes to define data flow between extensions. Subgraphs extend this architecture by providing a powerful reuse mechanism that allows you to break down complex graph structures into modular, reusable components. Subgraphs improve code organization and maintainability by enabling you to create self-contained graph modules that can be shared across different applications or used multiple times within the same application. This modular approach simplifies the development of complex systems while maintaining the same runtime performance characteristics as regular graphs. ## Understand the tech Subgraphs preserve the fundamental principle of TEN graphs: defining how data flows between extensions. Rather than introducing new runtime complexity, subgraphs act as a development-time abstraction that gets flattened into the larger graph structure during deployment. This design approach provides the organizational benefits of modularity while using the same graph execution mechanism that powers all TEN applications. ### Design principles Subgraphs are built on the following foundational principles that ensure they integrate seamlessly with the TEN framework while providing meaningful development benefits: * **Independence**: Every subgraph is a complete, self-contained graph that can run independently or be embedded as a component within larger graphs. This independence means you can develop, test, and validate subgraphs in isolation before integrating them into complex applications. * **Tool-friendly design**: Subgraphs provide structured metadata through `exposed_messages` and `exposed_properties` that development tools can use to offer intelligent suggestions, validation, and debugging capabilities. This enhanced tooling support improves the development experience without adding runtime overhead. * **Flattening mechanism**: At deployment time, subgraphs are flattened into standard graph structures, ensuring that runtime performance remains identical to regular graphs. This approach provides development-time modularity benefits while maintaining the proven performance characteristics of the core TEN graph engine. * **Simplicity**: Subgraph design prioritizes simplicity. Rather than providing patching mechanisms or complex configuration options at reference points, subgraphs use straightforward mapping and exposure patterns. If a subgraph needs modification, you modify the original definition directly, avoiding complexity cascades and maintenance overhead. This design approach allows you to treat subgraphs as black boxes. You can use them without understanding their internal complexity, while the framework handles the technical details of integration and execution. ### Key features Subgraphs provide four essential features that enable modular graph development and seamless integration within TEN applications: * **Message exposure**: Subgraphs declare their external interfaces through the `exposed_messages` field, specifying which message types (commands, data, audio, video) can be sent to or received from the subgraph. This exposure mechanism allows development tools to provide intelligent autocomplete and validation while hiding internal implementation details from external consumers. * **Property exposure**: Through the `exposed_properties` field, subgraphs can selectively expose internal extension properties to external configuration. This controlled exposure enables customization of subgraph behavior without revealing the entire internal structure, maintaining encapsulation while providing necessary flexibility. * **Namespace management**: Each referenced subgraph creates its own namespace, preventing naming conflicts between elements in different subgraphs. When you reference a subgraph with name `subgraph_1`, all internal elements become accessible through the namespace syntax (e.g., `subgraph_1:ext_name`), ensuring global uniqueness across complex graph hierarchies. * **Cross-graph connections**: Subgraphs enable sophisticated connection patterns that span multiple graph boundaries. You can connect directly to subgraph interfaces, connect to specific elements within subgraphs, or even establish connections between elements in different subgraphs. This flexibility supports complex data flow architectures while maintaining clear organizational boundaries. These features work together to provide a powerful abstraction layer that simplifies complex graph development without sacrificing the flexibility and performance that TEN applications require. ## Creating subgraphs ### Subgraph definition A subgraph is defined in a standard JSON file that contains the same basic structure as any TEN graph, with additional metadata for external interfaces. Here's a basic example: ```json { "nodes": [ { "type": "extension", "name": "audio_processor", "addon": "audio_enhancement" }, { "type": "extension", "name": "audio_filter", "addon": "noise_reduction" } ], "connections": [ { "extension": "audio_processor", "audio_frame": [ { "name": "processed_audio", "dest": [ { "extension": "audio_filter" } ] } ] } ], "exposed_messages": [ { "extension": "audio_processor", "type": "audio_frame_in", "name": "raw_audio" }, { "extension": "audio_filter", "type": "audio_frame_out", "name": "clean_audio" } ], "exposed_properties": [ { "extension": "audio_processor", "name": "sample_rate", "alias": "input_sample_rate" }, { "extension": "audio_filter", "name": "noise_threshold", "alias": "filter_sensitivity" } ] } ``` Following is the structure breakdown: * **Nodes section**: Defines the extensions that make up the subgraph, just like in a regular graph. Each extension specifies its type, name, and addon identifier. * **Connections section**: Establishes data flow between extensions within the subgraph. These internal connections define how the subgraph processes data internally. * **Exposed messages section**: Declares which message interfaces are available for external connections. This acts as the subgraph's public API, allowing other graphs to send data to or receive data from the subgraph without knowing its internal structure. * **Exposed properties section**: Specifies which extension properties can be configured externally. The `alias` field provides user-friendly names for external configuration, abstracting away internal implementation details. This sample subgraph can function as a standalone audio processing pipeline or be embedded within larger applications that need audio enhancement capabilities. ### Exposed interfaces Subgraphs use two interface types to control what external graphs can access and configure. These interfaces act as the subgraph's public API, providing controlled access while maintaining internal encapsulation. #### exposed\_messages The `exposed_messages` field defines which message interfaces are available for external connections. This allows other graphs to interact with your subgraph without needing to understand its internal structure. **Message exposure structure:** ```json { "exposed_messages": [ { "extension": "internal_extension_name", "type": "message_direction_and_type", "name": "message_name" } ] } ``` **Message types:** * `cmd_in` / `cmd_out`: Command messages for control and responses * `data_in` / `data_out`: General data messages * `audio_frame_in` / `audio_frame_out`: Audio stream data * `video_frame_in` / `video_frame_out`: Video stream data **Example:** ```json { "exposed_messages": [ { "extension": "speech_recognizer", "type": "audio_frame_in", "name": "audio_input" }, { "extension": "text_processor", "type": "data_out", "name": "transcribed_text" }, { "extension": "speech_recognizer", "type": "cmd_in", "name": "configure_language" } ] } ``` This example exposes an audio input, text output, and configuration command, allowing external graphs to send audio data and receive transcribed text while configuring the recognition language. #### exposed\_properties The `exposed_properties` field specifies which internal extension properties can be configured from outside the subgraph. This enables customization without exposing the entire internal configuration. **Property exposure structure:** ```json { "exposed_properties": [ { "extension": "internal_extension_name", "name": "internal_property_name", "alias": "external_property_name" } ] } ``` * `extension`: The internal extension that owns the property * `name`: The actual property name within the extension * `alias`: The external name used when referencing the subgraph (optional) **Example:** ```json { "exposed_properties": [ { "extension": "video_encoder", "name": "bitrate_kbps", "alias": "output_quality" }, { "extension": "frame_buffer", "name": "max_buffer_size", "alias": "buffer_limit" }, { "extension": "video_encoder", "name": "codec_type" } ] } ``` In this example, external graphs can configure video quality through the `output_quality` alias, set buffer limits through `buffer_limit`, and specify codec type directly. The aliases provide user-friendly names while hiding implementation details. Exposed interfaces provide the following benefits: * **Encapsulation**: Hide internal complexity while providing clean external APIs * **Selective exposure**: Allow control over which interfaces and properties can be accessed externally * **Interface documentation**: Serve as documentation for subgraph capabilities ## Using subgraphs ### Referencing subgraphs To use a subgraph in your main graph, reference it as a node with type `subgraph`. The subgraph becomes part of your graph's namespace and can be connected like any other component. **Basic subgraph reference:** ```json { "nodes": [ { "type": "extension", "name": "input_source", "addon": "audio_input" }, { "type": "extension", "name": "output_sink", "addon": "audio_output" }, { "type": "subgraph", "name": "audio_pipeline", "source_uri": "./ten_packages/extension/audio_processing/subgraph.json" } ], "connections": [ { "extension": "input_source", "audio_frame": [ { "name": "raw_audio", "dest": [ { "subgraph": "audio_pipeline" } ] } ] }, { "subgraph": "audio_pipeline", "audio_frame": [ { "name": "processed_audio", "dest": [ { "extension": "output_sink" } ] } ] } ] } ``` **Subgraph with properties:** You can configure exposed properties when referencing a subgraph: ```json { "type": "subgraph", "name": "video_processor", "source_uri": "./subgraphs/video_enhancement.json", "property": { "output_quality": "high", "buffer_limit": 1024, "codec_type": "h264" } } ``` The properties you specify must exist in the subgraph's `exposed_properties` field, otherwise the configuration is rejected. ### Connection patterns Subgraphs support multiple connection patterns that provide flexibility in how you route data between components. #### Direct connections to subgraphs Connect directly to a subgraph using its name when the subgraph exposes the appropriate message interface: ```json { "connections": [ { "extension": "data_source", "cmd": [ { "name": "process_data", "dest": [ { "subgraph": "data_processor" } ] } ] } ] } ``` This pattern works when `data_processor` exposes a `cmd_in` interface for `process_data`. The connection is clean and hides the subgraph's internal structure. #### Connections to elements within subgraphs For more granular control, connect directly to specific extensions within a subgraph using namespace syntax: ```json { "connections": [ { "extension": "external_source", "data": [ { "name": "raw_input", "dest": [ { "extension": "video_pipeline:frame_buffer" } ] } ] }, { "extension": "video_pipeline:encoder", "video_frame": [ { "name": "encoded_output", "dest": [ { "extension": "external_sink" } ] } ] } ] } ``` The `subgraph_name:extension_name` syntax allows you to bypass the subgraph's exposed interfaces and connect directly to internal elements. Use this pattern when you need fine-grained control over data flow. #### Advanced connection syntax You can mix different connection patterns within the same graph to create sophisticated data flow architectures: ```json { "connections": [ { "extension": "input_hub", "data": [ { "name": "sensor_data", "dest": [ { "subgraph": "processing_pipeline" }, { "extension": "backup_pipeline:data_logger" }, { "extension": "monitoring_extension" } ] } ] }, { "subgraph": "processing_pipeline", "cmd": [ { "name": "status_update", "dest": [ { "extension": "monitoring_extension" }, { "subgraph": "alert_system" } ] } ] } ] } ``` **Connection pattern guidelines:** * **Use direct subgraph connections** when working with well-defined interfaces * **Use namespace syntax** when you need to access specific internal functionality * **Mix patterns** when building complex data flow architectures * **Prefer exposed interfaces** for maintainability and clarity These patterns give you the flexibility to build everything from simple linear pipelines to complex multi-path processing architectures while maintaining clear organizational boundaries. ## Flattening process ### How flattening works Before your application runs, the TEN framework automatically flattens all subgraph references into a single, standard graph structure. This flattening process converts the modular development structure into the standard graph format that the TEN runtime executes. **Flattening transformations:** 1. **Namespace resolution**: Subgraph references are resolved and internal elements are prefixed with the subgraph name to ensure global uniqueness. 2. **Node expansion**: Subgraph nodes are replaced with their constituent extension nodes, maintaining all original properties and configurations. 3. **Connection mapping**: References using namespace syntax (`subgraph:extension`) are converted to direct extension names, and subgraph-level connections are mapped to specific internal extensions. 4. **Property inheritance**: Properties specified when referencing a subgraph are applied to the appropriate internal extensions based on the `exposed_properties` mapping. 5. **Interface removal**: The `exposed_messages` and `exposed_properties` metadata is removed since it's only needed during graph composition. **Naming convention:** The colon (`:`) is replaced with an underscore (`_`) to create valid extension names in the flattened structure. * Original: `subgraph_name:extension_name` * Flattened: `subgraph_name_extension_name` ### Flattened graph example Following is a complete example that shows how a graph with subgraphs transforms during the flattening process: **Before flattening:** ```json { "nodes": [ { "type": "extension", "name": "input_source", "addon": "microphone_input" }, { "type": "extension", "name": "output_sink", "addon": "speaker_output" }, { "type": "subgraph", "name": "audio_pipeline", "source_uri": "./subgraphs/audio_processing.json", "property": { "sample_rate": 48000, "noise_threshold": 0.3 } }, { "type": "subgraph", "name": "effects_chain", "source_uri": "./subgraphs/audio_effects.json" } ], "connections": [ { "extension": "input_source", "audio_frame": [ { "name": "raw_audio", "dest": [ { "subgraph": "audio_pipeline" } ] } ] }, { "extension": "audio_pipeline:processor", "audio_frame": [ { "name": "clean_audio", "dest": [ { "extension": "effects_chain:reverb" } ] } ] }, { "subgraph": "effects_chain", "audio_frame": [ { "name": "final_audio", "dest": [ { "extension": "output_sink" } ] } ] } ] } ``` **After flattening:** ```json { "nodes": [ { "type": "extension", "name": "input_source", "addon": "microphone_input" }, { "type": "extension", "name": "output_sink", "addon": "speaker_output" }, { "type": "extension", "name": "audio_pipeline_enhancer", "addon": "audio_enhancement", "property": { "sample_rate": 48000 } }, { "type": "extension", "name": "audio_pipeline_processor", "addon": "noise_reduction", "property": { "noise_threshold": 0.3 } }, { "type": "extension", "name": "effects_chain_reverb", "addon": "reverb_effect" }, { "type": "extension", "name": "effects_chain_compressor", "addon": "dynamic_compressor" } ], "connections": [ { "extension": "input_source", "audio_frame": [ { "name": "raw_audio", "dest": [ { "extension": "audio_pipeline_enhancer" } ] } ] }, { "extension": "audio_pipeline_enhancer", "audio_frame": [ { "name": "enhanced_audio", "dest": [ { "extension": "audio_pipeline_processor" } ] } ] }, { "extension": "audio_pipeline_processor", "audio_frame": [ { "name": "clean_audio", "dest": [ { "extension": "effects_chain_reverb" } ] } ] }, { "extension": "effects_chain_reverb", "audio_frame": [ { "name": "reverb_audio", "dest": [ { "extension": "effects_chain_compressor" } ] } ] }, { "extension": "effects_chain_compressor", "audio_frame": [ { "name": "final_audio", "dest": [ { "extension": "output_sink" } ] } ] } ] } ``` **Key changes during flattening:** * **Subgraph nodes removed**: `audio_pipeline` and `effects_chain` subgraph nodes are replaced with their constituent extensions * **Names prefixed**: Internal extensions get prefixed names (`audio_pipeline_processor`, `effects_chain_reverb`) * **Properties distributed**: Subgraph properties are applied to the correct internal extensions * **Connections resolved**: All namespace references and subgraph connections are converted to direct extension connections * **Internal connections preserved**: Connections that existed within the original subgraphs are included in the flattened result The flattened graph contains only standard extensions and direct connections, maintaining the organizational benefits you gained during development while using the standard TEN graph execution model. ## Advanced features ### Message conversion with subgraphs Subgraphs support the TEN framework's message conversion mechanism, allowing you to transform message formats when routing data between components. **Message conversion syntax:** You can apply message conversion rules when connecting to subgraphs, internal subgraph elements, or when subgraphs connect to external components: ```json { "connections": [ { "extension": "data_source", "cmd": [ { "name": "user_request", "dest": [ { "subgraph": "ai_processor", "msg_conversion": { "type": "per_property", "rules": [ { "path": "request_type", "conversion_mode": "fixed_value", "value": "chat_completion" }, { "path": "model_config.temperature", "conversion_mode": "from_original", "original_path": "settings.creativity" } ], "keep_original": true } }, { "extension": "nlp_pipeline:tokenizer", "msg_conversion": { "type": "per_property", "rules": [ { "path": "input_text", "conversion_mode": "from_original", "original_path": "user_message" } ], "keep_original": false } } ] } ] } ] } ``` **Conversion during flattening:** Message conversion rules are preserved during the flattening process, ensuring that your data transformations work correctly at runtime: ```json { "connections": [ { "extension": "data_source", "cmd": [ { "name": "user_request", "dest": [ { "extension": "ai_processor_chat_handler", "msg_conversion": { "type": "per_property", "rules": [ { "path": "request_type", "conversion_mode": "fixed_value", "value": "chat_completion" } ], "keep_original": true } }, { "extension": "nlp_pipeline_tokenizer", "msg_conversion": { "type": "per_property", "rules": [ { "path": "input_text", "conversion_mode": "from_original", "original_path": "user_message" } ], "keep_original": false } } ] } ] } ] } ``` ### Multi-graph connections Beyond subgraphs, TEN supports connections between separate graphs running in the same application or across different applications. This enables sophisticated distributed architectures. #### Connecting to predefined graphs You can reference and connect to other predefined graphs within the same TEN application: ```json { "nodes": [ { "type": "extension", "name": "local_processor", "addon": "data_processor" }, { "type": "graph", "graph_name": "analytics_pipeline", "singleton": true }, { "type": "graph", "graph_name": "monitoring_system", "singleton": false } ], "connections": [ { "extension": "local_processor", "data": [ { "name": "processed_data", "dest": [ { "extension": "analytics_pipeline:data_ingester" }, { "graph": "monitoring_system" } ] } ] }, { "graph": "analytics_pipeline", "cmd": [ { "name": "analysis_complete", "dest": [ { "extension": "local_processor" } ] } ] } ] } ``` #### Remote graph connections Connect to graphs running in different TEN applications across the network: ```json { "nodes": [ { "type": "extension", "name": "local_service", "addon": "api_handler" }, { "type": "graph", "graph_name": "ml_inference", "singleton": true, "app": "msgpack://ml-server.example.com:8002/" }, { "type": "graph", "graph_name": "data_warehouse", "singleton": false, "app": "msgpack://data-cluster.example.com:8003/" } ], "connections": [ { "extension": "local_service", "cmd": [ { "name": "inference_request", "dest": [ { "graph": "ml_inference" } ] } ] }, { "graph": "ml_inference", "data": [ { "name": "inference_result", "dest": [ { "extension": "local_service" }, { "graph": "data_warehouse" } ] } ] } ] } ``` #### Graph node types TEN supports three types of graph nodes for different connection scenarios: **Subgraph nodes:** ```json { "type": "subgraph", "name": "processing_module", "source_uri": "./subgraphs/data_processing.json", "property": { "batch_size": 100 } } ``` **Local graph nodes:** ```json { "type": "graph", "graph_name": "background_service", "singleton": true } ``` ```json { "type": "graph", "graph_name": "worker_pool", "singleton": false } ``` **Remote graph nodes:** ```json { "type": "graph", "graph_name": "external_api", "singleton": true, "app": "msgpack://api.service.com:8001/" } ``` ```json { "type": "graph", "graph_name": "distributed_cache", "singleton": false, "app": "msgpack://cache-cluster.internal:8004/" } ``` **Graph node properties:** * **graph\_name**: Identifies the target graph within its application * **singleton**: Controls whether multiple instances can exist (`true` = single instance, `false` = multiple instances allowed) * **app**: Specifies the target application URL for remote connections (omit for local graphs) These advanced features enable you to build sophisticated distributed systems where subgraphs, local graphs, and remote services work together seamlessly while maintaining clear architectural boundaries. ## Reference ### Graph node types TEN graphs support three types of node references that enable different architectural patterns and connection scopes. #### Subgraph nodes Reference external graph files as reusable components within your graph: **Basic subgraph node:** ```json { "type": "subgraph", "name": "unique_subgraph_name", "source_uri": "path/to/subgraph.json" } ``` **Subgraph node with properties:** ```json { "type": "subgraph", "name": "configurable_module", "source_uri": "./ten_packages/extension/module/subgraph.json", "property": { "param1": "value1", "param2": 42, "nested_config": { "enabled": true, "threshold": 0.85 } } } ``` **Remote subgraph node:** ```json { "type": "subgraph", "name": "shared_component", "source_uri": "https://cdn.example.com/subgraphs/v1.2/nlp_pipeline.json" } ``` **Required fields:** * `type`: Must be `"subgraph"` * `name`: Unique identifier within the current graph (serves as namespace) * `source_uri`: Path or URL to the subgraph definition file **Optional fields:** * `property`: Configuration values for exposed properties #### Local graph nodes Reference other predefined graphs within the same TEN application: **Singleton local graph:** ```json { "type": "graph", "graph_name": "predefined_graph_id", "singleton": true } ``` **Multi-instance local graph:** ```json { "type": "graph", "graph_name": "worker_template", "singleton": false } ``` **Required fields:** * `type`: Must be `"graph"` * `graph_name`: Identifier of the predefined graph in the same application * `singleton`: Boolean controlling instance multiplicity **Singleton behavior:** * `true`: Only one instance of the graph exists, shared across references * `false`: Each reference creates a separate graph instance #### Remote graph nodes Reference graphs running in different TEN applications across the network: **Singleton remote graph:** ```json { "type": "graph", "graph_name": "ml_service", "singleton": true, "app": "msgpack://ml-cluster.internal:8002/" } ``` **Multi-instance remote graph:** ```json { "type": "graph", "graph_name": "data_processor", "singleton": false, "app": "msgpack://processing-farm.example.com:8003/" } ``` **Required fields:** * `type`: Must be `"graph"` * `graph_name`: Identifier of the target graph in the remote application * `singleton`: Boolean controlling instance behavior * `app`: Network address of the target TEN application **Supported protocols:** * `msgpack://host:port/`: MessagePack-based communication * Additional protocols may be supported in future versions ### Connection syntax reference TEN provides flexible syntax for establishing connections between different types of graph nodes and their internal elements. #### subgraph\_name:element_name Access specific elements within subgraphs using namespace syntax: **Extension within subgraph:** ```json { "extension": "audio_pipeline:noise_filter", "audio_frame": [ { "name": "filtered_audio", "dest": [ { "extension": "output_device" } ] } ] } ``` **Connecting to subgraph element:** ```json { "extension": "microphone", "audio_frame": [ { "name": "raw_audio", "dest": [ { "extension": "processing_chain:input_buffer" } ] } ] } ``` **Syntax rules:** * Format: `subgraph_name:extension_name` * The colon (`:`) separates namespace from element name * Element name must exist within the referenced subgraph * Bypasses subgraph's exposed interface constraints #### Graph references Connect to graphs using their identifiers: **Direct graph connection:** ```json { "subgraph": "local_processor", "cmd": [ { "name": "process_complete", "dest": [ { "graph": "analytics_engine" } ] } ] } ``` **Graph element connection:** ```json { "graph": "background_service", "data": [ { "name": "status_update", "dest": [ { "extension": "monitor:logger" } ] } ] } ``` **Local graph with namespace:** ```json { "extension": "coordinator", "cmd": [ { "name": "task_assignment", "dest": [ { "extension": "worker_pool:task_manager" } ] } ] } ``` #### Advanced patterns Combine multiple connection types for sophisticated data flow architectures: **Multi-destination routing:** ```json { "extension": "data_ingester", "data": [ { "name": "incoming_data", "dest": [ { "subgraph": "validation_pipeline" }, { "extension": "processing_farm:load_balancer" }, { "graph": "audit_logger" }, { "extension": "backup_storage" } ] } ] } ``` **Cross-application message flow:** ```json { "graph": "local_analytics", "cmd": [ { "name": "analysis_request", "dest": [ { "extension": "ml_cluster:inference_engine" } ] } ] } ``` **Conditional routing with message conversion:** ```json { "extension": "request_router", "cmd": [ { "name": "user_request", "dest": [ { "subgraph": "fast_processing", "msg_conversion": { "type": "per_property", "rules": [ { "path": "priority", "conversion_mode": "fixed_value", "value": "high" } ] } }, { "extension": "slow_processing:queue_manager", "msg_conversion": { "type": "per_property", "rules": [ { "path": "priority", "conversion_mode": "fixed_value", "value": "normal" } ] } } ] } ] } ``` **Connection target types:** * `extension`: Direct extension name * `subgraph`: Subgraph name (uses exposed interfaces) * `graph`: Graph name (uses exposed interfaces) * `extension` with namespace: `namespace:extension_name` **Best practices:** * Use exposed interfaces when possible for better encapsulation * Use namespace syntax only when you need direct access to internal elements * Prefer subgraphs over direct graph references for reusable components * Document complex connection patterns for maintainability # Use MCP tools (/en/ai/build/mcp-tools) MCP is an open protocol for connecting LLMs to external tools and data sources. When you attach an MCP server to a Conversational AI agent and enable tool calling, the LLM can invoke the tools that the server exposes, such as looking up account data, running a search, or triggering an action in another system, and use the results to generate its response. This page explains how to attach an MCP server to an agent using the REST API and Agent SDKs. Info Building a no-code agent in the Agora Console? See [Manage integrations](../studio/build/integrations#mcp-servers). ## Understand the tech During a conversation, the LLM decides when a tool call is useful, the engine forwards that call to the configured MCP server over HTTP, and the server's response is fed back to the LLM as context for its next reply. Agora's Conversational AI Engine acts as the MCP client. To configure MCP tool-use: * **Specify an array of MCP servers**: Each entry defines the server's endpoint, transport, and which of its tools the agent can call. * **Enable tool invocation**: This allows the agent to call the available tools on the specified servers. ## Prerequisites * Implemented the basic logic for interacting with a conversational AI agent. See the [quickstart](../get-started/quickstart). * An MCP server reachable over HTTP or HTTPS. ## Implementation Take the following steps to give your agent access to an MCP server. ### Attach an MCP server and enable tool calling Add one or more servers then enable tool calling. MLLM agents currently support MCP tool-use, through the RESTful API only. The Agent SDKs don't yet expose an equivalent field on their MLLM vendor classes. For an MLLM agent, add `mcp_servers` under `mllm` instead of `llm`. Python SDK TypeScript SDK Go SDK REST API ```python from agora_agent import OpenAI # ... other agent configuration ... .with_llm(OpenAI( api_key='your-llm-api-key', model='gpt-4o-mini', mcp_servers=[ { 'name': 'deepwiki', 'endpoint': 'https://mcp.deepwiki.com/mcp', # Replace with your own MCP server 'transport': 'streamable_http', 'allowed_tools': ['ask_question'], # Replace with your server's tool names 'timeout_ms': 10000, }, ], )) .with_tools(True) # ... continue with .with_tts() and .create_session() ... ``` ```typescript import { OpenAI } from 'agora-agents'; // ... other agent configuration ... .withLlm(new OpenAI({ apiKey: 'your-llm-api-key', model: 'gpt-4o-mini', mcpServers: [ { name: 'deepwiki', endpoint: 'https://mcp.deepwiki.com/mcp', // Replace with your own MCP server transport: 'streamable_http', allowed_tools: ['ask_question'], // Replace with your server's tool names timeout_ms: 10000, }, ], })) .withTools(true) // ... continue with .withTts() and .createSession() ... ``` ```go // ... other agent configuration ... .WithLlm( vendors.NewOpenAI(vendors.OpenAIOptions{ APIKey: "your-llm-api-key", Model: "gpt-4o-mini", McpServers: []map[string]interface{}{ { "name": "deepwiki", "endpoint": "https://mcp.deepwiki.com/mcp", // Replace with your own MCP server "transport": "streamable_http", "allowed_tools": []string{"ask_question"}, // Replace with your server's tool names "timeout_ms": 10000, }, }, }), ) .WithTools(true) // ... continue with .WithTts() and .CreateSession() ... ``` ```json { "llm": { "url": "https://api.openai.com/v1/chat/completions", "api_key": "your-llm-api-key", "params": { "model": "gpt-4o-mini" }, "mcp_servers": [ { "name": "deepwiki", "endpoint": "https://mcp.deepwiki.com/mcp", "transport": "streamable_http", "allowed_tools": ["ask_question"], "timeout_ms": 10000 } ] }, "advanced_features": { "enable_tools": true } } ``` Update `endpoint` with your own MCP server's URL, and `allowed_tools` with the MCP tool names you want your agent to use. Attaching MCP servers and enabling tool invocation are independent settings. If you configure servers without enabling tool invocation, the LLM never calls any tools, with no error to signal the mismatch. ### Configure an MCP server Each item in the MCP servers array accepts the following fields. A unique identifier for the MCP server. Maximum 48 characters. Accepts only English letters and numbers. The MCP server's endpoint address. Transport protocol. Currently only `streamable_http` is supported. HTTP headers to send with requests to the MCP server. If your MCP server requires authentication, use it to pass credentials. For example: ```json { "Authorization": "Bearer your-mcp-server-token" } ``` The tools the agent is allowed to call: * Omit `allowed_tools`, or set it to `["*"]`: all tools on the server are enabled. * Set `allowed_tools` to a list of tool names, for example `["search_docs", "get_weather"]`: only those tools are enabled. Strings in `allowed_tools` must match the tool names the MCP server actually registers, not names from another integration's docs or a client config file. An empty array or a name that doesn't match any real tool doesn't fail cleanly: the agent can hang mid-response with no error, rather than answering or falling back. Confirm the exact names by calling the server's `tools/list` method directly, for example: ```bash curl --request POST \ --url https://your-mcp-server.example.com/mcp \ --header 'Content-Type: application/json' \ --header 'Accept: application/json, text/event-stream' \ --data '{"jsonrpc": "2.0", "id": 1, "method": "tools/list", "params": {}}' ``` Request timeout for the MCP server. After the timeout, the agent stops waiting for a response and continues with subsequent logic. To connect multiple MCP servers, add multiple entries to the MCP servers array. Each server must have a unique `name`. Filler words during tool calls Tool calls add latency while the MCP server processes the request. Configure [filler words](shape-the-conversation/filler-words) so the agent acknowledges the user's request while it waits for a tool result. ## Reference * [`llm.mcp_servers`](/en/api-reference/api-ref/conversational-ai/join#request-body-properties-llm-mcp-servers) * [`mllm.mcp_servers`](/en/api-reference/api-ref/conversational-ai/join#request-body-properties-mllm-mcp-servers) * [`advanced_features.enable_tools`](/en/api-reference/api-ref/conversational-ai/join#request-body-properties-advanced-features-enable-tools) * [`with_tools()` / `withTools()` / `WithTools()`](/en/api-reference/api-ref/server-sdk/python#other-builder-methods) # Send images to the agent (/en/ai/build/send-multimodal-messages) When interacting with an agent, you may need to upload images or send image messages from the client to help the agent better understand the user's intent. This page describes how to use the Conversational AI Engine toolkit to send image messages to the large language model from your app. The LLM can then automatically reference the image content in subsequent conversations and generate more relevant responses. ## Understand the tech Agora provides a flexible, scalable, and standardized conversational AI engine toolkit. The toolkit supports **iOS**, **Android**, and **Web** platforms, and encapsulates scenario-based APIs. You can use these APIs to integrate the capabilities of the [Agora Signaling SDK](/en/realtime-media/rtm) and [Agora RTC SDK](../../realtime-media/rtc/get-started-sdk) to enable the following features: * [Interrupt agents](shape-the-conversation/interrupt-agent) * [Display live transcripts](transcripts) * [Client-side events](handle-runtime-events/event-notifications) * [Set optimal audio parameters](../best-practices/audio-setup) for iOS and Android * [Send picture messages](send-multimodal-messages) Call the toolkit's `chat` API to send a picture message, and listen to `onMessageReceiptUpdated` to receive the picture message receipt. ## Prerequisites Before you begin, ensure the following: * You have implemented the Conversational AI Engine [quickstart](../get-started/quickstart). * Your app integrates Agora RTC SDK v4.5.1 or later and includes the [RTC SDK quickstart](../../realtime-media/rtc/get-started-sdk). * You have enabled Signaling in the Agora Console and completed the [Signaling quickstart](/en/realtime-media/rtm/quickstart) for basic messaging. * You maintain active and authenticated RTC and Signaling instances that persist beyond the component lifecycle. The toolkit does not manage RTC or Signaling initialization, lifecycle, or authentication. Info * Picture messaging is currently in beta and free for a limited time. * Image processing depends on the capabilities of the integrated LLM. Make sure the LLM you connect to Conversational AI Engine supports image input. ## Implementation This section explains how to send a picture message from your app. Android iOS Web 1. **Integrate the toolkit** Add the toolkit to your project using Maven or source code. See [Install the Android toolkit](/en/api-reference/api-ref/conversational-ai/client-toolkit/android#installation). Refer to [Folder structure](#reference) to understand the role of each file. 2. **Create a toolkit instance** Create a configuration object with the RTC SDK and Signaling engine instances. Use the configuration to create a toolkit instance. ```kotlin // Create configuration objects for the RTC and RTM instances val config = ConversationalAIAPIConfig( rtcEngine = rtcEngine, rtmClient = rtmClient, renderMode = TranscriptRenderMode.Word, enableLog = true, enableRenderModeFallback = true ) // Create component instance val api = ConversationalAIAPIImpl(config) ``` 3. **Register callback** ```kotlin api.addHandler(covEventHandler) ``` 4. **Subscribe to the channel** Agent-related events are delivered through Signaling channel messages. To receive these events, call `subscribeMessage` before starting the agent session. ```kotlin api.subscribeMessage("channelName") { error -> if (error != null) { // Handle error } } ``` 5. **Add a conversational AI agent to the channel** To [start a conversational AI agent](/en/api-reference/api-ref/conversational-ai/join), configure the following parameters in your `POST` request: | Parameter | Description | Required | | --------------------------------------- | -------------------------------------------------- | -------- | | `advanced_features.enable_rtm: true` | Starts the Signaling service | Yes | | `parameters.data_channel: "rtm"` | Enables Signaling as the data transmission channel | Yes | | `parameters.enable_metrics: true` | Enables agent performance data collection | Optional | | `parameters.enable_error_message: true` | Enables reporting of agent error events | Optional | After a successful response, the agent joins the specified RTC channel and is ready to interact with the user. 6. **Send an image** ```kotlin val uuid = "unique-image-id-123" val imageUrl = "https://example.com/image.jpg" api.chat("agentUserId", ImageMessage(uuid = uuid, imageUrl = imageUrl)) { error -> if (error != null) { Log.e("Chat", "Failed to send image: ${error.errorMessage}") } else { Log.d("Chat", "Image send request successful") } } ``` Info The `chat` completion callback only indicates whether the sending request is successful, and does not reflect the actual processing status of the message. 7. **Handle image sending status** Image send success is confirmed by `onMessageReceiptUpdated`. If sending fails, `onMessageError` is triggered. Use the `uuid` value in the callback to identify the uploaded picture. Image sent successfully: ```kotlin override fun onMessageReceiptUpdated(agentUserId: String, receipt: MessageReceipt) { if (receipt.chatMessageType == ChatMessageType.Image) { try { val json = JSONObject(receipt.message) if (json.has("uuid")) { val receivedUuid = json.getString("uuid") if (receivedUuid == "your-sent-uuid") { Log.d("ImageSend", "Image sent successfully: $receivedUuid") } } } catch (e: Exception) { Log.e("ImageSend", "Failed to parse message receipt: ${e.message}") } } } ``` Image sending failed: ```kotlin override fun onMessageError(agentUserId: String, error: MessageError) { if (error.chatMessageType == ChatMessageType.Image) { try { val json = JSONObject(error.message) if (json.has("uuid")) { val failedUuid = json.getString("uuid") if (failedUuid == "your-sent-uuid") { Log.e("ImageSend", "Image send failed: $failedUuid") } } } catch (e: Exception) { Log.e("ImageSend", "Failed to parse error message: ${e.message}") } } } ``` 8. **Unsubscribe from the channel** ```kotlin api.unsubscribeMessage("channelName") { error -> if (error != null) { // Handle the error } } ``` 9. **Release resources** ```kotlin api.destroy() ``` 1. **Integrate the toolkit** Add the toolkit to your project using CocoaPods, Swift Package Manager, or source code. See [Install the iOS toolkit](/en/api-reference/api-ref/conversational-ai/client-toolkit/ios#installation). Refer to [Folder structure](#reference) to understand the role of each file. 2. **Create a toolkit instance** Create a configuration object with the RTC SDK and Signaling engine instances, then use the configuration to create a toolkit instance. ```swift let config = ConversationalAIAPIConfig( rtcEngine: rtcEngine, rtmEngine: rtmEngine, renderMode: .words, enableLog: true, enableRenderModeFallback: true ) convoAIAPI = ConversationalAIAPIImpl(config: config) ``` 3. **Subscribe to the channel** Agent-related events are delivered through Signaling channel messages. To receive these events, call `subscribeMessage` before starting the agent session. ```swift convoAIAPI.subscribeMessage(channelName: channelName) { error in if let error = error { print("Subscription failed: \(error.message)") } else { print("Subscription successful") } } ``` 4. **Register callback** ```swift convoAIAPI.addHandler(handler: self) ``` 5. **Add a conversational AI agent to the channel** To [start a conversational AI agent](/en/api-reference/api-ref/conversational-ai/join), configure the following parameters in your `POST` request: | Parameter | Description | Required | | --------------------------------------- | -------------------------------------------------- | -------- | | `advanced_features.enable_rtm: true` | Starts the Signaling service | Yes | | `parameters.data_channel: "rtm"` | Enables Signaling as the data transmission channel | Yes | | `parameters.enable_metrics: true` | Enables agent performance data collection | Optional | | `parameters.enable_error_message: true` | Enables reporting of agent error events | Optional | 6. **Send an image** ```swift let uuid = UUID().uuidString let imageUrl = "https://example.com/image.jpg" let message = ImageMessage(uuid: uuid, url: imageUrl) self.convoAIAPI.chat(agentUserId: "\(agentUid)", message: message) { [weak self] error in if let error = error { print("send image failed, error: \(error.message)") } else { print("send image success") } } ``` Info The `chat` completion callback only indicates whether the sending request is successful, and does not reflect the actual processing status of the message. 7. **Receive image sending response** Success is confirmed by `onMessageReceiptUpdated`. If sending fails, `onMessageError` is triggered. Image sent successfully: ```swift struct PictureInfo: Codable { let uuid: String } public func onMessageReceiptUpdated(agentUserId: String, messageReceipt: MessageReceipt) { if messageReceipt.type == .context { guard let messageData = messageReceipt.message.data(using: .utf8) else { return } do { let imageInfo = try JSONDecoder().decode(PictureInfo.self, from: messageData) self.messageView.viewModel.updateImageMessage(uuid: imageInfo.uuid, state: .success) } catch { print("Failed to decode PictureInfo: \(error)") } } } ``` Image sending failed: ```swift struct ImageUploadError: Codable { let code: Int let message: String } struct ImageUploadErrorResponse: Codable { let uuid: String let success: Bool let error: ImageUploadError? } public func onMessageError(agentUserId: String, error: MessageError) { if let messageData = error.message.data(using: .utf8) { do { let errorResponse = try JSONDecoder().decode(ImageUploadErrorResponse.self, from: messageData) if !errorResponse.success { DispatchQueue.main.async { [weak self] in self?.messageView.viewModel.updateImageMessage(uuid: errorResponse.uuid, state: .failed) } } } catch { addLog("Failed to parse error message JSON: \(error)") } } } ``` 8. **Unsubscribe from the channel** ```swift convoAIAPI.unsubscribeMessage(channelName: channelName) { error in if let error = error { print("Unsubscription failed: \(error.message)") } else { print("Unsubscribed successfully") } } ``` 9. **Release resources** ```swift convoAIAPI.destroy() ``` 1. **Integrate the toolkit** Add the toolkit to your project using a package manager or source code. See [Install the Web toolkit](/en/api-reference/api-ref/conversational-ai/client-toolkit/web#installation). Refer to [Folder structure](#reference) to understand the role of each file. 2. **Create a toolkit instance** Before joining an RTC channel, create RTC SDK and Signaling engine instances and pass them into the toolkit instance. ```ts const config: IConversationalAIAPIConfig = { rtcEngine, rtmEngine, renderMode: ETranscriptHelperMode.WORD, enableLog: true, enableRenderModeFallback: true, } const conversationalAIAPI = await ConversationalAIAPI.init(config) ``` 3. **Subscribe to the channel** Agent-related events are delivered through Signaling messages. Before starting an agent session, call `subscribeMessage` to receive these events. ```ts conversationalAIAPI.subscribeMessage(channel_name) ``` 4. **Register callbacks** ```ts conversationalAIAPI.on(EConversationalAIAPIEvents.MESSAGE_RECEIPT_UPDATED, handleMessageReceiptUpdated) conversationalAIAPI.on(EConversationalAIAPIEvents.MESSAGE_ERROR, onAgentError) ``` 5. **Add a conversational AI agent to the channel** To [start a conversational AI agent](/en/api-reference/api-ref/conversational-ai/join), configure the following parameters in your `POST` request: | Parameter | Description | Required | | --------------------------------------- | -------------------------------------------------- | -------- | | `advanced_features.enable_rtm: true` | Starts the Signaling service | Yes | | `parameters.data_channel: "rtm"` | Enables Signaling as the data transmission channel | Yes | | `parameters.enable_metrics: true` | Enables agent performance data collection | Optional | | `parameters.enable_error_message: true` | Enables reporting of agent error events | Optional | 6. **Send an image** ```ts import { EChatMessageType } from '@/conversational-ai-api/type' await conversationalAIAPI.chat(`${agent_rtc_uid}`, { messageType: EChatMessageType.IMAGE, url: 'https://example.com/image.jpg', uuid: genUUID() }) ``` Info The `chat` completion callback only indicates whether the sending request is successful, and does not reflect the actual processing status of the message. 7. **Receive image sending response** Image sent successfully: ```ts conversationalAIAPI.on(EConversationalAIAPIEvents.MESSAGE_RECEIPT_UPDATED, (agentUserId: string, messageReceipt: TMessageReceipt) => { if (messageReceipt.moduleType !== EModuleType.CONTEXT) { return } try { const receiptMessage = JSON.parse(messageReceipt.message) const uuid = receiptMessage.uuid if (!uuid) return console.log(`Message sent successfully, UUID: ${uuid}`) } catch (error) { console.error('Failed to parse message:', error) } }) ``` Image sending failed: ```ts conversationalAIAPI.on(EConversationalAIAPIEvents.MESSAGE_ERROR, (agentUserId, error) => { console.error(`Message error for agent ${agentUserId}:`, error) if (error.type === EChatMessageType.IMAGE) { try { const errorData = JSON.parse(error.message) if (errorData?.uuid) { console.warn(`Image error for agent ${agentUserId} with UUID: ${errorData.uuid}`) } } catch (e) { console.error(`Failed to handle image error for agent ${agentUserId}:`, e) } } }) ``` 8. **Unsubscribe from the channel** ```ts conversationalAIAPI.unsubscribeMessage(channel_name) ``` 9. **Release resources** ```ts conversationalAIAPI.destroy() ``` ## Reference This section contains supporting information that completes the guidance on this page. Android iOS Web **Folder structure** * `IConversationalAIAPI.kt`: API interface, related data structures, and enumerations * `ConversationalAIAPIImpl.kt`: main implementation logic * `ConversationalAIUtils.kt`: utility functions and callback management * `v3/` * `TranscriptionController.kt`: transcript rendering and synchronization * `MessageParser.kt`: transcription and message parsing **API reference** * [`addHandler`](/en/api-reference/api-ref/conversational-ai/client-toolkit/android#addhandler) * [`subscribeMessage`](/en/api-reference/api-ref/conversational-ai/client-toolkit/android#subscribemessage) * [`unsubscribeMessage`](/en/api-reference/api-ref/conversational-ai/client-toolkit/android#unsubscribemessage) * [`destroy`](/en/api-reference/api-ref/conversational-ai/client-toolkit/android#destroy) * [`onMessageReceiptUpdated`](/en/api-reference/api-ref/conversational-ai/client-toolkit/android#onmessagereceiptupdated) * [`onMessageError`](/en/api-reference/api-ref/conversational-ai/client-toolkit/android#onmessageerror) * [`chat`](/en/api-reference/api-ref/conversational-ai/client-toolkit/android#chat) **Folder structure** * `ConversationalAIAPI.swift`: API interface, data structures, and enumerations * `ConversationalAIAPIImpl.swift`: main implementation logic * `Transcription/` * `TranscriptionController.swift`: transcript rendering and transcription control **API reference** * [`addHandler`](/en/api-reference/api-ref/conversational-ai/client-toolkit/ios#addhandler) * [`subscribeMessage`](/en/api-reference/api-ref/conversational-ai/client-toolkit/ios#subscribemessage) * [`unsubscribeMessage`](/en/api-reference/api-ref/conversational-ai/client-toolkit/ios#unsubscribemessage) * [`destroy`](/en/api-reference/api-ref/conversational-ai/client-toolkit/ios#destroy) * [`onMessageReceiptUpdated`](/en/api-reference/api-ref/conversational-ai/client-toolkit/ios#onmessagereceiptupdated) * [`onMessageError`](/en/api-reference/api-ref/conversational-ai/client-toolkit/ios#onmessageerror) * [`chat`](/en/api-reference/api-ref/conversational-ai/client-toolkit/ios#chat) **Folder structure** * `index.ts`: main API class * `type.ts`: API interfaces, data structures, and enumerations * `utils/` * `index.ts`: general utility functions * `events.ts`: event management * `sub-render.ts`: transcript rendering module **API reference** * [`IConversationalAIAPIEventHandler`](/en/api-reference/api-ref/conversational-ai/client-toolkit/web#iconversationalaiapieventhandlers-interface) * [`EConversationalAIAPIEvents`](/en/api-reference/api-ref/conversational-ai/client-toolkit/web#econversationalaiapievents) * [`subscribeMessage`](/en/api-reference/api-ref/conversational-ai/client-toolkit/web#subscribemessage) * [`unsubscribeMessage`](/en/api-reference/api-ref/conversational-ai/client-toolkit/web#unsubscribe) * [`destroy`](/en/api-reference/api-ref/conversational-ai/client-toolkit/web#destroy) * [`chat`](/en/api-reference/api-ref/conversational-ai/client-toolkit/web#chat) # Start and stop an agent (/en/ai/build/start-stop-agent) This page describes how to call the Conversational AI Engine APIs to start and stop an AI agent. ## Understand the tech A user joins a channel and triggers your business server to start an agent via the Conversational AI Engine API. The agent joins the same channel and communicates with the user through voice, using the specified LLM, TTS service, and Agora’s low-latency Software-Defined Real-Time Network (SDRTN®). When the conversation ends, the business server stops the agent and the user leaves the channel.
Conversational AI Engine workflow ![Conversational AI Engine workflow](https://assets-docs.agora.io/images/conversational-ai/ai-agent-tech.svg)
## Prerequisites Use the [Agora CLI](/en/introduction/agora-cli) to create or select an Agora project, verify project readiness, and export local credentials: ```bash agora login agora project create conv-ai-tutorial agora project use conv-ai-tutorial agora project doctor agora project env --format shell --with-secrets ``` You also need: * A client app that can join an Agora RTC channel, such as the [Voice Calling](../../realtime-media/rtc/voice-quickstart) or [Video Calling](../../realtime-media/rtc/get-started-sdk) quickstart. ## Install the SDK Select your preferred language and install the corresponding SDK. Python SDK TypeScript SDK Go SDK ```sh pip install agora-agents ``` ```sh npm install agora-agents ``` ```sh go get github.com/AgoraIO/agora-agents-go/v2@v2.2.0 ``` ## Implementation This section introduces the basic RESTful API requests you use to start and stop a conversational AI agent. In a production environment, implement these requests on your business server. ### Start a conversational AI agent Call the `join` endpoint to create an agent instance that joins an Agora channel. This example uses [managed mode](/en/ai/build/custom-model-integration/managed-mode) for ASR, LLM, and TTS, so you don't need to supply your own provider API keys. If you want to use your own API key (BYOK) instead, see the vendor pages under Models, for example [Deepgram](/en/ai/models/asr/deepgram). Python SDK TypeScript SDK Go SDK REST API ```python from agora_agent import Agent, Agora, Area, DeepgramSTT, OpenAI, MiniMaxTTS client = Agora( area=Area.US, app_id='your-app-id', app_certificate='your-app-certificate', ) # Omit API keys to use Agora-managed models for ASR, LLM, and TTS. agent = ( Agent(client) .with_stt(DeepgramSTT( model='nova-3', language='en-US', )) .with_llm(OpenAI( model='gpt-4o-mini', system_messages=[{'role': 'system', 'content': 'You are a helpful chatbot.'}], greeting_message='Hello, how can I help you?', failure_message="Sorry, I don't know how to answer this question.", max_history=10, )) .with_tts(MiniMaxTTS( model='speech-2.6-turbo', voice_id='English_captivating_female1', )) ) session = agent.create_session( channel='your_channel_name', agent_uid='0', remote_uids=['1002'], name='unique_name', idle_timeout=120, ) agent_id = session.start() print(f'Agent started: {agent_id}') ``` ```typescript import { Agent, AgoraClient, Area, DeepgramSTT, OpenAI, MiniMaxTTS } from 'agora-agents'; const client = new AgoraClient({ area: Area.US, appId: 'your-app-id', appCertificate: 'your-app-certificate', }); // Omit API keys to use Agora-managed models for ASR, LLM, and TTS. const agent = new Agent({ client }) .withStt(new DeepgramSTT({ model: 'nova-3', language: 'en-US', })) .withLlm(new OpenAI({ model: 'gpt-4o-mini', systemMessages: [{ role: 'system', content: 'You are a helpful chatbot.' }], greetingMessage: 'Hello, how can I help you?', failureMessage: 'Sorry, I don't know how to answer this question.', maxHistory: 10, })) .withTts(new MiniMaxTTS({ model: 'speech-2.6-turbo', voiceId: 'English_captivating_female1', })); const session = agent.createSession({ channel: 'your_channel_name', agentUid: '0', remoteUids: ['1002'], name: 'unique_name', idleTimeout: 120, }); const agentId = await session.start(); console.log('Agent started:', agentId); ``` ```go package main import ( "context" "fmt" "log" Agora "github.com/AgoraIO/agora-agents-go/v2" "github.com/AgoraIO/agora-agents-go/v2/agentkit" "github.com/AgoraIO/agora-agents-go/v2/agentkit/vendors" "github.com/AgoraIO/agora-agents-go/v2/option" ) func main() { ctx := context.Background() idleTimeout := 120 client := agentkit.NewAgoraClient(agentkit.AgoraClientOptions{ Area: option.AreaUS, AppID: "your-app-id", AppCertificate: "your-app-certificate", }) // Omit API keys to use Agora-managed models for ASR, LLM, and TTS. agent := agentkit.NewAgent(client).WithStt( vendors.NewDeepgramSTT(vendors.DeepgramSTTOptions{ Model: "nova-3", Language: "en-US", }), ).WithLlm( vendors.NewOpenAI(vendors.OpenAIOptions{ Model: "gpt-4o-mini", SystemMessages: []map[string]interface{}{ {"role": "system", "content": "You are a helpful chatbot."}, }, GreetingMessage: "Hello, how can I help you?", FailureMessage: "Sorry, I don't know how to answer this question.", MaxHistory: Agora.Int(10), }), ).WithTts( vendors.NewMiniMaxTTS(vendors.MiniMaxTTSOptions{ Model: "speech-2.6-turbo", VoiceID: "English_captivating_female1", }), ) session := agent.CreateSession(agentkit.CreateSessionOptions{ Channel: "your_channel_name", AgentUID: "0", RemoteUIDs: []string{"1002"}, Name: "unique_name", IdleTimeout: &idleTimeout, }) agentID, err := session.Start(ctx) if err != nil { log.Fatal(err) } fmt.Println("Agent started:", agentID) } ``` For the Curl request, pass in the `channel` name and `token` for agent authentication. To generate your base64-encoded credentials, see [RESTful authentication](/en/api-reference/api-ref/conversational-ai/authentication). Set `credential_mode` to `"managed"` in the `asr`, `llm`, and `tts` blocks to use Agora-managed credentials instead of your own API keys. ```bash curl --request POST \ --url https://api.agora.io/api/conversational-ai-agent/v2/projects/:appid/join \ --header 'Authorization: Basic ' \ --header 'Content-Type: application/json' \ --data ' { "name": "unique_name", "properties": { "channel": "", "token": "", "agent_rtc_uid": "0", "remote_rtc_uids": ["1002"], "enable_string_uid": false, "idle_timeout": 120, "asr": { "credential_mode": "managed", "vendor": "deepgram", "params": { "url": "wss://api.deepgram.com/v1/listen", "model": "nova-3", "language": "en-US" } }, "llm": { "credential_mode": "managed", "vendor": "openai", "style": "openai", "url": "https://api.openai.com/v1/chat/completions", "system_messages": [ { "role": "system", "content": "You are a helpful chatbot." } ], "greeting_message": "Hello, how can I help you?", "failure_message": "Sorry, I don't know how to answer this question.", "max_history": 10, "params": { "model": "gpt-4o-mini" } }, "tts": { "credential_mode": "managed", "vendor": "minimax", "params": { "url": "wss://api.minimax.io/ws/v1/t2a_v2", "model": "speech-2.6-turbo", "voice_setting": { "voice_id": "English_captivating_female1" } } } } }' ``` If the request is successful, you receive the following response: ```json // 200 OK { "agent_id": "1NT29X10YHxxxxxWJOXLYHNYB", "create_ts": 1737111452, "status": "RUNNING" } ``` If the request is successful, the agent ID is returned. Store the agent ID to manage the agent in subsequent calls. ### Stop the conversational AI agent To end the conversation with the AI agent, call the `leave` endpoint. This causes the agent to leave the Agora channel. Python SDK TypeScript SDK Go SDK REST API ```python session.stop() ``` ```typescript await session.stop(); ``` ```go if err := session.Stop(ctx); err != nil { log.Fatal(err) } ``` ```bash curl --request POST \ --url https://api.agora.io/api/conversational-ai-agent/v2/projects/:appid/agents/:agentId/leave \ --header 'Authorization: Basic ' ``` If the request is successful, the server responds with a `200 OK` status and an empty JSON object. ```json // 200 OK {} ``` Info The number of Peak Concurrent Users (PCU) allowed to call the server API under a single App ID is limited to 20. If you need to increase this limit, please [contact technical support](mailto\:support@agora.io). ## Reference This section contains content that completes the information on this page, or points you to documentation that explains other aspects of this product. * [Use managed mode](/en/ai/build/custom-model-integration/managed-mode) ### API reference * [Start a conversational AI agent](/en/api-reference/api-ref/conversational-ai/join) * [Stop a conversational AI agent](/en/api-reference/api-ref/conversational-ai/leave) * [Update agent configuration](/en/api-reference/api-ref/conversational-ai/update) * [Query agent status](/en/api-reference/api-ref/conversational-ai/query) * [Retrieve a list of agents](/en/api-reference/api-ref/conversational-ai/list) # Display live transcripts (/en/ai/build/transcripts) When interacting with conversational AI in real time, you can enable live transcripts to display the conversation content. This page explains how to implement live transcripts in your app. ## Understand the tech Agora provides a flexible, scalable, and standardized conversational AI engine toolkit. The toolkit supports **iOS**, **Android**, and **Web** platforms, and encapsulates scenario-based APIs. You can use these APIs to integrate the capabilities of the [Agora Signaling SDK](/en/realtime-media/rtm) and [Agora RTC SDK](../../realtime-media/rtc/get-started-sdk) to enable the following features: * [Interrupt agents](shape-the-conversation/interrupt-agent) * [Display live transcripts](transcripts) * [Client-side events](handle-runtime-events/event-notifications) * [Set optimal audio parameters](../best-practices/audio-setup) for iOS and Android * [Send picture messages](send-multimodal-messages) The toolkit receives transcript content through the `onTranscriptUpdated` callback and supports monitoring the following transcript data types: * **Agent transcript**: The agent's speech, including streaming updates and final results. * **User transcript**: The user's speech, including real-time display and status management. * **Transcript status**: State changes such as in progress, completed, or interrupted. The following diagram outlines the step-by-step process to integrate live transcript functionality into your application:
Transcript rendering workflow ![](https://assets-docs.agora.io/images/conversational-ai/transcript-workflow.svg)
## Prerequisites Before you begin, ensure the following: * You have implemented the Conversational AI Engine [quickstart](../get-started/quickstart). * Your app integrates Agora RTC SDK v4.5.1 or later and includes the [RTC SDK quickstart](../../realtime-media/rtc/get-started-sdk). * You have enabled Signaling in the Agora Console and completed the [Signaling quickstart](/en/realtime-media/rtm/quickstart) for basic messaging. * You maintain active and authenticated RTC and Signaling instances that persist beyond the component lifecycle. The toolkit does not manage RTC or Signaling initialization, lifecycle, or authentication. ## Implementation This section describes how to receive transcript content from the transcript processing module and display it in your app UI. Android iOS Web 1. **Integrate the toolkit** Add the toolkit to your project using Maven or source code. See [Install the Android toolkit](/en/api-reference/api-ref/conversational-ai/client-toolkit/android#installation). Refer to [Folder structure](#reference) to understand the role of each file. 2. **Create a toolkit instance** Create a configuration object with the RTC SDK and Signaling engine instances. Set the transcript rendering mode, then use the configuration to create a toolkit instance. ```kotlin // Create configuration objects for the RTC and RTM instances val config = ConversationalAIAPIConfig( rtcEngine = rtcEngineInstance, rtmClient = rtmClientInstance, // Set the transcript rendering mode. Options: // - TranscriptRenderMode.Word: render transcript word by word. // - TranscriptRenderMode.Text: render the full sentence at once. renderMode = TranscriptRenderMode.Word, enableLog = true ) // Create component instance val api = ConversationalAIAPIImpl(config) ``` 3. **Subscribe to the channel** Transcript data is delivered through Signaling channel messages. To receive transcript data, call `subscribeMessage` before starting the agent session. ```kotlin api.subscribeMessage("channelName") { error -> if (error != null) { // Handle error } } ``` 4. **Receive transcript** Call the `addHandler` method to register your implementation of the transcription callback. ```kotlin api.addHandler(covEventHandler) ``` 5. **Implement UI rendering logic** Inherit your UI module from the `IConversationalAIAPIEventHandler` interface. Implement the `onTranscriptUpdated` method to handle transcript rendering to the UI. ```kotlin private val covEventHandler = object : IConversationalAIAPIEventHandler { override fun onTranscriptUpdated(agentUserId: String, transcript: Transcript) { // Handle transcript data and update the UI here } } ``` 6. **Add a conversational AI agent to the channel** To [start a conversational AI agent](/en/api-reference/api-ref/conversational-ai/join), configure the following parameters in your `POST` request: | Parameter | Description | Required | | --------------------------------------- | -------------------------------------------------- | -------- | | `advanced_features.enable_rtm: true` | Starts the Signaling service | Yes | | `parameters.data_channel: "rtm"` | Enables Signaling as the data transmission channel | Yes | | `parameters.enable_metrics: true` | Enables agent performance data collection | Optional | | `parameters.enable_error_message: true` | Enables reporting of agent error events | Optional | After a successful response, the agent joins the specified RTC channel and is ready to interact with the user. 7. **Unsubscribe from the channel** After an agent session ends, unsubscribe from channel messages to release transcription resources. ```kotlin api.unsubscribeMessage("channelName") { error -> if (error != null) { // Handle the error } } ``` 8. **Release resources** At the end of each call, use the `destroy` method to clean up the cache. ```kotlin api.destroy() ``` 1. **Integrate the toolkit** Add the toolkit to your project using CocoaPods, Swift Package Manager, or source code. See [Install the iOS toolkit](/en/api-reference/api-ref/conversational-ai/client-toolkit/ios#installation). Refer to [Folder structure](#reference) to understand the role of each file. 2. **Create a toolkit instance** Create a configuration object with the RTC SDK and Signaling engine instances. Set the transcript rendering mode, then use the configuration to create a toolkit instance. ```swift // Create a configuration object for the RTC and RTM instances let config = ConversationalAIAPIConfig( rtcEngine: rtcEngine, rtmEngine: rtmEngine, /** * Set the transcript rendering mode. Available options: * - .words: Word-by-word rendering mode. * - .text: Sentence-by-sentence rendering mode. */ renderMode: .words, enableLog: true ) // Create the component instance convoAIAPI = ConversationalAIAPIImpl(config: config) ``` 3. **Subscribe to the channel** Transcript data is delivered through Signaling channel messages. To receive transcript data, call `subscribeMessage` before starting the agent session. ```swift convoAIAPI.subscribeMessage(channelName: channelName) { error in if let error = error { print("Subscription failed: \(error.message)") } else { print("Subscription successful") } } ``` 4. **Receive transcript** Call the `addHandler` method to register and implement the transcript callback. ```swift convoAIAPI.addHandler(handler: self) ``` 5. **Implement UI rendering logic** Implement the `ConversationalAIAPIEventHandler` protocol in your UI module, and use `onTranscriptUpdated` to handle and render transcript updates. ```swift extension ChatViewController: ConversationalAIAPIEventHandler { public func onTranscriptUpdated(agentUserId: String, transcript: Transcript) { // Handle transcript data and update the UI here } } ``` 6. **Add a conversational AI agent to the channel** To [start a conversational AI agent](/en/api-reference/api-ref/conversational-ai/join), configure the following parameters in your `POST` request: | Parameter | Description | Required | | --------------------------------------- | -------------------------------------------------- | -------- | | `advanced_features.enable_rtm: true` | Starts the Signaling service | Yes | | `parameters.data_channel: "rtm"` | Enables Signaling as the data transmission channel | Yes | | `parameters.enable_metrics: true` | Enables agent performance data collection | Optional | | `parameters.enable_error_message: true` | Enables reporting of agent error events | Optional | After a successful response, the agent joins the specified RTC channel and is ready to interact with the user. 7. **Unsubscribe from the channel** After each agent session ends, unsubscribe from channel messages to release transcript-related resources. ```swift convoAIAPI.unsubscribeMessage(channelName: channelName) { error in if let error = error { print("Unsubscription failed: \(error.message)") } else { print("Unsubscribed successfully") } } ``` 8. **Release resources** At the end of each call, use the `destroy` method to clean up the cache. ```swift convoAIAPI.destroy() ``` 1. **Integrate the toolkit** Add the toolkit to your project using a package manager or source code. See [Install the Web toolkit](/en/api-reference/api-ref/conversational-ai/client-toolkit/web#installation). Refer to [Folder structure](#reference) to understand the role of each file. 2. **Create a toolkit instance** Before joining an RTC channel, create RTC SDK and Signaling engine instances and pass them into the toolkit instance. ```ts // Initialize the component ConversationalAIAPI.init({ rtcEngine, rtmEngine, /** * Set the rendering mode for transcript. Available options: * - ETranscriptHelperMode.WORD: render transcript word by word. * - ETranscriptHelperMode.TEXT: render the full transcript at once. */ renderMode: ETranscriptHelperMode.WORD, }) // Get the API instance (singleton) const conversationalAIAPI = ConversationalAIAPI.getInstance() ``` 3. **Set audio parameters** In word-by-word rendering mode, you must receive audio timestamp metadata from RTC to synchronize subtitles with speech. Before creating the client object, configure the following parameter: ```ts AgoraRTC.setParameter('ENABLE_AUDIO_PTS_METADATA', true) const client = AgoraRTC.createClient({ mode: 'rtc', codec: 'vp8' }) ``` 4. **Subscribe to the channel** Agent-related events are delivered through Signaling messages. Before starting an agent session, call `subscribeMessage` to receive these events: ```ts conversationalAIAPI.subscribeMessage(channel_name) ``` 5. **Receive transcript** Register an event listener to receive transcript updates: ```tsx import * as React from 'react' import { type IUserTranscription, type IAgentTranscription, type ISubtitleHelperItem, EConversationalAIAPIEvents, } from '@/conversational-ai-api/type' import { ConversationalAIAPI } from '@/conversational-ai-api' export const ChatHistory = () => { const [chatHistory, setChatHistory] = React.useState< ISubtitleHelperItem>[] >([]) const conversationalAIAPI = ConversationalAIAPI.getInstance() conversationalAIAPI.on( EConversationalAIAPIEvents.TRANSCRIPT_UPDATED, setChatHistory ) return ( <> {chatHistory.map((message) => (
{message.uid}: {message.text}
))} ) } ``` 6. **Add a conversational AI agent to the channel** To [start a conversational AI agent](/en/api-reference/api-ref/conversational-ai/join), configure the following parameters in your `POST` request: | Parameter | Description | Required | | --------------------------------------- | -------------------------------------------------- | -------- | | `advanced_features.enable_rtm: true` | Starts the Signaling service | Yes | | `parameters.data_channel: "rtm"` | Enables Signaling as the data transmission channel | Yes | | `parameters.enable_metrics: true` | Enables agent performance data collection | Optional | | `parameters.enable_error_message: true` | Enables reporting of agent error events | Optional | After a successful response, the agent joins the specified RTC channel and is ready to interact with the user. 7. **Unsubscribe from the channel** After each agent session ends, unsubscribe from channel messages to release resources associated with callback events. ```ts conversationalAIAPI.unsubscribeMessage(channel_name) ``` 8. **Release resources** At the end of each call, use the `destroy` method to clean up the cache. ```ts conversationalAIAPI.destroy() ```
## Reference This section contains supporting information that completes the guidance on this page. Android iOS Web **Folder structure** * `IConversationalAIAPI.kt`: API interface and related data structures and enumerations * `ConversationalAIAPIImpl.kt`: ConversationalAI API main implementation logic * `ConversationalAIUtils.kt`: tool functions and event callback management * `subRender/` * `v3/`: transcription module * `TranscriptionController.kt`: transcription controller * `MessageParser.kt`: message parser **API reference** * [`addHandler`](/en/api-reference/api-ref/conversational-ai/client-toolkit/android#addhandler) * [`subscribeMessage`](/en/api-reference/api-ref/conversational-ai/client-toolkit/android#subscribemessage) * [`unsubscribeMessage`](/en/api-reference/api-ref/conversational-ai/client-toolkit/android#unsubscribemessage) * [`destroy`](/en/api-reference/api-ref/conversational-ai/client-toolkit/android#destroy) * [`onTranscriptUpdated`](/en/api-reference/api-ref/conversational-ai/client-toolkit/android#ontranscriptupdated) **Folder structure** * `ConversationalAIAPI.swift`: API interface and related data structures and enumerations * `ConversationalAIAPIImpl.swift`: ConversationalAI API main implementation logic * `Transcription/` * `TranscriptionController.swift`: transcription controller **API reference** * [`addHandler`](/en/api-reference/api-ref/conversational-ai/client-toolkit/ios#addhandler) * [`subscribeMessage`](/en/api-reference/api-ref/conversational-ai/client-toolkit/ios#subscribemessage) * [`unsubscribeMessage`](/en/api-reference/api-ref/conversational-ai/client-toolkit/ios#unsubscribemessage) * [`destroy`](/en/api-reference/api-ref/conversational-ai/client-toolkit/ios#destroy) * [`onTranscriptUpdated`](/en/api-reference/api-ref/conversational-ai/client-toolkit/ios#ontranscriptupdated) **Folder structure** * `index.ts`: API class * `type.ts`: API interface and related data structures and enumerations * `utils/` * `index.ts`: API utility functions * `events.ts`: event management class * `sub-render.ts`: transcription module **API reference** * [`IConversationalAIAPIEventHandler interface`](/en/api-reference/api-ref/conversational-ai/client-toolkit/web#iconversationalaiapieventhandlers-interface) * [`EConversationalAIAPIEvents`](/en/api-reference/api-ref/conversational-ai/client-toolkit/web#econversationalaiapievents) * [`subscribeMessage`](/en/api-reference/api-ref/conversational-ai/client-toolkit/web#subscribemessage) * [`unsubscribeMessage`](/en/api-reference/api-ref/conversational-ai/client-toolkit/web#unsubscribe) * [`destroy`](/en/api-reference/api-ref/conversational-ai/client-toolkit/web#destroy) # Convo AI Device Kit (/en/ai/device-kit) Convo AI Device Kit R1 is a device-oriented product built on top of Agora's IoT stack and Conversational AI Engine. It packages hardware, connectivity, audio processing, and software resources into a faster path for smart device teams. Use this product space when you need a hardware-first developer workflow instead of the general Conversational AI build path. ## Start here * [Quickstart](start-here/quickstart) ## Build * [Run the R1 demo](build/run-the-r1-demo) * [Run the demo server](build/run-the-demo-server) * [Configure device network](build/configure-device-network) * [Build and flash firmware](build/build-and-flash-firmware) ## Understand the system * [Architecture overview](build/architecture-overview) * [Specifications and compatibility](build/specifications-and-compatibility) ## Plan rollout * [Enable services](reference/enable-services) * [Pricing](reference/pricing) * [Release notes](reference/release-notes) # Integrate with MCP (/en/ai/get-started/mcp-integrate) ## Connect to the Agora Doc MCP Server Choose one of the following ways to connect to the MCP server: #### Trae **Manual configuration:** Add the following configuration manually in Trae under Settings > MCP: ```json title= { "mcpServers": { "shengwang-docs": { "type": "http", "url": "https://mcp.agora.io" } } } ``` #### Claude Code **Add with the `CLI`:** ```bash claude mcp add --transport http shengwang-docs https://mcp.agora.io ``` **Or configure it manually:** Create or edit `.mcp.json` in the project root, then add the following section: ```json { "mcpServers": { "shengwang-docs": { "type": "http", "url": "https://mcp.agora.io" } } } ``` #### Codex **Add with the `CLI`:** ```bash codex mcp add shengwang-docs --url https://mcp.agora.io ``` **Or configure it manually:** Create or edit `.codex/config.toml` in the project root, then add the following section: ```text [mcp_servers.shengwang-docs] url = "https://mcp.agora.io" ``` #### Cursor **Manual configuration:** Create or edit `.cursor/mcp.json` in the project root, then add the following section: ```json { "mcpServers": { "shengwang-docs": { "url": "https://mcp.agora.io" } } } ``` #### Kiro **Manual configuration:** Create or edit `.kiro/settings/mcp.json` in the project root, then add the following section: ```json { "mcpServers": { "shengwang-docs": { "url": "https://mcp.agora.io" } } } ``` #### VS Code & Copilot **Add with the `CLI`:** ```bash code --add-mcp '{"name":"shengwang-docs","type":"http","url":"https://mcp.agora.io"}' ``` **Or configure it manually:** Create or edit `.vscode/mcp.json` in the root directory, then add the following section: ```json { "servers": { "shengwang-docs": { "type": "http", "url": "https://mcp.agora.io" } } } ``` ## MCP tools * `search-docs`: Search for relevant documentation. * `list-docs`: Browse documentation categories and document lists. * `get-doc-content`: Read the full content of a specified document. ## Use MCP tools **Automatically called by the coding agent** In Agent Mode, you can describe your requirement directly and let the coding agent decide when to call MCP tools. For example: ```text Search the relevant Agora documentation, read the document content, and tell me how to build the simplest possible 1v1 real-time audio and video call. ``` **Specify tool calls in the prompt** You can also specify which MCP tools to use in the prompt through the tool-calling syntax supported by your AI IDE. For example: ```text #search-docs #get-doc-content Use search-docs to find relevant Agora documentation, then use get-doc-content to read the full content of the matched document and tell me how to build an interactive experience with an AI agent. ``` # Voice agent quickstart (/en/ai/get-started/quickstart) Set up a working voice agent in under five minutes. This page walks you through installing Agora Skills to give your AI coding assistant the official quickstarts and Agora CLI workflows. You then use the CLI to sign in, clone the official starter, and run it locally. You can follow the CLI steps yourself or paste the sample prompt and let your assistant handle setup for you. If you are new to Agora, read [Core concepts](/en/introduction/core-concepts) first to understand how RTC, Signaling, channels, UIDs, and tokens fit together. ### Install Agora Skills Agora Skills teach your AI coding assistant how to work with Conversational AI projects using official starter repos and the Agora CLI, including signing in, project binding, generating environment files, and running diagnostics. Install the CLI in the next section, or ask your assistant to run the installer for you. ```bash npx skills add AgoraIO/skills ``` Paste the following prompt into your assistant's chat. You can replace Python with TypeScript or Go depending on your language preference: ```text Set up and run the Agora Conversational AI Python starter project locally. ``` Your assistant installs the CLI, signs you in, scaffolds the official starter, and guides you through the remaining steps. Follow the manual CLI steps below if you prefer to run each command yourself. ### Install the Agora CLI The Agora CLI is a native Go binary available at [AgoraIO-Community/cli](https://github.com/AgoraIO-Community/cli). macOS and Linux Windows (PowerShell) ```bash curl -fsSL https://dl.agora.io/cli/install.sh | sh agora --help ``` If the `agora` command is not found after installation, re-run with `--add-to-path` or manually add the install directory to your shell profile. ```powershell irm https://dl.agora.io/cli/install.ps1 | iex agora --help ``` If your execution policy blocks inline scripts, download `install.ps1` and run: ```powershell powershell -ExecutionPolicy Bypass -File .\install.ps1 ``` ## Sign in, scaffold, and run Sign in with the Agora CLI, clone the starter project, and configure it for your chosen language. 1. Sign in to Agora Console. ```bash agora login ``` 2. Use `agora init` to clone the official starter for your chosen template, bind it to your Agora project, and write the runtime-specific environment file. Python TypeScript Go ```bash agora init my-python-demo --template python cd my-python-demo # Set up the web client and Python backend. bun run setup # Start the web client and Python backend. bun run dev ``` ```bash agora init my-nextjs-demo --template nextjs cd my-nextjs-demo pnpm install pnpm dev ``` ```bash agora init my-go-demo --template go cd my-go-demo make setup make dev ``` 3. Open `http://localhost:3000` and click **Start conversation**. If the agent does not join or transcripts do not appear, run `agora project doctor` to check credential validity, feature enablement, and network reachability. Conversational AI Engine is enabled by default for new projects and can't be disabled. If you're using an older project and the agent fails to start, verify that Conversational AI Engine is enabled on the **RTC Services** page in [Agora Console](https://console.agora.io/). ## What just happened The CLI cloned the selected starter, connected it to your Agora project, and wrote the environment configuration required to run the demo locally. The demo runs a local browser app and server-side agent controls: * **Browser app**: Captures microphone audio and plays the agent response. * **Server-side controls**: Generate tokens and start or stop the agent session. Agora provides the real-time path between the user and the agent. The user and agent join the same Agora channel, and Agora transports low-latency audio and session events between them. The configured models handle the speech pipeline: 1. ASR converts user speech to text. 2. The LLM reasons over the conversation and generates a response. 3. TTS converts the response back to speech. ## Next steps ### Build and customize * **[Manage agent sessions](../build/start-stop-agent)**: Start and stop agents with the SDKs or REST API. * **[Customize models](../build/custom-model-integration/managed-mode)**: Use managed mode, or connect your own LLM and TTS providers. * **[Improve conversation quality](../best-practices/optimize-latency)**: Tune latency, audio settings, interruptions, and turn taking. * **[Optimize audio setup](../best-practices/audio-setup)**: Set audio profiles, scenarios, and capture behavior for your target clients. * **[Review the API reference](/en/api-reference/api-ref/conversational-ai)**: Check REST and SDK parameters when you move beyond the starter. ### Observe and troubleshoot * **[Observe runtime behavior](../build/handle-runtime-events/monitor-agent-runtime)**: Monitor events, transcripts, session history, and failures. * **[Debug agent failures](../build/handle-runtime-events/debug-agent-failures)**: Correlate client callbacks, webhook events, and turn-level data. ### Prepare for production * **[Review security and privacy](/en/introduction/security-privacy)**: Check authentication, encryption, geofencing, compliance, and operational safeguards. * **[Restrict where agents run](../best-practices/regional-restrictions)**: Keep agent traffic within the regions your deployment requires. # Integrate with Skills (/en/ai/get-started/skills-integrate) ## Install Agora skills Choose one of the following ways to install the skills: #### Skills CLI Install with the `CLI`: ```bash npx skills add AgoraIO/skills ``` This is the most direct installation method. After the installation finishes, restart the session or refresh the skills list according to the instructions for your coding agent. #### Claude Code Plugin Marketplace Run the following command in `Claude Code`: ```bash plugin marketplace add AgoraIO/skills ``` #### OpenClaw Install through `ClawHub`: ```bash clawhub install voice-ai-integration clawhub update voice-ai-integration ``` Use `install` the first time and `update` for later upgrades. ## Repository * [Agora skills](https://github.com/AgoraIO/skills) ## Best practices After configuring the skills, you can directly describe tasks such as integration, code generation, or troubleshooting in your coding agent. The skills help the agent choose the right integration workflow and generation rules, and supplement responses with the latest Agora documentation, making generated code, configuration guidance, and issue diagnosis more accurate. * Clearly specify the Agora product, target platform, development language, and desired feature in your prompt. * Provide as much context as possible, such as your current tech stack, existing code, expected interaction flow, whether you need a token, and whether you need server-side examples. This helps the AI produce solutions that are easier to use directly. * When troubleshooting, include the observed issue, reproduction steps, logs, error codes, screenshots, or call stack details whenever possible so the coding agent can narrow down the cause faster. * For complex tasks, prefer an AI IDE with stronger model support and more complete tooling for more stable code generation and problem analysis. ## Prompt example After configuring the skills, you can send a prompt like the following to your coding agent to generate an AI voice conversation demo with real-time captions: ```markdown Build me an AI voice conversation demo. After the user opens the web page and clicks "Start conversation", they should be able to talk with the AI by voice, and the page should display real-time captions. ## Tech stack - Frontend: Next.js + React + TypeScript + Tailwind CSS - Backend: FastAPI (Python) ## Requirements 1. Start and end the voice conversation with one click 2. Show real-time captions for both the user and the AI 3. Mute and unmute the microphone 4. Display the AI status (listening, thinking, speaking) 5. Include a system log panel ``` # Notification event types (/en/ai/reference/event-types) After enabling Agora message notifications, the Agora notification server sends channel event notifications to your server through HTTPS POST requests. The data format is `JSON`, the character encoding is `UTF-8`, and the signature algorithm can be either `HMAC/SHA1` or `HMAC/SHA256`. This page explains the types of events returned in channel event callbacks and their meanings. If you are starting from a product goal instead of an event number, begin with [Monitor agent status, errors, and performance](../build/handle-runtime-events/monitor-agent-runtime). For webhook setup, request format, and signature verification, see [Webhooks](../build/handle-runtime-events/webhooks). ## Choose event types by goal | Goal | Use these event types | What they help you answer | | --------------------------------------------------- | ---------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Confirm that a session started or ended | [`101 agent joined`](#101-agent-joined), [`102 agent left`](#102-agent-left) | Did the agent enter the channel? How did the session end? | | Retrieve conversation history after a session | [`103 agent history`](#103-agent-history) | What was said during the session? See [Retrieve conversation history after a session ends](../build/handle-runtime-events/retrieve-session-history). | | Investigate failures and alert on production issues | [`110 agent error`](#110-agent-error), [`102 agent left`](#102-agent-left) | Which module failed? Did the session terminate because of that failure? See [Debug agent failures with runtime events](../build/handle-runtime-events/debug-agent-failures). | | Measure latency and runtime health | [`111 agent metrics`](#111-agent-metrics) | How long do LLM and TTS stages take? Are there regressions? | | Track telephony call progress | [`201 inbound call state`](#201-inbound-call-state), [`202 outbound call state`](#202-outbound-call-state) | Where is the SIP call in its lifecycle? | ## Request header The message notification callback header contains the following fields: Application/json The signature value generated by Agora using the customer key and `HMAC/SHA1` algorithm. Use the customer key and `HMAC/SHA1` algorithm to verify the signature value. See Verify the signature for details. The signature value generated by Agora using the customer key and `HMAC/SHA256` algorithm. Use the customer key and `HMAC/SHA256` algorithm to verify the signature value. See Verify the signature for details. ## Request body The message notification callback request body contains the following fields: Notification ID. Identifies an event notification from the Agora server. Business ID. A value of `17` indicates a Conversational AI Engine notification. The event type of the notification. See [Event types](#event-types) for details. The Unix timestamp (ms) indicating when the Agora message server sent the event notification to your server. This value is updated when the notification is retried. The session ID. The specific content of the notification event. `payload` varies depending on the event type. For details, see [Event types](#event-types). Following is an example of a message notification callback request body: ```json { "sid": "C866467GVJJ54687", "noticeId": "2000001428:4330:107", "productId": 17, "eventType": 101, "notifyMs": 1611566412672, "payload": {} } ``` ## Event types The Agora message notification service notifies the following Conversational AI Engine events: | Event Type | Event name | Event description | | :------------------------------ | :------------------ | :---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | [101](#101-agent-joined) | agent joined | The agent joins the channel. | | [102](#102-agent-left) | agent left | The agent leaves the channel. | | [103](#103-agent-history) | agent history | After an agent stops, this event notifies the stored history, which includes messages exchanged between the user and the agent and timestamps indicating when the agent was created and stopped. The maximum number of entries is determined by the `llm.max_history` parameter you can set when starting the agent. The default value is `32`. | | [104](#104-agent-expire) | agent expire | The agent's RTC token is about to expire. | | [110](#110-agent-error) | agent error | Agent error. | | [111](#111-agent-metrics) | agent metrics | The performance metrics of an agent. | | [112](#112-turns-finished) | turns finished | Batch callback of dialogue round data after the session ends. | | [201](#201-inbound-call-state) | inbound call state | Changes in the status of an incoming call. | | [202](#202-outbound-call-state) | outbound call state | Changes in the status of an outgoing call. | ### 101 agent joined An `eventType` of `101` indicates that an agent has joined a channel. The `payload` contains the following fields: Unique identifier of the agent. The agent name provided when calling [Start a conversational AI agent](/en/api-reference/api-ref/conversational-ai/join). Unique within a channel. Timestamp indicating when the agent was created. The name of the channel the agent was in. Custom labels in key-value pair format. Contains the same labels that were set when starting the agent. #### Payload example ```json { "agent_id": "1NT29X10YHxxxxxWJOXLYHNYB", "name": "my-agent", "start_ts": 1737111452, "channel": "xxxxx", "labels": { "campaign_id": "test_campaign", "customer_group": "vip" } } ``` ### 102 agent left An `eventType` of `102` indicates that an agent has left a channel. The `payload` contains the following fields: Unique identifier of the agent. The agent name provided when calling [Start a conversational AI agent](/en/api-reference/api-ref/conversational-ai/join). Unique within a channel. Timestamp indicating when the agent was created. Timestamp indicating when the agent left the channel. The name of the channel the agent was in. Agent status. The reason why the agent left the channel. Custom labels in key-value pair format. Contains the same labels that were set when starting the agent. #### Payload examples Following are some examples of the `payload` when an agent leaves a channel for different reasons. Agent manually stopped Agent idle timeout Agent error RTC connection error Task lifetime limit exceeded ```json { "agent_id": "1NT29X10YHxxxxxWJOXLYHNYB", "name": "my-agent", "start_ts": 1737111452, "stop_ts": 1737111455, "channel": "xxxxx", "status": "STOPPED", "message": "OK", "labels": { "campaign_id": "test_campaign", "customer_group": "vip" } } ``` ```json { "agent_id": "1NT29X10YHxxxxxWJOXLYHNYB", "name": "my-agent", "start_ts": 1737111452, "stop_ts": 1737111455, "channel": "xxxxx", "status": "STOPPED", "message": "Idle for too long", "labels": { "campaign_id": "test_campaign", "customer_group": "vip" } } ``` ```json { "agent_id": "1NT29X10YHxxxxxWJOXLYHNYB", "name": "my-agent", "start_ts": 1737111452, "stop_ts": 1737111455, "channel": "xxxxx", "status": "FAILED", "message": "Connecting for too long", "labels": { "campaign_id": "test_campaign", "customer_group": "vip" } } ``` ```json { "agent_id": "1NT29X10YHxxxxxWJOXLYHNYB", "name": "my-agent", "start_ts": 1737111452, "stop_ts": 1737111455, "channel": "xxxxx", "status": "FAILED", "message": "RTC connection error", "labels": { "campaign_id": "test_campaign", "customer_group": "vip" } } ``` ```json { "agent_id": "1NT29X10YHxxxxxWJOXLYHNYB", "name": "my-agent", "start_ts": 1737111452, "stop_ts": 1737111455, "channel": "xxxxx", "status": "STOPPED", "message": "Task lifetime limit exceeded", "labels": { "campaign_id": "test_campaign", "customer_group": "vip" } } ``` ### 103 agent history An `eventType` of `103` notifies the history of a user and agent dialogue. The notification `payload` contains the following fields: Unique identifier of the agent. The agent name provided when calling [Start a conversational AI agent](/en/api-reference/api-ref/conversational-ai/join). Unique within a channel. The name of the channel the agent was in. The timestamp indicating when the agent started and joined the channel. The timestamp indicating when the agent stopped and left the channel. Agent history. The message sender. * `user`: User * `assistant`: AI agent Message content. Unix timestamp in milliseconds indicating when the user started speaking or the agent started TTS playback. Only returned when `llm.vendor` is `custom`. Unix timestamp in milliseconds indicating when the user stopped speaking, or when TTS playback completed or was interrupted. Only returned when `llm.vendor` is `custom`. The total delay in milliseconds introduced by audio processing algorithms, including noise reduction, background voice suppression, and voiceprint locking, after audio is captured from the user's microphone. Use this value to align timestamps with cloud recording audio. Only returned when: * `llm.vendor` is `custom` * `contents[].role` is `user` * Actual voice input is present Custom labels in key-value pair format. Contains the same labels that were set when starting the agent. `speech_start_ms`, `speech_end_ms`, and `speech_algorithmic_delay` are returned only in `103 agent history` events for sessions that use `llm.vendor` set to `custom`. #### Payload examples Basic Custom LLM ```json { "agent_id": "xxxx", "name": "my-agent", "channel": "xxxx", "start_ts": 123, "stop_ts": 123, "contents": [ { "role": "user", "content": "hello." }, { "role": "assistant", "content": "hi, how can I help you?" } ], "labels": { "campaign_id": "test_campaign", "customer_group": "vip" } } ``` ```json { "agent_id": "xxxx", "name": "my-agent", "channel": "xxxx", "start_ts": 123, "stop_ts": 123, "contents": [ { "role": "user", "content": "Hello, I need help with my recent order.", "speech_start_ms": 1715000001200, "speech_end_ms": 1715000003400, "speech_algorithmic_delay": 120 }, { "role": "assistant", "content": "Sure, could you please provide your order number?", "speech_start_ms": 1715000004100, "speech_end_ms": 1715000005000 }, { "role": "user", "content": "I'd like to check the status of my package, tracking number 12345.", "speech_start_ms": 1715000005200, "speech_end_ms": 1715000007600, "speech_algorithmic_delay": 120 }, { "role": "assistant", "content": "Your package with tracking number 12345 is currently out for delivery.", "speech_start_ms": 1715000009500, "speech_end_ms": 1715000013200 }, { "role": "user", "content": "[think API injected] User level: VIP" } ], "labels": { "campaign_id": "test_campaign", "customer_group": "vip" } } ``` ### 104 agent expire An `eventType` of `104` indicates that the agent's RTC token is about to expire. The `payload` contains the following fields: Unique identifier of the agent. The agent name provided when calling [Start a conversational AI agent](/en/api-reference/api-ref/conversational-ai/join). Unique within a channel. Timestamp indicating when the agent was created. The name of the channel the agent is in. Token expiry warning message. Fixed value: `Task exceeded maximum lifetime`. Info This event is a warning only. The agent does not exit the channel or interrupt the session when this event is triggered. Upon receiving this event, call the [Update agent](/en/api-reference/api-ref/conversational-ai/update) API with a new `properties.token` to refresh the token before it expires. #### Payload example ```json { "agent_id": "1NT29X10YHxxxxxWJOXLYHNYB", "name": "my-agent", "start_ts": 1737111452, "channel": "xxxxx", "message": "Task exceeded maximum lifetime" } ``` ### 110 agent error An `eventType` of `110` indicates that an agent has encountered an error. The `payload` contains the following fields: Unique identifier of the agent. The agent name provided when calling [Start a conversational AI agent](/en/api-reference/api-ref/conversational-ai/join). Unique within a channel. Timestamp indicating when the agent was created. The name of the channel the agent was in. Transcript conversation turn. For details, see [Display live transcripts](../build/transcripts#reference). An array of error messages. Each object contains the following fields: The module where the error occurred. Transcript dialogue turn. Error code. Refer to the error code document of the vendor corresponding to the error module for detailed information. The error message. Custom labels in key-value pair format. Contains the same labels that were set when starting the agent. #### Payload examples LLM module error Greeting audio URL error {/* SIP module error */} ```json { "agent_id": "1NT29X10YHxxxxxWJOXLYHNYB", "name": "my-agent", "start_ts": 1737111452, "channel": "xxxxx", "turn_id": 2, "errors": [{"module": "llm", "turn_id":2, "code":503, "message": "Insufficient balance."}], "labels": { "campaign_id": "test_campaign", "customer_group": "vip" } } ``` When the greeting audio file fails to download, times out, is in an unsupported format, or fails to decode, the system reports the error via a `110 agent error` event and automatically falls back to TTS synthesis using `greeting_message`. In these errors, `errors[].module` is `agent`. ```json { "agent_id": "1NT29X10YHxxxxxWJOXLYHNYB", "name": "outbound_voice_agent", "start_ts": 1737111452, "channel": "outbound_call_channel_001", "turn_id": 0, "errors": [ { "module": "agent", "turn_id": 0, "code": 4001, "message": "Greeting audio URL unreachable, fallback to TTS. audio_url: https://cdn.example.com/audio/welcome.mp3" } ], "labels": {} } ``` {/* ```json { "agent_id": "1NT29X10YHxxxxxWJOXLYHNYB", "name": "my-agent", "start_ts": 1737111452, "channel": "xxxxx", "turn_id": "", "errors": [{"module": "sip", "turn_id": "", "code":50, "message": "Insufficient balance."}], "labels": { "campaign_id": "test_campaign", "customer_group": "vip" } } ``` */} Info If `parameters.enable_error_message` is set to `true` when starting the agent, greeting audio URL errors are also delivered to the client via the `onMessageError` RTM callback, in addition to the server-side webhook event. ### 111 agent metrics An `eventType` of `111` notifies the performance metrics of an agent. The `payload` contains the following fields: Unique identifier of the agent. The agent name provided when calling [Start a conversational AI agent](/en/api-reference/api-ref/conversational-ai/join). Unique within a channel. Timestamp indicating when the agent was created and joined the channel. Timestamp indicating when the agent stopped and left the channel. The name of the channel where the agent is located. An array of performance metrics. Each object contains the following fields: Dialogue round ID. The ttlw (Time To Last Word) metric of the ASR module. Represents the delay (in milliseconds) from when the user finishes speaking until the last word is output by the ASR. The ttfb (Time To First Byte) metric for the LLM module. Represents the latency (in milliseconds) from the start of an LLM request to the receipt of the first byte of the response. The ttfs (Time To First Sentence) metric for the LLM module. Represents the time (in milliseconds) elapsed from the start of an LLM request to the receipt of the first complete sentence response. The ttfb (Time To First Byte) metric for the TTS module. Represents the latency (in milliseconds) from the start of a TTS request to the receipt of the first byte of the response. Custom labels in key-value pair format. Contains the same labels that were set when starting the agent. #### Payload example ```json { "agent_id": "A42AC47Hxxxxxxxx4PK27ND25E", "name": "my-agent", "start_ts": 1000, "stop_ts": 1672531200, "channel": "test-channel", "metrics": [ { "turn_id": 1, "asr_ttlw": 503, "llm_ttfb": 1104, "tts_ttfb": 85 }, { "turn_id": 2, "asr_ttlw": 2385, "llm_ttfb": 980, "tts_ttfb": 78 } ], "labels": { "campaign_id": "test_campaign", "customer_group": "vip" } } ``` ### 112 turns finished An `eventType` of `112` indicates a batch callback of conversation turn data after the session ends. This event returns the conversation turns for the current session in a single callback, making it convenient for post-session analysis, archiving, or offline processing. When a session has more than 200 turns, the callback data may be truncated. Use the `is_truncated` field to check whether the current callback contains the complete turn data. The `payload` contains the following fields: Unique identifier of the agent. The agent name provided when calling [Start a conversational AI agent](/en/api-reference/api-ref/conversational-ai/join). Unique within a channel. The name of the channel where the agent is located. Timestamp indicating when the agent was created and joined the channel. Timestamp indicating when the agent stopped and left the channel. The total number of conversation turns in the session. Whether the turn data in the current callback is truncated. A value of `true` indicates that the current callback did not return all turns. Custom labels in key-value pair format. Contains the same labels that were set when starting the agent. Only returned if labels were passed in when creating the agent. A list of conversation turns for the session. Unique identifier of the agent. The name of the channel where the agent is located. The sequential index of the turn within the session. Starts at `1`. Details about the start of the turn. The Unix timestamp in milliseconds (UTC time) when the turn started. The type of event that initiated the turn. Additional context about the turn start event. Included fields depend on the value of the `type` field. The duration of the user's voice input in milliseconds. Included only when `type` is `voice_input`. The minimum voice duration in milliseconds required to trigger an interruption. Included only when `type` is `voice_input`. Details about the end of the turn. The Unix timestamp in milliseconds (UTC time) when the turn ended. The type of event that ended the turn. Additional context about the turn end event. Latency metrics for the turn. The end-to-end latency in milliseconds for the turn. Represents the time from when the user finishes speaking to when the agent starts speaking. A breakdown of latency by segment. Algorithm processing delay in milliseconds. The ASR Time To Last Word (TTLW) in milliseconds. Represents the delay from when the user finishes speaking to when the ASR module outputs the last word. The LLM Time To First Token (TTFT) in milliseconds. Represents the delay from when the LLM receives the request to when it outputs the first token. The LLM First Token To First Sentence (FTFS) in milliseconds. Represents the delay from when the LLM outputs the first token to when it outputs the first complete sentence. The TTS Time To First Byte (TTFB) in milliseconds. Represents the delay from when the TTS module receives a text request to when it outputs the first audio byte. Network transmission delay in milliseconds. Not returned when the user is connected using the RTC Web SDK. #### Payload example ```json { "agent_id": "xxxx", "name": "support_agent_001", "channel": "xxxx", "start_ts": 1715000000000, "stop_ts": 1715000015000, "total_turn_count": 250, "is_truncated": true, "labels": { "campaign_id": "test_campaign", "customer_group": "vip" }, "turns": [ { "agent_id": "xxxx", "channel": "xxxx", "turn_id": 1, "start": { "start_at": 1774579820147, "type": "voice_input" }, "end": { "end_at": 1774579822412, "type": "ok" }, "metrics": { "e2e_latency_ms": 1500, "segmented_latency_ms": { "algorithm_processing": 300, "asr_ttlw": 200, "llm_ttft": 600, "tts_ttfb": 200, "transport": 300 } } } ] } ``` ### 201 inbound call state An `eventType` of `201` notifies changes in the status of an incoming call. The `payload` contains the following fields: Unique identifier of the agent. The agent name provided when calling [Start a conversational AI agent](/en/api-reference/api-ref/conversational-ai/join). Unique within a channel. The name of the channel where the agent is located. Incoming call status: * `START`: Call received * `ANSWERED`: Call answered * `TRANSFERED`: Transfer to human operator * `HANGUP`: Call hung up Timestamp (in milliseconds) of the state change. Custom labels in key-value pair format. Contains the same labels that were set when starting the agent. #### Payload example ```json { "agent_id": "1NT29X10YHxxxxxWJOXLYHNYB", "name": "my-agent", "channel": "xxxxx", "state": "START", "report_ms": 1737111452000, "labels": { "campaign_id": "test_campaign", "customer_group": "vip" } } ``` ### 202 outbound call state An `eventType` of `202` notifies changes in the status of an outgoing call. The `payload` contains the following fields: Unique identifier of the agent. The agent name provided when calling [Start a conversational AI agent](/en/api-reference/api-ref/conversational-ai/join). Unique within a channel. The name of the channel where the agent is located. Outgoing call status: * `START`: Call begins * `CALLING`: Making a phone call * `RINGING`: The telephone is ringing * `ANSWERED`: The call was connected * `HANGUP`: Call hung up Timestamp (in milliseconds) of the state change. Custom labels in key-value pair format. Contains the same labels that were set when starting the agent. #### Payload example ```json { "agent_id": "1NT29X10YHxxxxxWJOXLYHNYB", "name": "my-agent", "channel": "xxxxx", "state": "CALLING", "report_ms": 1737111452000, "labels": { "campaign_id": "test_campaign", "customer_group": "vip" } } ``` # OpenAI Realtime integration (/en/ai/reference/openai-realtime-integration) This guide explains how to combine Agora's real-time audio transport with the OpenAI Realtime API. Use this integration when you want Agora to handle the live audio session and network transport, while OpenAI Realtime handles the model-side conversational behavior. This is not the same as building on Agora's managed Conversational AI Engine. If you want Agora to manage the voice-agent runtime, use the main Conversational AI build path instead. ## When to use this path * You already know you want to build around the OpenAI Realtime API. * You need Agora RTC SDK transport in front of that model stack. * You are building a browser demo, device-oriented flow, or backend-mediated prototype around OpenAI Realtime. ## When not to use this path * You want the default Agora-managed voice-agent stack. * You are looking for the main onboarding path for Conversational AI Engine. * You want Agent Studio or the Device Kit route instead of a custom integration. ## Core pieces * **Agora App ID**: Identifies your Agora project. * **App certificate and token model**: Used for secure join flows where needed. * **Channel and user ID**: Define the live session and participants. * **SD-RTN**: Provides Agora's low-latency transport layer. * **OpenAI Realtime API**: Provides the model-side real-time conversation flow. ## Recommended quickstart flow 1. Prepare your Agora account and project. 2. Decide how your client joins the Agora session and how the OpenAI-side session is coordinated. 3. Configure the OpenAI Realtime model and audio flow. 4. Run the round-trip voice path locally and validate latency, auth, and media behavior. ## Security and network checks * Keep Agora credentials and provider API keys on the server side. * Separate local, staging, and production credentials. * Review firewall requirements before testing in restricted networks. * Re-test the full join and media path after changing transport or security settings. ## Related pages * [Conversational AI quickstart](/en/ai/get-started/quickstart) * [MCP integration](/en/ai/get-started/mcp-integrate) * [Agora skills](/en/ai/get-started/skills-integrate) * [Security and privacy](/en/introduction/security-privacy) * [Firewall requirements](/en/introduction/firewall) * [Glossary](/en/introduction/glossary) # Pricing (/en/ai/reference/pricing) This page describes how Agora calculates and bills for Conversational AI Engine usage. When you use Conversational AI Engine in your project, Agora charges a monthly fee based on your usage across all projects under your developer account. At the end of each month, the free quota is subtracted from your total usage, and the remaining minutes are multiplied by the unit price to calculate your bill, rounded to two decimal places. For general billing information, see [Billing](../../introduction/billing). Info If you have signed a sales contract with Agora, your billing follows the terms in your contract. ## Unit price Creating a Conversational AI Engine instance using the REST API and joining a channel incurs an audio task fee at the following rate: | Usage Type | Pricing (USD / minute) | Free Minutes | | ----------------------------------- | ---------------------: | -------------------------- | | Conversational AI Engine Audio Task | 0.10\* | First 300 minutes are free | \* The unit price includes usage of selected ASR, LLM, and TTS models. You are charged the same price even if you bring your own key (BYOK). Usage of ASR, LLM, and TTS providers is included in the unit price when using an Agora managed key. Agora provides and manages the API keys for the following providers: * **ASR** * ARES * Deepgram nova-2 * Deepgram nova-3 * **LLM** * OpenAI GPT-4o-mini * OpenAI GPT-4.1-mini * OpenAI GPT-5-nano * OpenAI GPT-5-mini * **TTS** * MiniMax 2.6 Turbo * MiniMax 2.8 Turbo * OpenAI TTS-1 ## Examples The following examples demonstrate how billing is calculated for different Conversational AI Engine configurations. ### Using ASR, LLM, and TTS managed by Agora User A joins a channel and starts a voice conversation with an instance created by Conversational AI Engine. The interaction lasts for 10 minutes. User A and the Conversational AI Engine instance exit the channel at the same time. Agora calculates the cost for this session as follows: | Usage Type | Duration (minutes) | Unit Price | Service Cost (USD) | | ---------------------------------------------------------- | -----------------: | ---------: | -----------------: | | User A: Audio RTC | 10 | 0.00099 | 0.0099 | | Conversational AI Engine Audio Task | 10 | 0.10 | 1.00 | | ASR: ARES, LLM: OpenAI GPT-4o-mini, TTS: MiniMax 2.8 Turbo | 10 | 0.00 | 0.00 | Total cost: 1.0099 USD # Overview (/en/ai/studio) The no-code agent builder is a visual workspace in Agora Console for building and operating conversational AI voice agents. You configure agent behavior, connect telephony, and monitor production calls from a single interface, without writing integration code. It abstracts the underlying AI stack. You configure your ASR, LLM, and TTS providers, or a single realtime MLLM; the console handles the orchestration between them. Instead of managing API integrations across multiple services, you focus on what your agent should say and do. ## How it's organized The no-code agent builder is structured around four areas: **Concierge:** Explore, configure, and troubleshoot your agents using natural language, instead of navigating each page individually. See [Concierge](build/concierge). **Build:** Create and configure agents. Define the system prompt, greeting, models, and behavior. Connect credentials, knowledge bases, MCP servers, Custom Tools, and connectors through **Integrations**. These resources are stored centrally so they can be reused across multiple agents. See [Customize your agent](build/customize-agent), [Manage integrations](build/integrations), [Custom Tools](build/custom-tools), and [HubSpot Connector](build/hubspot-connector). **Deploy:** Connect agents to telephony. Import phone numbers using Elastic SIP Trunk and configure inbound routing or outbound campaigns. See [Set up SIP trunk](deploy/sip-trunk), [Import a phone number](deploy/import), [Handle inbound calls](deploy/inbound), [Publish your agent](deploy/deploy-agent), and [Set up a campaign](deploy/campaign). **Insights:** Monitor production usage. Review call history, transcripts, and analytics to understand agent performance and debug issues. See [Agent analytics](observe/analytics) and [Call History](observe/call-history). ## How it works The no-code agent builder sits between your telephony provider and the AI models that power your agent. For inbound calls, it receives the call from your carrier via Elastic SIP Trunk, processes the conversation through your configured ASR, LLM, and TTS providers (or a realtime MLLM), and delivers the agent's response back to the caller. For outbound calls, it initiates the call through the same SIP trunk, connects to the recipient, and handles the conversation in the same way. In both cases, it manages the full call lifecycle. No SIP servers to manage, no voice recognition to train, no telephony protocols to debug. ## Key concepts ### Agents Agents are the primary objects you configure. Create an agent, then configure it across four tabs in the agent editor: * **Prompt**: The system prompt, greeting message, and failure message that define your agent's identity and behavior. * **Models**: Your ASR, LLM, and TTS providers, or a single realtime MLLM. * **Advanced**: Turn detection, speech detection, selective attention locking, filler words, voice format, and conversation history. * **Actions**: Knowledge bases, Custom Tools, connectors, and MCP servers attached to the agent. After configuring, test the agent using the **Test** tab, then publish it when ready. See [Customize your agent](build/customize-agent) and [Test your agent](build/test-agent). ### Integrations **Integrations** is a centralized resource library. Store credentials, knowledge bases, MCP servers, Custom Tools, and connectors here once, then attach them to any agent. This eliminates repeated setup when you create new agents or update provider settings. See [Manage integrations](build/integrations). ### Phone numbers Import phone numbers from your SIP trunk provider and assign them to agents for inbound calls, or use them as caller IDs for outbound campaigns. See [Import a phone number](deploy/import). ### Campaigns For outbound use cases, create campaigns with contact lists, scheduling, and call transfer settings. See [Set up a campaign](deploy/campaign). ## Who should use this The no-code agent builder is designed for developers and technical users who want to build and operate voice agents without managing the underlying AI infrastructure. If you need capabilities beyond what it exposes, such as advanced API parameters or custom orchestration logic, you can use agent SDKs or the [Conversational AI REST API](/en/api-reference/api-ref/conversational-ai) for full programmatic control. ## Get started To get started, follow the [Quickstart](quickstart), which walks you through creating and testing your first agent in about 10 minutes using a pre-built template. # Create your first agent (/en/ai/studio/quickstart) This page guides you through creating and testing your first AI voice agent. In less than 10 minutes, you will have a working agent that you can speak to in the browser. No telephony setup is required. Prefer natural language? Use [Concierge](build/concierge) to build and test an agent by describing what you want instead of following the steps below. ## Create an agent To create your first agent: 1. Log in to [Agora Console](https://console.agora.io). 2. Select your project from the dropdown at the top of the page. The agent uses this project's credentials to connect to Agora. Agora bills the usage to this project. 3. In the navigation sidebar, select **Agents**. ![List of agents](https://assets-docs.agora.io/images/console/agent-list.png) 4. Select **Create Agent**. Create a new agent 5. Enter a name for your agent. 6. Under **Choose a template**, select the **Blank Template**. 7. Select **Create Agent**. The agent editor opens with a system prompt and pre-configured model settings. ![](https://assets-docs.agora.io/images/console/agent-editor-prompt.png) ## Configure your agent The agent editor is organized into multiple tabs. For the quickstart, you only need to review the **Prompt** tab and the **Models** tab. ### Review the prompt The agent editor opens with the following prompts: * **System prompt**: The base instructions that shape how your agent thinks, speaks, and escalates. * **Greeting**: The first assistant message before the live exchange starts. Type in a greeting such as `Hello, how can I help you today?` * **Failure message**: The fallback message used when the agent needs a moment or cannot proceed. ### Review selected models Models determine which ASR, LLM, and TTS services your agent uses to process speech, generate responses, and synthesize voice output. 1. Select the **Models** tab. ![](https://assets-docs.agora.io/images/console/agent-editor-models.png) 2. Review the pre-selected models for **Automatic Speech Recognition (ASR)**, **Large Language Model (LLM)**, and **Text-to-Speech (TTS)**. Use the default values to create your first agent. The default models support **Agora Managed Key**. When you use a managed key, Agora provides the API credentials for your selected vendor. You do not need to obtain an API key directly from the vendor. See [Pricing](../reference/pricing) for details. To use your own API key or use a vendor that does not support a managed key, see [Customize your agent](build/customize-agent). ## Test the agent 1. Select the **Test** tab in the right panel. 2. Select **Start Call** to start a test call. 3. Allow microphone access when prompted. 4. Speak to the agent to verify it responds correctly. For guidance on what to test and how to troubleshoot issues, see [Test your agent](build/test-agent). ## Next steps You now have a working voice agent. From here you can: * [Customize your agent](build/customize-agent): Fully configure models, prompts, and advanced settings for your use case * [Manage integrations](build/integrations): Manage API credentials, knowledge bases, MCPs, custom tools, and connectors for reuse across agents # TEN Framework core concepts (/en/ai/ten-agent/core-concepts) ## TEN runtime The TEN runtime is an execution environment designed for running applications within the TEN Framework. It provides a flexible architecture that supports extensions developed in multiple programming languages and handles various types of data streams. By managing the lifecycle of extensions, data flows, and threads, the TEN runtime offers developers a powerful tool for building a wide range of applications and services. ![TEN Framework](https://assets-docs.agora.io/images/ten-framework/ten-architecture.png) ## App An app in the TEN Framework can operate as a standalone process or as a thread within an existing process. This flexibility allows for versatile deployment options depending on the needs of the application. ## Extension group An extension group is a mechanism that designates a specific thread for execution. All extensions within a given group are executed on the same thread, ensuring consistent and synchronized processing. ## Extension An extension is the fundamental building block of the TEN Framework. Developers can create extensions in various programming languages and combine them to build different applications and scenarios. The framework's design supports cross-language collaboration, enabling extensions written in different languages to work together seamlessly within the same application or service. For example, a developer might use C++ to create an extension for real-time communication (RTC) due to its performance benefits in handling audio and video data, while developing an AI extension in Python to take advantage of its robust libraries for data analysis and machine learning. These extensions can then be integrated into a single application, leveraging the strengths of each language. ## Graph A graph in the TEN Framework describes the data flow between extensions. It orchestrates how data moves from one extension to another, defining the participants and the flow of data between them. For example, you can route the output of a speech-to-text (STT) extension to a large language model (LLM) extension for further processing. The TEN Framework supports four main types of data flows between extensions: * Command * Data * Video frame * Audio frame By defining these data flows within a graph, you can create inter-extension communication and unidirectional data streams, particularly useful for handling audio and video data. ## Component hierarchy The TEN Framework uses a nested structure where apps contain graphs, graphs contain extension groups, and extension groups contain extensions. Understanding this hierarchy is essential for building and configuring TEN applications. ![Hierarchical relationship of concepts](https://assets-docs.agora.io/images/ten-framework/hierarchical_relationship_of_concepts.png) * **App**: An app can execute multiple graphs, which can be either statically predefined or dynamically assembled. * **Graph**: A graph is formed by multiple extensions working together to create a meaningful scenario. Each graph instance operates as a session within the app. * **Extension group**: The concept of an extension group is analogous to a thread. Extensions written in the same language and within the same extension group run on the same thread during runtime. You do not need to manage threads directly; simply declare the group to which each extension belongs. * **Extension**: Each extension within the framework is assigned a unique ID, structured as: `app-uri/graph-name/group-name/extension-name` ## TEN cloud store The TEN cloud store functions similarly to Google Play Store or Apple's App Store, providing a marketplace for extensions. You can share your own extensions or download those created by others. Integrate these extensions into TEN apps to facilitate development and expand functionality. ![TEN cloud store](https://assets-docs.agora.io/images/ten-framework/ten_cloud_store.png) ## TEN manager The TEN manager is a tool that simplifies the management of extensions. It handles tasks such as uploading, sharing, and installing extensions, automatically managing dependencies between them and their environment. This makes the installation and publication of extensions convenient and efficient, and streamlines the development process within the TEN Framework. # TEN Framework overview (/en/ai/ten-agent/framework-overview) The TEN Framework, or Transformative Extensions Network, is an open-source framework for building real-time multimodal AI agents. It helps you create applications that process voice, video, data streams, images, and text at the same time. TEN is designed for agents that need to think, listen, see, and respond in real time. It provides a graph-based runtime, reusable extensions, and multi-language development support so you can assemble complex AI systems without rebuilding every capability from scratch. ## What you can build Use the TEN Framework to build real-time AI applications such as: * Voice and video AI agents * Simultaneous interpretation systems * Speech-to-text and text-to-speech pipelines * Multilingual communication experiences * Audio-visual assistants and virtual companions * AI-generated meeting minutes * Language tutoring and coaching applications ## Key capabilities ### Real-time multimodal interactions TEN optimizes interaction between extensions to support low-latency AI applications. A graph can process multiple data types, including audio, video, commands, structured data, images, and text. ### Multi-language and cross-platform development You can build modular extensions in C++, Go, and Python. TEN applications can run across Windows, macOS, Linux, and mobile environments, which gives you flexibility when choosing where each capability should execute. ### Modular extension architecture TEN applications are composed from extensions. Each extension owns a focused capability, such as audio capture, ASR, LLM processing, TTS, tool calling, or output rendering. This modular architecture makes it easier to prototype, replace providers, and add new capabilities over time. ### Edge-cloud integration TEN supports applications that combine edge and cloud components. You can run latency-sensitive or privacy-sensitive work near the user while using cloud-based models for heavier reasoning or generation tasks. ### Flexible orchestration beyond model limits TEN lets you combine AI models with databases, retrieval systems, monitoring tools, and external services. Instead of depending on one model to handle the whole workflow, you can orchestrate multiple specialized components through the graph. ### Real-time agent state management TEN graphs can manage changing agent state so applications can adapt behavior during a session. This is useful for interactive agents that need to respond to user context, media state, or tool results while the conversation is still running. ## How it relates to TEN Agent TEN Agent is built on the TEN Framework. The framework provides the runtime, graph model, extension system, schema system, and package mechanics. TEN Agent applies those framework capabilities to conversational AI workflows. Start with [Project overview](./project-overview) when you want to understand the TEN Agent repository layout, runtime configuration, extension folders, and local server API. For implementation details, see: * [Subgraphs](./architecture/subgraphs) * [Schema system](./architecture/schema-system) * [Type system](./architecture/type-system) * [Development workflow](./develop/development-workflow) ## Samples Explore TEN Framework examples in the [TEN Framework repository](https://github.com/TEN-framework/ten-framework/tree/main/ai_agents/agents/examples). # Project overview (/en/ai/ten-agent/project-overview) The TEN Agent project is built on the TEN Framework and follows a modular architecture. This page describes the project structure, configuration system, and web server API. TEN Agent uses a graph-based configuration system built on TEN Framework concepts. The project organizes code into modular extensions that can be orchestrated through configuration files without modifying source code. The folder structure follows this pattern: ```text ├── agents/ │ └── ten_packages/ │ ├── extension/ │ │ ├── openai_chatgpt_python/ │ │ │ ├── extension.py │ │ │ ├── .. │ │ │ └── requirements.txt │ │ ├── elevenlabs_tts │ │ └── .. │ └── property.json ├── playground ├── demo └── server ``` It contains the following important folders and files: * `property.json`: This file contains the orchestration of extensions. It is the main runtime configuration file. * `ten_packages/extension`: This folder contains the extension modules. Each extension module is a separate Python/Golang/C++ package. * `server`: This folder contains the web server code. It is responsible for handling the incoming requests and start/stop of agent processes. * `playground`: This folder contains the UI code for the playground. It is a web-based interface to interact with the agent. ## Runtime configuration The main runtime configuration file is `property.json`. It contains the orchestration of extensions. The file is structured as follows: ```json "predefined_graphs": [ { "name": "va_openai_azure_fashionai", "auto_start": false, "connections": [ // ... ], "nodes": [ // ... ] }, // ... ] ``` The file contains the following orchestration info: * **Graphs**: Collections of nodes and connections that determine agent behavior * **Nodes**: Individual extension instances with specific configurations * **Connections**: Data flow paths between nodes ```json { "predefined_graphs": [ { "name": "va_openai_azure_fashionai", "auto_start": false, "connections": [...], "nodes": [...] } ] } ``` ### Graphs The `predefined_graphs` property contains a list of available graphs. Each graph defines how the agent behaves in a specific scenario. Each graph contains: * `name`: Unique identifier for the graph * `auto_start`: Whether to start this graph automatically * `nodes`: List of extension instances * `connections`: Data routing between nodes ### Nodes The `nodes` section contains the list of extensions that are part of the graph. Each node includes the following: * `name`: Unique identifier of a node within the graph * `addon`: Specifies the extension module to use. You can create multiple instances of the same extension within a graph. For example, you might include several `chatgpt_openai_python` nodes, each using the same `addon` property but with unique `name` properties. This allows you to run parallel instances of an extension with different configurations or behaviors. * `property`: Extension-specific configuration ```json { "name": "chatgpt_openai_python", "addon": "chatgpt_openai_python", "property": { "api_key": "${env:OPENAI_API_KEY|}", "model": "gpt-3.5-turbo", "temperature": 0.5, "max_tokens": 100, "prompt": "You are a helpful assistant" } } ``` #### Node properties The `property` section of a node configures the extension's behavior. Each extension defines its available properties in the `manifest.json` file within its folder. You can customize an extension by setting these runtime properties. #### Read environment variables Many extensions require API keys to function. Instead of hardcoding sensitive keys in `property.json`, use environment variables with the syntax `${env:|}`. The following example shows how to read and use the `OPENAI_API_KEY` environment variable: ```json { "name": "chatgpt_openai_python", "addon": "chatgpt_openai_python", "property": { "api_key": "${env:OPENAI_API_KEY|}" } } ``` ### Connections The connections section defines how data flows between nodes. Each connection specifies: * **Source node**: Identified by `extension_group` and `extension` properties * **Data protocols**: Types of data that can flow: `audio_frame`, `video_frame`, `data`, or `cmd` * **Destinations**: Where each data type is routed For each protocol, you define destinations with: * `name`: The data property key * `dest`: List of target nodes The following example connects `agora_rtc` extension to `deepgram_asr` extension. The `agora_rtc` extension sends `pcm_frame` data to `deepgram_asr` extension. ```json { "extension_group": "default", "extension": "agora_rtc", "audio_frame": [ { "name": "pcm_frame", "dest": [ { "extension_group": "default", "extension": "deepgram_asr" } ] } ] } ``` ## Extension folder The `ten_packages/extension` folder contains extension modules written in Python, Go, or C++. Each extension is a self-contained package with its own structure. Extension folder names typically match the module name, though the definitive module name is specified in the extension's `manifest.json` file. Use this module name as the `addon` property value in your configuration. The extension folder is structured as follows: ```text project_root/ ├── ten_packages/ │ └── extension/ │ ├── sample_python_extension/ │ │ ├── ... │ │ ├── extension.py │ │ ├── manifest.json │ │ └── property.json │ ├── sample_go_extension/ │ │ ├── ... │ │ ├── example_extension.go │ │ ├── manifest.json │ │ └── property.json │ └── sample_cpp_extension/ │ ├── ... │ ├── src/ │ │ ├── main.cc │ │ └── ... │ ├── BUILD.gn │ ├── manifest.json │ └── property.json ├── manifest.json └── property.json ``` ### Extension common files * `manifest.json`: Defines the extension's metadata including name, version, available properties, and supported APIs such as `data`, `audio_frame`, and `video_frame` * `property.json`: Specifies default configuration values for the extension ### Extension-specific files Each language has its own set of required files for extension development: | Language | Main logic file | Dependencies/Build file | | -------- | ---------------------- | ------------------------------------------------------------------------------------------------------------------------------------ | | Python | `extension.py` | `requirements.txt`: Python dependencies required by the extension. Dependencies are installed automatically when you run `task use`. | | Go | `example_extension.go` | `go.mod`: Go module definition that specifies the module name and dependencies. | | C++ | `src/main.cc` | `BUILD.gn`: Build configuration that specifies the target name and dependencies. | The main logic file contains the core implementation of the extension for each language. ## Web server The Go-based web server manages agent processes and handles HTTP requests from clients. Both the playground and demo interfaces communicate with this server, though you can also interact directly using curl or any HTTP client. ### API endpoints The server exposes three endpoints for agent management: * [POST /start](#post-start): Start an agent with specified configuration * [POST /stop](#post-stop): Stop a running agent * [POST /ping](#post-ping): Maintain agent connection #### POST /start Starts an agent with a specified graph and property overrides. The agent joins the specified RTC channel and subscribes to the user's audio stream. | Parameter | Description | | -------------- | --------------------------------------------------------------------------------------------------------------------- | | `request_id` | UUID for request tracking | | `channel_name` | RTC channel name (must match the channel your client joins) | | `user_uid` | User's RTC identifier for audio subscription | | `bot_uid` | Optional agent RTC identifier | | `graph_name` | Graph configuration name from `property.json` | | `properties` | Property overrides (temporary, doesn't modify `property.json`) | | `timeout` | Inactivity timeout in seconds (`-1` disables timeout, default: 60s, configurable using `WORKER_QUIT_TIMEOUT_SECONDS`) | Example: ```bash curl 'http://localhost:8080/start' \ -H 'Content-Type: application/json' \ --data-raw '{ "request_id": "c1912182-924c-4d15-a8bb-85063343077c", "channel_name": "test", "user_uid": 176573, "graph_name": "camera_va_openai_azure", "properties": { "openai_chatgpt": { "model": "gpt-4o" } } }' ``` #### POST /stop Stops a running agent instance. | Parameter | Description | | -------------- | ----------------------------------------- | | `request_id` | UUID for request tracking | | `channel_name` | Channel name used when starting the agent | Example: ```bash curl 'http://localhost:8080/stop' \ -H 'Content-Type: application/json' \ --data-raw '{ "request_id": "c1912182-924c-4d15-a8bb-85063343077c", "channel_name": "test" }' ``` #### POST /ping Maintains the agent connection by sending a keepalive signal. Not required if the agent was started with `timeout: -1`. Without pings, agents terminate after the configured timeout period. | Param | Description | | -------------- | -------------------------------- | | `request_id` | UUID for request tracking | | `channel_name` | Channel name of the active agent | Example: ```bash curl 'http://localhost:8080/ping' \ -H 'Content-Type: application/json' \ --data-raw '{ "request_id": "c1912182-924c-4d15-a8bb-85063343077c", "channel_name": "test" }' ``` # Account settlement (/en/introduction/billing/account-settlement) To ensure billing transparency and smooth service continuity, Agora implements real-time account balance reservation based on estimated usage across all Agora products for SSP customers. This does not apply to customers with a signed contract. For uninterrupted access to Agora's products and services, ensure at least one of the following: * Add a valid credit card for auto-recharge to your Agora account, **or** * Maintain sufficient funds in your Agora Console balance to cover your estimated usage. Agora employs the following policies to manage your account balance and ensure uninterrupted access to its products and services. ### Real-time usage and estimated bill You can view your real-time usage and estimated monthly bill at any time by selecting **Billing** in the Agora Console sidebar. ![bill estimate preview](https://assets-docs.agora.io/images/console/bill-estimate-preview.png) ### Real-time reservation of balance Your reserved balance is updated continuously based on your real-time bill estimation. The corresponding estimated amount is reserved from your available balance at all times. ![Available and reserved balance](https://assets-docs.agora.io/images/console/available-reserved-balance.png) ### Auto-recharge and account suspension On the **7th, 14th, 21st, and 28th** of each calendar month (excluding February 28), if your available balance is in arrears by more than **$50**: * If a valid credit card is linked, the system automatically recharges your balance to zero. * If no credit card is linked, or if auto-recharge fails, a notification email is sent to you. * Top up your wallet within the **24-hour grace period** to avoid account suspension. * If your balance remains in arrears after 24 hours, your account is suspended until the balance is restored to zero or above. ### Monthly billing Your final bill is issued monthly. After bill finalization: * Any overpaid amount is refunded to your balance. * Any undercharged amount is deducted accordingly. ### Balance withdrawal You can withdraw your available balance for payments completed within the past 90 days directly through the Agora Console. Funds will be returned to the original payment method. For further inquiries, please email [billing@agora.io](mailto\:billing@agora.io). ![Withdraw balance](https://assets-docs.agora.io/images/console/withdraw-balance.png) # Billing policies (/en/introduction/billing/billing-policies) This page explains billing, account settlement, end-of-life policies, and any applicable free-of-charge policies for this product. ## Billing and account policies Agora Console provides billing information, fee deduction details, and account suspension notices based on your account type. If you have signed a contract with Agora, the contract terms override all billing, deduction, and suspension details described on this page. ### Paid accounts Your account is a paid account if you have registered with Agora and completed any of the following: * Added a credit card to your account or topped up your balance using a bank account. * Made a recent payment. * Signed a contract with Agora. ### Billing cycle Agora provides each account with [10,000 free minutes](#free-of-charge-policy) every month. On the first day of each month, Agora issues your bill for the previous calendar month. To view billing information for your projects: 1. In [Agora Console](https://console.agora.io/), click home. 2. Click **Billing**. You see the detailed billing information for your projects, including billing period, due date, and amount. ![View bills](https://assets-docs.agora.io/images/video-sdk/bills.png) #### Additional charges This section describes the additional charges applicable to your account. #### Singapore goods and services tax As of November 2021, Agora Singapore charges [9% Singapore Goods and Services Tax (GST)](https://www.iras.gov.sg/taxes/goods-services-tax-%28gst%29/basics-of-gst/goods-and-services-tax-%28gst%29-what-it-is-and-how-it-works) on invoices for all Agora services provided to accounts located in Singapore. Agora determines the account location based on the tax identification number, contact address, or billing address that you have provided. All GST collected from Singapore accounts is paid to the Singapore tax authority. If you have any questions, contact [support@agora.io](mailto\:support@agora.io). #### Fee deduction On the sixth day of each month, Agora automatically deducts the fee for the previous month and notifies you by email. No fee is deducted if you meet both of the following conditions: * Your monthly usage does not exceed the free quota. * You do not use any other charged Agora services or products. If your account balance is negative after the deduction, Agora sends you an email, reminding you to top up your account at your earliest convenience and avoid account suspension. #### Account suspension If your account balance remains negative for 5 days after the deduction date, Agora suspends your account and notifies you by email. During suspension, none of your projects can access Agora services. To restore access, top up your account as soon as possible. Once your balance is zero or greater, Agora unfreezes your account. ### Free accounts Your account is a free account if you have registered with Agora and have not completed any of the following: * Added a credit card to your account or topped up your balance using a bank account. * Made a recent payment. * Signed a contract with Agora. Free accounts include 10,000 free minutes per month. For details, see [Free-of-charge policy](#free-of-charge-policy). #### Free account suspension Agora suspends your account on the second day after any of the following occur: * Your total usage exceeds the free quota of a service or product. * You use Agora services or products not included in the free quota. After account suspension, none of your projects can access Agora services. To restore access: 1. Add a credit card to your account or top up your balance using a bank account. This unfreezes your account and upgrades it to a paid account. 2. Purchase a [pre-paid monthly package](/en/introduction/billing/subscription-packages) or [top-up package](/en/introduction/billing/subscription-packages#top-up-package). This option is highly recommended. ## Free-of-charge policy Agora offers a new pricing model. If your account uses this model, you receive a monthly free usage package by default instead of the 10,000 free minutes described below. Agora provides each account with 10,000 free minutes per month and deducts them in the following order: * Broadcast Streaming audio minutes * Voice call, video call, and Interactive Live Streaming audio minutes * On-premise recording audio minutes * Cloud recording audio non-transcoding minutes * Cloud recording audio minutes * Web page recording audio minutes * Cloud Proxy audio minutes * Broadcast Streaming HD video minutes * Voice call, video call, and Interactive Live Streaming HD video minutes * On-premise recording HD video minutes * Cloud recording HD video minutes * Web page recording HD video minutes * Cloud Proxy HD video minutes * Broadcast Streaming Full HD\* video minutes * Voice call, video call, and Interactive Live Streaming Full HD\* video minutes * On-premise recording Full HD\* video minutes * Cloud recording Full HD\* video minutes * Web page recording Full HD\* video minutes * Cloud Proxy Full HD\* video minutes * Broadcast Streaming 2K video minutes * Voice call, video call, and Interactive Live Streaming 2K video minutes * On-premise recording 2K video minutes * Cloud recording 2K video minutes * Broadcast Streaming 2K+ video minutes * Voice call, video call, Interactive Live Streaming 2K+ video minutes * On-premise recording 2K+ video minutes * Cloud recording 2K+ video minutes * Cloud proxy 2K+ video minutes \* HD+ minutes are also included in the shared 10,000 free minutes. If your total service minutes do not exceed 10,000 minutes, the service is free-of-charge. After the 10,000 free-of-charge minutes are fully deducted, Agora charges you for the additional service minutes. Agora clears any remaining free-of-charge minutes at the end of each calendar month. * The 10,000 free-of-charge minutes policy does not apply to the Signaling SDK, the Chat SDK, or the IoT SDK. * Conversational AI Engine has its own free minutes policy. For details, see [Conversational AI pricing](/en/ai/reference/pricing). ### How service minutes are calculated Service minutes are calculated either by the number of users or by the number of streams. Agora calculates service minutes **by the number of users**. #### Approach 1: calculate by the number of users Suppose N users talk for M minutes in a channel, the total service minutes = N \* M. * If two users talk for 10 minutes, the total service minutes are: 2 \* 10 = 20. * If five users talk for 10 minutes, the total service minutes are: 5 \* 10 = 50. * If 10 users talk for 10 minutes, the total service minutes are: 10 \* 10 = 100. In this approach, service minutes depend only on the number of users in the channel, regardless of how many streams each user subscribes to. #### Approach 2: calculate by the number of streams Suppose N users talk for M minutes in a channel, and each user subscribes to all remote streams in the channel, the total service minutes = N \* (N-1) \* M. * If two users talk for 10 minutes, the total service minutes are: 2 \* (2-1) \* 10 = 20. * If five users talk for 10 minutes, the total service minutes are: 5 \* (5-1) \* 10 = 200. * If 10 users talk for 10 minutes, the total service minutes are: 10 \* (10-1) \* 10 = 900. In this approach, every remote stream that a user subscribes to is counted separately. #### Comparison of calculation approaches The following table compares service minutes under each approach: | Use-case | Service minutes by the number of users | Service minutes by the number of streams | | ------------------------------- | -------------------------------------- | ---------------------------------------- | | Two users talk for 10 minutes. | 20 minutes | 20 minutes | | Five users talk for 10 minutes. | 50 minutes | 200 minutes | | 10 users talk for 10 minutes. | 100 minutes | 900 minutes | The difference between the two approaches increases as more users join the channel. ### Agora's calculation method Agora calculates service minutes **by the number of users**. Service minutes are also calculated based on aggregate video resolution. ## End-of-life policy Agora is committed to providing regular updates to core products, extensions, and tools. These updates include new features, updated APIs, bug fixes, security patches, and documentation improvements. Agora strongly encourages developers to update to the latest product releases to benefit from new features, security enhancements, and other improvements. This section outlines the stages that an Agora product or service moves through, from pre-GA Beta to general availability to retirement, and the support Agora provides during each phase. ### Default introduction period Agora guarantees a minimum introduction period of 12 months from the release of a product on the Agora Developer Center. During this period, Agora does not initiate any end-of-support or end-of-life actions. After the 12-month period, Agora may transition a product into the Maintenance phase, which marks the beginning of the end-of-support phase. ### SDK lifecycle | Phase | Description | | ------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Beta | During this phase, SDKs are intended solely for early access and feedback purposes. They are not recommended for use in production environments. Beta products fall outside of the EOL policy and are intended for evaluation purposes only. For more information, see the Beta Service Agreement. | | General Availability (GA) | SDKs in this phase are fully supported by Agora. Agora provides support for new services, API updates, feature enhancements, bug fixes, and security patches. Agora guarantees a minimum of 12 months of support for GA SDKs. For details on client SDK support periods, see [Client SDK support periods](#client-sdk-support-periods). | | Retirement (EOL) SDKs: (Maintenance) | When an SDK enters the Retirement phase, Agora announces its retirement with a minimum notice period of 60 days, which is approximately 2 months. This announcement will include crucial timelines and guidelines to assist developers in migrating to the latest recommended SDK version. Agora communicates retirement announcements through email, the Agora Console notifications center, and SDK documentation, and may also post announcements on social media or the [Agora blog](https://www.agora.io/en/blog/). When a product or service reaches the end-of-support or end-of-life stage, Agora stops providing security updates, non-security updates, and assisted support. | ### Client SDK support periods Agora provides the following support periods for client SDKs: | Release | Support | | ------------- | ------------------------------------------------------------------------------------------ | | Major release | Agora guarantees support for at least 12 months starting from the date of a major release. | | Minor release | For minor releases, Agora ensures support for at least 6 months from the release date. | ### Maintenance support for retired SDKs Occasionally, Agora may transition a product SDK into a maintenance support phase focused primarily on bug fixes and security patches. In such cases, Agora notifies customers at least 60 days, approximately 2 months, in advance of the retirement period for the affected SDK. During the Retirement phase, products entering the EOS/EOL phase will continue to be supported as follows: | Duration | Support | | ---------------------------------- | ------------------------------------------------------------------------------------------------------------------------------- | | Months 1–6 after EOL announcement | Agora provides bug fixes and security updates. New feature requests are not accepted. | | Months 7–12 after EOL announcement | Agora provides security updates only. | | After 12 months | Agora no longer supports the product or service, unless a support extension was arranged before the end of the 12-month period. | Using an SDK beyond its maintenance support cycle is not recommended and is entirely at the developer's discretion and risk. Agora is committed to delivering the best SDK experience to its developers. This policy ensures that you have access to the latest features and security updates with clear guidance on the support timelines. ### Support extensions Customers may apply to extend support for a retired SDK for a temporary period, subject to Agora's discretion and commercial terms. Agora reviews each request and decides whether to grant the customer a license to use the product or service for a limited period beyond the EOL timeline. # Billing (/en/introduction/billing) This is the single place to understand how Agora charges for usage and how to manage payment. For product-specific rates, see the **Pricing** page of each product. ## Billing center In Agora Console, the **Billing** page lets you check your balance, add funds, and view or export invoices and transactions. Access is limited to members assigned to **Admin**, **Finance**, or an authorized custom team. ## How usage is billed Most Agora products bill by usage (for example, by minutes, message volume, storage, or peak concurrent users), with a free tier before charges begin. The exact unit and rate depend on the product, so check the product's own **Pricing** page for current numbers. Before you ship to production, confirm: * the billing unit makes sense for your expected usage pattern * the right people on your team can access billing and financial records ## Billing policies and management * [Billing policies](/en/introduction/billing/billing-policies) — free-usage allowance and product lifecycle rules. * [Account settlement](/en/introduction/billing/account-settlement) — balance reservation, auto-recharge, and suspension. * [Subscription packages](/en/introduction/billing/subscription-packages) — package tiers and upgrades. # Subscription packages (/en/introduction/billing/subscription-packages) A subscription package is a prepaid billing method. You can purchase a package in the [`Agora Console`](https://console.agora.io/) to cover usage for the current month. This page explains how to purchase, and manage subscription and top-up packages. ### Purchase and upgrade By default, a new account may be assigned a Free package when the first eligible project is created. You can upgrade to a paid package at any time. Available package tiers, discounts, and unit prices vary by product. To upgrade: 1. Log in to [Agora Console](https://console.agora.io/). 2. Click the username at the bottom of the navigation sidebar to open the account popup menu. 3. Select **Subscriptions**. 4. Select the tab for the product subscription you want to manage, such as Realtime, Signaling, or Chat. 5. Select your package and click **Upgrade** or **Downgrade**. ![Subscription plans](https://assets-docs.agora.io/images/console/subscription-plans.png) For detailed pricing for a specific product, see the product's pricing page. * For non-contracted customers, most packages can be purchased directly from the Agora Console. Some enterprise or contracted packages may require contacting [Agora sales](mailto\:sales@agora.io). * You can upgrade sequentially or skip levels when the product supports it. * Package upgrades take effect immediately and apply to the entire calendar month. If you make multiple upgrades on the same day, only the last upgrade is applied. * If your account has a negative balance, purchasing a prepaid package first charges your credit card for the outstanding balance, then for the package fee. #### Upgrade rules When you upgrade to a paid package, it becomes effective for the entire calendar month, regardless of the purchase date. You receive the full monthly allocation of the included quota. The cost of the previous package is deducted from the new package price, so you only pay the difference. If you request an upgrade after auto-renewal has triggered, you may upgrade for the next month as well. ### Renewal By default, your subscription package renews automatically 48 hours before it expires. Ensure your account has sufficient balance or enable auto-pay to avoid renewal failure. You can unsubscribe at any time to prevent renewal. See [Cancel subscription](#cancel-subscription). If renewal fails: * The system sends a notification and retries the renewal. * If renewal still fails by the time the package expires, the package downgrades to the Free package. * To restore the previous package, see [Purchase and Upgrade](#purchase-and-upgrade). ### Downgrade You can downgrade by selecting and purchasing a lower-tier package. A downgrade takes effect when your current package expires if you request it before auto-renewal is triggered. You are charged for the new package when you request the downgrade. You can downgrade only once per month. * If your downgrade for the next month is successful, auto-renewal for the current package is not triggered. * If a package is already active, you must wait until the 1st day of that month to initiate a downgrade for the following month. #### Example * **Requested more than 48 hours before the current package expires**: The downgrade takes effect after the current package expires. * **Requested less than 48 hours before the current package expires**: Since a package is already active for the next month, you must wait until the new package starts to downgrade for the following month. ### Cancel subscription To cancel a prepaid package, downgrade to the Free package. Downgrade rules apply. ## Top-up package A top-up package is a pay-as-you-go plan. You can purchase a top-up package from the [`Agora Console`](https://console.agora.io/subscriptions/rtc-plans?tab=top-up) to cover usage that exceeds your monthly package quota. ### Purchase You can purchase multiple top-up packages. They are deducted in the order of purchase. #### Example: * Purchased 250,000-minutes top-up packages on **August 15** and **August 16**. On **September 1**, minutes consumed in August are deducted from the first package, then from the second. ### Validity Each top-up package is valid for one year, starting from the purchase date to the end of the same month of the following year. #### Example: * Purchased on **August 15, 2025** → valid until **August 31, 2026** Unused minutes expire after the validity period and cannot be carried forward. ### Cancel top-up package Cancelling a top-up package is not supported. ## Free tier overage For Free tier users, exceeding the included minutes and any purchased top-up minutes results in service suspension. Service resumes when you purchase additional top-up minutes or upgrade to a prepaid subscription package. # Start building (/en/introduction/get-started/build-it-yourself) Most teams do not start by asking which Agora product to read first. They start by asking what kind of real-time problem they need to solve. Use this page to map your product goal to the most useful documentation path. ## When to use this path This is the right page when: * you already know the product problem you want to solve * you want to choose the right capability family before reading deep product docs * you do not need an AI-assisted setup workflow first * you want the shortest route from problem statement to relevant quickstart or product area If you want an AI agent to help choose the route and bootstrap a starter, use [Start with AI](/en/introduction/start-with-ai) instead. ## Start from the problem you are solving ### Voice-first AI interaction If users need to talk to an AI agent in real time, start with: * [Conversational AI](/en/introduction/conversational-ai) * [AI quickstart](/en/ai/get-started/quickstart) * [Voice agent quickstart](/en/ai/choose-your-path/quickstart-coding) This path is usually right for assistants, tutors, companions, service agents, and voice-enabled devices. ### Live voice, video, or interactive sessions If users need to join the same session and interact through live media, start with: * [Real-Time Voice & Video](/en/introduction/realtime-audio-video) * [Audio & Video](/en/realtime-media/rtc) * [Voice & Video quickstart](/en/realtime-media/rtc/android/quick-start/build-from-scratch) This path is usually right for calling, meetings, classrooms, social rooms, telehealth, and interactive livestreaming. ### Room messaging, state sync, or workflow coordination If the product depends on chat, presence, metadata, or event coordination, start with: * [Messaging & Presence](/en/introduction/messaging-presence) * [RTM / Signaling](/en/realtime-media/rtm) * [IM / Chat](/en/realtime-media/im) This path is usually right for in-room communication, control flows, collaborative state, and backend-triggered room actions. ### Cloud-side media workflows If the media must be captured, processed, routed, or redistributed beyond the live session itself, start with: * [Cloud Media Services](/en/introduction/cloud-media-services) * [Cloud Recording](/en/realtime-media/cloud-recording) * [Real-time Transcription & Translation](/en/realtime-media/speech-to-text) * [Transcoding](/en/realtime-media/transcoding) * [Server Media Processing](/en/realtime-media/rtc-server-sdk) This path is usually right for archive, replay, moderation, server-side automation, and downstream content pipelines. ## Common combinations Many production systems combine several of these paths: * voice or video sessions plus [Messaging & Presence](/en/introduction/messaging-presence) * AI interaction plus [Cloud Media Services](/en/introduction/cloud-media-services) * live sessions plus [Cloud Media Services](/en/introduction/cloud-media-services) * broadcast products plus [Cloud Media Services](/en/introduction/cloud-media-services) ## What to do next If you are still evaluating the platform, read [About Agora](/en/introduction/about-agora) first. If you want the shared mental model before choosing a path, read [Core Concepts](/en/introduction/core-concepts). If you want an AI-assisted setup workflow, switch to [Start with AI](/en/introduction/start-with-ai). # Service activation (/en/realtime-media/agora-analytics/activation) To activate Agora Analytics, follow these steps: 1. Contact your account executive, solutions architect, or email [support@agora.io](mailto\:support@agora.io) to learn about the available Agora Analytics plans. 2. The account team or your solutions architect will open a support ticket with [Technical support](mailto\:support@agora.io) to request activation. Make sure the ticket includes your Customer ID and the specific Agora Analytics plan you want activated. Activation typically takes up to 2 business days. # Agora Analytics overview (/en/realtime-media/agora-analytics/product-overview) Agora Analytics tracks and analyzes the usage, quality, and performance of real-time voice and video live streams and other Agora products. Designed as a companion to Agora's Voice Calling, Video Calling, Broadcast Streaming, Interactive Live Streaming, and Chat it helps locate quality issues, identify root causes, and resolve problems to enhance the end-user experience. Monitor sessions in real-time with Call Inspector or use RESTful APIs to analyze usage trends and performance. Extend Agora Analytics with the Datadog integration to bring Agora Analytics data into your Datadog account. Visualize and monitor metrics, set up custom alerts, and manage Agora data alongside other system metrics for a unified view of performance and quality. ## Start building ## Product Features Identify, analyze, and respond to performance quality issues with advanced search, detailed visibility into call and user metrics, and diagnostics. Provides periodic call usage and quality statistics. It is designed to help customers understand the usage and quality of calls in apps. Visualized data for multiple call metrics in real time. Sends alerts to you when abnormal metrics or events are detected in your Agora projects. Use RESTful APIs to retrieve call statistics and quality of experience metrics, use them in your own application or DataOps workflow. Easily embed Agora Analytics pages in internal web portals using a low-code approach. # Core concepts (/en/realtime-media/broadcast-streaming/core-concepts) RTC (Real-Time Communication) refers to real-time communication technology, which allows almost instant exchange of audio, video, and other data between the sender and the receiver. Agora SDKs provide real-time audio and video interaction services, with multi-platform and multi-device support. This includes high-definition video calls, voice-only calls, interactive live streaming, as well as one-on-one and multi-group chats. This guide introduces the key processes and concepts you need to know to use Video SDK. Agora relies on the following fundamental concepts to enable seamless real-time communication:
### Agora SDRTN® Agora's core engagement services are powered by its Software-Defined Real-Time Network (SDRTN®), a global infrastructure accessible anytime, anywhere. Unlike traditional networks, Agora SDRTN® is not restricted by devices, phone numbers, or telecom coverage areas. With data centers in over 200 countries and regions, it ensures sub-second latency and high availability for real-time media. Agora SDRTN® enables live user engagement through real-time communication (RTC), offering: * Unmatched quality of service * High availability and accessibility * True scalability * Low cost ## Channel concepts Agora uses channels to group users together, enabling seamless communication and interaction. Channels serve as the foundation for transmitting real-time data, whether audio, video, or signaling, and play a crucial role in connecting users and services. ### Channel A channel organizes users into a group and is identified by a unique channel name. Users who connect to the same channel are able to communicate with each other. A channel is created when the first user joins and ceases to exist when the last user leaves. Channels are created by calling the methods for transmitting real-time data. Agora uses different channels to transmit different types of data: * A Video SDK channel is used for transmitting audio or video data. * A Signaling channel is used for transmitting messaging or signaling data. These channels are independent of each other. Additional services provided by Agora, such as Cloud Recording and Speech to Text, join the Video SDK channel to provide real-time recording, transmission acceleration, media playback, and content moderation. ### Channel profile The Video SDK applies different optimization methods according to the selected channel profile. Agora supports the following channel profiles: | Channel profile | Description | | --------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | **Communication** | This profile is suitable for one-on-one or group calls, where all users in the channel talk freely. | | **Live Broadcasting** | In a live streaming channel, users have two client roles: *host* and *audience*. The *host* sends and receives streams, while the *audience* only receives streams with the sending function disabled. | ### Stream A stream is a sequence of digitally encoded, coherent signals that contain media data. Users in a channel [publish](#publish) local streams and [subscribe](#subscribe) to remote streams from other users. ### User role The user role defines whether a user in a channel has the permission to publish streams. There are two user roles: * **Host**: A user who can publish streams to a channel. * **Audience**: A User who can only subscribe to remote media streams. A user with this role cannot publish streams. ### Publish Publishing is the act of sending a user’s audio or video data to the channel. Usually, the published stream is created by the audio data sampled from a microphone or the video data captured by a camera. You can also publish media streams from other sources, such as an online music file or the user’s screen. After successfully publishing a stream, the SDK uses it to send media data to other users in the channel. Users communicate with each other in real-time by publishing local streams and subscribing to remote streams. ### Subscribe Subscribing is the act of receiving media streams published by remote users to the channel. A user receives audio and video data from other users by subscribing to one or more of their streams. You either directly play the subscribed streams or process incoming data for other purposes such as recording or capturing screenshots. ### User ID In Broadcast Streaming, the UID is an integer value that uniquely identifies a user within the context of a channel. When joining a channel, you have the option to either assign a specific UID to the user or pass `0` or `null` and allow Agora to automatically generate and assign a UID to the user. If two users attempt to join the same channel with the same UID, it can lead to unexpected behavior. The UID is used by Agora's services and components to identify and manage users within a channel. Ensure that UIDs are properly assigned to prevent conflicts. ### RTC connection The connection between the SDK and the channel. When publishing or subscribing to multiple streams in multiple channels, a connection is used to specify the target channel. ## Credentials To ensure reliable access and secure communication, Agora uses credentials such as the App ID, App Certificate, and tokens to identify applications, authenticate their requests, and authorize their access on its platform. ### App ID The App ID is a unique key generated by Agora to identify each project and provide billing and other statistical data services. The App ID is critical for connecting users within your app. It is used to initialize the Agora Engine in your app, and as one of the required keys to create authentication tokens for secure communication. Retrieve the App ID for your project using the [Agora Console](https://console.agora.io/projects). App IDs are stored on the front-end client and do not provide access control. Projects using only an App ID allow any user with the App ID to join. For access control, especially in production environments, choose the **App ID + Token** mechanism for user authentication when creating a new project. Without authentication tokens, your environment is open to anyone with access to your App ID. ### App Certificate An App Certificate is a unique key generated by the Agora Console to secure projects through token authentication. It is required, along with the App ID, to generate a token that proves authorization between your systems and Agora's network. App Certificates are used to generate Broadcast Streaming authentication tokens. Store the App Certificate securely in your backend systems. If your App Certificate is compromised or to meet security compliance requirements, you can invalidate certificates and create new ones through the Agora Console. ### Tokens A token is a dynamic key generated using the App ID, App Certificate, user ID, and expiration timestamp. Tokens authenticate and secure access to Agora's services, ensuring only authorized users can join a channel and participate in real-time communication. Tokens are generated on your server and passed to the client for use in Broadcast Streaming. The token generation process involves digitally signing the App ID, App Certificate, user ID, and expiration timestamp using a specific algorithm, preventing tampering or forgery. During development and testing, use the Agora Console to generate temporary tokens. For production environments, implement a token server as part of your security infrastructure to control access to your channels. ### Agora Console [Agora Console](https://console.agora.io) is the main dashboard where you manage your Agora projects and services. Before you can use Agora's SDKs, you must first create a project in the Agora Console. See [Agora account management](/en/introduction/account) for details. ![Create project in Agora Console](https://assets-docs.agora.io/images/common/create-project.svg) Agora Console provides an intuitive interface for developers to query and manage their Agora account. After registering an Agora account, you use the Agora Console to perform the following tasks: * Manage your account * Create and configure Agora projects and services * Get an App ID and the App certificate * Generate temporary tokens for development and testing * Manage members and roles * Check call quality and usage * Check bills and make payments * Access product resources See [Agora account management](/en/introduction/account) for details on how to manage all aspects of your Agora account. Agora also provides RESTful APIs that you use to implement features such as creating a project and fetching usage numbers programmatically. ## Audio and video concepts ### Audio and video interaction workflow The following figure illustrates the workflow of using the Video SDK to implement basic audio and video interaction. ![orientation\_adaptive\_locked\_landscape](https://assets-docs.agora.io/images/common/basic-audio-and-video.svg) Agora relies on the following fundamental concepts to enable seamless real-time communication: ### Audio module In audio interaction, the main functions of the audio module are as shown in the figure below: ![Audio module functions](https://assets-docs.agora.io/images/common/audio-module.svg) After you call `registerAudioFrameObserver`, you can obtain the raw audio data at the following observation points in the audio transmission process: 1. Obtain the raw audio data of ear monitoring through the `onEarMonitoringAudioFrame` callback. 2. Obtain the captured raw audio data through the `onRecordAudioFrame` callback. 3. Obtain the raw audio playback data of each individual stream through the `onPlaybackAudioFrameBeforeMixing` callback. 4. Obtain the raw audio playback data of all mixed streams through the `onPlaybackAudioFrame` callback. 5. Obtain the raw audio data after mixing the captured and playback audio through the `onMixedAudioFrame` callback. (5) `onMixedAudioFrame` = (2) `onRecordAudioFrame` + (4) `onPlaybackAudioFrame` ### Audio routing The audio output device used by the app when playing audio. Common audio routes include wired headphones, earpieces, speakers, Bluetooth headphones, and others. The APIs used by the audio module are as follows: * Enable local audio collection: `enableLocalAudio` * Set local playback device: `setPlaybackDevice` * Set up audio routing: `setDefaultAudioRouteToSpeakerphone` ### Video module The following diagram shows the main functions of the video module in video interaction: ![Video module functions](https://assets-docs.agora.io/images/common/video-module.svg) The figure shows the following observation points: 1. `POSITION_POST_CAPTURER_ORIGIN`. 2. `POSITION_POST_CAPTURER`, corresponds to the `onCaptureVideoFrame` callback. 3. `POSITION_PRE_ENCODER`, corresponds to the `onPreEncodeVideoFrame` callback. 4. `POSITION_PRE_RENDERER`, corresponds to the `onRenderVideoFrame` callback. The APIs used by the video module are as follows: * Enable local video collection: `enableLocalVideo` * Local preview: `setupLocalVideo` → `startPreview` * Video rendering shows: `setupRemoteVideo` # Broadcast Streaming overview (/en/realtime-media/broadcast-streaming) Agora's Broadcast Streaming API delivers low-latency, high-definition live video streaming at scale for one-to-many broadcast delivery. With support for cross-platform integration and adaptive streaming technologies, it ensures reliable, high-quality viewing experiences across a wide range of network conditions and devices. Reach global audiences with smooth, uninterrupted broadcasts for large-scale events such as concerts, sports, conferences, live shows, and other live production experiences. Enhance Agora's Video SDK with broadcast production capabilities such as recording, stream management, content moderation, and audience analytics, or leverage the Extensions Marketplace to enable AI-powered features like noise cancellation, video effects, and more. ## Start building ## Product Features Agora’s Software-Defined Real-Time Network (SDRTN®) supports video users in over 200 countries and regions. Consistent high-quality video from few to thousands of concurrent users, even under challenging network conditions. Publish multiple audio and video tracks to one or more channels from a single instance, with support for multi-channel capture cameras and microphones. Enable screen sharing or interactive whiteboards that allow users to draw, annotate, and share content from multiple devices simultaneously. Support for high quality audio with 3D spatial audio, AI noise suppression, and gain control to provide an immersive audio experience. Record video sessions in the cloud or on premises with control over the format, path of storage, and quality. # Agora MCP (/en/realtime-media/broadcast-streaming/mcp) The Agora MCP server gives your AI assistant direct access to Agora's documentation, so it can look up APIs, SDK methods, and platform-specific details in real time. The Agora MCP server is included when you install Agora Skills. If you prefer to install only the MCP server, it is available at: ```text https://mcp.agora.io ``` ### Installation Refer to the installation instructions for your coding assistant. Cursor Claude Codex Gemini CLI Manual installation Click the button below to install the MCP server in [Cursor](https://www.cursor.com/) Install Agora MCP Server in Cursor or add it manually with the following JSON: ```json { "mcpServers": { "agora-docs": { "url": "https://mcp.agora.io" } } } ``` * **Claude Code** Run the following command in your terminal to install the MCP server in [Claude Code](https://claude.com/product/claude-code): ```bash claude mcp add --transport http agora-docs https://mcp.agora.io ``` * **Claude Desktop** In **Settings**, select **Connectors** and then choose **Add custom connector**. Enter the following values and click **Add**: * **Name**: `agora-docs` * **Remote MCP server URL**: `https://mcp.agora.io` Run the following command in your terminal to install the server in [OpenAI Codex](https://openai.com/codex/): ```bash codex mcp add --url https://mcp.agora.io agora-docs ``` Run the following command in your terminal to install the server in [Gemini CLI](https://github.com/google-gemini/gemini-cli): ```bash gemini mcp add --transport http agora-docs https://mcp.agora.io ``` Add the server URL `https://mcp.agora.io` to your MCP client of choice. If prompted, set the transport to `http` or "Streamable HTTP". ### Getting started Once installed, your coding assistant has access to Agora's documentation through the MCP server. The assistant will intelligently use this resource when relevant to your questions. For more targeted results, mention Agora along with your target product and platform, such as 'iOS', 'Web', 'Conversational AI', 'Video Calling' in your prompts. ### System prompt This MCP works with all LLMs that support MCP, but performs best when the assistant understands facet-based exploration. Add the following prompt to your LLMs custom instructions: **System prompt for LLMs** ```markdown # Agora MCP Markdown - System Prompt You have access to Agora's documentation search via three tools: - `algolia_search_index_docs_platform_aware_markdown` - Full-text search with facets - `algolia_search_for_facet_values` - Browse products/platforms - `algolia_recommendations` - Find related documentation ## Key Behaviors **1. Use facets for discovery** - When users ask about "what's available", explore facets first - Example: User asks "what video products exist?" → Use `algolia_search_for_facet_values(facetName: "product", facetQuery: "video")` → Shows video-calling is the main product **2. Platform-aware searching** - Include target platform in search queries (iOS, Android, Web, etc.) - Results are intelligently ranked by platform relevance **3. Interpret facet results for refinement** - Search results include facet breakdowns by product and platform - Use these to suggest filtering: "15 results are for video-calling, 8 for voice-calling" **4. Maintain session continuity** - Use same sessionId (UUID) across related searches in a conversation - Improves ranking for follow-up queries ## Available Products & Platforms **Key Products:** conversational-ai, video-calling, broadcast-streaming, interactive-live-streaming, agora-chat, voice-calling, signaling, extensions-marketplace, interactive-whiteboard **Key Platforms:** android, web, ios, windows, macos, unity, flutter, react-native, electron, unreal ## Workflow Example User: "I'm building an iOS app with Video Calling. Where do I start?" 1. Search: "iOS video calling quickstart" 2. Results show facets indicating video-calling is the best match 3. Suggest: "Most results are for Video Calling, here's the quickstart" 4. Follow-up searches: Keep same sessionId for context ``` #### What This Enables * Intelligent facet exploration before deep searches * Platform-aware result refinement suggestions * Cross-search session continuity for better context Without the prompt, searches still work but won't leverage faceted browsing patterns. ## AGENTS.md To get the most out of Agora Skills and the MCP server, Agora recommends that you include an [AGENTS.md](https://agents) or similar file in your repository. This file ensures that your agent always consults Agora docs to produce reliable, working code. Add the following instructions to your own `AGENTS.md` file: ```markdown ## Agora Documentation Agora documentation is updated frequently. You should always refer to the latest documentation when working with Agora products. Agora offers an MCP server that you can use to browse and search its documentation. If the developer has not yet installed this server, recommend that they install it at https://mcp.agora.io. If the Agora Skills files are available in your context, use them for platform-specific API guidance and code examples. ``` # Broadcast Streaming overview (/en/realtime-media/broadcast-streaming/product-overview) Agora's Broadcast Streaming API delivers low-latency, high-definition live video streaming at scale for one-to-many broadcast delivery. With support for cross-platform integration and adaptive streaming technologies, it ensures reliable, high-quality viewing experiences across a wide range of network conditions and devices. Reach global audiences with smooth, uninterrupted broadcasts for large-scale events such as concerts, sports, conferences, live shows, and other live production experiences. Enhance Agora's Video SDK with broadcast production capabilities such as recording, stream management, content moderation, and audience analytics, or leverage the Extensions Marketplace to enable AI-powered features like noise cancellation, video effects, and more. ## Start building ## Product Features * **Global coverage** - Agora’s Software-Defined Real-Time Network (SDRTN®) supports video users in over 200 countries and regions. * **High-quality video at scale** - Consistent high-quality video from few to thousands of concurrent users, even under challenging network conditions. * **Multiple audio and video tracks** - Publish multiple audio and video tracks to one or more channels from a single instance, with support for multi-channel capture cameras and microphones. * **Screen sharing and collaboration** - Enable screen sharing or interactive whiteboards that allow users to draw, annotate, and share content from multiple devices simultaneously. * **AI-powered audio enhancement** - Support for high quality audio with 3D spatial audio, AI noise suppression, and gain control to provide an immersive audio experience. * **Recording** - Record video sessions in the cloud or on premises with control over the format, path of storage, and quality. # Quickstart (/en/realtime-media/broadcast-streaming/quickstart) This page provides a step-by-step guide on how to create a basic Broadcast Streaming app using the Agora Video SDK. ## Understand the tech To start a Broadcast Streaming session, implement the following steps in your app: * **Initialize the Agora Engine**: Before calling other APIs, create and initialize an Agora Engine instance. * **Join a channel**: Call methods to create and join a channel. * **Join as a host**: A live streaming event has one or more hosts. A host publishes audio and video to the channel. Hosts can also subscribe to streams from other hosts. * **Join as audience**: Audience members can only subscribe to streams published by hosts. * **Send and receive audio and video**: Hosts publish streams to the channel. Audience members subscribe to audio and video streams published by hosts. ![Streaming workflow](https://assets-docs.agora.io/images/video-sdk/get-started-ils-bs.svg) ## Prerequisites * [Android Studio](https://developer.android.com/studio) 4.2 or higher. * Android SDK API Level 21 or higher. * Two mobile devices running Android 5.0 or higher. * A camera and a microphone * A valid Agora account and project. Please refer to [Agora account management](/en/introduction/account) for details. ## Set up your project This section shows you how to set up your Android project and install the Agora Video SDK. **Create a new project** 1. Create a [new project](https://developer.android.com/studio/projects/create-project). 1. Open Android Studio and select **File > New > New Project...**. 2. Select **Phone and Tablet** > **Empty Activity** and click **Next**. 3. Set the project name and storage path. 4. Select **Java** or **Kotlin** as the language, and click **Finish** to create the Android project. After you create a project, Android Studio automatically starts gradle sync. Ensure that the synchronization is successful before proceeding to the next step. **Add to an existing project** 1. Add a new activity to your project. 1. Open your project in Android Studio. 2. Right-click on the `app/src/main/java/` folder. 3. Select **New → Activity → Empty Activity**. 4. Enter an activity name and click **Finish**. This guide uses `MainActivity` as the activity name in the sample code. Replace it with your activity name where required. 2. Add a layout file for your activity. Set up two container elements in your activity to display local and remote video streams. Refer to [Create a user interface](#create-a-user-interface) to get a bare bones sample layout. ### Install the SDK Use either of the following methods to add Video SDK to your project. **Maven Central** 1. Open the `settings.gradle` file in the project's root directory and add the Maven Central dependency, if it doesn't already exist: ```text repositories { mavenCentral() } ``` If your Android project uses dependencyResolutionManagement, the method of adding the Maven Central dependency may differ. 2. To integrate the Video SDK into your Android project, add the following to the `dependencies` block in your project module `build.gradle` file: * Groovy `build.gradle` ```json implementation 'io.agora.rtc:full-sdk:x.y.z' ``` * Kotlin `build.gradle.kts` ```kotlin implementation("io.agora.rtc:full-sdk:x.y.z") ``` Replace `x.y.z` with the specific SDK version number, such as `4.5.0`. To get the latest version number, check the [Release notes](reference/release-notes). To integrate the Lite SDK, use `io.agora.rtc:lite-sdk` instead. 3. Prevent code obfuscation Open the `/app/proguard-rules.pro` file and add the following lines to prevent the Video SDK code from being obfuscated: ```java -keep class io.agora.** { *; } -dontwarn io.agora.** ``` **Manual integration** 1. Download the latest version of Video SDK from the [SDKs](/en/api-reference/sdks?product=video\&platform=android) page and unzip it. 2. Open the unzipped file and copy the following files or subfolders to your project path. | File or folder | Project path | | :----------------------------------- | :----------------------- | | `agora-rtc-sdk.jar` file | `/app/libs/` | | `arm64-v8a` folder | `/app/src/main/jniLibs/` | | `armeabi-v7a` folder | `/app/src/main/jniLibs/` | | `x86` folder | `/app/src/main/jniLibs/` | | `x86_64` folder | `/app/src/main/jniLibs/` | | `high_level_api` in `include` folder | `/app/src/main/jniLibs/` | 3. Select the file `/app/libs/agora-rtc-sdk.jar` in the left navigation bar of Android Studio project files, right-click, and select **add as a library** from the drop-down menu. 4. Prevent code obfuscation Open the `/app/proguard-rules.pro` file and add the following lines to prevent the Video SDK code from being obfuscated: ```java -keep class io.agora.** { *; } -dontwarn io.agora.** ``` ## Implement Broadcast Streaming This section guides you through the implementation of basic real-time audio and video interaction in your app. The following figure illustrates the essential steps: ![Quick start sequence](https://assets-docs.agora.io/images/video-sdk/quick-start-sequence.svg) This guide includes [complete sample code](#complete-sample-code) that demonstrates implementing basic real-time interaction. To understand the core API calls in the sample code, review the following implementation steps and use the code in your `MainActivity` file. ### Import Agora classes Import the relevant Agora classes and interfaces: Java Kotlin ```java import io.agora.rtc2.Constants; import io.agora.rtc2.IRtcEngineEventHandler; import io.agora.rtc2.RtcEngine; import io.agora.rtc2.RtcEngineConfig; import io.agora.rtc2.video.VideoCanvas; import io.agora.rtc2.ChannelMediaOptions; ``` ```kotlin import io.agora.rtc2.Constants import io.agora.rtc2.IRtcEngineEventHandler import io.agora.rtc2.RtcEngine import io.agora.rtc2.RtcEngineConfig import io.agora.rtc2.video.VideoCanvas import io.agora.rtc2.ChannelMediaOptions ``` ### Initialize the engine For real-time communication, initialize an `RtcEngine` instance and set up event handlers to manage user interactions within the channel. Use `RtcEngineConfig` to specify the application context, [App ID](/en/introduction/account), and custom [event handler](#subscribe-to--events), then call `RtcEngine.create(config)` to initialize the engine, enabling further channel operations. In your `MainActivity` file, add the following code: Java Kotlin ```java // Fill in the app ID from Agora Console private String myAppId = ""; private RtcEngine mRtcEngine; private void initializeAgoraVideoSDK() { try { RtcEngineConfig config = new RtcEngineConfig(); config.mContext = getBaseContext(); config.mAppId = myAppId; config.mEventHandler = mRtcEventHandler; mRtcEngine = RtcEngine.create(config); } catch (Exception e) { throw new RuntimeException("Error initializing RTC engine: " + e.getMessage()); } } ``` ```kotlin // Fill in the App ID obtained from the Agora Console private val myAppId = "" private var mRtcEngine: RtcEngine? = null private fun initializeRtcEngine() { try { val config = RtcEngineConfig().apply { mContext = applicationContext mAppId = myAppId mEventHandler = mRtcEventHandler } mRtcEngine = RtcEngine.create(config) } catch (e: Exception) { throw RuntimeException("Error initializing RTC engine: ${e.message}") } } ``` ### Join a channel To join a channel, call `joinChannel` with the following parameters: * **Channel name**: The name of the channel to join. Clients that pass the same channel name join the same channel. If a channel with the specified name does not exist, it is created when the first user joins. * **Authentication token**: A dynamic key that authenticates a user when the client joins a channel. In a production environment, you obtain a token from a [token server](build/authenticate-users/deploy-token-server.mdx) in your security infrastructure. For the purpose of this guide [Generate a temporary token](/en/introduction/account). * **User ID**: A 32-bit signed integer that identifies a user in the channel. You can specify a unique user ID for each user yourself. If you set the user ID to `0` when joining a channel, the SDK generates a random number for the user ID and returns the value in the `onJoinChannelSuccess` callback. * **Channel media options**: Configure `ChannelMediaOptions` to define publishing and subscription settings, optimize performance for your specific use-case, and set optional parameters. For Broadcast Streaming, set the `channelProfile` to `CHANNEL_PROFILE_LIVE_BROADCASTING`, the `clientRoleType` to `CLIENT_ROLE_BROADCASTER` (host) or `CLIENT_ROLE_AUDIENCE`, and the `audienceLatencyLevel` to `AUDIENCE_LATENCY_LEVEL_LOW_LATENCY`. Java Kotlin ```java // Fill in the channel name private String channelName = ""; // Fill in the temporary token generated from Agora Console private String token = ""; private void joinChannel() { // Create an instance of ChannelMediaOptions and configure it ChannelMediaOptions options = new ChannelMediaOptions(); // Set the user role to BROADCASTER or AUDIENCE according to the use-case options.clientRoleType = Constants.CLIENT_ROLE_BROADCASTER; // In the broadcast streaming use-case, set the channelProfile to BROADCASTING options.channelProfile = Constants.CHANNEL_PROFILE_LIVE_BROADCASTING; // Set the latency level for audience options.audienceLatencyLevel = Constants.AUDIENCE_LATENCY_LEVEL_LOW_LATENCY; // Publish local media options.publishCameraTrack = true; options.publishMicrophoneTrack = true; mRtcEngine.joinChannel(token, channelName, 0, options); } ``` ```kotlin // Fill in the channel name private val channelName = "" // Fill in the temporary token generated from Agora Console private val token = "" private fun joinChannel() { // Create an instance of ChannelMediaOptions and configure it val options = ChannelMediaOptions().apply { // Set the user role to BROADCASTER or AUDIENCE according to the use-case clientRoleType = Constants.CLIENT_ROLE_BROADCASTER // In the broadcast streaming use-case, set the channelProfile to BROADCASTING channelProfile = Constants.CHANNEL_PROFILE_LIVE_BROADCASTING // Set the latency level for audience audienceLatencyLevel = Constants.AUDIENCE_LATENCY_LEVEL_LOW_LATENCY // Publish local media publishMicrophoneTrack = true publishCameraTrack = true } mRtcEngine?.joinChannel(token, channelName, 0, options) } ``` ### Subscribe to Video SDK events The Video SDK provides an interface for subscribing to channel events. To use it, create an instance of `IRtcEngineEventHandler` and implement the event methods you want to handle. To ensure that you receive all Video SDK events, set the Agora Engine event handler before joining a channel. Java Kotlin ```java private final IRtcEngineEventHandler mRtcEventHandler = new IRtcEngineEventHandler() { // Triggered when the local user successfully joins the specified channel. @Override public void onJoinChannelSuccess(String channel, int uid, int elapsed) { super.onJoinChannelSuccess(channel, uid, elapsed); showToast("Joined channel " + channel); } // Triggered when a remote user/host joins the channel. @Override public void onUserJoined(int uid, int elapsed) { super.onUserJoined(uid, elapsed); runOnUiThread(() -> { // Initialize and display remote video view for the new user. setupRemoteVideo(uid); showToast("User joined: " + uid); }); } // Triggered when a remote user/host leaves the channel. @Override public void onUserOffline(int uid, int reason) { super.onUserOffline(uid, reason); runOnUiThread(() -> { showToast("User offline: " + uid); }); } }; ``` ```kotlin private val mRtcEventHandler = object : IRtcEngineEventHandler() { override fun onJoinChannelSuccess(channel: String?, uid: Int, elapsed: Int) { super.onJoinChannelSuccess(channel, uid, elapsed) runOnUiThread { showToast("Joined channel $channel") } } override fun onUserJoined(uid: Int, elapsed: Int) { runOnUiThread { showToast("User joined: $uid") } } override fun onUserOffline(uid: Int, reason: Int) { super.onUserOffline(uid, reason) runOnUiThread { showToast("User offline: $uid") } } } ``` ### Enable the video module Follow these steps to enable the video module: 1. Call `enableVideo` to enable the video module. 2. Call `startPreview` to enable local video preview. Java Kotlin ```java private void enableVideo() { mRtcEngine.enableVideo(); mRtcEngine.startPreview(); } ``` ```kotlin private fun enableVideo() { mRtcEngine?.apply { enableVideo() startPreview() } } ``` ### Display the local video Call `setupLocalVideo` to initialize the local view and set the local video display properties. Java Kotlin ```java private void setupLocalVideo() { FrameLayout container = findViewById(R.id.local_video_view_container); SurfaceView surfaceView = new SurfaceView(getBaseContext()); container.addView(surfaceView); mRtcEngine.setupLocalVideo(new VideoCanvas(surfaceView, VideoCanvas.RENDER_MODE_FIT, 0)); } ``` ```kotlin /** * Initializes the local video view and sets the display properties. * This method adds a SurfaceView to the local video container and configures it. */ private fun setupLocalVideo() { val container: FrameLayout = findViewById(R.id.local_video_view_container) val surfaceView = SurfaceView(baseContext) container.addView(surfaceView) mRtcEngine.setupLocalVideo(VideoCanvas(surfaceView, VideoCanvas.RENDER_MODE_FIT, 0)) } ``` ### Display remote video When a remote user joins the channel, call `setupRemoteVideo` and pass in the remote user's `uid`, obtained from the `onUserJoined` callback, to display the remote video. Java Kotlin ```java private void setupRemoteVideo(int uid) { FrameLayout container = findViewById(R.id.remote_video_view_container); SurfaceView surfaceView = new SurfaceView(getBaseContext()); surfaceView.setZOrderMediaOverlay(true); container.addView(surfaceView); mRtcEngine.setupRemoteVideo(new VideoCanvas(surfaceView, VideoCanvas.RENDER_MODE_FIT, uid)); } ``` ```kotlin private fun setupRemoteVideo(uid: Int) { val container = findViewById(R.id.remote_video_view_container) val surfaceView = SurfaceView(baseContext).apply { setZOrderMediaOverlay(true) } container.addView(surfaceView) mRtcEngine.setupRemoteVideo(VideoCanvas(surfaceView, VideoCanvas.RENDER_MODE_FIT, uid)) } ``` ### Handle permissions To access the camera and microphone on Android devices, declare the necessary permissions in the app's manifest and ensure that the user grants these permissions when the app starts. 1. Open your project's `AndroidManifest.xml` file and add the following permissions before ``: ```xml ``` 2. Use the following code to handle runtime permissions in your Android app. The logic ensures that the necessary permissions are granted before starting Broadcast Streaming. In your `MainActivity` file, add the following code: Java Kotlin ```java private boolean checkPermissions() { for (String permission : getRequiredPermissions()) { if (ContextCompat.checkSelfPermission(this, permission) != PackageManager.PERMISSION_GRANTED) { return false; } } return true; } private String[] getRequiredPermissions() { if (android.os.Build.VERSION.SDK_INT >= android.os.Build.VERSION_CODES.S) { return new String[]{ Manifest.permission.RECORD_AUDIO, Manifest.permission.CAMERA, Manifest.permission.READ_PHONE_STATE, Manifest.permission.BLUETOOTH_CONNECT }; } else { return new String[]{ Manifest.permission.RECORD_AUDIO, Manifest.permission.CAMERA }; } } @Override public void onRequestPermissionsResult(int requestCode, @NonNull String[] permissions, @NonNull int[] grantResults) { if (requestCode == PERMISSION_REQ_ID && checkPermissions()) { startBroadcastStreaming(); } } ``` ```kotlin private val PERMISSION_REQ_ID = 22 private fun requestPermissions() { ActivityCompat.requestPermissions(this, getRequiredPermissions(), PERMISSION_REQ_ID) } private fun checkPermissions(): Boolean { for (permission in getRequiredPermissions()) { if (ContextCompat.checkSelfPermission(this, permission) != PackageManager.PERMISSION_GRANTED) { return false } } return true } private fun getRequiredPermissions(): Array { return if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.S) { arrayOf( Manifest.permission.RECORD_AUDIO, Manifest.permission.CAMERA, Manifest.permission.READ_PHONE_STATE, Manifest.permission.BLUETOOTH_CONNECT ) } else { arrayOf( Manifest.permission.RECORD_AUDIO, Manifest.permission.CAMERA ) } } override fun onRequestPermissionsResult( requestCode: Int, permissions: Array, grantResults: IntArray ) { super.onRequestPermissionsResult(requestCode, permissions, grantResults) if (requestCode == PERMISSION_REQ_ID && checkPermissions()) { startBroadcastStreaming() } } ``` ### Start and close the app When a user launches your app, start real-time interaction. When a user closes the app, stop the interaction. 1. In the `onCreate` callback, check whether the app has been granted the required permissions. If the permissions have not been granted, request the required permissions from the user. If permissions are granted, initialize `RtcEngine` and join a channel. Java Kotlin ```java @Override protected void onCreate(Bundle savedInstanceState) { super.onCreate(savedInstanceState); setContentView(R.layout.activity_main); if (checkPermissions()) { startBroadcastStreaming(); } else { requestPermissions(); } } ``` ```kotlin override fun onCreate(savedInstanceState: Bundle?) { super.onCreate(savedInstanceState) setContentView(R.layout.activity_main) if (checkPermissions()) { startBroadcastStreaming() } else { requestPermissions() } } ``` 2. When a user closes the app, or switches the app to the background, call `stopPreview` to stop the video preview and then call `leaveChannel` to leave the current channel and release all session-related resources. Java Kotlin ```java private void cleanupAgoraEngine() { if (mRtcEngine != null) { mRtcEngine.stopPreview(); mRtcEngine.leaveChannel(); mRtcEngine = null; } } ``` ```kotlin private fun cleanupAgoraEngine() { mRtcEngine?.apply { stopPreview() leaveChannel() } mRtcEngine = null } ``` ### Complete sample code A complete code sample demonstrating the basic process of real-time interaction is provided for your reference. To use the sample code, copy the following lines into the `MainActivity` file in your project. Then, replace `` in package `com.example.` with your project's name. Java Kotlin ```java package com.example. import android.Manifest; import android.content.pm.PackageManager; import android.os.Bundle; import android.view.SurfaceView; import android.widget.FrameLayout; import android.widget.Toast; import androidx.annotation.NonNull; import androidx.appcompat.app.AppCompatActivity; import androidx.core.app.ActivityCompat; import androidx.core.content.ContextCompat; import io.agora.rtc2.ChannelMediaOptions; import io.agora.rtc2.Constants; import io.agora.rtc2.IRtcEngineEventHandler; import io.agora.rtc2.RtcEngine; import io.agora.rtc2.RtcEngineConfig; import io.agora.rtc2.video.VideoCanvas; public class MainActivity extends AppCompatActivity { private static final int PERMISSION_REQ_ID = 22; // Fill in the app ID from Agora Console private String myAppId = ""; // Fill in the channel name private String channelName = ""; // Fill in the temporary token generated from Agora Console private String token = ""; private RtcEngine mRtcEngine; private final IRtcEngineEventHandler mRtcEventHandler = new IRtcEngineEventHandler() { // Callback when successfully joining the channel @Override public void onJoinChannelSuccess(String channel, int uid, int elapsed) { super.onJoinChannelSuccess(channel, uid, elapsed); showToast("Joined channel " + channel); } // Callback when a remote user or host joins the current channel @Override public void onUserJoined(int uid, int elapsed) { super.onUserJoined(uid, elapsed); runOnUiThread(() -> { // When a remote user joins the channel, display the remote video stream for the specified uid setupRemoteVideo(uid); showToast("User joined: " + uid); // Show toast for user joining }); } // Callback when a remote user or host leaves the current channel @Override public void onUserOffline(int uid, int reason) { super.onUserOffline(uid, reason); runOnUiThread(() -> { showToast("User offline: " + uid); // Show toast for user going offline }); } }; @Override protected void onCreate(Bundle savedInstanceState) { super.onCreate(savedInstanceState); setContentView(R.layout.activity_main); if (checkPermissions()) { startBroadcastStreaming(); } else { requestPermissions(); } } private void requestPermissions() { ActivityCompat.requestPermissions(this, getRequiredPermissions(), PERMISSION_REQ_ID); } private boolean checkPermissions() { for (String permission : getRequiredPermissions()) { if (ContextCompat.checkSelfPermission(this, permission) != PackageManager.PERMISSION_GRANTED) { return false; } } return true; } private String[] getRequiredPermissions() { if (android.os.Build.VERSION.SDK_INT >= android.os.Build.VERSION_CODES.S) { return new String[]{ Manifest.permission.RECORD_AUDIO, Manifest.permission.CAMERA, Manifest.permission.READ_PHONE_STATE, Manifest.permission.BLUETOOTH_CONNECT }; } else { return new String[]{ Manifest.permission.RECORD_AUDIO, Manifest.permission.CAMERA }; } } @Override public void onRequestPermissionsResult(int requestCode, @NonNull String[] permissions, @NonNull int[] grantResults) { super.onRequestPermissionsResult(requestCode, permissions, grantResults); if (requestCode == PERMISSION_REQ_ID && checkPermissions()) { startBroadcastStreaming(); } } private void startBroadcastStreaming() { initializeAgoraVideoSDK(); enableVideo(); setupLocalVideo(); joinChannel(); } private void initializeAgoraVideoSDK() { try { RtcEngineConfig config = new RtcEngineConfig(); config.mContext = getBaseContext(); config.mAppId = myAppId; config.mEventHandler = mRtcEventHandler; mRtcEngine = RtcEngine.create(config); } catch (Exception e) { throw new RuntimeException("Error initializing RTC engine: " + e.getMessage()); } } private void enableVideo() { mRtcEngine.enableVideo(); mRtcEngine.startPreview(); } private void setupLocalVideo() { FrameLayout container = findViewById(R.id.local_video_view_container); SurfaceView surfaceView = new SurfaceView(getBaseContext()); container.addView(surfaceView); mRtcEngine.setupLocalVideo(new VideoCanvas(surfaceView, VideoCanvas.RENDER_MODE_FIT, 0)); } private void joinChannel() { // Create an instance of ChannelMediaOptions and configure it ChannelMediaOptions options = new ChannelMediaOptions(); // Set the user role to BROADCASTER or AUDIENCE according to the use-case options.clientRoleType = Constants.CLIENT_ROLE_BROADCASTER; // In the live broadcast use-case, set the channel profile to BROADCASTING (live broadcast use-case) options.channelProfile = Constants.CHANNEL_PROFILE_LIVE_BROADCASTING; // Set the audience latency level options.audienceLatencyLevel = Constants.AUDIENCE_LATENCY_LEVEL_LOW_LATENCY; // Publish local media options.publishCameraTrack = true; options.publishMicrophoneTrack = true; mRtcEngine.joinChannel(token, channelName, 0, options); } private void setupRemoteVideo(int uid) { FrameLayout container = findViewById(R.id.remote_video_view_container); SurfaceView surfaceView = new SurfaceView(getBaseContext()); surfaceView.setZOrderMediaOverlay(true); container.addView(surfaceView); mRtcEngine.setupRemoteVideo(new VideoCanvas(surfaceView, VideoCanvas.RENDER_MODE_FIT, uid)); } @Override protected void onDestroy() { super.onDestroy(); cleanupAgoraEngine(); } private void cleanupAgoraEngine() { if (mRtcEngine != null) { mRtcEngine.stopPreview(); mRtcEngine.leaveChannel(); mRtcEngine = null; } } private void showToast(String message) { runOnUiThread(() -> Toast.makeText(MainActivity.this, message, Toast.LENGTH_SHORT).show()); } } ``` ```kotlin package com.example. import android.Manifest import android.content.pm.PackageManager import android.os.Build import android.os.Bundle import android.view.SurfaceView import android.widget.FrameLayout import android.widget.Toast import androidx.appcompat.app.AppCompatActivity import androidx.core.app.ActivityCompat import androidx.core.content.ContextCompat import io.agora.rtc2.ChannelMediaOptions import io.agora.rtc2.Constants import io.agora.rtc2.IRtcEngineEventHandler import io.agora.rtc2.RtcEngine import io.agora.rtc2.RtcEngineConfig import io.agora.rtc2.video.VideoCanvas class MainActivity : AppCompatActivity() { private val PERMISSION_REQ_ID = 22 private val myAppId = "" private val channelName = "" private val token = "" private var mRtcEngine: RtcEngine? = null private val mRtcEventHandler = object : IRtcEngineEventHandler() { override fun onJoinChannelSuccess(channel: String?, uid: Int, elapsed: Int) { super.onJoinChannelSuccess(channel, uid, elapsed) runOnUiThread { showToast("Joined channel $channel") } } override fun onUserJoined(uid: Int, elapsed: Int) { runOnUiThread { setupRemoteVideo(uid) } } override fun onUserOffline(uid: Int, reason: Int) { super.onUserOffline(uid, reason) runOnUiThread { showToast("User offline: $uid") } } } override fun onCreate(savedInstanceState: Bundle?) { super.onCreate(savedInstanceState) setContentView(R.layout.activity_main) if (checkPermissions()) { startBroadcastStreaming() } else { requestPermissions() } } private fun requestPermissions() { ActivityCompat.requestPermissions(this, getRequiredPermissions(), PERMISSION_REQ_ID) } private fun checkPermissions(): Boolean { for (permission in getRequiredPermissions()) { if (ContextCompat.checkSelfPermission(this, permission) != PackageManager.PERMISSION_GRANTED) { return false } } return true } private fun getRequiredPermissions(): Array { return if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.S) { arrayOf( Manifest.permission.RECORD_AUDIO, Manifest.permission.CAMERA, Manifest.permission.READ_PHONE_STATE, Manifest.permission.BLUETOOTH_CONNECT ) } else { arrayOf( Manifest.permission.RECORD_AUDIO, Manifest.permission.CAMERA ) } } override fun onRequestPermissionsResult( requestCode: Int, permissions: Array, grantResults: IntArray ) { super.onRequestPermissionsResult(requestCode, permissions, grantResults) if (requestCode == PERMISSION_REQ_ID && checkPermissions()) { startBroadcastStreaming() } } private fun startBroadcastStreaming() { initializeAgoraVideoSDK() enableVideo() setupLocalVideo() joinChannel() } private fun initializeAgoraVideoSDK() { try { val config = RtcEngineConfig().apply { mContext = applicationContext mAppId = myAppId mEventHandler = mRtcEventHandler } mRtcEngine = RtcEngine.create(config) } catch (e: Exception) { throw RuntimeException("Error initializing RTC engine: ${e.message}") } } private fun enableVideo() { mRtcEngine?.apply { enableVideo() startPreview() } } private fun setupLocalVideo() { val container = findViewById(R.id.local_video_view_container) val surfaceView = SurfaceView(this) container.addView(surfaceView) mRtcEngine?.setupLocalVideo(VideoCanvas(surfaceView, VideoCanvas.RENDER_MODE_FIT, 0)) } private fun joinChannel() { val options = ChannelMediaOptions().apply { clientRoleType = Constants.CLIENT_ROLE_BROADCASTER channelProfile = Constants.CHANNEL_PROFILE_LIVE_BROADCASTING audienceLatencyLevel = Constants.AUDIENCE_LATENCY_LEVEL_LOW_LATENCY publishMicrophoneTrack = true publishCameraTrack = true } mRtcEngine?.joinChannel(token, channelName, 0, options) } private fun setupRemoteVideo(uid: Int) { val container = findViewById(R.id.remote_video_view_container) val surfaceView = SurfaceView(this).apply { setZOrderMediaOverlay(true) } container.addView(surfaceView) mRtcEngine?.setupRemoteVideo(VideoCanvas(surfaceView, VideoCanvas.RENDER_MODE_FIT, uid)) } override fun onDestroy() { super.onDestroy() cleanupAgoraEngine() } private fun cleanupAgoraEngine() { mRtcEngine?.apply { stopPreview() leaveChannel() } mRtcEngine = null } private fun showToast(message: String) { runOnUiThread { Toast.makeText(this, message, Toast.LENGTH_SHORT).show() } } } ``` For the `myAppId` and `token` variables, replace the placeholders with the values you obtained from Agora Console. Ensure you enter the same `channelName` you used when generating the temporary token. ### Create a user interface To connect the sample code to your existing UI, ensure that your XML layout includes the container UI element IDs used to [Display the local video](#display-the-local-video) and [Display remote video](#display-remote-video). Alternatively, use the following sample code to generate a basic user interface. Replace the existing content in `/app/src/main/res/layout/activity_main.xml` with this code. ![UI design](https://assets-docs.agora.io/images/video-sdk/quickstart-ui-android-design.png) **Sample code to create the user interface** ```xml ``` ## Test the sample code Take the following steps to test the sample code: 1. In `MainActivity` update the values for `myAppId`, and `token` with values from Agora Console. Fill in the same `channelName` you used to generate the token. 2. Enable developer options on your Android test device. Turn on USB debugging, connect the Android device to your development machine through a USB cable, and check that your device appears in the Android device options. 3. In Android Studio, click ![image](https://assets-docs.agora.io/images/video-sdk/icon_android_gradle_sync.png) **Sync Project with Gradle Files** to resolve project dependencies and update the configuration. 4. After synchronization is successful, click ![image](https://assets-docs.agora.io/images/video-sdk/icon_android_run.png) **Run app**. Android Studio starts compilation. After a few moments, the app is installed on your Android device. 5. Launch the App, grant recording and camera permissions. If you set the user role to host, you will see yourself in the local view. 6. On a second Android device, repeat the previous steps to install and launch the app. Alternatively, use the [Web demo](https://webdemo-global.agora.io/example/quickStart/videoAndVoiceCalling/index.html) to join the same channel and test the following use-cases: * If users on both devices join the channel as hosts, they can see and hear each other. * If one user joins as host and the other as audience, the host can see themselves in the local video window; the audience can see the host in the remote video window and hear the host. ## Reference This section contains content that completes the information on this page, or points you to documentation that explains other aspects to this product. * If a firewall is deployed in your network environment, refer to [Connect with Cloud Proxy](build/optimize-quality-and-connection/cloud-proxy.mdx) to use Agora services normally. ### Next steps After implementing the quickstart sample, read the following documents to learn more: * To ensure communication security in a test or production environment, best practice is to obtain and use a token from an authentication server. For details, see [Secure authentication with tokens](build/authenticate-users/use-tokens.mdx). ### Sample project Agora provides open source sample projects on [GitHub](https://github.com/AgoraIO/API-Examples) for your reference. Download or view the [JoinChannelVideo](https://github.com/AgoraIO-Community/Agora-RTC-QuickStart/tree/main/Android/Agora-RTC-QuickStart-Android) project for a more detailed example. ### API reference * [`RtcEngineConfig`](https://api-ref.agora.io/en/video-sdk/android/4.x/API/class_rtcengineconfig.html) * [`create`](https://api-ref.agora.io/en/video-sdk/android/4.x/API/class_irtcengine.html#api_irtcengine_create) * [`ChannelMediaOptions`](https://api-ref.agora.io/en/video-sdk/android/4.x/API/class_channelmediaoptions.html) * [`joinChannel`](https://api-ref.agora.io/en/video-sdk/android/4.x/API/class_irtcengine.html#api_irtcengine_joinchannel2) * [`enableVideo`](https://api-ref.agora.io/en/video-sdk/android/4.x/API/class_irtcengine.html#api_irtcengine_enablevideo) * [`startPreview`](https://api-ref.agora.io/en/video-sdk/android/4.x/API/class_irtcengine.html#api_irtcengine_startpreview) * [`leaveChannel`](https://api-ref.agora.io/en/video-sdk/android/4.x/API/class_irtcengine.html#api_irtcengine_leavechannel) * [`IRtcEngineEventHandler`](https://api-ref.agora.io/en/video-sdk/android/4.x/API/class_irtcengineeventhandler.html#class_irtcengineeventhandler) ### Frequently asked questions * [How can I fix black screen issues?](/en/api-reference/faq/quality/video_blank) * [Why can't I turn on the camera?](/en/api-reference/faq/quality/video_camera) * [How can I listen for audience joining or leaving a channel?](/en/api-reference/faq/integration/audience_event) * [How can I solve channel-related issues?](/en/api-reference/faq/integration/channel) * [How can I set the log file?](/en/api-reference/faq/integration/set_log_file) * [Why do apps on some Android versions fail to capture audio and video after screen locking or switching to the background?](/en/api-reference/faq/quality/android_background) ### See also * [Error codes](reference/error-codes.md) * [Connection status management](build/optimize-quality-and-connection/connection-status-management.mdx) ## Platform-specific versions - [Android](/en/realtime-media/broadcast-streaming/quickstart/android.md) - [iOS](/en/realtime-media/broadcast-streaming/quickstart/ios.md) - [macOS](/en/realtime-media/broadcast-streaming/quickstart/macos.md) - [Web](/en/realtime-media/broadcast-streaming/quickstart/web.md) - [Windows](/en/realtime-media/broadcast-streaming/quickstart/windows.md) - [Electron](/en/realtime-media/broadcast-streaming/quickstart/electron.md) - [Flutter](/en/realtime-media/broadcast-streaming/quickstart/flutter.md) - [React Native](/en/realtime-media/broadcast-streaming/quickstart/react-native.md) - [JavaScript](/en/realtime-media/broadcast-streaming/quickstart/javascript.md) - [Unity](/en/realtime-media/broadcast-streaming/quickstart/unity.md) - [Unreal Engine](/en/realtime-media/broadcast-streaming/quickstart/unreal.md) - [Unreal Blueprint](/en/realtime-media/broadcast-streaming/quickstart/blueprint.md) - [Python](/en/realtime-media/broadcast-streaming/quickstart/python.md) # Agora skills (/en/realtime-media/broadcast-streaming/skills) Agora skills is a set of structured reference files that give AI coding assistants deep knowledge of Agora's platform. When you ask your assistant to build something with Agora, it loads the relevant skill files covering products, APIs, and platform-specific code examples, so it can generate working code without guessing. Skills includes integration with the [Agora MCP server](#agora-mcp-server), which gives your assistant access to live Agora documentation. For installation instructions and supported tools, see the [Agora Skills repository](https://github.com/AgoraIO/skills). ### Installation Install Agora Skills using one of the following methods: #### Skills CLI (recommended) Run the following command: ```bash npx skills add github:AgoraIO/skills ``` Skills activate automatically when your agent detects relevant tasks, for example, "build a voice agent", "integrate Agora RTC", or "generate a token". #### Manual installation Clone the repository once and point your AI coding assistant to the skill files directly. 1. Clone the [Agora Skills repo](https://github.com/AgoraIO/skills.git): ```bash git clone https://github.com/AgoraIO/skills.git ~/agora-skills ``` 2. Point your AI assistant to `skills/agora/`. Follow the instructions for your AI coding assistant: Claude Code Cursor Windsurf GitHub Copilot Other tools Run the following commands inside Claude Code: * **User-level** (available across all your projects) ```bash /plugin marketplace add AgoraIO/skills /plugin install agora ``` * **Project-level** (shared with your team via version control) ```bash /plugin marketplace add AgoraIO/skills /plugin install agora --scope project ``` Copy or symlink `skills/agora/` into `.cursor/rules/`. For more information, see [Cursor skill directories](https://cursor.com/docs/skills#skill-directories). Add `skills/agora/` to your Cascade context. For more information, see [Windsurf skills](https://docs.windsurf.com/windsurf/cascade/skills). Reference via `@workspace` or add to `.github/copilot-instructions.md`. For more information, see [Create skills for Copilot in the CLI](https://docs.github.com/en/copilot/how-tos/copilot-cli/customize-copilot/create-skills) or [Create skills for the Copilot coding agent](https://docs.github.com/en/copilot/how-tos/use-copilot-agents/coding-agent/create-skills). The skill files are plain markdown. Point your tool to `skills/agora/` or load individual files directly. Use `SKILL.md` as the entry point. # Subscription packages (/en/realtime-media/broadcast-streaming/subscription-packages) A subscription package is a prepaid billing method. You can purchase a package in the [`Agora Console`](https://console.agora.io/subscriptions/rtc-plans?tab=monthly) to cover usage for the current month. This page explains how to purchase, and manage subscription and top-up packages. ### Purchase and upgrade By default, a new account is assigned the Free package when the first project is created. You can upgrade at any time to the Starter, Pro, Business, Business Advanced, or Business Plus. Higher-tier packages provide greater discounts and lower unit prices. To select your package: 1. Log in to [Agora Console](https://console.agora.io/). 2. Click the username at the bottom of the navigation sidebar to open the account popup menu. 3. Select **Subscriptions**. 4. Switch to the **RTC** tab. 5. Select your package and click **Upgrade** or **Downgrade**. ![Subscription plans](https://assets-docs.agora.io/images/console/subscription-plans.png) * For non-contracted customers, all packages, except the Enterprise package can be purchased directly from the Agora Console. To upgrade to the Enterprise package, contact [Agora sales](mailto\:sales@agora.io). * You can upgrade sequentially or skip levels. For example, you may upgrade from Starter to Business Plus directly. * Package upgrades take effect immediately and apply to the entire calendar month. If you make multiple upgrades on the same day, only the last upgrade is applied. * If your account has a negative balance, purchasing a prepaid package first charges your credit card for the outstanding balance, then for the package fee. #### Upgrade rules When you upgrade to a paid package, it becomes effective for the entire calendar month, regardless of the purchase date. You receive the full monthly allocation of included minutes. The cost of the previous package is deducted from the new package price, so you only pay the difference. If you request an upgrade after auto-renewal has triggered, you may upgrade for the next month as well. ### Renewal By default, your subscription package renews automatically 48 hours before it expires. Ensure your account has sufficient balance or enable auto-pay to avoid renewal failure. You can unsubscribe at any time to prevent renewal. See [Cancel subscription](#cancel-subscription). If renewal fails: * The system sends a notification and retries the renewal. * If renewal still fails by the time the package expires, the package downgrades to the Free package. * To restore the previous package, see [Purchase and Upgrade](#purchase-and-upgrade). ### Downgrade You can downgrade by selecting and purchasing a lower-tier package. A downgrade takes effect when your current package expires if you request it before auto-renewal is triggered. You are charged for the new package when you request the downgrade. You can downgrade only once per month. * If your downgrade for the next month is successful, auto-renewal for the current package is not triggered. * If a package is already active, you must wait until the 1st day of that month to initiate a downgrade for the following month. #### Example * **Requested more than 48 hours before the current package expires**: The downgrade takes effect after the current package expires. * **Requested less than 48 hours before the current package expires**: Since a package is already active for the next month, you must wait until the new package starts to downgrade for the following month. ### Cancel subscription To cancel a prepaid package, downgrade to the Free package. Downgrade rules apply. ## Top-up package A top-up package is a pay-as-you-go plan. You can purchase a top-up package from the [`Agora Console`](https://console.agora.io/subscriptions/rtc-plans?tab=top-up) to cover usage that exceeds your monthly package quota. ### Purchase You can purchase multiple top-up packages. They are deducted in the order of purchase. #### Example: * Purchased 250,000-minutes top-up packages on **August 15** and **August 16**. On **September 1**, minutes consumed in August are deducted from the first package, then from the second. ### Validity Each top-up package is valid for one year, starting from the purchase date to the end of the same month of the following year. #### Example: * Purchased on **August 15, 2025** → valid until **August 31, 2026** Unused minutes expire after the validity period and cannot be carried forward. ### Cancel top-up package Cancelling a top-up package is not supported. ## Free tier overage For Free tier users, exceeding the included minutes and any purchased top-up minutes results in service suspension. Service resumes when you purchase additional top-up minutes or upgrade to a prepaid subscription package. # Core concepts (/en/realtime-media/cloud-recording/core-concepts) Agora Cloud Recording enables you to record video and voice calls or streams in the cloud for storage or on-demand viewing. Cloud Recording works with Voice Calling, Video Calling, Broadcast Streaming and Interactive Live Streaming. This page introduces the key processes and concepts you need to know to use Cloud Recording. ## General concepts Agora relies on the following fundamental concepts to enable seamless real-time communication: ### Agora SDRTN® Agora's core engagement services are powered by its Software-Defined Real-Time Network (SDRTN®), a global infrastructure accessible anytime, anywhere. Unlike traditional networks, Agora SDRTN® is not restricted by devices, phone numbers, or telecom coverage areas. With data centers in over 200 countries and regions, it ensures sub-second latency and high availability for real-time media. Agora SDRTN® enables live user engagement through real-time communication (RTC), offering: * Unmatched quality of service * High availability and accessibility * True scalability * Low cost ### App ID The App ID is a unique key generated by Agora to identify each project and provide billing and other statistical data services. The App ID is critical for connecting users within your app. It is used to initialize the ENGINE in your app, and as one of the required keys to create authentication tokens for secure communication. Retrieve the App ID for your project using the [Agora Console](https://console.agora.io/legacy/project-management). App IDs are stored on the front-end client and do not provide access control. Projects using only an App ID allow any user with the App ID to join. For access control, especially in production environments, choose the **App ID + Token** mechanism for user authentication when creating a new project. Without authentication tokens, your environment is open to anyone with access to your App ID. ### App Certificate An App Certificate is a unique key generated by the Agora Console to secure projects through token authentication. It is required, along with the App ID, to generate a token that proves authorization between your systems and Agora's network. App Certificates are used to generate Cloud Recording authentication tokens. Store the App Certificate securely in your backend systems. If your App Certificate is compromised or to meet security compliance requirements, you can invalidate certificates and create new ones through the Agora Console. ### Tokens A token is a dynamic key generated using the App ID, App Certificate, user ID, and expiration timestamp. Tokens authenticate and secure access to Agora's services, ensuring only authorized users can join a channel and participate in real-time communication. Tokens are generated on your server and passed to the client for use in Cloud Recording. The token generation process involves digitally signing the App ID, App Certificate, user ID, and expiration timestamp using a specific algorithm, preventing tampering or forgery. During development and testing, use the Agora Console to generate temporary tokens. For production environments, implement a token server as part of your security infrastructure to control access to your channels. For information on setting up a token server for generating and managing tokens, refer to [Deploy a token server](/en/realtime-media/cloud-recording/build/set-up-authentication/authentication-workflow). ### Channel A channel organizes users into a group and is identified by a unique channel name. Users who connect to the same channel are able to communicate with each other. A channel is created when the first user joins and ceases to exist when the last user leaves. Channels are created by calling the methods for transmitting real-time data. Agora uses different channels to transmit different types of data: * An RTC channel is used for transmitting audio or video data. * A MESS channel is used for transmitting messaging or signaling data. These channels are independent of each other. Additional services provided by Agora, such as Cloud Recording and STT, join the SDK channel to provide real-time recording, transmission acceleration, media playback, and content moderation. ### User ID In Cloud Recording, the UID is an integer value that uniquely identifies a user within the context of a channel. When joining a channel, you have the option to either assign a specific UID to the user or pass `0` or `null` and allow Agora to automatically generate and assign a UID to the user. If two users attempt to join the same channel with the same UID, it can lead to unexpected behavior. The UID is used by Agora's services and components to identify and manage users within a channel. Ensure that UIDs are properly assigned to prevent conflicts. ### Agora Console To use Agora Cloud Recording, create a project in the [Agora Console](https://console.agora.io/) first. ![Create project in Agora Console](https://assets-docs.agora.io/images/common/create-project.svg) Agora Console provides an intuitive interface for developers to query and manage their Agora account. After registering an Agora account, you use the Agora Console to perform the following tasks: * Manage your account * Create and configure Agora projects and services * Get an App ID and the App certificate * Generate temporary tokens for development and testing * Manage members and roles * Check call quality and usage * Check bills and make payments * Access product resources See [Agora account management](/en/introduction/account) for details on how to manage all aspects of your Agora account. Agora also provides RESTful APIs that you use to implement features such as creating a project and fetching usage numbers programmatically. ## Cloud recording concepts ### Recording modes Agora Cloud Recording supports three recording modes: * Individual recording * Composite recording * Web page recording After the recording is complete, the recorded content is uploaded as a `TS` file to the third-party cloud storage you specified. An `M3U8` file is also generated to serve as an index for the corresponding `TS` file The working principles of different recording modes and the types of files generated by Cloud Recording are as follows: #### Individual recording In individual recording, the recording service records the audio and video streams of each UID in the channel separately. After the recording is complete, the recording service generates the corresponding audio and video files for each UID. For example, if there are 3 UIDs in the channel and each UID sends audio and video, then in the individual recording mode, 3 audio files and 3 video files are generated. #### Composite recording In mixed recording, the recording service combines the audio and video of multiple UIDs in the channel into a single audio and video file. For example, if there are 3 UIDs in the channel and each sends audio and video, the mixed recording mode generates one recording file that includes the audio and video of all UIDs. #### Web page recording In web page recording, the recording service combines the page content and audio of a specified web page into an audio and video file. Web page recording is commonly used in the following use-cases: * In online classrooms, to record the teacher and student audio and video along with courseware, whiteboard, and other visuals. * In video conferences, to capture participants' audio and video, as well as whiteboard, PPT, and other visuals. ### Transcoding and non-transcoding modes In individual recording, audio transcoding and non-transcoding modes have different use cases and characteristics. **Individual recording with transcoding**: This mode is used in use-cases where unified audio encoding parameters are needed to ensure consistent recording file formats and parameters for easier post-processing and playback. It is commonly used in cases requiring high compatibility and standardized output, such as wide player support and standardized storage. **Individual recording without transcoding**: This mode is used when the original audio encoding parameters must be preserved to maintain the sound quality and performance. It is often used in use-cases with high demands for real-time performance and original sound quality, such as high-fidelity audio recording. | Feature | Individual recording with transcoding | Individual recording without transcoding | | :--------------------------------- | :------------------------------------------------------------------------------------------------------ | :-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Transcoding during audio encoding | Yes | No | | Raw audio data | The sampling rate, number of channels, and bitrate are fixed at 48 kHz, mono, and 48 Kbps respectively. | The bitrate, sampling rate and number of channels are determined by the audio encoding parameters of the streaming end `AudioProfile`. | | Audio encoding format | LC-AAC | Determined by the configuration of the source end `AudioProfile` | | Generated recording files | Each UID generates an audio file in `M3U8` format and multiple audio files in `TS` format. | Same as transcoding recording. If the user stops streaming using `muteLocalAudioStream` or `leaveChannel` audio recording stops immediately, and there is no 15 seconds of silent data. | | Player compatibility | The recorded file can be played by any mainstream player that supports the HLS protocol. | The audio encoding format is determined by the configuration of the streaming end `AudioProfile`. Different audio encoding formats have different compatibility. | ### Delayed transcoding Delayed transcoding is designed for audio-only recording use-cases. When you enable this mode, the recording service merges and transcodes the audio files of all users in the specified channel into an `MP3`, `M4A`, or `AAC` file within 24 hours after the recording ends (or up to 48 hours in special cases) and uploads it to the specified third-party cloud storage. ### Delayed audio mixing Delayed audio mixing is used for individual audio recording use-cases. To obtain a mixed recording file of all users in the channel after recording, you enable the delayed audio mixing feature when starting individual audio recording without transcoding. Once enabled, the recording service merges and transcodes the audio files of all users in the specified channel into an `MP3`, `M4A`, or `AAC` file within 24 hours after the recording is complete (or up to 48 hours in special cases) and uploads it to the specified third-party cloud storage. ### Slicing Slicing involves cutting audio and video data according to specific rules during the recording process to generate multiple recording files. After slicing, several slice files (such as `TS` or `WebM` files) are created, along with `M3U8` files that store the indexes of these slice files. # Cloud Recording overview (/en/realtime-media/cloud-recording) Agora's Cloud Recording is a RESTful API-based solution for recording real-time voice, video, and interactive streaming sessions directly to your cloud storage. It supports leading CDN and cloud hosting providers, offers customizable recording layouts, and supports multiple file formats for playback and distribution. Easily launch multiple recorders within a single channel to ensure redundancy or to capture different layouts and formats simultaneously. This makes Cloud Recording a flexible and reliable solution for recording and preserving real-time communication. ## Features Supports globally distributed cluster deployment and automatically backs up files on Agora’s cloud server when the third-party cloud storage fails. End-to-end security mechanisms for voice and video calls, data transmission, and storage. Supports third-party cloud storage, including Amazon S3, Alibaba Cloud, Tencent Cloud, Kingsoft Cloud, Qiniu Cloud, Microsoft Azure, Google Cloud, Huawei Cloud, and Baidu AI Cloud. Simple implementation and easy to learn with RESTful API calls that allow you to start, stop, and query the recording. # Agora MCP (/en/realtime-media/cloud-recording/mcp) The Agora MCP server gives your AI assistant direct access to Agora's documentation, so it can look up APIs, SDK methods, and platform-specific details in real time. The Agora MCP server is included when you install Agora Skills. If you prefer to install only the MCP server, it is available at: ```text https://mcp.agora.io ``` ## Installation Refer to the installation instructions for your coding assistant. Cursor Claude Codex Gemini CLI Manual installation Click the button below to install the MCP server in [Cursor](https://www.cursor.com/) Install Agora MCP Server in Cursor or add it manually with the following JSON: ```json { "mcpServers": { "agora-docs": { "url": "https://mcp.agora.io" } } } ``` * **Claude Code** Run the following command in your terminal to install the MCP server in [Claude Code](https://claude.com/product/claude-code): ```bash claude mcp add --transport http agora-docs https://mcp.agora.io ``` * **Claude Desktop** In **Settings**, select **Connectors** and then choose **Add custom connector**. Enter the following values and click **Add**: * **Name**: `agora-docs` * **Remote MCP server URL**: `https://mcp.agora.io` Run the following command in your terminal to install the server in [OpenAI Codex](https://openai.com/codex/): ```bash codex mcp add --url https://mcp.agora.io agora-docs ``` Run the following command in your terminal to install the server in [Gemini CLI](https://github.com/google-gemini/gemini-cli): ```bash gemini mcp add --transport http agora-docs https://mcp.agora.io ``` Add the server URL `https://mcp.agora.io` to your MCP client of choice. If prompted, set the transport to `http` or "Streamable HTTP". ## Getting started Once installed, your coding assistant has access to Agora's documentation through the MCP server. The assistant will intelligently use this resource when relevant to your questions. For more targeted results, mention Agora along with your target product and platform, such as `Cloud Recording`, `Web`, and `REST API` in your prompts. ## AGENTS.md To get the most out of Agora Skills and the MCP server, Agora recommends that you include an [AGENTS.md](https://agents.md) or similar file in your repository. This file ensures that your agent always consults Agora docs to produce reliable, working code. Add the following instructions to your own `AGENTS.md` file: ```markdown ## Agora Documentation Agora documentation is updated frequently. You should always refer to the latest documentation when working with Agora products. Agora offers an MCP server that you can use to browse and search its documentation. If the developer has not yet installed this server, recommend that they install it at https://mcp.agora.io. If the Agora Skills files are available in your context, use them for platform-specific API guidance and code examples. ``` # Quickstart using middleware (/en/realtime-media/cloud-recording/middleware-quickstart) To streamline the use of Agora RESTful APIs within your infrastructure, Agora’s developer community offers the open-source [Agora Go Backend Middleware](https://github.com/AgoraIO-Community/agora-go-backend-middleware). This backend provides RESTful APIs for tasks such as token generation, cloud recording management, and real-time transcription. It simplifies the integration of Agora’s cloud services into your real-time voice and video applications. This guide shows you how to implement cloud recording using the community middleware. ## Understand the tech The following figure illustrates the architecture of the middleware cloud recording micro service. ![](https://assets-docs.agora.io/images/cloud-recording/cloud-recording-middleware.svg) ## Set up and run the Go backend middleware Take the following steps to set up and run the middleware project: 1. **Clone the repository** ```bash git clone https://github.com/AgoraIO-Community/agora-go-backend-middleware.git ``` 2. **Install dependencies** Ensure you have [Go](https://go.dev/doc/install) installed on your system. Navigate to the project directory and install the project dependencies: ```bash cd agora-go-backend-middleware go mod download ``` 3. **Configure environment variables** 1. Copy the example `.env` file. ```bash cp .env.example .env ``` 2. Update the following variables in the `.env` file: * `APP_ID`: Your Agora App ID. * `APP_CERTIFICATE`: Your Agora App Certificate. * `CUSTOMER_ID`: Your customer ID * `CUSTOMER_SECRET`: Your Customer Secret * `STORAGE_VENDOR`: Cloud storage vendor (e.g., AWS, GCP). * `STORAGE_REGION`: Region of your cloud storage. * `STORAGE_BUCKET`: Bucket name. * `STORAGE_ACCESS_KEY`: Cloud storage access key. * `STORAGE_SECRET_KEY`: Cloud storage secret key. 4. **Run the middleware:** Start the middleware server using the following command: ```bash go run cmd/main.go ``` The middleware runs on the default port, for example `localhost:8080`. ## Implement Cloud Recording using middleware This section explains the RESTful API calls to the backend middleware for starting, managing, and stopping a Cloud Recording session. ### Start recording To start a Cloud Recording session refer to the following examples: The command-line examples in this guide are for demonstration purposes only. Do not use them directly in a production environment. Implement RESTful API requests through your application server. * Basic example ```bash curl -X POST http://localhost:8080/cloud_recording/start \ -H "Content-Type: application/json" \ -d '{ "channelName": "test_channel", "sceneMode": "realtime", "recordingMode": "mix", "excludeResourceIds": [] }' ``` * Advanced example ```bash curl -X POST http://localhost:8080/cloud_recording/start \ -H "Content-Type: application/json" \ -d '{ "channelName": "testChannel", "sceneMode": "realtime", "recordingMode": "mix", "excludeResourceIds": [], "recordingConfig": { "channelType": 0, "decryptionMode": 1, "secret": "your_secret", "salt": "your_salt", "maxIdleTime": 120, "streamTypes": 2, "videoStreamType": 0, "subscribeAudioUids": ["#allstream#"], "unsubscribeAudioUids": [], "subscribeVideoUids": ["#allstream#"], "unsubscribeVideoUids": [], "subscribeUidGroup": 0, "streamMode": "individual", "audioProfile": 1, "transcodingConfig": { "width": 640, "height": 360, "fps": 15, "bitrate": 500, "maxResolutionUid": "1", "layoutConfig": [ { "x_axis": 0, "y_axis": 0, "width": 640, "height": 360, "alpha": 1, "render_mode": 1 } ] } } }' ``` ### Stop recording To stop an ongoing cloud recording session: ```bash curl -X POST http://localhost:8080/cloud_recording/stop \ -H "Content-Type: application/json" \ -d '{ "cname": "test_channel", "uid": "uid-from-start-response", "resourceId": "resource-id-from-start-response", "sid": "sid-from-start-response", "recordingMode": "mix", "async_stop": false }' ``` ### Get Recording Status During the recording, call the `status` endpoint to query the recording status as required. ```bash curl -X GET "http://localhost:8080/cloud_recording/status?resourceId=your-resource-id&sid=your-sid&mode=mix" ``` ### Update Subscriber List To update the subscriber list during a recording session, refer to the following example: ```bash curl -X POST http://localhost:8080/cloud_recording/update/subscriber-list \ -H "Content-Type: application/json" \ -d '{ "cname": "test_channel", "uid": "uid-from-start-response", "resourceId": "your-resource-id", "sid": "your-sid", "recordingMode": "mix", "recordingConfig": { "streamSubscribe": { "audioUidList": { "subscribeAudioUids": ["2345", "3456"] }, "videoUidList": { "subscribeVideoUids": ["2345", "3456"] } } } }' ``` ### Update Layout To update the layout of a recording session, refer to the following example: ```bash curl -X POST http://localhost:8080/cloud_recording/update/layout \ -H "Content-Type: application/json" \ -d '{ "cname": "test_channel", "uid": "uid-from-start-response", "resourceId": "your-resource-id", "sid": "your-sid", "recordingMode": "mix", "recordingConfig": { "mixedVideoLayout": 1, "backgroundColor": "#000000", "layoutConfig": [ { "uid": "2345", "x_axis": 0, "y_axis": 0, "width": 360, "height": 640, "alpha": 1, "render_mode": 1 } ] } }' ``` ## Cloud Recording Middleware API Reference This section provides details about the Go middleware Cloud Recording API endpoints. ### Start Recording Starts a cloud recording session. ##### Endpoint **POST:** `/cloud_recording/start` #### Request Body ```json { "channelName": "string", "uid": "string", "recordingConfig": { // RecordingConfig fields }, "storageConfig": { // StorageConfig fields } } ``` #### Response ```json { "resourceId": "string", "sid": "string", "timestamp": "string" } ``` ### Stop Recording Stops an ongoing cloud recording session. #### Endpoint **POST:** `/cloud_recording/stop` #### Request Body ```json { "cname": "string", "uid": "string", "resourceId": "string", "sid": "string", "recordingMode": "string", "async_stop": boolean } ``` #### Response ```json { "resourceId": "string", "sid": "string", "serverResponse": { "fileListMode": "string", "fileList": [ { "fileName": "string", "trackType": "string", "uid": "string", "mixedAllUser": boolean, "isPlayable": boolean, "sliceStartTime": number } ] }, "timestamp": "string" } ``` ### Get Recording Status Retrieves the status of a cloud recording session. #### Endpoint **GET:** `/cloud_recording/status` #### Query Parameters * `resourceId`: string * `sid`: string * `mode`: string #### Response ```json { "resourceId": "string", "sid": "string", "serverResponse": { "fileListMode": "string", "fileList": [ { "fileName": "string", "trackType": "string", "uid": "string", "mixedAllUser": boolean, "isPlayable": boolean, "sliceStartTime": number } ] }, "timestamp": "string" } ``` ### Update Subscriber List Updates the subscriber list for a cloud recording session. #### Endpoint **POST:** `/cloud_recording/update/subscriber-list` #### Request Body ```json { "cname": "string", "uid": "string", "resourceId": "string", "sid": "string", "recordingMode": "string", "recordingConfig": { // UpdateSubscriptionClientRequest fields } } ``` #### Response ```json { "cname": "string", "uid": "string", "resourceId": "string", "sid": "string", "timestamp": "string" } ``` ### Update Layout Updates the layout of a cloud recording session. #### Endpoint **POST:** `/cloud_recording/update/layout` #### Request Body ```json { "cname": "string", "uid": "string", "resourceId": "string", "sid": "string", "recordingMode": "string", "recordingConfig": { // UpdateLayoutClientRequest fields } } ``` #### Response ```json { "cname": "string", "uid": "string", "resourceId": "string", "sid": "string", "timestamp": "string" } ``` Replace `localhost:8080` with your server's address, if it is different. # REST quickstart (/en/realtime-media/cloud-recording/rest-quickstart) Cloud Recording enables you to record and store real-time audio and video streams from channels. To implement Cloud Recording, you set up a self-hosted backend that interacts with Agora servers to manage recording tasks. This guide introduces the essential Cloud Recording RESTful APIs you use to manage the recording process. The command-line examples in this guide are for demonstration purposes only. Do not use them directly in a production environment. Implement RESTful API requests through your application server or use Agora's [Go backend middleware](https://github.com/AgoraIO-Community/agora-go-backend-middleware). For details, see [Quickstart using middleware](/en/realtime-media/cloud-recording/middleware-quickstart) ## Understand the tech The basic process of implementing Cloud Recording is as follows: ![](https://assets-docs.agora.io/images/cloud-recording/cloud-recording-tech.svg) 1. Get a resource ID Before starting a cloud recording, call the [`acquire`](#acquire) method to obtain a cloud recording resource ID. After calling this method successfully, you get a resource ID in the response body. 2. Start cloud recording Call the [`start`](#start) method to join the channel and start a cloud recording. After calling this method successfully, you get a recording ID from the response body to identify the current recording process. 3. Query the recording status Call the [`query`](#query) method to check the recording status during the recording. 4. Stop cloud recording Call the [`stop`](#stop) method to stop the cloud recording. 5. Upload the recording file After the recording ends, the cloud recording service uploads the recording file to the [third-party cloud storage](/en/api-reference/api-ref/cloud-recording#storageconfig) you specify. ## Prerequisites To implement Cloud Recording, ensure that you have: * A valid Agora account and project. Obtain the following parameters from [Agora Console](https://console.agora.io): * The App ID and App certificate for your project. See [Get Started with Agora](/en/introduction/account#get-the-app-id-and-app-certificate) * A valid temporary token * Customer ID and Customer key for RESTful API. See [Authenticate REST calls](/en/api-reference/api-ref/cloud-recording/authentication) * Set up and enabled a supported third-party cloud storage service. Obtain the following parameters for your storage: * Bucket name * Access key * Secret key - [Alibaba Cloud](https://www.alibabacloud.com/product/object-storage-service) - [Amazon S3](https://aws.amazon.com/s3/?nc1=h_ls) - [Baidu AI Cloud](https://intl.cloud.baidu.com/product/bos.html) - [Google Cloud](https://cloud.google.com/storage) - [Huawei Cloud](https://www.huaweicloud.com/intl/en-us/product/obs) - [Kingsoft Cloud](https://en.ksyun.com/nv/product/KS3) - [Microsoft Azure](https://azure.microsoft.com/en-us/services/storage/blobs/) - [Qiniu Cloud](https://www.qiniu.com/en/products/kodo) - [Tencent Cloud](https://intl.cloud.tencent.com/product/cos) * Enabled the Agora Cloud Recording service for your project. In [Agora Console](https://console.agora.io), select **RTC Services** from the sidebar, then locate the **Cloud Recording** card and turn on its toggle. You can now use Agora Cloud Recording and see the usage statistics on the **Usage** page. * If your network has a firewall, follow the instructions in [Firewall Requirements](/en/realtime-media/cloud-recording/reference/firewall). ## Implement cloud recording The following figure shows the API call sequence to start and manage cloud recording: ![](https://assets-docs.agora.io/images/cloud-recording/cloud-recording-sequence.svg) The following APIs are optional and can be called multiple times. However, they must be called during a recording session, that is, after recording starts and before it ends: * [`query`](#query): Query the recording status * [`update`](/en/api-reference/api-ref/cloud-recording#update): Update the subscription list * [`updateLayout`](/en/api-reference/api-ref/cloud-recording#updatelayout): Update the video layout ### Use basic HTTP authentication The Cloud Recording RESTful APIs require basic HTTP authentication. You need to set the `Authorization` parameter in every HTTP request header. For details, see [Basic HTTP authentication](/en/api-reference/api-ref/cloud-recording/authentication). ### Get a resource ID Call the [`acquire`](/en/api-reference/api-ref/cloud-recording#acquire) method to request a resource ID for Cloud Recording. After calling this method successfully, you receive a resource ID in the response body. The resource ID is valid for five minutes. Start recording with this resource ID within the validity period. One resource ID can only be used for a single recording session. * The recording service is equivalent to a non-streaming client in the channel. The `uid` parameter in the request body is used to identify the recording service and cannot be the same as any existing user ID in the channel. For example, if there are two users already in the channel and their user IDs are 123 and 456, the `uid` cannot be `"123"` or `"456"`. * Agora Cloud Recording does not support user IDs in string format (User Accounts). Ensure that every user in the channel has an integer user ID. The string content of the `uid` parameter must also be an integer. #### Sample code For testing purposes, use the following command in the terminal to call the `acquire` method. ```bash # Replace with the App ID of your Agora project curl --location --request POST 'https://api.agora.io/v1/apps//cloud_recording/acquire' \ # Replace with the Base64-encoded credential in basic HTTP authentication --header 'Authorization: Basic ' \ --header 'Content-Type: application/json' \ --data-raw '{ # Replace with the name of the channel you need to record "cname": "", # Replace with your user ID "uid": "", "clientRequest":{} }' ``` ### Start recording Call the [`start`](/en/api-reference/api-ref/cloud-recording#start) method within five minutes of getting a resource ID to join a channel and start recording. You can choose either [individual recording](/en/realtime-media/cloud-recording/build/start-a-recording/individual-mode) or [composite recording](/en/realtime-media/cloud-recording/build/start-a-recording/composite-mode) as the recording mode. If this method call succeeds, you receive a recording ID (sid) in the HTTP response body. This ID identifies the current recording. After you obtain the recording ID `sid`, you can call the `query`, `updateLayout`, and `stop` methods before the time set (in hours) by `resourceExpiredHour` has passed. #### Sample code For testing purposes, use the following command in the terminal to call the `start` method. ```bash # Replace with the App ID of your Agora project # Replace with the resource ID obtained through the acquire method # Replace "" with "individual" for individual recording or "composite" for composite recording curl --location --request POST 'https://api.agora.io/v1/apps//cloud_recording/resourceid//mode//start' \ # Replace with the Base64-encoded credential in basic HTTP authentication --header 'Authorization: Basic ' \ --header 'Content-Type: application/json' \ --data-raw '{ # Replace with the name of the channel you need to record. "cname": "", # Replace with your user ID that identifies the recording service. "uid": "", "clientRequest": { # Replace with the temporary token you obtain from the console. "token": "", # Set the storageConfig related parameters. "storageConfig": { "secretKey": "", "vendor": 0, "region": 0, "bucket": "", "accessKey": "" }, # Set the recordingConfig related parameters. "recordingConfig": { # Which is consistent with the "channelType" of the Agora RTC SDK. "channelType": 0 } } }' ``` ### Query recording status During a recording session, can call the [`query`](/en/api-reference/api-ref/cloud-recording#query) method to query the recording status. You can call this API multiple times. When you call this method successfully, you receive the current recording status and related information about the recording file in the response body. See [Best Practices in Integrating Cloud Recording](/en/realtime-media/cloud-recording/build/best-practices/integration-best-practices) for details about how to [Monitor service status during a recording](../best-practices/integration-best-practices#monitor-service-status-during-a-recording) and[ Obtain the M3U8 file name](../best-practices/integration-best-practices#obtain-the-m3u8-file-name). #### Sample code For testing purposes, use the following command in the terminal to call the `query` method. ```bash # Replace with the App ID of your Agora project # Replace with the resource ID obtained through the acquire method # Replace with the sid obtained through the start method # Replace "" with "individual" for individual recording or "composite" for composite recording curl --location --request GET 'https://api.agora.io/v1/apps//cloud_recording/resourceid//sid//mode//query' \ # Replace with the Base64-encoded credential in basic HTTP authentication --header 'Authorization: Basic ' \ --header 'Content-Type: application/json' ``` ### Stop recording Call the [`stop`](/en/api-reference/api-ref/cloud-recording#stop) API to end the recording session. When you call this method successfully, you receive the status of the recording file upload and information about the recording file in the response body. #### Sample code For testing purposes, use the following command in the terminal to call the `stop` method. ```bash # Replace with the App ID of your Agora project # Replace with the resource ID obtained through the acquire method # Replace with the sid obtained through the start method # Replace "" with "individual" for individual recording or "composite" for composite recording curl --location --request POST 'https://api.agora.io/v1/apps//cloud_recording/resourceid//sid//mode//stop' \ --header 'Content-Type: application/json;charset=utf-8' \ # Replace with the Base64-encoded credential in basic HTTP authentication --header 'Authorization: Basic ' \ --data-raw '{ # Replace with your user ID that identifies the recording service. "uid": "", # Replace with the name of the channel you are recording. "cname": "", "clientRequest":{ } }' ``` Parameter settings * If the `uid` parameter in the request body is the same as a user ID in the channel, or if you use a non-integer user ID, the recording fails. For details, see the notes on the `uid` parameter in the section [Get a cloud recording resource](#acquire). * When the `start` request returns `200`, it only means that the RESTful API request is successful. To ensure that the recording has started successfully and continues normally, call `query` to check the recording status. Errors such as unreasonable `transcodingConfig` parameter settings, incorrect third-party cloud storage information, or incorrect token information cause the `query` method to return `404`. See [Why do I get a 404 error when I call query after successfully starting a cloud recording? ](../reference/common-errors#errors). * Set the `maxIdleTime` parameter based on your business needs. Within the time range set by `maxIdleTime`, the recording continues and billing is generated even if the channel is idle. ## Reference This section contains content that completes the information on this page, or points you to documentation that explains other aspects to this product. ### Sample project Agora provides a [Postman collection](https://documenter.getpostman.com/view/6319646/SVSLr9AM), which contains sample requests of RESTful API for a cloud recording. You can use the collection to quickly grasp the basic functionalities of the Cloud Recording RESTful APIs. You only need to import the collection to Postman and set your environment variables. You can also use Postman to generate code snippets written in various programming languages. To do so, select a request, click **Code**, and select the desired language in **GENERATE CODE SNIPPETS**. ![img](https://web-cdn.agora.io/docs-files/1588737379230) ### See also * To streamline the use of Agora RESTful APIs within your infrastructure, see [Quickstart using middleware](/en/realtime-media/cloud-recording/middleware-quickstart). The community middleware project provides RESTful APIs for tasks such as token generation and cloud recording management. * To update the subscription lists during the recording, call [`update`](/en/api-reference/api-ref/cloud-recording#update). You can call this method multiple times. See [Set up subscription lists](/en/realtime-media/cloud-recording/build/customize-the-recording/subscription) for details. * To set or update the video layout during the recording, call the [`updateLayout`](/en/api-reference/api-ref/cloud-recording#updatelayout) method. See [Set Video Layout](/en/realtime-media/cloud-recording/build/customize-the-recording/layout) for details. * [Common errors in cloud recording](/en/realtime-media/cloud-recording/reference/common-errors) lists common error codes and error messages in the response body. * [Agora Cloud Recording RESTful API Callback Service](../reference/rest-api-overview) lists all the callback events of cloud recording. * To learn more about the implementation steps and details of basic functions, you can refer to the following documents: * [Individual recording](/en/realtime-media/cloud-recording/build/start-a-recording/individual-mode) * [Composite recording](/en/realtime-media/cloud-recording/build/start-a-recording/composite-mode) * [Web page recording](/en/realtime-media/cloud-recording/build/start-a-recording/webpage-mode) * [Capture screenshots](/en/realtime-media/cloud-recording/build/start-a-recording/screen-capture) ## Next steps ### Manage recorded files After the recording starts, the Agora server splits the recorded content into multiple TS/WebM files and keeps uploading them to the third-party cloud storage until the recording stops. You can refer to [Manage Recorded Files](/en/realtime-media/cloud-recording/build/process-recorded-files/manage-files) to learn about the naming rules, file sizes, and slicing rules of recording files. ### Token authentication To ensure communication security, in a formal production environment, you need to generate tokens on your app server. See [Authenticate Your Users with Token](/en/realtime-media/cloud-recording/build/set-up-authentication/authentication-workflow). # Agora skills (/en/realtime-media/cloud-recording/skills) Agora skills is a set of structured reference files that give AI coding assistants deep knowledge of Agora's platform. When you ask your assistant to build something with Agora, it loads the relevant skill files covering products, APIs, and platform-specific code examples, so it can generate working code without guessing. Skills includes integration with the [Agora MCP server](./mcp), which gives your assistant access to live Agora documentation. For installation instructions and supported tools, see the [Agora Skills repository](https://github.com/AgoraIO/skills). ### Installation Install Agora Skills using one of the following methods: #### Skills CLI (recommended) Run the following command: ```bash npx skills add github:AgoraIO/skills ``` Skills activate automatically when your agent detects relevant tasks, for example, "build cloud recording workflows", "integrate Agora RTC", or "generate a token". #### Manual installation Clone the repository once and point your AI coding assistant to the skill files directly. 1. Clone the [Agora Skills repo](https://github.com/AgoraIO/skills.git): ```bash git clone https://github.com/AgoraIO/skills.git ~/agora-skills ``` 2. Point your AI assistant to `skills/agora/`. Follow the instructions for your AI coding assistant: Claude Code Cursor Windsurf GitHub Copilot Other tools Run the following commands inside Claude Code: * **User-level** (available across all your projects) ```bash /plugin marketplace add AgoraIO/skills /plugin install agora ``` * **Project-level** (shared with your team via version control) ```bash /plugin marketplace add AgoraIO/skills /plugin install agora --scope project ``` Copy or symlink `skills/agora/` into `.cursor/rules/`. For more information, see [Cursor skill directories](https://cursor.com/docs/skills#skill-directories). Add `skills/agora/` to your Cascade context. For more information, see [Windsurf skills](https://docs.windsurf.com/windsurf/cascade/skills). Reference via `@workspace` or add to `.github/copilot-instructions.md`. For more information, see [Create skills for Copilot in the CLI](https://docs.github.com/en/copilot/how-tos/copilot-cli/customize-copilot/create-skills) or [Create skills for the Copilot coding agent](https://docs.github.com/en/copilot/how-tos/use-copilot-agents/coding-agent/create-skills). The skill files are plain markdown. Point your tool to `skills/agora/` or load individual files directly. Use `SKILL.md` as the entry point. # SDK quickstart (/en/realtime-media/im/get-started-sdk) Instant messaging enhances user engagement by enabling users to connect and form a community within the app. Increased engagement can lead to increased user satisfaction and loyalty to your app. An instant messaging feature can also provide real-time support to users, allowing them to get help and answers to their questions quickly. The Chat SDK enables you to embed real-time messaging in any app, on any device, anywhere. This page guides you through implementing peer-to-peer messaging into your app using the Chat SDK. ## Understand the tech The following figure shows the workflow of sending and receiving peer-to-peer messages using Chat SDK.
Chat SDK workflow ![understand](https://assets-docs.agora.io/images/im/get-started-sdk-understand.png)
1. Clients retrieve an authentication token from your app server. 2. Users log in to Chat using the App ID, their user ID, and token. 3. Clients send and receive messages through Chat as follows: 1. Client A sends a message to Client B. The message is sent to the Agora Chat server. 2. The server delivers the message to Client B. When Client B receives a message, the SDK triggers an event. 3. Client B listens for the event to read and display the message. ## Prerequisites In order to follow the procedure on this page, you must have: * A valid [Agora account](/en/introduction/account#sign-up-for-an-agora-account). * An [Agora project](/en/introduction/account#your-first-agora-project) for which you have [enabled Chat](./get-started/enable#enable-chat). * The [App Key](./get-started/enable#get-chat-project-information) for the project. * Internet access. Ensure that no firewall is blocking your network communication. * An Android emulator or a physical Android device. * Android Studio 3.6 or higher. * Java Development Kit (JDK). You can refer to the [Android User Guide](https://developer.android.com/studio/write/java8-support) for applicable versions. ## Project setup To integrate Chat into your app, do the following: 1. Method 1: Use `mavenCentral` to automatically integrate: 1. In Android Studio, create a new **Phone and Tablet** [Android project](https://developer.android.com/studio/projects/create-project) with an **Empty Activity**. Choose **Java** as the project language, and ensure that the **Minimum SDK** version is set to `21` or higher. Android Studio automatically starts gradle sync. Wait for the sync to succeed before you continue. 2. Add Chat SDK to your project dependencies. To add the SDK to your project: 1. In `/Gradle Scripts/build.gradle (Module: .app)`, add the following line under `dependencies`: ```text dependencies { ... implementation 'io.agora.rtc:chat-sdk:' } ``` Replace `` with the version number for the latest Chat SDK release, for example `1.3.1`. You can obtain the latest version information using [Maven Central Repository Search](https://central.sonatype.com/artifact/io.agora.rtc/chat-sdk/versions). 2. To download the SDK from Maven Central, press **Sync Now**. 3. Add permissions for network and device access. To add the necessary permissions, in `/app/Manifests/AndroidManifest.xml`, add the following permissions after `
`: ```xml ``` 4. Prevent code obfuscation. In `/Gradle Scripts/proguard-rules.pro`, add the following line: ```java -keep class io.agora.** {*;} -dontwarn io.agora.** ``` 2. Method 2: Dynamically load .so library files In order to reduce the size of the application installation package, the SDK provides `ChatOptions#setNativeLibBasePath` method to support dynamic loading of the `.so` files required by the SDK. Taking SDK 1.3.0 as an example, the `.so` file includes two files: `libcipherdb.so` and `.libhyphenate.so`. The steps to implement this function are as follows: 1. Download the latest version of the SDK and unzip it. 2. Integrate the `agorachat_1.3.0.jar` file into your project. 3. Upload `.so` files for all schemas to your server and ensure that the application can download `.so` files for the target schemas over the network. 4. When the application runs, it checks whether the `.so` file exists. If not found, the app downloads the `.so` file and saves it to your custom app's private directory. 5. When calling `ChatClient#init`, set the app private directory where the `.so` file is located as a parameter into the `ChatOptions#setNativeLibBasePath` method. 6. The SDK will automatically load `.so` files from the specified path upon initialization. ```java // Assume that you have put two .so libraries, `libcipherdb.so` and `libagora-chat-sdk.so` in the /data/data/packagename/files directory of the app. String filesPath = mContext.getFilesDir().getAbsolutePath(); ChatOptions options = new ChatOptions(); options.setNativeLibBasePath(filesPath); ChatClient.getInstance().init(mContext, options); ``` This method is applicable when you integrate the SDK manually but not when you integrate the SDK with Maven Central. You can specify the path of `.so` files with the path parameter in the `ChatOptions#setNativeLibBasePath` method. The path must be a valid and private directory of the app. If you set this parameter, the SDK will use `System.load` to load the `.so` library from the directory you specify, so that the app dynamically loads the required `.so` files when it runs, thereby reducing the package size. If you do not set this parameter or set it to `null`, the SDK will use `System.load` to load the `.so` library from the default path when compiling the app, thus increasing the package size compared to the previous method. Since March 6, 2024, using this method to reduce the app size no longer meets the requirements of Google Play. For details, refer to the [Developer Program Policies](https://support.google.com/googleplay/android-developer/answer/14906471?hl=en\&visit_id=638555962315571301-1274648059\&rd=1) issued by Google. If your app needs to be listed on Google Play, please try other ways to reduce the app size. For details, see [App size optimization](/en/realtime-media/rtc/best-practices/app-size-optimization#dynamically-load-so-files). ## Implement peer-to-peer messaging This section shows how to use the Chat SDK to implement peer-to-peer messaging in your app, step by step. ### Create the UI In the quickstart app, you create a simple UI that consists of the following elements: * A `Button` to log in or out of Agora Chat. * An `EditText` box to specify the recipient user ID. * An `EditText` box to enter a text message. * A `Button` to send the text message. * A scrollable layout to display sent and received messages. To add the UI framework to your device project, open `app/res/layout/activity_main.xml` and replace the content with the following: ```xml