OpenAI Realtime integration
Updated
Use Agora RTC transport with the OpenAI Realtime API when you need a custom voice stack instead of the managed Conversational AI Engine.
This guide explains how to combine Agora's real-time audio transport with the OpenAI Realtime API.
Use this integration when you want Agora to handle the live audio session and network transport, while OpenAI Realtime handles the model-side conversational behavior.
This is not the same as building on Agora's managed Conversational AI Engine. If you want Agora to manage the voice-agent runtime, use the main Conversational AI build path instead.
When to use this path
- You already know you want to build around the OpenAI Realtime API.
- You need Agora RTC or Video SDK transport in front of that model stack.
- You are building a browser demo, device-oriented flow, or backend-mediated prototype around OpenAI Realtime.
When not to use this path
- You want the default Agora-managed voice-agent stack.
- You are looking for the main onboarding path for Conversational AI Engine.
- You want Agent Studio or the Device Kit route instead of a custom integration.
Core pieces
- Agora App ID: Identifies your Agora project.
- App certificate and token model: Used for secure join flows where needed.
- Channel and user ID: Define the live session and participants.
- SD-RTN: Provides Agora's low-latency transport layer.
- OpenAI Realtime API: Provides the model-side real-time conversation flow.
Recommended quickstart flow
- Prepare your Agora account and project.
- Decide how your client joins the Agora session and how the OpenAI-side session is coordinated.
- Configure the OpenAI Realtime model and audio flow.
- Run the round-trip voice path locally and validate latency, auth, and media behavior.
Security and network checks
- Keep Agora credentials and provider API keys on the server side.
- Separate local, staging, and production credentials.
- Review firewall requirements before testing in restricted networks.
- Re-test the full join and media path after changing transport or security settings.