OpenAI Realtime API
Updated
Integrate OpenAI Realtime MLLM into Conversational AI Engine.
OpenAI Realtime provides multimodal large language model capabilities with real-time audio processing, enabling natural voice conversations without separate ASR/TTS components. Enabling MLLM automatically disables ASR, LLM, and TTS since the MLLM handles end-to-end voice processing directly.
Sample configuration
The following examples show how to configure OpenAI Realtime MLLM when starting a conversational AI agent.
from agora_agent import Agent
from agora_agent.agentkit.vendors import OpenAIRealtime
# client is your configured Agora client
agent = (
Agent(client)
.with_mllm(OpenAIRealtime(
api_key='your-openai-key',
url='wss://api.openai.com/v1/realtime',
model='gpt-realtime',
voice='coral',
instructions='You are a Conversational AI Agent, developed by Agora.',
))
)import { Agent, OpenAIRealtime } from 'agora-agents';
// client is your configured Agora client
const agent = new Agent({ client })
.withMllm(new OpenAIRealtime({
apiKey: 'your-openai-key',
url: 'wss://api.openai.com/v1/realtime',
model: 'gpt-realtime',
voice: 'coral',
instructions: 'You are a Conversational AI Agent, developed by Agora.',
}));import (
"github.com/AgoraIO/agora-agents-go/v2/agentkit"
"github.com/AgoraIO/agora-agents-go/v2/agentkit/vendors"
)
// client is your configured Agora client
agent := agentkit.NewAgent(client).WithMllm(
vendors.NewOpenAIRealtime(vendors.OpenAIRealtimeOptions{
APIKey: "your-openai-key",
URL: "wss://api.openai.com/v1/realtime",
Model: "gpt-realtime",
Voice: "coral",
Instructions: "You are a Conversational AI Agent, developed by Agora.",
}),
)Use the following mllm configuration in your request:
"mllm": {
"enable": true,
"url": "wss://api.openai.com/v1/realtime",
"api_key": "<openai_api_key>",
"params": {
"model": "gpt-realtime",
"voice": "coral",
"instructions": "You are a Conversational AI Agent, developed by Agora.",
"input_audio_transcription": {
"language": "<language>",
"model": "gpt-4o-mini-transcribe",
"prompt": "expect words related to real-time engagement"
}
},
"turn_detection": {
// see details below
},
"greeting_message": "<greetings>",
"output_modalities": ["text", "audio"],
"vendor": "openai"
}Turn detection
To set up turn detection, add a turn_detection block inside the mllm object when you Start a conversational AI agent.
Info
When mllm.turn_detection is defined, the top-level turn_detection object has no effect.
The following examples show the supported turn_detection configurations for OpenAI Realtime API.
-
Server VAD
"turn_detection": { "mode": "server_vad", "server_vad_config": { "prefix_padding_ms": 800, "silence_duration_ms": 640, "threshold": 0.5 } } -
Semantic VAD
"turn_detection": { "mode": "semantic_vad", "semantic_vad_config": { "eagerness": "auto" } } -
Agora VAD
"turn_detection": { "mode": "agora_vad", "agora_vad_config": { "interrupt_duration_ms": 160, "prefix_padding_ms": 800, "silence_duration_ms": 640, "threshold": 0.5 } }
Key parameters
| Parameter | Description |
|---|---|
enableboolean | Enables the MLLM module. Replaces the deprecated |
urlstring | The WebSocket URL for OpenAI Realtime API. |
api_keystring | The API key used for authentication. Get your API key from the OpenAI Console. |
messagesarray[object] | An array of conversation history items passed to the model as context. Each item represents a single message in the conversation history.
|
paramsobject | Additional MLLM configuration parameters. See the MLLM provider pages in this section for details.
|
turn_detectionobject | Turn detection configuration for the MLLM module. For a full list of
|
input_modalitiesarray[string] |
MLLM input modalities:
|
output_modalitiesarray[string] |
Output format options: |
greeting_messagestring | Initial message the agent speaks when a user joins the channel. |
vendorstring | MLLM provider identifier. Set to |
For comprehensive API reference, real-time capabilities, and detailed parameter descriptions, see the OpenAI Realtime API documentation.
