Azure OpenAI Realtime API
Updated
Integrate Azure OpenAI Realtime API MLLM into Conversational AI Engine.
Azure OpenAI Realtime API provides multimodal large language model capabilities with real-time audio processing through Microsoft Azure's infrastructure, enabling natural voice conversations without separate ASR/TTS components. Enabling MLLM automatically disables ASR, LLM, and TTS since the MLLM handles end-to-end voice processing directly.
Sample configuration
The following examples show how to configure Azure OpenAI Realtime MLLM when starting a conversational AI agent.
from agora_agent import Agent
from agora_agent.agentkit.vendors import AzureOpenAIRealtime
# client is your configured Agora client
agent = (
Agent(client)
.with_mllm(AzureOpenAIRealtime(
api_key='your-azure-api-key',
url='wss://your-resource-name.openai.azure.com/openai/v1/realtime?model=gpt-realtime-2',
model='gpt-realtime-2',
voice='alloy',
instructions='You are a Conversational AI Agent, developed by Agora.',
output_modalities=['audio'],
turn_detection={'mode': 'server_vad'},
))
)import { Agent, AzureOpenAIRealtime } from 'agora-agents';
// client is your configured Agora client
const agent = new Agent({ client })
.withMllm(new AzureOpenAIRealtime({
apiKey: 'your-azure-api-key',
url: 'wss://your-resource-name.openai.azure.com/openai/v1/realtime?model=gpt-realtime-2',
model: 'gpt-realtime-2',
voice: 'alloy',
instructions: 'You are a Conversational AI Agent, developed by Agora.',
outputModalities: ['audio'],
turnDetection: { mode: 'server_vad' },
}));import (
"github.com/AgoraIO/agora-agents-go/v2/agentkit"
"github.com/AgoraIO/agora-agents-go/v2/agentkit/vendors"
)
// client is your configured Agora client
agent := agentkit.NewAgent(client).WithMllm(
vendors.NewAzureOpenAIRealtime(vendors.AzureOpenAIRealtimeOptions{
APIKey: "your-azure-api-key",
URL: "wss://your-resource-name.openai.azure.com/openai/v1/realtime?model=gpt-realtime-2",
Model: "gpt-realtime-2",
Voice: "alloy",
Instructions: "You are a Conversational AI Agent, developed by Agora.",
OutputModalities: []string{"audio"},
TurnDetection: &agora.MllmTurnDetection{
Mode: agora.MllmTurnDetectionModeServerVad.Ptr(),
},
}),
)Use the following mllm configuration in your request:
"mllm": {
"enable": true,
"vendor": "azure",
"url": "wss://your-resource-name.openai.azure.com/openai/v1/realtime?model=gpt-realtime-2",
"api_key": "<azure_api_key>",
"output_modalities": ["audio"],
"turn_detection": {
"mode": "server_vad"
},
"params": {
"model": "gpt-realtime-2",
"voice": "alloy",
"instructions": "You are a Conversational AI Agent, developed by Agora."
}
}Turn detection
Add a turn_detection block inside the mllm object when you Start a conversational AI agent. Unlike other MLLM vendors, turn_detection is required for Azure OpenAI Realtime API.
Info
When mllm.turn_detection is defined, the top-level turn_detection object has no effect.
The following examples show the supported turn_detection configurations for Azure OpenAI Realtime API.
-
Server VAD
"turn_detection": { "mode": "server_vad", "server_vad_config": { "prefix_padding_ms": 800, "silence_duration_ms": 640, "threshold": 0.5 } } -
Semantic VAD
"turn_detection": { "mode": "semantic_vad", "semantic_vad_config": { "eagerness": "auto" } } -
Agora VAD
"turn_detection": { "mode": "agora_vad", "agora_vad_config": { "interrupt_duration_ms": 160, "prefix_padding_ms": 800, "silence_duration_ms": 640, "threshold": 0.5 } }
Key parameters
| Parameter | Description |
|---|---|
api_keystring | The API key used to authenticate with your Azure OpenAI resource. |
urlstring | The WebSocket URL for your Azure OpenAI Realtime deployment. For example, |
messagesarray[object] | An array of conversation history items passed to the model as context. Each item represents a single message in the conversation history.
|
paramsobject | Additional Azure OpenAI Realtime configuration parameters.
|
turn_detectionobject | Turn detection configuration for the MLLM module. Required for Azure OpenAI Realtime API. For a full list of
|
output_modalitiesarray[string] |
Output format options: |
max_historyinteger | The number of conversation history messages to cache. |
greeting_messagestring | Initial message the agent speaks when a user joins the channel. |
failure_messagestring | The message the agent speaks when an error occurs. |
vendorstring | MLLM provider identifier. Set to |
For comprehensive API reference, real-time capabilities, and detailed parameter descriptions, see the Azure OpenAI Realtime API documentation.
