Azure OpenAI Realtime API

Azure OpenAI Realtime API provides multimodal large language model capabilities with real-time audio processing through Microsoft Azure's infrastructure, enabling natural voice conversations without separate ASR/TTS components.

Info

Enabling MLLM automatically disables ASR, LLM, and TTS since the MLLM handles end-to-end voice processing directly.

Sample configuration

The following examples show how to configure Azure OpenAI Realtime MLLM when starting a conversational AI agent.

from agora_agent import Agent
from agora_agent.agentkit.vendors import AzureOpenAIRealtime

# client is your configured Agora client
agent = (
    Agent(client)
    .with_mllm(AzureOpenAIRealtime(
        api_key='your-azure-api-key',
        url='wss://your-resource-name.openai.azure.com/openai/v1/realtime?model=gpt-realtime-2',
        model='gpt-realtime-2',
        voice='alloy',
        instructions='You are a Conversational AI Agent, developed by Agora.',
        output_modalities=['audio'],
        turn_detection={'mode': 'server_vad'},
    ))
)

import { Agent, AzureOpenAIRealtime } from 'agora-agents';

// client is your configured Agora client
const agent = new Agent({ client })
  .withMllm(new AzureOpenAIRealtime({
    apiKey: 'your-azure-api-key',
    url: 'wss://your-resource-name.openai.azure.com/openai/v1/realtime?model=gpt-realtime-2',
    model: 'gpt-realtime-2',
    voice: 'alloy',
    instructions: 'You are a Conversational AI Agent, developed by Agora.',
    outputModalities: ['audio'],
    turnDetection: { mode: 'server_vad' },
  }));

import "github.com/AgoraIO/agora-agents-go/v2/agentkit/vendors"

// client is your configured Agora client
agent := agentkit.NewAgent(client).WithMllm(
    vendors.NewAzureOpenAIRealtime(vendors.AzureOpenAIRealtimeOptions{
        APIKey:           "your-azure-api-key",
        URL:              "wss://your-resource-name.openai.azure.com/openai/v1/realtime?model=gpt-realtime-2",
        Model:            "gpt-realtime-2",
        Voice:            "alloy",
        Instructions:     "You are a Conversational AI Agent, developed by Agora.",
        OutputModalities: []string{"audio"},
        TurnDetection: &agora.MllmTurnDetection{
            Mode: agora.MllmTurnDetectionModeServerVad.Ptr(),
        },
    }),
)

Use the following mllm configuration in your request:

"mllm": {
  "enable": true,
  "vendor": "azure",
  "url": "wss://your-resource-name.openai.azure.com/openai/v1/realtime?model=gpt-realtime-2",
  "api_key": "<azure_api_key>",
  "output_modalities": ["audio"],
  "turn_detection": {
    "mode": "server_vad"
  },
  "params": {
    "model": "gpt-realtime-2",
    "voice": "alloy",
    "instructions": "You are a Conversational AI Agent, developed by Agora."
  }
}

Turn detection

Add a turn_detection block inside the mllm object when you Start a conversational AI agent. Unlike other MLLM vendors, turn_detection is required for Azure OpenAI Realtime API.

Info

When mllm.turn_detection is defined, the top-level turn_detection object has no effect.

The following examples show the supported turn_detection configurations for Azure OpenAI Realtime API.

Server VAD

"turn_detection": {
  "mode": "server_vad",
  "server_vad_config": {
    "prefix_padding_ms": 800,
    "silence_duration_ms": 640,
    "threshold": 0.5
  }
}

Semantic VAD

"turn_detection": {
  "mode": "semantic_vad",
  "semantic_vad_config": {
    "eagerness": "auto"
  }
}

Agora VAD

"turn_detection": {
  "mode": "agora_vad",
  "agora_vad_config": {
    "interrupt_duration_ms": 160,
    "prefix_padding_ms": 800,
    "silence_duration_ms": 640,
    "threshold": 0.5
  }
}

Key parameters

mllmrequired

api_keystring

required

The API key used to authenticate with your Azure OpenAI resource.

urlstring

required

The WebSocket URL for your Azure OpenAI Realtime deployment. For example, wss://your-resource-name.openai.azure.com/openai/v1/realtime?model=gpt-realtime-2.

messagesarray[object]

optional

An array of conversation history items passed to the model as context. Each item represents a single message in the conversation history.

rolestring

required

The role of the message author. For example, system or user.

contentstring

required

The content of the message.

paramsobject

optional

Additional Azure OpenAI Realtime configuration parameters.

Modalities override: The modalities setting in params is overridden by output_modalities.
Turn detection override: The turn_detection setting in params is overridden by mllm.turn_detection.

modelstring

optional

The model or deployment name. For example, gpt-realtime-2.

voicestring

optional

The voice identifier for audio output. For example, alloy.

instructionsstring

optional

System instructions that define the assistant's behavior and personality.

input_audio_transcriptionobject

optional

Configuration for audio input transcription.

languagestring

optional

The language of the input audio. Supplying the input language in ISO-639-1 format (For example en) improves accuracy and latency.

modelstring

optional

The model to use for transcription.

promptstring

optional

An optional text to guide the model's style or continue a previous audio segment.

turn_detectionobject

required

Turn detection configuration for the MLLM module. Required for Azure OpenAI Realtime API. For a full list of turn_detection parameters, see mllm.turn_detection.

modestring

optional

Possible values: agora_vad
server_vad
semantic_vad

agora_vad: Agora VAD-based detection.
server_vad: Vendor-side VAD-based detection.
semantic_vad: Semantic-based detection.

agora_vad_configobject

optional

Configuration for Agora VAD-based turn detection. Applicable when mode is agora_vad.

interrupt_duration_msinteger

optional

Minimum duration of speech in milliseconds required to trigger an interruption.

prefix_padding_msinteger

optional

Duration of audio in milliseconds to include before the detected speech start.

silence_duration_msinteger

optional

Duration of silence in milliseconds required to determine end of speech.

thresholdnumber

optional

VAD sensitivity threshold. A higher value reduces false positives.

server_vad_configobject

optional

Configuration for vendor-side VAD-based turn detection. Applicable when mode is server_vad. Parameters are passed through to the vendor.

prefix_padding_msinteger

optional

Duration of audio in milliseconds to include before the detected speech start.

silence_duration_msinteger

optional

Duration of silence in milliseconds required to determine end of speech.

thresholdnumber

optional

VAD sensitivity threshold.

semantic_vad_configobject

optional

Configuration for semantic-based turn detection. Applicable when mode is semantic_vad.

eagernessstring

optional

Possible values: auto
low
medium
high

Controls how eagerly the model ends its turn.

output_modalitiesarray[string]

optional

Default value: ["text", "audio"]

Output format options: ["text", "audio"] for both text and voice responses.

max_historyinteger

optional

The number of conversation history messages to cache.

greeting_messagestring

optional

Initial message the agent speaks when a user joins the channel.

failure_messagestring

optional

The message the agent speaks when an error occurs.

vendorstring

required

MLLM provider identifier. Set to azure for Azure OpenAI Realtime API.

For comprehensive API reference, real-time capabilities, and detailed parameter descriptions, see the Azure OpenAI Realtime API documentation.

Azure OpenAI Realtime API

Sample configuration

Turn detection

Key parameters

Contact Us

Contact Us