Azure OpenAI Realtime API

Updated

Integrate Azure OpenAI Realtime API MLLM into Conversational AI Engine.

Azure OpenAI Realtime API provides multimodal large language model capabilities with real-time audio processing through Microsoft Azure's infrastructure, enabling natural voice conversations without separate ASR/TTS components. Enabling MLLM automatically disables ASR, LLM, and TTS since the MLLM handles end-to-end voice processing directly.

Sample configuration

The following examples show how to configure Azure OpenAI Realtime MLLM when starting a conversational AI agent.

from agora_agent import Agent
from agora_agent.agentkit.vendors import AzureOpenAIRealtime

# client is your configured Agora client
agent = (
    Agent(client)
    .with_mllm(AzureOpenAIRealtime(
        api_key='your-azure-api-key',
        url='wss://your-resource-name.openai.azure.com/openai/v1/realtime?model=gpt-realtime-2',
        model='gpt-realtime-2',
        voice='alloy',
        instructions='You are a Conversational AI Agent, developed by Agora.',
        output_modalities=['audio'],
        turn_detection={'mode': 'server_vad'},
    ))
)
import { Agent, AzureOpenAIRealtime } from 'agora-agents';

// client is your configured Agora client
const agent = new Agent({ client })
  .withMllm(new AzureOpenAIRealtime({
    apiKey: 'your-azure-api-key',
    url: 'wss://your-resource-name.openai.azure.com/openai/v1/realtime?model=gpt-realtime-2',
    model: 'gpt-realtime-2',
    voice: 'alloy',
    instructions: 'You are a Conversational AI Agent, developed by Agora.',
    outputModalities: ['audio'],
    turnDetection: { mode: 'server_vad' },
  }));
import (
    "github.com/AgoraIO/agora-agents-go/v2/agentkit"
    "github.com/AgoraIO/agora-agents-go/v2/agentkit/vendors"
)

// client is your configured Agora client
agent := agentkit.NewAgent(client).WithMllm(
    vendors.NewAzureOpenAIRealtime(vendors.AzureOpenAIRealtimeOptions{
        APIKey:           "your-azure-api-key",
        URL:              "wss://your-resource-name.openai.azure.com/openai/v1/realtime?model=gpt-realtime-2",
        Model:            "gpt-realtime-2",
        Voice:            "alloy",
        Instructions:     "You are a Conversational AI Agent, developed by Agora.",
        OutputModalities: []string{"audio"},
        TurnDetection: &agora.MllmTurnDetection{
            Mode: agora.MllmTurnDetectionModeServerVad.Ptr(),
        },
    }),
)

Use the following mllm configuration in your request:

"mllm": {
  "enable": true,
  "vendor": "azure",
  "url": "wss://your-resource-name.openai.azure.com/openai/v1/realtime?model=gpt-realtime-2",
  "api_key": "<azure_api_key>",
  "output_modalities": ["audio"],
  "turn_detection": {
    "mode": "server_vad"
  },
  "params": {
    "model": "gpt-realtime-2",
    "voice": "alloy",
    "instructions": "You are a Conversational AI Agent, developed by Agora."
  }
}

Turn detection

Add a turn_detection block inside the mllm object when you Start a conversational AI agent. Unlike other MLLM vendors, turn_detection is required for Azure OpenAI Realtime API.

Info

When mllm.turn_detection is defined, the top-level turn_detection object has no effect.

The following examples show the supported turn_detection configurations for Azure OpenAI Realtime API.

  • Server VAD

    "turn_detection": {
      "mode": "server_vad",
      "server_vad_config": {
        "prefix_padding_ms": 800,
        "silence_duration_ms": 640,
        "threshold": 0.5
      }
    }
  • Semantic VAD

    "turn_detection": {
      "mode": "semantic_vad",
      "semantic_vad_config": {
        "eagerness": "auto"
      }
    }
  • Agora VAD

    "turn_detection": {
      "mode": "agora_vad",
      "agora_vad_config": {
        "interrupt_duration_ms": 160,
        "prefix_padding_ms": 800,
        "silence_duration_ms": 640,
        "threshold": 0.5
      }
    }

Key parameters

mllm required
ParameterDescription
api_keystring
required

The API key used to authenticate with your Azure OpenAI resource.

urlstring
required

The WebSocket URL for your Azure OpenAI Realtime deployment. For example, wss://your-resource-name.openai.azure.com/openai/v1/realtime?model=gpt-realtime-2.

messagesarray[object]
optional

An array of conversation history items passed to the model as context. Each item represents a single message in the conversation history.

  • role: string

    The role of the message author. For example, system or user.

  • content: string

    The content of the message.

paramsobject
optional

Additional Azure OpenAI Realtime configuration parameters.

  • Modalities override: The modalities setting in params is overridden by output_modalities.
  • Turn detection override: The turn_detection setting in params is overridden by mllm.turn_detection.
  • Optional model?: string

    The model or deployment name. For example, gpt-realtime-2.

  • Optional voice?: string

    The voice identifier for audio output. For example, alloy.

  • Optional instructions?: string

    System instructions that define the assistant's behavior and personality.

  • Optional input_audio_transcription?: object

    Configuration for audio input transcription.

    • Optional language?: string

      The language of the input audio. Supplying the input language in ISO-639-1 format (For example en) improves accuracy and latency.

    • Optional model?: string

      The model to use for transcription.

    • Optional prompt?: string

      An optional text to guide the model's style or continue a previous audio segment.

turn_detectionobject
required

Turn detection configuration for the MLLM module. Required for Azure OpenAI Realtime API. For a full list of turn_detection parameters, see mllm.turn_detection.

  • Optional mode?: string
    Possible values
    • agora_vad
    • server_vad
    • semantic_vad
    • agora_vad: Agora VAD-based detection.
    • server_vad: Vendor-side VAD-based detection.
    • semantic_vad: Semantic-based detection.
  • Optional agora_vad_config?: object

    Configuration for Agora VAD-based turn detection. Applicable when mode is agora_vad.

    • Optional interrupt_duration_ms?: integer

      Minimum duration of speech in milliseconds required to trigger an interruption.

    • Optional prefix_padding_ms?: integer

      Duration of audio in milliseconds to include before the detected speech start.

    • Optional silence_duration_ms?: integer

      Duration of silence in milliseconds required to determine end of speech.

    • Optional threshold?: number

      VAD sensitivity threshold. A higher value reduces false positives.

  • Optional server_vad_config?: object

    Configuration for vendor-side VAD-based turn detection. Applicable when mode is server_vad. Parameters are passed through to the vendor.

    • Optional prefix_padding_ms?: integer

      Duration of audio in milliseconds to include before the detected speech start.

    • Optional silence_duration_ms?: integer

      Duration of silence in milliseconds required to determine end of speech.

    • Optional threshold?: number

      VAD sensitivity threshold.

  • Optional semantic_vad_config?: object

    Configuration for semantic-based turn detection. Applicable when mode is semantic_vad.

    • Optional eagerness?: string
      Possible values
      • auto
      • low
      • medium
      • high

      Controls how eagerly the model ends its turn.

output_modalitiesarray[string]
optional
Default value
["text", "audio"]

Output format options: ["text", "audio"] for both text and voice responses.

max_historyinteger
optional

The number of conversation history messages to cache.

greeting_messagestring
optional

Initial message the agent speaks when a user joins the channel.

failure_messagestring
optional

The message the agent speaks when an error occurs.

vendorstring
required

MLLM provider identifier. Set to azure for Azure OpenAI Realtime API.

The table extends beyond the current width. Scroll horizontally to view more columns.

For comprehensive API reference, real-time capabilities, and detailed parameter descriptions, see the Azure OpenAI Realtime API documentation.