OpenAI Realtime API

OpenAI Realtime provides multimodal large language model capabilities with real-time audio processing, enabling natural voice conversations without separate ASR/TTS components.

Info

Enabling MLLM automatically disables ASR, LLM, and TTS since the MLLM handles end-to-end voice processing directly.

Sample configuration

The following examples show how to configure OpenAI Realtime MLLM when starting a conversational AI agent.

from agora_agent import Agent
from agora_agent.agentkit.vendors import OpenAIRealtime

# client is your configured Agora client
agent = (
    Agent(client)
    .with_mllm(OpenAIRealtime(
        api_key='your-openai-key',
        url='wss://api.openai.com/v1/realtime',
        model='gpt-realtime',
        voice='coral',
        instructions='You are a Conversational AI Agent, developed by Agora.',
    ))
)

import { Agent, OpenAIRealtime } from 'agora-agents';

// client is your configured Agora client
const agent = new Agent({ client })
  .withMllm(new OpenAIRealtime({
    apiKey: 'your-openai-key',
    url: 'wss://api.openai.com/v1/realtime',
    model: 'gpt-realtime',
    voice: 'coral',
    instructions: 'You are a Conversational AI Agent, developed by Agora.',
  }));

import "github.com/AgoraIO/agora-agents-go/v2/agentkit/vendors"

// client is your configured Agora client
agent := agentkit.NewAgent(client).WithMllm(
    vendors.NewOpenAIRealtime(vendors.OpenAIRealtimeOptions{
        APIKey:       "your-openai-key",
        URL:          "wss://api.openai.com/v1/realtime",
        Model:        "gpt-realtime",
        Voice:        "coral",
        Instructions: "You are a Conversational AI Agent, developed by Agora.",
    }),
)

Use the following mllm configuration in your request:

"mllm": {
  "enable": true,
  "url": "wss://api.openai.com/v1/realtime",
  "api_key": "<openai_api_key>",
  "params": {
    "model": "gpt-realtime",
    "voice": "coral",
    "instructions": "You are a Conversational AI Agent, developed by Agora.",
    "input_audio_transcription": {
      "language": "<language>",
      "model": "gpt-4o-mini-transcribe",
      "prompt": "expect words related to real-time engagement"
    }
  },
  "turn_detection": {
    // see details below
  },
  "greeting_message": "<greetings>",
  "output_modalities": ["text", "audio"],
  "vendor": "openai"
}

Turn detection

To set up turn detection, add a turn_detection block inside the mllm object when you Start a conversational AI agent.

Info

When mllm.turn_detection is defined, the top-level turn_detection object has no effect.

The following examples show the supported turn_detection configurations for OpenAI Realtime API.

Server VAD

"turn_detection": {
  "mode": "server_vad",
  "server_vad_config": {
    "prefix_padding_ms": 800,
    "silence_duration_ms": 640,
    "threshold": 0.5
  }
}

Semantic VAD

"turn_detection": {
  "mode": "semantic_vad",
  "semantic_vad_config": {
    "eagerness": "auto"
  }
}

Agora VAD

"turn_detection": {
  "mode": "agora_vad",
  "agora_vad_config": {
    "interrupt_duration_ms": 160,
    "prefix_padding_ms": 800,
    "silence_duration_ms": 640,
    "threshold": 0.5
  }
}

Key parameters

mllmrequired

enableboolean

optional

Enables the MLLM module. Replaces the deprecated advanced_features.enable_mllm.

urlstring

required

The WebSocket URL for OpenAI Realtime API.

api_keystring

required

The API key used for authentication. Get your API key from the OpenAI Console.

messagesarray[object]

optional

An array of conversation history items passed to the model as context. Each item represents a single message in the conversation history.

rolestring

required

The role of the message author. For example, system or user.

contentstring

required

The content of the message.

paramsobject

optional

Additional MLLM configuration parameters. See the MLLM provider pages in this section for details.

Modalities override: The modalities setting in params is overridden by input_modalities and output_modalities.
Turn detection override: The turn_detection setting in params is overridden by mllm.turn_detection.

modelstring

optional

The model identifier.

voicestring

optional

The voice identifier for audio output.

instructionsstring

optional

System instructions that define the assistant's behavior and personality.

input_audio_transcriptionobject

optional

Configuration for audio input transcription.

languagestring

optional

The language of the input audio. Supplying the input language in ISO-639-1 format (For example en) improves accuracy and latency.

modelstring

optional

The model to use for transcription. Current options are gpt-4o-transcribe, gpt-4o-mini-transcribe, and whisper-1.

promptstring

optional

An optional text to guide the model's style or continue a previous audio segment. For whisper-1, the prompt is a list of keywords. For gpt-4o-transcribe models, the prompt is a free text string, for example "expect words related to technology".

turn_detectionobject

optional

Turn detection configuration for the MLLM module. For a full list of turn_detection parameters, see mllm.turn_detection.

modestring

optional

Possible values: agora_vad
server_vad
semantic_vad

agora_vad: Agora VAD-based detection.
server_vad: Vendor-side VAD-based detection.
semantic_vad: Semantic-based detection.

agora_vad_configobject

optional

Configuration for Agora VAD-based turn detection. Applicable when mode is agora_vad.

interrupt_duration_msinteger

optional

Minimum duration of speech in milliseconds required to trigger an interruption.

prefix_padding_msinteger

optional

Duration of audio in milliseconds to include before the detected speech start.

silence_duration_msinteger

optional

Duration of silence in milliseconds required to determine end of speech.

thresholdnumber

optional

VAD sensitivity threshold. A higher value reduces false positives.

server_vad_configobject

optional

Configuration for vendor-side VAD-based turn detection. Applicable when mode is server_vad. Parameters are passed through to the vendor.

prefix_padding_msinteger

optional

Duration of audio in milliseconds to include before the detected speech start.

silence_duration_msinteger

optional

Duration of silence in milliseconds required to determine end of speech.

thresholdnumber

optional

VAD sensitivity threshold.

idle_timeout_msinteger

optional

Idle timeout in milliseconds.

semantic_vad_configobject

optional

Configuration for semantic-based turn detection. Applicable when mode is semantic_vad.

eagernessstring

optional

Possible values: auto
low
medium
high

Controls how eagerly the model ends its turn.

input_modalitiesarray[string]

optional

Default value: ["audio"]

MLLM input modalities:

["audio"]: Audio only
["audio", "text"]: Audio plus text

output_modalitiesarray[string]

optional

Default value: ["text", "audio"]

Output format options: ["text", "audio"] for both text and voice responses.

greeting_messagestring

optional

Initial message the agent speaks when a user joins the channel.

vendorstring

optional

MLLM provider identifier. Set to openai for OpenAI Realtime API.

For comprehensive API reference, real-time capabilities, and detailed parameter descriptions, see the OpenAI Realtime API documentation.

Sample configuration

Turn detection

Key parameters

Contact Us

Contact Us