OpenAI Realtime API

Updated

Integrate OpenAI Realtime MLLM into Conversational AI Engine.

OpenAI Realtime provides multimodal large language model capabilities with real-time audio processing, enabling natural voice conversations without separate ASR/TTS components. Enabling MLLM automatically disables ASR, LLM, and TTS since the MLLM handles end-to-end voice processing directly.

Sample configuration

The following examples show how to configure OpenAI Realtime MLLM when starting a conversational AI agent.

from agora_agent import Agent
from agora_agent.agentkit.vendors import OpenAIRealtime

# client is your configured Agora client
agent = (
    Agent(client)
    .with_mllm(OpenAIRealtime(
        api_key='your-openai-key',
        url='wss://api.openai.com/v1/realtime',
        model='gpt-realtime',
        voice='coral',
        instructions='You are a Conversational AI Agent, developed by Agora.',
    ))
)
import { Agent, OpenAIRealtime } from 'agora-agents';

// client is your configured Agora client
const agent = new Agent({ client })
  .withMllm(new OpenAIRealtime({
    apiKey: 'your-openai-key',
    url: 'wss://api.openai.com/v1/realtime',
    model: 'gpt-realtime',
    voice: 'coral',
    instructions: 'You are a Conversational AI Agent, developed by Agora.',
  }));
import (
    "github.com/AgoraIO/agora-agents-go/v2/agentkit"
    "github.com/AgoraIO/agora-agents-go/v2/agentkit/vendors"
)

// client is your configured Agora client
agent := agentkit.NewAgent(client).WithMllm(
    vendors.NewOpenAIRealtime(vendors.OpenAIRealtimeOptions{
        APIKey:       "your-openai-key",
        URL:          "wss://api.openai.com/v1/realtime",
        Model:        "gpt-realtime",
        Voice:        "coral",
        Instructions: "You are a Conversational AI Agent, developed by Agora.",
    }),
)

Use the following mllm configuration in your request:

"mllm": {
  "enable": true,
  "url": "wss://api.openai.com/v1/realtime",
  "api_key": "<openai_api_key>",
  "params": {
    "model": "gpt-realtime",
    "voice": "coral",
    "instructions": "You are a Conversational AI Agent, developed by Agora.",
    "input_audio_transcription": {
      "language": "<language>",
      "model": "gpt-4o-mini-transcribe",
      "prompt": "expect words related to real-time engagement"
    }
  },
  "turn_detection": {
    // see details below
  },
  "greeting_message": "<greetings>",
  "output_modalities": ["text", "audio"],
  "vendor": "openai"
}

Turn detection

To set up turn detection, add a turn_detection block inside the mllm object when you Start a conversational AI agent.

Info

When mllm.turn_detection is defined, the top-level turn_detection object has no effect.

The following examples show the supported turn_detection configurations for OpenAI Realtime API.

  • Server VAD

    "turn_detection": {
      "mode": "server_vad",
      "server_vad_config": {
        "prefix_padding_ms": 800,
        "silence_duration_ms": 640,
        "threshold": 0.5
      }
    }
  • Semantic VAD

    "turn_detection": {
      "mode": "semantic_vad",
      "semantic_vad_config": {
        "eagerness": "auto"
      }
    }
  • Agora VAD

    "turn_detection": {
      "mode": "agora_vad",
      "agora_vad_config": {
        "interrupt_duration_ms": 160,
        "prefix_padding_ms": 800,
        "silence_duration_ms": 640,
        "threshold": 0.5
      }
    }

Key parameters

mllm required
ParameterDescription
enableboolean
optional

Enables the MLLM module. Replaces the deprecated advanced_features.enable_mllm.

urlstring
required

The WebSocket URL for OpenAI Realtime API.

api_keystring
required

The API key used for authentication. Get your API key from the OpenAI Console.

messagesarray[object]
optional

An array of conversation history items passed to the model as context. Each item represents a single message in the conversation history.

  • role: string

    The role of the message author. For example, system or user.

  • content: string

    The content of the message.

paramsobject
optional

Additional MLLM configuration parameters. See the MLLM provider pages in this section for details.

  • Modalities override: The modalities setting in params is overridden by input_modalities and output_modalities.
  • Turn detection override: The turn_detection setting in params is overridden by mllm.turn_detection.
  • Optional model?: string

    The model identifier.

  • Optional voice?: string

    The voice identifier for audio output.

  • Optional instructions?: string

    System instructions that define the assistant's behavior and personality.

  • Optional input_audio_transcription?: object

    Configuration for audio input transcription.

    • Optional language?: string

      The language of the input audio. Supplying the input language in ISO-639-1 format (For example en) improves accuracy and latency.

    • Optional model?: string

      The model to use for transcription. Current options are gpt-4o-transcribe, gpt-4o-mini-transcribe, and whisper-1.

    • Optional prompt?: string

      An optional text to guide the model's style or continue a previous audio segment. For whisper-1, the prompt is a list of keywords. For gpt-4o-transcribe models, the prompt is a free text string, for example "expect words related to technology".

turn_detectionobject
optional

Turn detection configuration for the MLLM module. For a full list of turn_detection parameters, see mllm.turn_detection.

  • Optional mode?: string
    Possible values
    • agora_vad
    • server_vad
    • semantic_vad
    • agora_vad: Agora VAD-based detection.
    • server_vad: Vendor-side VAD-based detection.
    • semantic_vad: Semantic-based detection.
  • Optional agora_vad_config?: object

    Configuration for Agora VAD-based turn detection. Applicable when mode is agora_vad.

    • Optional interrupt_duration_ms?: integer

      Minimum duration of speech in milliseconds required to trigger an interruption.

    • Optional prefix_padding_ms?: integer

      Duration of audio in milliseconds to include before the detected speech start.

    • Optional silence_duration_ms?: integer

      Duration of silence in milliseconds required to determine end of speech.

    • Optional threshold?: number

      VAD sensitivity threshold. A higher value reduces false positives.

  • Optional server_vad_config?: object

    Configuration for vendor-side VAD-based turn detection. Applicable when mode is server_vad. Parameters are passed through to the vendor.

    • Optional prefix_padding_ms?: integer

      Duration of audio in milliseconds to include before the detected speech start.

    • Optional silence_duration_ms?: integer

      Duration of silence in milliseconds required to determine end of speech.

    • Optional threshold?: number

      VAD sensitivity threshold.

    • Optional idle_timeout_ms?: integer

      Idle timeout in milliseconds.

  • Optional semantic_vad_config?: object

    Configuration for semantic-based turn detection. Applicable when mode is semantic_vad.

    • Optional eagerness?: string
      Possible values
      • auto
      • low
      • medium
      • high

      Controls how eagerly the model ends its turn.

input_modalitiesarray[string]
optional
Default value
["audio"]

MLLM input modalities:

  • ["audio"]: Audio only
  • ["audio", "text"]: Audio plus text
output_modalitiesarray[string]
optional
Default value
["text", "audio"]

Output format options: ["text", "audio"] for both text and voice responses.

greeting_messagestring
optional

Initial message the agent speaks when a user joins the channel.

vendorstring
optional

MLLM provider identifier. Set to openai for OpenAI Realtime API.

The table extends beyond the current width. Scroll horizontally to view more columns.

For comprehensive API reference, real-time capabilities, and detailed parameter descriptions, see the OpenAI Realtime API documentation.