# OpenAI Realtime API (/en/ai/models/mllm/openai)

> For AI agents: see the complete documentation index at [llms.txt](/llms.txt).

OpenAI Realtime provides multimodal large language model capabilities with real-time audio processing, enabling natural voice conversations without separate ASR/TTS components.

<CalloutContainer type="info">
  <CalloutTitle>
    Info
  </CalloutTitle>

  <CalloutDescription>
    Enabling MLLM automatically disables ASR, LLM, and TTS since the MLLM handles end-to-end voice processing directly.
  </CalloutDescription>
</CalloutContainer>

### Sample configuration [#sample-configuration]

The following examples show how to configure OpenAI Realtime MLLM when starting a conversational AI agent.

<Tabs defaultValue="python" groupId="ai-sdk-language">
  <TabsList>
    <TabsTrigger value="python">
      Python SDK
    </TabsTrigger>

    <TabsTrigger value="typescript">
      TypeScript SDK
    </TabsTrigger>

    <TabsTrigger value="go">
      Go SDK
    </TabsTrigger>

    <TabsTrigger value="rest-api">
      REST API
    </TabsTrigger>
  </TabsList>

  <TabsContent value="python">
    ```python
    from agora_agent import Agent
    from agora_agent.agentkit.vendors import OpenAIRealtime

    # client is your configured Agora client
    agent = (
        Agent(client)
        .with_mllm(OpenAIRealtime(
            api_key='your-openai-key',
            url='wss://api.openai.com/v1/realtime',
            model='gpt-realtime',
            voice='coral',
            instructions='You are a Conversational AI Agent, developed by Agora.',
        ))
    )
    ```
  </TabsContent>

  <TabsContent value="typescript">
    ```typescript
    import { Agent, OpenAIRealtime } from 'agora-agents';

    // client is your configured Agora client
    const agent = new Agent({ client })
      .withMllm(new OpenAIRealtime({
        apiKey: 'your-openai-key',
        url: 'wss://api.openai.com/v1/realtime',
        model: 'gpt-realtime',
        voice: 'coral',
        instructions: 'You are a Conversational AI Agent, developed by Agora.',
      }));
    ```
  </TabsContent>

  <TabsContent value="go">
    ```go
    import "github.com/AgoraIO/agora-agents-go/v2/agentkit/vendors"

    // client is your configured Agora client
    agent := agentkit.NewAgent(client).WithMllm(
        vendors.NewOpenAIRealtime(vendors.OpenAIRealtimeOptions{
            APIKey:       "your-openai-key",
            URL:          "wss://api.openai.com/v1/realtime",
            Model:        "gpt-realtime",
            Voice:        "coral",
            Instructions: "You are a Conversational AI Agent, developed by Agora.",
        }),
    )
    ```
  </TabsContent>

  <TabsContent value="rest-api">
    Use the following `mllm` configuration in your request:

    ```json
    "mllm": {
      "enable": true,
      "url": "wss://api.openai.com/v1/realtime",
      "api_key": "<openai_api_key>",
      "params": {
        "model": "gpt-realtime",
        "voice": "coral",
        "instructions": "You are a Conversational AI Agent, developed by Agora.",
        "input_audio_transcription": {
          "language": "<language>",
          "model": "gpt-4o-mini-transcribe",
          "prompt": "expect words related to real-time engagement"
        }
      },
      "turn_detection": {
        // see details below
      },
      "greeting_message": "<greetings>",
      "output_modalities": ["text", "audio"],
      "vendor": "openai"
    }
    ```
  </TabsContent>
</Tabs>

### Turn detection [#turn-detection]

To set up turn detection, add a `turn_detection` block inside the `mllm` object when you [Start a conversational AI agent](/en/api-reference/api-ref/conversational-ai/join).

<CalloutContainer type="info">
  <CalloutTitle>
    Info
  </CalloutTitle>

  <CalloutDescription>
    When `mllm.turn_detection` is defined, the top-level `turn_detection` object has no effect.
  </CalloutDescription>
</CalloutContainer>

The following examples show the supported `turn_detection` configurations for OpenAI Realtime API.

* **Server VAD**

  ```json
  "turn_detection": {
    "mode": "server_vad",
    "server_vad_config": {
      "prefix_padding_ms": 800,
      "silence_duration_ms": 640,
      "threshold": 0.5
    }
  }
  ```

* **Semantic VAD**

  ```json
  "turn_detection": {
    "mode": "semantic_vad",
    "semantic_vad_config": {
      "eagerness": "auto"
    }
  }
  ```

* **Agora VAD**

  ```json
  "turn_detection": {
    "mode": "agora_vad",
    "agora_vad_config": {
      "interrupt_duration_ms": 160,
      "prefix_padding_ms": 800,
      "silence_duration_ms": 640,
      "threshold": 0.5
    }
  }
  ```

### Key parameters [#key-parameters]

<ParameterList title="mllm" required="true">
  <Parameter name="enable" type="boolean" required="false">
    Enables the MLLM module. Replaces the deprecated `advanced_features.enable_mllm`.
  </Parameter>

  <Parameter name="url" type="string" required="true">
    The WebSocket URL for OpenAI Realtime API.
  </Parameter>

  <Parameter name="api_key" type="string" required="true">
    The API key used for authentication. Get your API key from the [OpenAI Console](https://platform.openai.com/api-keys).
  </Parameter>

  <Parameter name="messages" type="array[object]" required="false">
    An array of conversation history items passed to the model as context. Each item represents a single message in the conversation history.

    <Parameter name="role" type="string" required="true">
      The role of the message author. For example, `system` or `user`.
    </Parameter>

    <Parameter name="content" type="string" required="true">
      The content of the message.
    </Parameter>
  </Parameter>

  <Parameter name="params" type="object" required="false">
    Additional MLLM configuration parameters. See the MLLM provider pages in this section for details.

    * **Modalities override**: The `modalities` setting in params is overridden by `input_modalities` and `output_modalities`.
    * **Turn detection override**: The `turn_detection` setting in `params` is overridden by [`mllm.turn_detection`](/en/api-reference/api-ref/conversational-ai/join#properties-mllm-turn-detection).

    <Parameter name="model" type="string" required="false">
      The model identifier.
    </Parameter>

    <Parameter name="voice" type="string" required="false">
      The voice identifier for audio output.
    </Parameter>

    <Parameter name="instructions" type="string" required="false">
      System instructions that define the assistant's behavior and personality.
    </Parameter>

    <Parameter name="input_audio_transcription" type="object" required="false">
      Configuration for audio input transcription.

      <Parameter name="language" type="string" required="false">
        The language of the input audio. Supplying the input language in ISO-639-1 format (For example `en`) improves accuracy and latency.
      </Parameter>

      <Parameter name="model" type="string" required="false">
        The model to use for transcription. Current options are `gpt-4o-transcribe`, `gpt-4o-mini-transcribe`, and `whisper-1`.
      </Parameter>

      <Parameter name="prompt" type="string" required="false">
        An optional text to guide the model's style or continue a previous audio segment. For `whisper-1`, the prompt is a list of keywords. For `gpt-4o-transcribe` models, the prompt is a free text string, for example "expect words related to technology".
      </Parameter>
    </Parameter>
  </Parameter>

  <Parameter name="turn_detection" type="object" required="false">
    Turn detection configuration for the MLLM module. For a full list of `turn_detection` parameters, see [`mllm.turn_detection`](/en/api-reference/api-ref/conversational-ai/join#properties-mllm-turn-detection).

    <Parameter name="mode" type="string" required="false" possibleValues="agora_vad, server_vad, semantic_vad">
      * `agora_vad`: Agora VAD-based detection.
      * `server_vad`: Vendor-side VAD-based detection.
      * `semantic_vad`: Semantic-based detection.
    </Parameter>

    <Parameter name="agora_vad_config" type="object" required="false">
      Configuration for Agora VAD-based turn detection. Applicable when `mode` is `agora_vad`.

      <Parameter name="interrupt_duration_ms" type="integer" required="false">
        Minimum duration of speech in milliseconds required to trigger an interruption.
      </Parameter>

      <Parameter name="prefix_padding_ms" type="integer" required="false">
        Duration of audio in milliseconds to include before the detected speech start.
      </Parameter>

      <Parameter name="silence_duration_ms" type="integer" required="false">
        Duration of silence in milliseconds required to determine end of speech.
      </Parameter>

      <Parameter name="threshold" type="number" required="false">
        VAD sensitivity threshold. A higher value reduces false positives.
      </Parameter>
    </Parameter>

    <Parameter name="server_vad_config" type="object" required="false">
      Configuration for vendor-side VAD-based turn detection. Applicable when `mode` is `server_vad`. Parameters are passed through to the vendor.

      <Parameter name="prefix_padding_ms" type="integer" required="false">
        Duration of audio in milliseconds to include before the detected speech start.
      </Parameter>

      <Parameter name="silence_duration_ms" type="integer" required="false">
        Duration of silence in milliseconds required to determine end of speech.
      </Parameter>

      <Parameter name="threshold" type="number" required="false">
        VAD sensitivity threshold.
      </Parameter>

      <Parameter name="idle_timeout_ms" type="integer" required="false">
        Idle timeout in milliseconds.
      </Parameter>
    </Parameter>

    <Parameter name="semantic_vad_config" type="object" required="false">
      Configuration for semantic-based turn detection. Applicable when `mode` is `semantic_vad`.

      <Parameter name="eagerness" type="string" required="false" possibleValues="auto, low, medium, high">
        Controls how eagerly the model ends its turn.
      </Parameter>
    </Parameter>
  </Parameter>

  <Parameter name="input_modalities" type="array[string]" defaultValue="[&#x22;audio&#x22;]" required="false">
    MLLM input modalities:

    * `["audio"]`: Audio only
    * `["audio", "text"]`: Audio plus text
  </Parameter>

  <Parameter name="output_modalities" type="array[string]" defaultValue="[&#x22;text&#x22;, &#x22;audio&#x22;]" required="false">
    Output format options: `["text", "audio"]` for both text and voice responses.
  </Parameter>

  <Parameter name="greeting_message" type="string" required="false">
    Initial message the agent speaks when a user joins the channel.
  </Parameter>

  <Parameter name="vendor" type="string" required="false">
    MLLM provider identifier. Set to `openai` for OpenAI Realtime API.
  </Parameter>
</ParameterList>

For comprehensive API reference, real-time capabilities, and detailed parameter descriptions, see the [OpenAI Realtime API documentation](https://platform.openai.com/docs/guides/realtime).