# xAI Grok (/en/ai/models/mllm/xai)

> For AI agents: see the complete documentation index at [llms.txt](/llms.txt).

xAI Grok provides multimodal large language model capabilities with real-time audio processing, enabling natural voice conversations without separate ASR/TTS components. This page covers integration using the xAI Realtime API, authenticated with an API key obtained from the xAI developer console.

<CalloutContainer type="info">
  <CalloutTitle>
    Info
  </CalloutTitle>

  <CalloutDescription>
    Enabling MLLM automatically disables ASR, LLM, and TTS since the MLLM handles end-to-end voice processing directly.
  </CalloutDescription>
</CalloutContainer>

### Sample configuration [#sample-configuration]

The following examples show how to configure xAI Grok MLLM when starting a conversational AI agent.

<Tabs defaultValue="python" groupId="ai-sdk-language">
  <TabsList>
    <TabsTrigger value="python">
      Python SDK
    </TabsTrigger>

    <TabsTrigger value="typescript">
      TypeScript SDK
    </TabsTrigger>

    <TabsTrigger value="go">
      Go SDK
    </TabsTrigger>

    <TabsTrigger value="rest-api">
      REST API
    </TabsTrigger>
  </TabsList>

  <TabsContent value="python">
    ```python
    from agora_agent import Agent
    from agora_agent.agentkit.vendors import XaiGrok

    # client is your configured Agora client
    agent = (
        Agent(client)
        .with_mllm(XaiGrok(
            api_key='your-xai-key',
            voice='eve',
            language='en',
            sample_rate=24000,
        ))
    )
    ```
  </TabsContent>

  <TabsContent value="typescript">
    ```typescript
    import { Agent, XaiGrok } from 'agora-agents';

    // client is your configured Agora client
    const agent = new Agent({ client })
      .withMllm(new XaiGrok({
        apiKey: 'your-xai-key',
        voice: 'eve',
        language: 'en',
        sampleRate: 24000,
      }));
    ```
  </TabsContent>

  <TabsContent value="go">
    ```go
    import "github.com/AgoraIO/agora-agents-go/v2/agentkit/vendors"

    // client is your configured Agora client
    agent := agentkit.NewAgent(client).WithMllm(
        vendors.NewXaiGrok(vendors.XaiGrokOptions{
            APIKey:     "your-xai-key",
            Voice:      "eve",
            Language:   "en",
            SampleRate: agora.Int(24000),
        }),
    )
    ```
  </TabsContent>

  <TabsContent value="rest-api">
    Use the following `mllm` configuration in your request:

    ```json
    "mllm": {
      "enable": true,
      "vendor": "xai",
      "url": "wss://api.x.ai/v1/realtime",
      "api_key": "<XAI_API_KEY>",
      "messages": [
        {
          "role": "user",
          "content": "<HISTORY_CONTENT>"
        }
      ],
      "output_modalities": [
        "audio",
        "text"
      ],
      "params": {
        "voice": "eve",
        "language": "en",
        "sample_rate": 24000
      },
      "turn_detection": {
        // see details below
      },
      "greeting_message": "Hello, how can I help?"
    }
    ```
  </TabsContent>
</Tabs>

### Turn detection [#turn-detection]

To set up turn detection, add a `turn_detection` block inside the `mllm` object when you [Start a conversational AI agent](/en/api-reference/api-ref/conversational-ai/join).

<CalloutContainer type="info">
  <CalloutTitle>
    Info
  </CalloutTitle>

  <CalloutDescription>
    When `mllm.turn_detection` is defined, the top-level `turn_detection` object has no effect.
  </CalloutDescription>
</CalloutContainer>

The following examples show the supported configurations for xAI Grok.

* **Server VAD**

  ```json
  "turn_detection": {
    "mode": "server_vad",
    "server_vad_config": {
      "threshold": 0.5,
      "prefix_padding_ms": 640,
      "silence_duration_ms": 900
    }
  }
  ```

* **Agora VAD**

  ```json
  "turn_detection": {
    "mode": "agora_vad",
    "agora_vad_config": {
      "threshold": 0.5,
      "interrupt_duration_ms": 160,
      "prefix_padding_ms": 800,
      "silence_duration_ms": 640
    }
  }
  ```

### Key parameters [#key-parameters]

<ParameterList title="mllm" required="true">
  <Parameter name="enable" type="boolean" required="false">
    Enables the MLLM module. Replaces the deprecated `advanced_features.enable_mllm`.
  </Parameter>

  <Parameter name="vendor" type="string" required="true">
    The MLLM provider identifier. Set to `"xai"` to use xAI Grok.
  </Parameter>

  <Parameter name="url" type="string" required="true">
    The WebSocket endpoint for the xAI Realtime API. Set to `"wss://api.x.ai/v1/realtime"`.
  </Parameter>

  <Parameter name="api_key" type="string" required="true">
    The xAI API key used to authenticate requests. Get your API key from the [xAI Console](https://console.x.ai/home).
  </Parameter>

  <Parameter name="messages" type="array[object]" required="false">
    An array of conversation history items passed to the model as context. Each item represents a single message in the conversation history.

    <Parameter name="role" type="string" required="true">
      The role of the message author. For example, `system` or `user`.
    </Parameter>

    <Parameter name="content" type="string" required="true">
      The content of the message.
    </Parameter>
  </Parameter>

  <Parameter name="params" type="object" required="true">
    Configuration object for the xAI Grok model.

    <Parameter name="voice" type="string" required="false">
      The voice identifier for audio output. For example, `eve` or `rex`.
    </Parameter>

    <Parameter name="language" type="string" required="false">
      The language code for speech recognition and synthesis. For example, `en`.
    </Parameter>

    <Parameter name="sample_rate" type="integer" required="false">
      The audio sample rate in Hz. For example, `24000`.
    </Parameter>
  </Parameter>

  <Parameter name="turn_detection" type="object" required="false">
    Turn detection configuration for the MLLM module. For a full list of `turn_detection` parameters, see [`mllm.turn_detection`](/en/api-reference/api-ref/conversational-ai/join#properties-mllm-turn-detection).

    <Parameter name="mode" type="string" required="false" possibleValues="agora_vad, server_vad">
      * `agora_vad`: Agora VAD-based detection.
      * `server_vad`: Vendor-side VAD-based detection.
    </Parameter>

    <Parameter name="agora_vad_config" type="object" required="false">
      Configuration for Agora VAD-based turn detection. Applicable when `mode` is `agora_vad`.

      <Parameter name="interrupt_duration_ms" type="integer" required="false">
        Minimum duration of speech in milliseconds required to trigger an interruption.
      </Parameter>

      <Parameter name="prefix_padding_ms" type="integer" required="false">
        Duration of audio in milliseconds to include before the detected speech start.
      </Parameter>

      <Parameter name="silence_duration_ms" type="integer" required="false">
        Duration of silence in milliseconds required to determine end of speech.
      </Parameter>

      <Parameter name="threshold" type="number" required="false">
        VAD sensitivity threshold. A higher value reduces false positives.
      </Parameter>
    </Parameter>

    <Parameter name="server_vad_config" type="object" required="false">
      Configuration for vendor-side VAD-based turn detection. Applicable when `mode` is `server_vad`. Parameters are passed through to the vendor.

      <Parameter name="threshold" type="number" required="false">
        VAD sensitivity threshold. A higher value reduces false positives.
      </Parameter>

      <Parameter name="prefix_padding_ms" type="integer" required="false">
        Duration of audio in milliseconds to include before the detected speech start.
      </Parameter>

      <Parameter name="silence_duration_ms" type="integer" required="false">
        Duration of silence in milliseconds required to determine end of speech.
      </Parameter>
    </Parameter>
  </Parameter>

  <Parameter name="output_modalities" type="array[string]" defaultValue="[&#x22;audio&#x22;]" required="false">
    Output modalities for the MLLM.

    * `["audio"]`: Audio-only output
    * `["text", "audio"]`: Combined text and audio output
  </Parameter>

  <Parameter name="greeting_message" type="string" required="false">
    The message the agent speaks when a user joins the channel.
  </Parameter>

  <Parameter name="failure_message" type="string" required="false">
    The message the agent speaks when an error occurs.
  </Parameter>
</ParameterList>

For comprehensive API reference, real-time capabilities, and detailed parameter descriptions, see the [xAI Voice Agent API](https://docs.x.ai/developers/model-capabilities/audio/voice-agent).