xAI Grok

xAI Grok provides multimodal large language model capabilities with real-time audio processing, enabling natural voice conversations without separate ASR/TTS components. This page covers integration using the xAI Realtime API, authenticated with an API key obtained from the xAI developer console.

Info

Enabling MLLM automatically disables ASR, LLM, and TTS since the MLLM handles end-to-end voice processing directly.

Sample configuration

The following examples show how to configure xAI Grok MLLM when starting a conversational AI agent.

from agora_agent import Agent
from agora_agent.agentkit.vendors import XaiGrok

# client is your configured Agora client
agent = (
    Agent(client)
    .with_mllm(XaiGrok(
        api_key='your-xai-key',
        voice='eve',
        language='en',
        sample_rate=24000,
    ))
)

import { Agent, XaiGrok } from 'agora-agents';

// client is your configured Agora client
const agent = new Agent({ client })
  .withMllm(new XaiGrok({
    apiKey: 'your-xai-key',
    voice: 'eve',
    language: 'en',
    sampleRate: 24000,
  }));

import "github.com/AgoraIO/agora-agents-go/v2/agentkit/vendors"

// client is your configured Agora client
agent := agentkit.NewAgent(client).WithMllm(
    vendors.NewXaiGrok(vendors.XaiGrokOptions{
        APIKey:     "your-xai-key",
        Voice:      "eve",
        Language:   "en",
        SampleRate: agora.Int(24000),
    }),
)

Use the following mllm configuration in your request:

"mllm": {
  "enable": true,
  "vendor": "xai",
  "url": "wss://api.x.ai/v1/realtime",
  "api_key": "<XAI_API_KEY>",
  "messages": [
    {
      "role": "user",
      "content": "<HISTORY_CONTENT>"
    }
  ],
  "output_modalities": [
    "audio",
    "text"
  ],
  "params": {
    "voice": "eve",
    "language": "en",
    "sample_rate": 24000
  },
  "turn_detection": {
    // see details below
  },
  "greeting_message": "Hello, how can I help?"
}

Turn detection

To set up turn detection, add a turn_detection block inside the mllm object when you Start a conversational AI agent.

Info

When mllm.turn_detection is defined, the top-level turn_detection object has no effect.

The following examples show the supported configurations for xAI Grok.

Server VAD

"turn_detection": {
  "mode": "server_vad",
  "server_vad_config": {
    "threshold": 0.5,
    "prefix_padding_ms": 640,
    "silence_duration_ms": 900
  }
}

Agora VAD

"turn_detection": {
  "mode": "agora_vad",
  "agora_vad_config": {
    "threshold": 0.5,
    "interrupt_duration_ms": 160,
    "prefix_padding_ms": 800,
    "silence_duration_ms": 640
  }
}

Key parameters

mllmrequired

enableboolean

optional

Enables the MLLM module. Replaces the deprecated advanced_features.enable_mllm.

vendorstring

required

The MLLM provider identifier. Set to "xai" to use xAI Grok.

urlstring

required

The WebSocket endpoint for the xAI Realtime API. Set to "wss://api.x.ai/v1/realtime".

api_keystring

required

The xAI API key used to authenticate requests. Get your API key from the xAI Console.

messagesarray[object]

optional

An array of conversation history items passed to the model as context. Each item represents a single message in the conversation history.

rolestring

required

The role of the message author. For example, system or user.

contentstring

required

The content of the message.

paramsobject

required

Configuration object for the xAI Grok model.

voicestring

optional

The voice identifier for audio output. For example, eve or rex.

languagestring

optional

The language code for speech recognition and synthesis. For example, en.

sample_rateinteger

optional

The audio sample rate in Hz. For example, 24000.

turn_detectionobject

optional

Turn detection configuration for the MLLM module. For a full list of turn_detection parameters, see mllm.turn_detection.

modestring

optional

Possible values: agora_vad
server_vad

agora_vad: Agora VAD-based detection.
server_vad: Vendor-side VAD-based detection.

agora_vad_configobject

optional

Configuration for Agora VAD-based turn detection. Applicable when mode is agora_vad.

interrupt_duration_msinteger

optional

Minimum duration of speech in milliseconds required to trigger an interruption.

prefix_padding_msinteger

optional

Duration of audio in milliseconds to include before the detected speech start.

silence_duration_msinteger

optional

Duration of silence in milliseconds required to determine end of speech.

thresholdnumber

optional

VAD sensitivity threshold. A higher value reduces false positives.

server_vad_configobject

optional

Configuration for vendor-side VAD-based turn detection. Applicable when mode is server_vad. Parameters are passed through to the vendor.

thresholdnumber

optional

VAD sensitivity threshold. A higher value reduces false positives.

prefix_padding_msinteger

optional

Duration of audio in milliseconds to include before the detected speech start.

silence_duration_msinteger

optional

Duration of silence in milliseconds required to determine end of speech.

output_modalitiesarray[string]

optional

Default value: ["audio"]

Output modalities for the MLLM.

["audio"]: Audio-only output
["text", "audio"]: Combined text and audio output

greeting_messagestring

optional

The message the agent speaks when a user joins the channel.

failure_messagestring

optional

The message the agent speaks when an error occurs.

For comprehensive API reference, real-time capabilities, and detailed parameter descriptions, see the xAI Voice Agent API.

Sample configuration

Turn detection

Key parameters

Contact Us

Contact Us