xAI Grok
Updated
Integrate xAI Grok with the Conversational AI Engine using the xAI Realtime API.
xAI Grok provides multimodal large language model capabilities with real-time audio processing, enabling natural voice conversations without separate ASR/TTS components. This page covers integration using the xAI Realtime API, authenticated with an API key obtained from the xAI developer console. Enabling MLLM automatically disables ASR, LLM, and TTS since the MLLM handles end-to-end voice processing directly.
Sample configuration
The following examples show how to configure xAI Grok MLLM when starting a conversational AI agent.
from agora_agent import Agent
from agora_agent.agentkit.vendors import XaiGrok
# client is your configured Agora client
agent = (
Agent(client)
.with_mllm(XaiGrok(
api_key='your-xai-key',
voice='eve',
language='en',
sample_rate=24000,
))
)import { Agent, XaiGrok } from 'agora-agents';
// client is your configured Agora client
const agent = new Agent({ client })
.withMllm(new XaiGrok({
apiKey: 'your-xai-key',
voice: 'eve',
language: 'en',
sampleRate: 24000,
}));import (
Agora "github.com/AgoraIO/agora-agents-go/v2"
"github.com/AgoraIO/agora-agents-go/v2/agentkit"
"github.com/AgoraIO/agora-agents-go/v2/agentkit/vendors"
)
// client is your configured Agora client
agent := agentkit.NewAgent(client).WithMllm(
vendors.NewXaiGrok(vendors.XaiGrokOptions{
APIKey: "your-xai-key",
Voice: "eve",
Language: "en",
SampleRate: Agora.Int(24000),
}),
)Use the following mllm configuration in your request:
"mllm": {
"enable": true,
"vendor": "xai",
"url": "wss://api.x.ai/v1/realtime",
"api_key": "<XAI_API_KEY>",
"messages": [
{
"role": "user",
"content": "<HISTORY_CONTENT>"
}
],
"output_modalities": [
"audio",
"text"
],
"params": {
"voice": "eve",
"language": "en",
"sample_rate": 24000
},
"turn_detection": {
// see details below
},
"greeting_message": "Hello, how can I help?"
}Turn detection
To set up turn detection, add a turn_detection block inside the mllm object when you Start a conversational AI agent.
Info
When mllm.turn_detection is defined, the top-level turn_detection object has no effect.
The following examples show the supported configurations for xAI Grok.
-
Server VAD
"turn_detection": { "mode": "server_vad", "server_vad_config": { "threshold": 0.5, "prefix_padding_ms": 640, "silence_duration_ms": 900 } } -
Agora VAD
"turn_detection": { "mode": "agora_vad", "agora_vad_config": { "threshold": 0.5, "interrupt_duration_ms": 160, "prefix_padding_ms": 800, "silence_duration_ms": 640 } }
Key parameters
| Parameter | Description |
|---|---|
enableboolean | Enables the MLLM module. Replaces the deprecated |
vendorstring | The MLLM provider identifier. Set to |
urlstring | The WebSocket endpoint for the xAI Realtime API. Set to |
api_keystring | The xAI API key used to authenticate requests. Get your API key from the xAI Console. |
messagesarray[object] | An array of conversation history items passed to the model as context. Each item represents a single message in the conversation history.
|
paramsobject | Configuration object for the xAI Grok model.
|
turn_detectionobject | Turn detection configuration for the MLLM module. For a full list of
|
output_modalitiesarray[string] |
Output modalities for the MLLM.
|
greeting_messagestring | The message the agent speaks when a user joins the channel. |
failure_messagestring | The message the agent speaks when an error occurs. |
For comprehensive API reference, real-time capabilities, and detailed parameter descriptions, see the xAI Voice Agent API.
