OpenAI GPT-Live

OpenAI GPT-Live provides multimodal large language model capabilities with real-time audio processing, enabling natural voice conversations without separate ASR/TTS components.

Info

Enabling MLLM automatically disables ASR, LLM, and TTS since the MLLM handles end-to-end voice processing directly.

Caution

GPT-Live is an alpha preview built on gpt-live-1-diamond-alpha. It is not intended for production traffic.

SDK version

The SDK examples require Agora Agent SDK v2.8.0 or later. The SDK automatically routes GPT-Live sessions to the preview endpoint and includes the agora-feature: live-models header. Direct REST API requests must use the preview URL and header shown in the REST API tab.

Sample configuration

The following examples show how to configure OpenAI GPT-Live MLLM when starting a conversational AI agent.

from agora_agent import Agent, OpenAIGPTLive

# client is your configured Agora client
agent = (
    Agent(client)
    .with_mllm(OpenAIGPTLive(
        api_key='your-openai-key',
        greeting="Hello! I'm GPT-live. How can I help you today?",
        model='gpt-live-1-diamond-alpha',
        voice='marin',
        prompt='your-system-prompt',
    ))
)

import { Agent, OpenAIGPTLive } from 'agora-agents';

// client is your configured Agora client
const agent = new Agent({ client })
  .withMllm(new OpenAIGPTLive({
    apiKey: 'your-openai-key',
    greeting: "Hello! I'm GPT-live. How can I help you today?",
    model: 'gpt-live-1-diamond-alpha',
    voice: 'marin',
    prompt: 'your-system-prompt',
  }));

import (
    "github.com/AgoraIO/agora-agents-go/v2/agentkit"
    "github.com/AgoraIO/agora-agents-go/v2/agentkit/vendors"
)

// client is your configured Agora client
agent := agentkit.NewAgent(client).WithMllm(
    vendors.NewOpenAIGPTLive(vendors.OpenAIGPTLiveOptions{
        APIKey:          "your-openai-key",
        GreetingMessage: "Hello! I'm GPT-live. How can I help you today?",
        Model:           "gpt-live-1-diamond-alpha",
        Voice:           "marin",
        Prompt:          "your-system-prompt",
    }),
)

Info

OpenAI GPT-Live is available as an early access preview. Send your Start a conversational AI agent request to the following preview endpoint instead of the standard endpoint, and include the required header:

URL: https://partner.ai.agora.io/preview/api/conversational-ai-agent/v2/projects/<APP_ID>/join
Header: agora-feature: live-models

Use the following mllm configuration in your request:

"mllm": {
  "enable": true,
  "vendor": "openai_gpt_live",
  "api_key": "<openai_api_key>",
  "url": "wss://api.openai.com/v1/live/sessions",
  "greeting_message": "Hello! I'm GPT-live. How can I help you today?",
  "params": {
    "model": "gpt-live-1-diamond-alpha",
    "alpha_selector": "quicksilver=v3",
    "voice": "marin",
    "prompt": "<system_prompt>"
  }
}

Tool calling

GPT-Live can call tools defined directly on the MLLM configuration. Set mllm.mcp_servers, enable advanced_features.enable_tools, and set params.tool_enabled to true to advertise those tools to GPT-Live:

"mllm": {
  "mcp_servers": [
    {
      "name": "lookup",
      "endpoint": "https://tools.example/mcp",
      "transport": "streamable_http"
    }
  ],
  "params": {
    "tool_enabled": true
  }
},
"advanced_features": {
  "enable_tools": true
}

Key parameters

mllmrequired

enableboolean

optional

Enables the MLLM module. Replaces the deprecated advanced_features.enable_mllm.

vendorstring

required

MLLM provider identifier. Set to openai_gpt_live for OpenAI GPT-Live.

api_keystring

required

The API key used for authentication. Requires an alpha-enabled OpenAI key.

urlstring

optional

Default value: wss://api.openai.com/v1/live/sessions

The WebSocket URL for OpenAI GPT-Live.

messagesarray[object]

optional

An array of conversation history items passed to the model as context. Each item represents a single message in the conversation history.

rolestring

required

The role of the message author. For example, system or user.

contentstring

required

The content of the message.

input_modalitiesarray[string]

optional

Default value: ["audio"]

MLLM input modalities:

["audio"]: Audio only
["audio", "text"]: Audio plus text

output_modalitiesarray[string]

optional

Default value: ["text", "audio"]

Output format options: ["text", "audio"] for both text and voice responses.

greeting_messagestring

optional

Initial message the agent speaks when a user joins the channel.

failure_messagestring

optional

The message the agent speaks when an error occurs.

mcp_serversarray[object]

optional

MCP servers whose tools GPT-Live can call. See Tool calling.

paramsobject

optional

Additional MLLM configuration parameters.

modelstring

optional

Default value: gpt-live-1-diamond-alpha

The model identifier.

alpha_selectorstring

optional

Default value: quicksilver=v3

The OpenAI-Alpha selector required by the GPT-Live v3 preview contract. Agent SDK v2.8.0 or later sends this value automatically. Include it explicitly in direct REST API requests.

voicestring

optional

Default value: marin

The voice identifier for audio output.

promptstring

optional

Session instructions that define the assistant's behavior and personality.

tool_enabledboolean

optional

Default value: false

Advertises the agent's graph tools to GPT-Live for tool calling. See Tool calling. Does not control delegate built-in tools.

Info

GPT-Live does not support turn detection and input audio transcription.

Caution

The parameters listed on this page are validated for use with Conversational AI Engine. Required parameters must be provided as documented. Any additional parameters are passed through directly to the underlying vendor without validation.

Sample configuration

Tool calling

Key parameters

Contact Us

Contact Us