OpenAI GPT-Live

OpenAI GPT-Live provides multimodal large language model capabilities with real-time audio processing, enabling natural voice conversations without separate ASR/TTS components. Enabling MLLM automatically disables ASR, LLM, and TTS since the MLLM handles end-to-end voice processing directly.

Sample configuration

The following examples show how to configure OpenAI GPT-Live MLLM when starting a conversational AI agent.

from agora_agent import Agent, OpenAIGPTLive

# client is your configured Agora client
agent = (
    Agent(client)
    .with_mllm(OpenAIGPTLive(
        api_key='your-openai-key',
        greeting="Hello! I'm GPT-live. How can I help you today?",
        model='gpt-live-1',
        voice='marin',
        prompt='your-system-prompt',
    ))
)

import { Agent, OpenAIGPTLive } from 'agora-agents';

// client is your configured Agora client
const agent = new Agent({ client })
  .withMllm(new OpenAIGPTLive({
    apiKey: 'your-openai-key',
    greeting: "Hello! I'm GPT-live. How can I help you today?",
    model: 'gpt-live-1',
    voice: 'marin',
    prompt: 'your-system-prompt',
  }));

import (
    "github.com/AgoraIO/agora-agents-go/v2/agentkit"
    "github.com/AgoraIO/agora-agents-go/v2/agentkit/vendors"
)

// client is your configured Agora client
agent := agentkit.NewAgent(client).WithMllm(
    vendors.NewOpenAIGPTLive(vendors.OpenAIGPTLiveOptions{
        APIKey:          "your-openai-key",
        GreetingMessage: "Hello! I'm GPT-live. How can I help you today?",
        Model:           "gpt-live-1",
        Voice:           "marin",
        Prompt:          "your-system-prompt",
    }),
)

Use the following mllm configuration in your request:

"mllm": {
  "enable": true,
  "vendor": "openai_gpt_live",
  "api_key": "<openai_api_key>",
  "url": "wss://api.openai.com/v1/live/sessions",
  "greeting_message": "Hello! I'm GPT-live. How can I help you today?",
  "params": {
    "model": "gpt-live-1",
    "voice": "marin",
    "prompt": "<system_prompt>"
  }
}

Tool calling

GPT-Live can call tools defined directly on the MLLM configuration. Set mllm.mcp_servers, enable advanced_features.enable_tools, and set params.tool_enabled to true to advertise those tools to GPT-Live:

"mllm": {
  "mcp_servers": [
    {
      "name": "lookup",
      "endpoint": "https://tools.example/mcp",
      "transport": "streamable_http"
    }
  ],
  "params": {
    "tool_enabled": true
  }
},
"advanced_features": {
  "enable_tools": true
}

Key parameters

mllmrequired

enableboolean

optional

Enables the MLLM module. Replaces the deprecated advanced_features.enable_mllm.

vendorstring

required

MLLM provider identifier. Set to openai_gpt_live for OpenAI GPT-Live.

api_keystring

required

The OpenAI API key used for authentication.

urlstring

optional

Default value: wss://api.openai.com/v1/live/sessions

The WebSocket URL for OpenAI GPT-Live.

messagesarray[object]

optional

An array of conversation history items passed to the model as context. Each item represents a single message in the conversation history.

rolestring

required

The role of the message author. For example, system or user.

contentstring

required

The content of the message.

input_modalitiesarray[string]

optional

Default value: ["audio"]

MLLM input modalities:

["audio"]: Audio only
["audio", "text"]: Audio plus text

output_modalitiesarray[string]

optional

Default value: ["text", "audio"]

Output format options: ["text", "audio"] for both text and voice responses.

greeting_messagestring

optional

Initial message the agent speaks when a user joins the channel.

failure_messagestring

optional

The message the agent speaks when an error occurs.

mcp_serversarray[object]

optional

MCP servers whose tools GPT-Live can call. See Tool calling.

paramsobject

optional

Additional MLLM configuration parameters.

modelstring

optional

Default value: gpt-live-1

The model identifier. Specify another supported model to override the default.

voicestring

optional

Default value: marin

The voice identifier for audio output.

promptstring

optional

Session instructions that define the assistant's behavior and personality.

tool_enabledboolean

optional

Default value: false

Advertises the agent's graph tools to GPT-Live for tool calling. See Tool calling. Does not control delegate built-in tools.

Limitations

GPT-Live doesn't support turn detection or input audio transcription.

The parameters listed on this page are validated for use with Conversational AI Engine. Required parameters must be provided as documented. Additional parameters are passed through directly to the underlying vendor without validation.

Sample configuration

Tool calling

Key parameters

Contact Us

Contact Us