Google Gemini Live

Updated

Integrate Google Gemini Live with the Conversational AI Engine using the Gemini Developer API.

Agora has partnered with Google Gemini Live to bring frontier multimodal large language model capabilities to your app. Gemini Live provides real-time audio processing, enabling natural voice conversations without separate ASR and TTS components. This page explains how to integrate Gemini Live through the Gemini Developer API. Enabling MLLM automatically disables ASR, LLM, and TTS since the MLLM handles end-to-end voice processing directly.

Supported models

The following Gemini Live models are supported:

ModelDescription
models/gemini-3.8-liveGeneral-purpose Gemini Live model.
models/gemini-3.8-live-extended-thinkingGemini Live model with an adjustable reasoning budget.

Use models/gemini-3.8-live-extended-thinking when the conversation benefits from additional reasoning. Set thinking_level to low, medium, or high. Omit thinking_level when using models/gemini-3.8-live.

Sample configuration

The following examples show how to configure Google Gemini Live MLLM (Gemini Developer API) when starting a conversational AI agent.

from agora_agent import Agent
from agora_agent.agentkit.vendors import GeminiLive

# client is your configured Agora client
agent = (
    Agent(client)
    .with_mllm(GeminiLive(
        api_key='your-google-api-key',
        model='models/gemini-3.8-live-extended-thinking',
        thinking_level='medium',
        voice='Charon',
        instructions='You are a friendly assistant.',
        transcribe_agent=True,
        transcribe_user=True,
    ))
)
import { Agent, GeminiLive } from 'agora-agents';

// client is your configured Agora client
const agent = new Agent({ client })
  .withMllm(new GeminiLive({
    apiKey: 'your-google-api-key',
    model: 'models/gemini-3.8-live-extended-thinking',
    thinkingLevel: 'medium',
    voice: 'Charon',
    instructions: 'You are a friendly assistant.',
    transcribeAgent: true,
    transcribeUser: true,
  }));
import (
    Agora "github.com/AgoraIO/agora-agents-go/v2"
    "github.com/AgoraIO/agora-agents-go/v2/agentkit"
    "github.com/AgoraIO/agora-agents-go/v2/agentkit/vendors"
)

// client is your configured Agora client
agent := agentkit.NewAgent(client).WithMllm(
    vendors.NewGeminiLive(vendors.GeminiLiveOptions{
        APIKey:         "your-google-api-key",
        Model:          "models/gemini-3.8-live-extended-thinking",
        ThinkingLevel:  vendors.GeminiThinkingLevelMedium,
        Voice:          "Charon",
        Instructions:   "You are a friendly assistant.",
        TranscribeAgent: Agora.Bool(true),
        TranscribeUser:  Agora.Bool(true),
    }),
)

Use the following mllm configuration in your request:

"mllm": {
  "enable": true,
  "api_key": "<GOOGLE_GEMINI_API_KEY>",
  "messages": [
    {
      "role": "user",
      "content": "<HISTORY_CONTENT>"
    }
  ],
  "params": {
    "model": "models/gemini-3.8-live-extended-thinking",
    "thinking_level": "medium",
    "instructions": "You are a friendly assistant.",
    "voice": "Charon",
    "affective_dialog": false,
    "proactive_audio": false,
    "transcribe_agent": true,
    "transcribe_user": true,
    "http_options": {
      "api_version": "v1beta"
    }
  },
  "turn_detection": {
    // see details below
  },
  "input_modalities": [
    "audio"
  ],
  "output_modalities": [
    "audio"
  ],
  "greeting_message": "Hi, how can I assist you today?",
  "failure_message": "Sorry, I encountered an issue. Please try again.",
  "vendor": "gemini"
}

Turn detection

To set up turn detection, add a turn_detection block inside the mllm object when you Start a conversational AI agent.

Info

When mllm.turn_detection is defined, the top-level turn_detection object has no effect.

The following examples show the supported configurations for Google Gemini Live.

  • Server VAD

    "turn_detection": {
      "mode": "server_vad",
      "server_vad_config": {
        "prefix_padding_ms": 800,
        "silence_duration_ms": 640,
        "start_of_speech_sensitivity": "START_SENSITIVITY_HIGH",
        "end_of_speech_sensitivity": "END_SENSITIVITY_HIGH"
      }
    }
  • Agora VAD

    "turn_detection": {
      "mode": "agora_vad",
      "agora_vad_config": {
        "interrupt_duration_ms": 160,
        "prefix_padding_ms": 800,
        "silence_duration_ms": 640,
        "threshold": 0.5
      }
    }

Key parameters

mllm required
ParameterDescription
enableboolean
optional

Enables the MLLM module. Replaces the deprecated advanced_features.enable_mllm.

api_keystring
required

The Google Gemini API key used to authenticate requests. You can generate an API key in Google AI Studio.

messagesarray[object]
optional

An array of conversation history items passed to the model as context. Each item represents a single message in the conversation history.

  • role: string

    The role of the message author. For example, system or user.

  • content: string

    The content of the message.

paramsobject
required

Configuration object for the Gemini Live model.

  • model: string
    Possible values
    • models/gemini-3.8-live
    • models/gemini-3.8-live-extended-thinking

    The Gemini Live model identifier. Set to models/gemini-3.8-live-extended-thinking to enable Extended Thinking. See Supported models for all available options.

  • Optional thinking_level?: string
    Possible values
    • low
    • medium
    • high

    The reasoning budget for models/gemini-3.8-live-extended-thinking. Do not set this parameter with models/gemini-3.8-live.

  • Optional instructions?: string

    System instructions that define the agent's behavior or tone.

  • Optional voice?: string

    The voice identifier for audio output. For example, Aoede, Puck, Charon, Kore, Fenrir, Leda, Orus, or Zephyr.

  • Optional affective_dialog?: boolean

    Whether to enable affective dialog, which allows the model to adapt its tone based on the user's emotional cues.

  • Optional proactive_audio?: boolean

    When enabled, the model may choose not to respond if the user's input does not require a reply, such as background speech or incomplete requests.

  • Optional transcribe_agent?: boolean

    Whether to transcribe the agent's speech in real time.

  • Optional transcribe_user?: boolean

    Whether to transcribe the user's speech in real time.

  • Optional http_options?: object

    HTTP request options for the Gemini Live API.

    • Optional api_version?: string

      The API version to use. For example, v1beta.

turn_detectionobject
optional

Turn detection configuration for the MLLM module. For a full list of turn_detection parameters, see mllm.turn_detection.

  • Optional mode?: string
    Possible values
    • agora_vad
    • server_vad
    • agora_vad: Agora VAD-based detection.
    • server_vad: Vendor-side VAD-based detection.
  • Optional agora_vad_config?: object

    Configuration for Agora VAD-based turn detection. Applicable when mode is agora_vad.

    • Optional interrupt_duration_ms?: integer

      Minimum duration of speech in milliseconds required to trigger an interruption.

    • Optional prefix_padding_ms?: integer

      Duration of audio in milliseconds to include before the detected speech start.

    • Optional silence_duration_ms?: integer

      Duration of silence in milliseconds required to determine end of speech.

    • Optional threshold?: number

      VAD sensitivity threshold. A higher value reduces false positives.

  • Optional server_vad_config?: object

    Configuration for vendor-side VAD-based turn detection. Applicable when mode is server_vad. Parameters are passed through to the vendor.

    • Optional prefix_padding_ms?: integer

      Duration of audio in milliseconds to include before the detected speech start.

    • Optional silence_duration_ms?: integer

      Duration of silence in milliseconds required to determine end of speech.

    • Optional start_of_speech_sensitivity?: string
      Possible values
      • START_SENSITIVITY_HIGH
      • START_SENSITIVITY_LOW

      Sensitivity for start of speech detection.

    • Optional end_of_speech_sensitivity?: string
      Possible values
      • END_SENSITIVITY_HIGH
      • END_SENSITIVITY_LOW

      Sensitivity for end of speech detection.

input_modalitiesarray[string]
optional
Default value
["audio"]

Input modalities for the MLLM.

  • ["audio"]: Audio-only input
  • ["audio", "text"]: Accept both audio and text input
output_modalitiesarray[string]
optional
Default value
["audio"]

Output modalities for the MLLM.

  • ["audio"]: Audio-only output
  • ["text", "audio"]: Combined text and audio output
greeting_messagestring
optional

The message the agent speaks when a user joins the channel.

failure_messagestring
optional

The message the agent speaks when an error occurs.

vendorstring
required

The MLLM provider identifier. Set to "gemini" to use Google Gemini Live with the Gemini Developer API.

The table extends beyond the current width. Scroll horizontally to view more columns.

For comprehensive API reference, real-time capabilities, and detailed parameter descriptions, see the Google Gemini Live API.