Google Gemini Live (Vertex AI)

Updated

Integrate Google Gemini Live with the Conversational AI Engine using Google Cloud Vertex AI.

Google Gemini Live provides multimodal large language model capabilities with real-time audio processing, enabling natural voice conversations without separate ASR/TTS components. This page covers integration using Vertex AI, authenticated with Google Cloud Application Default Credentials (ADC). Enabling MLLM automatically disables ASR, LLM, and TTS since the MLLM handles end-to-end voice processing directly.

Sample configuration

The following examples show how to configure Google Gemini Live (Vertex AI) MLLM when starting a conversational AI agent.

from agora_agent import Agent
from agora_agent.agentkit.vendors import VertexAI

# client is your configured Agora client
agent = (
    Agent(client)
    .with_mllm(VertexAI(
        model='gemini-live-2.5-flash-native-audio',
        project_id='<GOOGLE_CLOUD_PROJECT_ID>',
        location='<GOOGLE_CLOUD_REGION>',
        adc_credentials_string='<GOOGLE_APPLICATION_CREDENTIALS_STRING>',
        voice='Aoede',
        instructions='You are a friendly assistant.',
        transcribe_agent=True,
        transcribe_user=True,
    ))
)
import { Agent, VertexAI } from 'agora-agents';

// client is your configured Agora client
const agent = new Agent({ client })
  .withMllm(new VertexAI({
    model: 'gemini-live-2.5-flash-native-audio',
    projectId: '<GOOGLE_CLOUD_PROJECT_ID>',
    location: '<GOOGLE_CLOUD_REGION>',
    adcCredentialsString: '<GOOGLE_APPLICATION_CREDENTIALS_STRING>',
    voice: 'Aoede',
    instructions: 'You are a friendly assistant.',
    transcribeAgent: true,
    transcribeUser: true,
  }));
import (
    Agora "github.com/AgoraIO/agora-agents-go/v2"
    "github.com/AgoraIO/agora-agents-go/v2/agentkit"
    "github.com/AgoraIO/agora-agents-go/v2/agentkit/vendors"
)

// client is your configured Agora client
agent := agentkit.NewAgent(client).WithMllm(
    vendors.NewVertexAI(vendors.VertexAIOptions{
        Model:               "gemini-live-2.5-flash-native-audio",
        ProjectID:           "<GOOGLE_CLOUD_PROJECT_ID>",
        Location:            "<GOOGLE_CLOUD_REGION>",
        ADCredentialsString: "<GOOGLE_APPLICATION_CREDENTIALS_STRING>",
        Voice:               "Aoede",
        Instructions:        "You are a friendly assistant.",
        TranscribeAgent:     Agora.Bool(true),
        TranscribeUser:      Agora.Bool(true),
    }),
)

Use the following mllm configuration in your request:

"mllm": {
  "enable": true,
  "messages": [
    {
      "role": "user",
      "content": "<HISTORY_CONTENT>"
    }
  ],
  "params": {
    "model": "gemini-live-2.5-flash-native-audio",
    "instructions": "<YOUR_SYSTEM_PROMPT>",
    "voice": "Aoede",
    "adc_credentials_string": "<GOOGLE_APPLICATION_CREDENTIALS_STRING>",
    "project_id": "<GOOGLE_CLOUD_PROJECT_ID>",
    "location": "<GOOGLE_CLOUD_REGION>",    
    "transcribe_agent": true,
    "transcribe_user": true
  },
  "turn_detection": {
    // see details below
  },
  "greeting_message": "Hi, how can I assist you today?",
  "input_modalities": [
    "audio"
  ],
  "output_modalities": [
    "audio"
  ],
  "vendor": "vertexai"
}

Turn detection

To set up turn detection, add a turn_detection block inside the mllm object when you Start a conversational AI agent.

Info

When mllm.turn_detection is defined, the top-level turn_detection object has no effect.

The following examples show the supported configurations for Google Gemini Live (Vertex AI).

  • Server VAD

    "turn_detection": {
      "mode": "server_vad",
      "server_vad_config": {
        "prefix_padding_ms": 800,
        "silence_duration_ms": 640,
        "start_of_speech_sensitivity": "START_SENSITIVITY_HIGH",
        "end_of_speech_sensitivity": "END_SENSITIVITY_HIGH"
      }
    }
  • Agora VAD

    "turn_detection": {
      "mode": "agora_vad",
      "agora_vad_config": {
        "interrupt_duration_ms": 160,
        "prefix_padding_ms": 800,
        "silence_duration_ms": 640,
        "threshold": 0.5
      }
    }

Key parameters

mllm required
ParameterDescription
enableboolean
optional

Enables the MLLM module. Replaces the deprecated advanced_features.enable_mllm.

messagesarray[object]
optional

An array of conversation history items passed to the model as context. Each item represents a single message in the conversation history.

  • role: string

    The role of the message author. For example, user.

  • content: string

    The content of the message.

paramsobject
required

Main configuration object for the Gemini Live model.

  • model: string

    The Gemini Live model identifier. See supported models.

  • Optional instructions?: string

    System instructions that define the agent’s behavior or tone.

  • Optional voice?: string

    The voice identifier for audio output. For example, Aoede, Puck, Charon, Kore, Fenrir, Leda, Orus, or Zephyr.

  • adc_credentials_string: string

    Base64-encoded Google Cloud Application Default Credentials (ADC).

  • project_id: string

    Your Google Cloud project ID for Vertex AI access.

  • location: string

    The Google Cloud region hosting the Gemini Live model. Check the Google Cloud documentation for the full list of available regions.

  • Optional transcribe_agent?: boolean

    Whether to transcribe the agent’s speech in real time.

  • Optional transcribe_user?: boolean

    Whether to transcribe the user’s speech in real time.

turn_detectionobject
optional

Turn detection configuration for the MLLM module. For a full list of turn_detection parameters, see mllm.turn_detection.

  • Optional mode?: string
    Possible values
    • agora_vad
    • server_vad
    • agora_vad: Agora VAD-based detection.
    • server_vad: Vendor-side VAD-based detection.
  • Optional agora_vad_config?: object

    Configuration for Agora VAD-based turn detection. Applicable when mode is agora_vad.

    • Optional interrupt_duration_ms?: integer

      Minimum duration of speech in milliseconds required to trigger an interruption.

    • Optional prefix_padding_ms?: integer

      Duration of audio in milliseconds to include before the detected speech start.

    • Optional silence_duration_ms?: integer

      Duration of silence in milliseconds required to determine end of speech.

    • Optional threshold?: number

      VAD sensitivity threshold. A higher value reduces false positives.

  • Optional server_vad_config?: object

    Configuration for vendor-side VAD-based turn detection. Applicable when mode is server_vad. Parameters are passed through to the vendor.

    • Optional prefix_padding_ms?: integer

      Duration of audio in milliseconds to include before the detected speech start.

    • Optional silence_duration_ms?: integer

      Duration of silence in milliseconds required to determine end of speech.

    • Optional start_of_speech_sensitivity?: string
      Possible values
      • START_SENSITIVITY_HIGH
      • START_SENSITIVITY_LOW

      Sensitivity for start of speech detection.

    • Optional end_of_speech_sensitivity?: string
      Possible values
      • END_SENSITIVITY_HIGH
      • END_SENSITIVITY_LOW

      Sensitivity for end of speech detection.

input_modalitiesarray[string]
optional
Default value
["audio"]

Input modalities for the MLLM.

  • ["audio"]: Audio-only input
  • ["audio", "text"]: Accept both audio and text input
output_modalitiesarray[string]
optional
Default value
["audio"]

Output modalities for the MLLM.

  • ["audio"]: Audio-only response
  • ["text", "audio"]: Combined text and audio output
greeting_messagestring
optional

The message the agent speaks when a user joins the channel.

vendorstring
required

The MLLM provider identifier. Set to "vertexai" to use Google Gemini Live with Vertex AI.

The table extends beyond the current width. Scroll horizontally to view more columns.

For comprehensive API reference, real-time capabilities, and detailed parameter descriptions, see the Google Gemini Live API.