Google Gemini Live (Vertex AI)

Google Gemini Live provides multimodal large language model capabilities with real-time audio processing, enabling natural voice conversations without separate ASR/TTS components. This page covers integration using Vertex AI, authenticated with Google Cloud Application Default Credentials (ADC).

Info

Enabling MLLM automatically disables ASR, LLM, and TTS since the MLLM handles end-to-end voice processing directly.

Sample configuration

The following examples show how to configure Google Gemini Live (Vertex AI) MLLM when starting a conversational AI agent.

from agora_agent import Agent
from agora_agent.agentkit.vendors import VertexAI

# client is your configured Agora client
agent = (
    Agent(client)
    .with_mllm(VertexAI(
        model='gemini-live-2.5-flash-native-audio',
        project_id='<GOOGLE_CLOUD_PROJECT_ID>',
        location='<GOOGLE_CLOUD_REGION>',
        adc_credentials_string='<GOOGLE_APPLICATION_CREDENTIALS_STRING>',
        voice='Aoede',
        instructions='You are a friendly assistant.',
        transcribe_agent=True,
        transcribe_user=True,
    ))
)

import { Agent, VertexAI } from 'agora-agents';

// client is your configured Agora client
const agent = new Agent({ client })
  .withMllm(new VertexAI({
    model: 'gemini-live-2.5-flash-native-audio',
    projectId: '<GOOGLE_CLOUD_PROJECT_ID>',
    location: '<GOOGLE_CLOUD_REGION>',
    adcCredentialsString: '<GOOGLE_APPLICATION_CREDENTIALS_STRING>',
    voice: 'Aoede',
    instructions: 'You are a friendly assistant.',
    transcribeAgent: true,
    transcribeUser: true,
  }));

import "github.com/AgoraIO/agora-agents-go/v2/agentkit/vendors"

// client is your configured Agora client
agent := agentkit.NewAgent(client).WithMllm(
    vendors.NewVertexAI(vendors.VertexAIOptions{
        Model:               "gemini-live-2.5-flash-native-audio",
        ProjectID:           "<GOOGLE_CLOUD_PROJECT_ID>",
        Location:            "<GOOGLE_CLOUD_REGION>",
        ADCredentialsString: "<GOOGLE_APPLICATION_CREDENTIALS_STRING>",
        Voice:               "Aoede",
        Instructions:        "You are a friendly assistant.",
        TranscribeAgent:     agora.Bool(true),
        TranscribeUser:      agora.Bool(true),
    }),
)

Use the following mllm configuration in your request:

"mllm": {
  "enable": true,
  "messages": [
    {
      "role": "user",
      "content": "<HISTORY_CONTENT>"
    }
  ],
  "params": {
    "model": "gemini-live-2.5-flash-native-audio",
    "instructions": "<YOUR_SYSTEM_PROMPT>",
    "voice": "Aoede",
    "adc_credentials_string": "<GOOGLE_APPLICATION_CREDENTIALS_STRING>",
    "project_id": "<GOOGLE_CLOUD_PROJECT_ID>",
    "location": "<GOOGLE_CLOUD_REGION>",    
    "transcribe_agent": true,
    "transcribe_user": true
  },
  "turn_detection": {
    // see details below
  },
  "greeting_message": "Hi, how can I assist you today?",
  "input_modalities": [
    "audio"
  ],
  "output_modalities": [
    "audio"
  ],
  "vendor": "vertexai"
}

Turn detection

To set up turn detection, add a turn_detection block inside the mllm object when you Start a conversational AI agent.

Info

When mllm.turn_detection is defined, the top-level turn_detection object has no effect.

The following examples show the supported configurations for Google Gemini Live (Vertex AI).

Server VAD

"turn_detection": {
  "mode": "server_vad",
  "server_vad_config": {
    "prefix_padding_ms": 800,
    "silence_duration_ms": 640,
    "start_of_speech_sensitivity": "START_SENSITIVITY_HIGH",
    "end_of_speech_sensitivity": "END_SENSITIVITY_HIGH"
  }
}

Agora VAD

"turn_detection": {
  "mode": "agora_vad",
  "agora_vad_config": {
    "interrupt_duration_ms": 160,
    "prefix_padding_ms": 800,
    "silence_duration_ms": 640,
    "threshold": 0.5
  }
}

Key parameters

mllmrequired

enableboolean

optional

Enables the MLLM module. Replaces the deprecated advanced_features.enable_mllm.

messagesarray[object]

optional

An array of conversation history items passed to the model as context. Each item represents a single message in the conversation history.

rolestring

required

The role of the message author. For example, user.

contentstring

required

The content of the message.

paramsobject

required

Main configuration object for the Gemini Live model.

modelstring

required

The Gemini Live model identifier. See supported models.

instructionsstring

optional

System instructions that define the agent’s behavior or tone.

voicestring

optional

The voice identifier for audio output. For example, Aoede, Puck, Charon, Kore, Fenrir, Leda, Orus, or Zephyr.

adc_credentials_stringstring

required

Base64-encoded Google Cloud Application Default Credentials (ADC).

project_idstring

required

Your Google Cloud project ID for Vertex AI access.

locationstring

required

The Google Cloud region hosting the Gemini Live model. Check the Google Cloud documentation for the full list of available regions.

transcribe_agentboolean

optional

Whether to transcribe the agent’s speech in real time.

transcribe_userboolean

optional

Whether to transcribe the user’s speech in real time.

turn_detectionobject

optional

Turn detection configuration for the MLLM module. For a full list of turn_detection parameters, see mllm.turn_detection.

modestring

optional

Possible values: agora_vad
server_vad

agora_vad: Agora VAD-based detection.
server_vad: Vendor-side VAD-based detection.

agora_vad_configobject

optional

Configuration for Agora VAD-based turn detection. Applicable when mode is agora_vad.

interrupt_duration_msinteger

optional

Minimum duration of speech in milliseconds required to trigger an interruption.

prefix_padding_msinteger

optional

Duration of audio in milliseconds to include before the detected speech start.

silence_duration_msinteger

optional

Duration of silence in milliseconds required to determine end of speech.

thresholdnumber

optional

VAD sensitivity threshold. A higher value reduces false positives.

server_vad_configobject

optional

Configuration for vendor-side VAD-based turn detection. Applicable when mode is server_vad. Parameters are passed through to the vendor.

prefix_padding_msinteger

optional

Duration of audio in milliseconds to include before the detected speech start.

silence_duration_msinteger

optional

Duration of silence in milliseconds required to determine end of speech.

start_of_speech_sensitivitystring

optional

Possible values: START_SENSITIVITY_HIGH
START_SENSITIVITY_LOW

Sensitivity for start of speech detection.

end_of_speech_sensitivitystring

optional

Possible values: END_SENSITIVITY_HIGH
END_SENSITIVITY_LOW

Sensitivity for end of speech detection.

input_modalitiesarray[string]

optional

Default value: ["audio"]

Input modalities for the MLLM.

["audio"]: Audio-only input
["audio", "text"]: Accept both audio and text input

output_modalitiesarray[string]

optional

Default value: ["audio"]

Output modalities for the MLLM.

["audio"]: Audio-only response
["text", "audio"]: Combined text and audio output

greeting_messagestring

optional

The message the agent speaks when a user joins the channel.

vendorstring

required

The MLLM provider identifier. Set to "vertexai" to use Google Gemini Live with Vertex AI.

For comprehensive API reference, real-time capabilities, and detailed parameter descriptions, see the Google Gemini Live API.

Google Gemini Live (Vertex AI)

Sample configuration

Turn detection

Key parameters

Contact Us

Contact Us