Google Gemini Live (Vertex AI)
Updated
Integrate Google Gemini Live with the Conversational AI Engine using Google Cloud Vertex AI.
Google Gemini Live provides multimodal large language model capabilities with real-time audio processing, enabling natural voice conversations without separate ASR/TTS components. This page covers integration using Vertex AI, authenticated with Google Cloud Application Default Credentials (ADC). Enabling MLLM automatically disables ASR, LLM, and TTS since the MLLM handles end-to-end voice processing directly.
Sample configuration
The following examples show how to configure Google Gemini Live (Vertex AI) MLLM when starting a conversational AI agent.
from agora_agent import Agent
from agora_agent.agentkit.vendors import VertexAI
# client is your configured Agora client
agent = (
Agent(client)
.with_mllm(VertexAI(
model='gemini-live-2.5-flash-native-audio',
project_id='<GOOGLE_CLOUD_PROJECT_ID>',
location='<GOOGLE_CLOUD_REGION>',
adc_credentials_string='<GOOGLE_APPLICATION_CREDENTIALS_STRING>',
voice='Aoede',
instructions='You are a friendly assistant.',
transcribe_agent=True,
transcribe_user=True,
))
)import { Agent, VertexAI } from 'agora-agents';
// client is your configured Agora client
const agent = new Agent({ client })
.withMllm(new VertexAI({
model: 'gemini-live-2.5-flash-native-audio',
projectId: '<GOOGLE_CLOUD_PROJECT_ID>',
location: '<GOOGLE_CLOUD_REGION>',
adcCredentialsString: '<GOOGLE_APPLICATION_CREDENTIALS_STRING>',
voice: 'Aoede',
instructions: 'You are a friendly assistant.',
transcribeAgent: true,
transcribeUser: true,
}));import (
Agora "github.com/AgoraIO/agora-agents-go/v2"
"github.com/AgoraIO/agora-agents-go/v2/agentkit"
"github.com/AgoraIO/agora-agents-go/v2/agentkit/vendors"
)
// client is your configured Agora client
agent := agentkit.NewAgent(client).WithMllm(
vendors.NewVertexAI(vendors.VertexAIOptions{
Model: "gemini-live-2.5-flash-native-audio",
ProjectID: "<GOOGLE_CLOUD_PROJECT_ID>",
Location: "<GOOGLE_CLOUD_REGION>",
ADCredentialsString: "<GOOGLE_APPLICATION_CREDENTIALS_STRING>",
Voice: "Aoede",
Instructions: "You are a friendly assistant.",
TranscribeAgent: Agora.Bool(true),
TranscribeUser: Agora.Bool(true),
}),
)Use the following mllm configuration in your request:
"mllm": {
"enable": true,
"messages": [
{
"role": "user",
"content": "<HISTORY_CONTENT>"
}
],
"params": {
"model": "gemini-live-2.5-flash-native-audio",
"instructions": "<YOUR_SYSTEM_PROMPT>",
"voice": "Aoede",
"adc_credentials_string": "<GOOGLE_APPLICATION_CREDENTIALS_STRING>",
"project_id": "<GOOGLE_CLOUD_PROJECT_ID>",
"location": "<GOOGLE_CLOUD_REGION>",
"transcribe_agent": true,
"transcribe_user": true
},
"turn_detection": {
// see details below
},
"greeting_message": "Hi, how can I assist you today?",
"input_modalities": [
"audio"
],
"output_modalities": [
"audio"
],
"vendor": "vertexai"
}Turn detection
To set up turn detection, add a turn_detection block inside the mllm object when you Start a conversational AI agent.
Info
When mllm.turn_detection is defined, the top-level turn_detection object has no effect.
The following examples show the supported configurations for Google Gemini Live (Vertex AI).
-
Server VAD
"turn_detection": { "mode": "server_vad", "server_vad_config": { "prefix_padding_ms": 800, "silence_duration_ms": 640, "start_of_speech_sensitivity": "START_SENSITIVITY_HIGH", "end_of_speech_sensitivity": "END_SENSITIVITY_HIGH" } } -
Agora VAD
"turn_detection": { "mode": "agora_vad", "agora_vad_config": { "interrupt_duration_ms": 160, "prefix_padding_ms": 800, "silence_duration_ms": 640, "threshold": 0.5 } }
Key parameters
| Parameter | Description |
|---|---|
enableboolean | Enables the MLLM module. Replaces the deprecated |
messagesarray[object] | An array of conversation history items passed to the model as context. Each item represents a single message in the conversation history.
|
paramsobject | Main configuration object for the Gemini Live model.
|
turn_detectionobject | Turn detection configuration for the MLLM module. For a full list of
|
input_modalitiesarray[string] |
Input modalities for the MLLM.
|
output_modalitiesarray[string] |
Output modalities for the MLLM.
|
greeting_messagestring | The message the agent speaks when a user joins the channel. |
vendorstring | The MLLM provider identifier. Set to |
For comprehensive API reference, real-time capabilities, and detailed parameter descriptions, see the Google Gemini Live API.
