Gemini
Updated
Use Gemini for speech recognition (ASR) in a Conversational AI Platform agent.
Preview documentation
Google Gemini provides real-time streaming transcription. Use it as the ASR model in a cascading pipeline with any supported LLM and TTS vendor.
Sample configuration
The following example shows how to configure Gemini as the agent's ASR when you create or update an agent. For a complete example, see Create an agent and start a session.
Gemini supports BYOK only. It isn't available in managed mode. Store your Gemini API key in a secret set, and refer to it from the agent. For details, see Use your own provider keys.
from agora_agent import Agent, GeminiSTT, secret_ref
# client is your configured Agora client
agent = (
Agent(client=client)
.with_stt(
GeminiSTT(
api_key=secret_ref("support_prod", "gemini"),
model="gemini-3.5-transcribe-live",
language="en-US",
sample_rate=16000,
word_timestamp=True,
)
)
.with_llm(...) # configure your LLM vendor
.with_tts(...) # configure your TTS vendor
)import { Agent, GeminiSTT, secretRef } from 'agora-agents';
// client is your configured Agora client
const agent = new Agent({ client })
.withStt(
new GeminiSTT({
apiKey: secretRef('support_prod', 'gemini'),
model: 'gemini-3.5-transcribe-live',
language: 'en-US',
sampleRate: 16000,
wordTimestamp: true,
}),
)
.withLlm(/* configure your LLM vendor */)
.withTts(/* configure your TTS vendor */);wordTimestamp := true
// client is your configured Agora client
agent := agentkit.NewAgent(client).
WithStt(vendors.NewGeminiSTT(vendors.GeminiSTTOptions{
APIKey: agentkit.SecretRef("support_prod", "gemini"),
Model: "gemini-3.5-transcribe-live",
Language: "en-US",
SampleRate: 16000,
WordTimestamp: &wordTimestamp,
})).
WithLlm( /* configure your LLM vendor */ ).
WithTts( /* configure your TTS vendor */ )Set credential.mode to byok and credential.fields.api_key to a reference to your Gemini API key in a secret set. In the following example, the key is stored as gemini in the support_prod set:
"asr": {
"credential": {
"mode": "byok",
"fields": {
"api_key": "$secrets.support_prod.gemini"
}
},
"vendor": "gemini",
"model": "gemini-3.5-transcribe-live",
"language": "en-US",
"params": {
"sample_rate": 16000,
"word_timestamp": true
}
}Key parameters
The asr block takes the following fields. In the SDKs, GeminiSTT sets them from its options, and passes any other Gemini parameters from its additional parameters option to params.
vendorstringSet to gemini.
credentialobjectHow the agent authenticates with Gemini. Set mode to byok and fields.api_key to a $secrets.<set>.<key> reference to your Google Gemini API key. You can generate an API key in Google AI Studio.
modelstringThe Gemini transcription model identifier. Set to gemini-3.5-transcribe-live to use Gemini 3.5 Transcribe Live.
languagestringThe BCP-47 language tag identifying the primary language used for agent interaction, for example, en-US.
paramsobjectsample_rateintegerThe audio sample rate in Hz. For example, 16000.
word_timestampbooleanWhether to include word-level timestamps in the transcription results.
Caution
The parameters listed on this page are validated for use with the Conversational AI Platform. Required parameters must be provided as documented. Any additional parameters in params are passed through directly to Gemini without validation. For a full list of supported options, refer to the Google Gemini API documentation.