OpenAI
Updated
Use OpenAI for speech recognition (ASR) in a Conversational AI Platform agent.
Preview documentation
OpenAI provides real-time speech-to-text with low latency and reliable performance, making it ideal for conversational AI applications.
Sample configuration
The following example shows how to configure OpenAI ASR as the agent's ASR when you create or update an agent. For a complete example, see Create an agent and start a session.
OpenAI ASR supports BYOK only. It isn't available in managed mode. Store your OpenAI API key in a secret set, and refer to it from the agent. For details, see Use your own provider keys.
from agora_agent import Agent, OpenAISTT, secret_ref
# client is your configured Agora client
agent = (
Agent(client=client)
.with_stt(
OpenAISTT(
api_key=secret_ref("support_prod", "openai"),
model="gpt-4o-mini-transcribe",
prompt="Please transcribe the following audio into text. Output in English.",
language="en",
)
)
.with_llm(...) # configure your LLM vendor
.with_tts(...) # configure your TTS vendor
)import { Agent, OpenAISTT, secretRef } from 'agora-agents';
// client is your configured Agora client
const agent = new Agent({ client })
.withStt(
new OpenAISTT({
apiKey: secretRef('support_prod', 'openai'),
model: 'gpt-4o-mini-transcribe',
prompt: 'Please transcribe the following audio into text. Output in English.',
language: 'en',
}),
)
.withLlm(/* configure your LLM vendor */)
.withTts(/* configure your TTS vendor */);// client is your configured Agora client
agent := agentkit.NewAgent(client).
WithStt(vendors.NewOpenAISTT(vendors.OpenAISTTOptions{
APIKey: agentkit.SecretRef("support_prod", "openai"),
Model: "gpt-4o-mini-transcribe",
Prompt: "Please transcribe the following audio into text. Output in English.",
Language: "en",
})).
WithLlm( /* configure your LLM vendor */ ).
WithTts( /* configure your TTS vendor */ )Set credential.mode to byok and credential.fields.api_key to a reference to your OpenAI API key in a secret set. In the following example, the key is stored as openai in the support_prod set:
"asr": {
"credential": {
"mode": "byok",
"fields": {
"api_key": "$secrets.support_prod.openai"
}
},
"vendor": "openai",
"language": "en-US",
"params": {
"input_audio_transcription": {
"model": "gpt-4o-mini-transcribe",
"prompt": "Please transcribe the following audio into text. Output in English.",
"language": "en"
}
}
}Key parameters
The asr block takes the following fields. In the SDKs, OpenAISTT sets the transcription model, prompt, and language in params.input_audio_transcription, and passes any other OpenAI ASR parameters from its additional parameters option to params.
vendorstringSet to openai.
credentialobjectHow the agent authenticates with OpenAI. Set mode to byok and fields.api_key to a $secrets.<set>.<key> reference to your OpenAI API key.
languagestringThe language of the user's speech, for example, en-US.
paramsobjectinput_audio_transcriptionobjectThe configuration object for audio transcription. Use this object to specify the model, prompt, and language for the transcription task.
modelstringThe OpenAI ASR model to use for transcription. For example, gpt-4o-mini-transcribe.
promptstringA prompt that guides the transcription process. Use this parameter to provide context or instructions for how the audio should be transcribed.
languagestringThe language code to use for transcription. For example, use en for English.
Caution
The parameters listed on this page are validated for use with the Conversational AI Platform. Required parameters must be provided as documented. Any additional parameters in params are passed through directly to OpenAI without validation. For a full list of supported options, refer to the OpenAI documentation.
This integration is fully supported for use with the Conversational AI Platform. While it has completed functional validation, it is newer to the platform, and additional provider-specific edge cases may be identified as usage scales across a broader range of applications and workloads.