OpenAI
Updated
Use OpenAI for text-to-speech (TTS) in a Conversational AI Platform agent.
Preview documentation
OpenAI provides natural-sounding text-to-speech with customizable voice instructions and multiple voice options.
Sample configuration
The following example shows how to configure OpenAI as the agent's TTS when you create or update an agent. For a complete example, see Create an agent and start a session.
OpenAI supports managed mode and BYOK. In managed mode, Agora provides access to OpenAI, and you don't need a OpenAI account. For BYOK, store your OpenAI API key in a secret set, and refer to it from the agent. For details, see Use managed mode and Use your own provider keys.
from agora_agent import Agent, OpenAITTS, secret_ref
# client is your configured Agora client
agent = (
Agent(client=client)
.with_stt(...) # configure your ASR vendor
.with_llm(...) # configure your LLM vendor
.with_tts(
OpenAITTS(
api_key=secret_ref("support_prod", "openai"), # omit to use managed mode
base_url="https://api.openai.com/v1/audio/speech", # omit to use managed mode
model="tts-1",
voice="coral",
instructions="Please use standard American English, natural tone, moderate pace, and steady intonation",
speed=1,
)
)
)import { Agent, OpenAITTS, secretRef } from 'agora-agents';
// client is your configured Agora client
const agent = new Agent({ client })
.withStt(/* configure your ASR vendor */)
.withLlm(/* configure your LLM vendor */)
.withTts(
new OpenAITTS({
apiKey: secretRef('support_prod', 'openai'), // omit to use managed mode
baseUrl: 'https://api.openai.com/v1/audio/speech', // omit to use managed mode
model: 'tts-1',
voice: 'coral',
instructions: 'Please use standard American English, natural tone, moderate pace, and steady intonation',
speed: 1,
}),
);speed := 1.0
// client is your configured Agora client
agent := agentkit.NewAgent(client).
WithStt( /* configure your ASR vendor */ ).
WithLlm( /* configure your LLM vendor */ ).
WithTts(vendors.NewOpenAITTS(vendors.OpenAITTSOptions{
APIKey: agentkit.SecretRef("support_prod", "openai"), // omit to use managed mode
BaseURL: "https://api.openai.com/v1/audio/speech", // omit to use managed mode
Model: "tts-1",
Voice: "coral",
Instructions: "Please use standard American English, natural tone, moderate pace, and steady intonation",
Speed: &speed,
}))-
Managed mode
Set
credential.modetomanagedin thettsblock. Agora provides access to OpenAI, and you don't need a OpenAI account."tts": { "credential": { "mode": "managed" }, "vendor": "openai", "params": { "model": "tts-1", "voice": "coral", "instructions": "Please use standard American English, natural tone, moderate pace, and steady intonation", "speed": 1 } } -
BYOK
Set
credential.modetobyokandcredential.fields.api_keyto a reference to your OpenAI API key in a secret set. In the following example, the key is stored asopenaiin thesupport_prodset:"tts": { "credential": { "mode": "byok", "fields": { "api_key": "$secrets.support_prod.openai" } }, "vendor": "openai", "url": "https://api.openai.com/v1/audio/speech", "params": { "model": "gpt-4o-mini-tts", "voice": "coral", "instructions": "Please use standard American English, natural tone, moderate pace, and steady intonation", "speed": 1 } }
Key parameters
The tts block takes the following fields. In the SDKs, the OpenAI TTS class sets them from its options.
vendorstringSet to openai.
credentialobjectHow the agent authenticates with OpenAI. Set mode to managed, or to byok with fields.api_key set to a $secrets.<set>.<key> reference to your OpenAI API key. Get your API key from the OpenAI Console.
urlstringThe full endpoint URL for the OpenAI speech API, for example https://api.openai.com/v1/audio/speech. Not needed in managed mode. See How to use data residency.
paramsobjectmodelstringIdentifier of the model to be used. For managed mode, use tts-1.
voicestringThe voice identifier for speech synthesis.
instructionsstringCustom instructions for voice style, accent, pace, and tone. Helps fine-tune the speech characteristics.
speednumber- Default value
- 1.0
Speaking rate multiplier. Values between 0.25 and 4.0, where 1.0 is normal speed.
Info
OpenAI TTS models do not support changing the sample rate. The audio output is fixed at 24,000 Hz.
Caution
The parameters listed on this page are validated for use with the Conversational AI Platform. Required parameters must be provided as documented. Any additional parameters in params are passed through directly to OpenAI without validation. For a full list of supported options, refer to the OpenAI TTS documentation.
This integration is fully supported for use with the Conversational AI Platform. While it has completed functional validation, it is newer to the platform, and additional provider-specific edge cases may be identified as usage scales across a broader range of applications and workloads.