Generic TTS
Updated
Use a custom TTS service that implements the OpenAI TTS protocol in a Conversational AI Platform agent.
The generic_http TTS vendor lets you integrate a custom TTS service that implements the OpenAI TTS protocol into a Conversational AI Platform agent. For a walkthrough of preparing and verifying a compatible TTS service, see Connect your own TTS service.
Sample configuration
The following example shows how to configure Generic TTS as the agent's TTS when you create or update an agent. For a complete example, see Create an agent and start a session.
Generic TTS uses your own TTS service. Store the API key for your service in a secret set, and refer to it from the agent. For details, see Use your own provider keys.
from agora_agent import Agent, GenericTTS, secret_ref
# client is your configured Agora client
agent = (
Agent(client=client)
.with_stt(...) # configure your ASR vendor
.with_llm(...) # configure your LLM vendor
.with_tts(
GenericTTS(
url="https://your-tts-service.example.com/v1/audio/speech",
api_key=secret_ref("support_prod", "custom_tts"),
model="gpt-4o-mini-tts",
voice="alloy",
speed=1.0,
sample_rate=16000,
response_format="pcm",
additional_params={"instructions": "calm", "enable_request_id": False},
)
)
)import { Agent, GenericTTS, secretRef } from 'agora-agents';
// client is your configured Agora client
const agent = new Agent({ client })
.withStt(/* configure your ASR vendor */)
.withLlm(/* configure your LLM vendor */)
.withTts(
new GenericTTS({
url: 'https://your-tts-service.example.com/v1/audio/speech',
apiKey: secretRef('support_prod', 'custom_tts'),
model: 'gpt-4o-mini-tts',
voice: 'alloy',
speed: 1.0,
sampleRate: 16000,
responseFormat: 'pcm',
additionalParams: { instructions: "calm", enable_request_id: false },
}),
);// client is your configured Agora client
speed, sampleRate := 1.0, vendors.SampleRate16kHz
agent := agentkit.NewAgent(client).
WithStt( /* configure your ASR vendor */ ).
WithLlm( /* configure your LLM vendor */ ).
WithTts(vendors.NewGenericTTS(vendors.GenericTTSOptions{
URL: "https://your-tts-service.example.com/v1/audio/speech",
APIKey: agentkit.SecretRef("support_prod", "custom_tts"),
Model: "gpt-4o-mini-tts",
Voice: "alloy",
Speed: &speed,
SampleRate: &sampleRate,
ResponseFormat: "pcm",
AdditionalParams: map[string]interface{}{"instructions": "calm", "enable_request_id": false},
}))Set url to your service endpoint. Set credential.mode to byok and credential.fields.api_key to a reference to the API key for your service. In the following example, the key is stored as custom_tts in the support_prod set:
"tts": {
"credential": {
"mode": "byok",
"fields": {
"api_key": "$secrets.support_prod.custom_tts"
}
},
"vendor": "generic_http",
"url": "https://your-tts-service.example.com/v1/audio/speech",
"params": {
"model": "gpt-4o-mini-tts",
"voice": "alloy",
"speed": 1.0,
"sample_rate": 16000,
"response_format": "pcm",
"instructions": "calm",
"enable_request_id": false
}
}To send other custom headers to your service, add a headers object to the block. Headers are write-only and are never returned.
Key parameters
The tts block takes the following fields. In the SDKs, GenericTTS sets them from its options.
vendorstringSet to generic_http.
credentialobjectSet mode to byok and fields.api_key to a $secrets.<set>.<key> reference to the API key for your TTS service.
urlstringThe OpenAI TTS protocol compatible service endpoint.
headersobjectCustom request headers to include in requests to the TTS service.
paramsobjectmodelstringThe name of the TTS model.
voicestringThe voice name. Omit this parameter if your TTS service does not support multiple voices.
speednumberThe speech rate.
sample_rateinteger- Default value
- 16000
The sample rate, in Hz, of the output audio. If your TTS service does not support multiple sample rates, make sure the sample rate of the returned audio matches this value.
response_formatstring- Default value
- pcm
The output audio format. Currently only pcm is supported.
instructionsstringInstructions for voice style, emotion, or other playback directives.
enable_request_idboolean- Default value
- false
Whether to include request_id, request_seq_id, and request_end in the TTS request body.
Caution
Your TTS service must implement the OpenAI TTS protocol. The parameters listed on this page are validated for use with the Conversational AI Platform. Any additional fields in params are passed through directly to your TTS service without validation.
This integration is fully supported for use with the Conversational AI Platform. While it has completed functional validation, it is newer to the platform, and additional provider-specific edge cases may be identified as usage scales across a broader range of applications and workloads.