Generic TTS

Updated

Use a custom TTS service that implements the OpenAI TTS protocol in a Conversational AI Platform agent.

The generic_http TTS vendor lets you integrate a custom TTS service that implements the OpenAI TTS protocol into a Conversational AI Platform agent. For a walkthrough of preparing and verifying a compatible TTS service, see Connect your own TTS service.

Sample configuration

The following example shows how to configure Generic TTS as the agent's TTS when you create or update an agent. For a complete example, see Create an agent and start a session.

Generic TTS uses your own TTS service. Store the API key for your service in a secret set, and refer to it from the agent. For details, see Use your own provider keys.

from agora_agent import Agent, GenericTTS, secret_ref

# client is your configured Agora client
agent = (
    Agent(client=client)
    .with_stt(...)  # configure your ASR vendor
    .with_llm(...)  # configure your LLM vendor
    .with_tts(
        GenericTTS(
            url="https://your-tts-service.example.com/v1/audio/speech",
            api_key=secret_ref("support_prod", "custom_tts"),
            model="gpt-4o-mini-tts",
            voice="alloy",
            speed=1.0,
            sample_rate=16000,
            response_format="pcm",
            additional_params={"instructions": "calm", "enable_request_id": False},
        )
    )
)
import { Agent, GenericTTS, secretRef } from 'agora-agents';

// client is your configured Agora client
const agent = new Agent({ client })
  .withStt(/* configure your ASR vendor */)
  .withLlm(/* configure your LLM vendor */)
  .withTts(
    new GenericTTS({
      url: 'https://your-tts-service.example.com/v1/audio/speech',
      apiKey: secretRef('support_prod', 'custom_tts'),
      model: 'gpt-4o-mini-tts',
      voice: 'alloy',
      speed: 1.0,
      sampleRate: 16000,
      responseFormat: 'pcm',
      additionalParams: { instructions: "calm", enable_request_id: false },
    }),
  );
// client is your configured Agora client
speed, sampleRate := 1.0, vendors.SampleRate16kHz
agent := agentkit.NewAgent(client).
	WithStt( /* configure your ASR vendor */ ).
	WithLlm( /* configure your LLM vendor */ ).
	WithTts(vendors.NewGenericTTS(vendors.GenericTTSOptions{
		URL:              "https://your-tts-service.example.com/v1/audio/speech",
		APIKey:           agentkit.SecretRef("support_prod", "custom_tts"),
		Model:            "gpt-4o-mini-tts",
		Voice:            "alloy",
		Speed:            &speed,
		SampleRate:       &sampleRate,
		ResponseFormat:   "pcm",
		AdditionalParams: map[string]interface{}{"instructions": "calm", "enable_request_id": false},
	}))

Set url to your service endpoint. Set credential.mode to byok and credential.fields.api_key to a reference to the API key for your service. In the following example, the key is stored as custom_tts in the support_prod set:

"tts": {
  "credential": {
    "mode": "byok",
    "fields": {
      "api_key": "$secrets.support_prod.custom_tts"
    }
  },
  "vendor": "generic_http",
  "url": "https://your-tts-service.example.com/v1/audio/speech",
  "params": {
    "model": "gpt-4o-mini-tts",
    "voice": "alloy",
    "speed": 1.0,
    "sample_rate": 16000,
    "response_format": "pcm",
    "instructions": "calm",
    "enable_request_id": false
  }
}

To send other custom headers to your service, add a headers object to the block. Headers are write-only and are never returned.

Key parameters

The tts block takes the following fields. In the SDKs, GenericTTS sets them from its options.

tts
ParameterDescription
vendorstring
required

Set to generic_http.

credentialobject
required

Set mode to byok and fields.api_key to a $secrets.<set>.<key> reference to the API key for your TTS service.

urlstring
required

The OpenAI TTS protocol compatible service endpoint.

headersobject
optional

Custom request headers to include in requests to the TTS service.

paramsobject
required
  • Optional model?: string

    The name of the TTS model.

  • Optional voice?: string

    The voice name. Omit this parameter if your TTS service does not support multiple voices.

  • Optional speed?: number

    The speech rate.

  • Optional sample_rate?: integer
    Default value
    16000

    The sample rate, in Hz, of the output audio. If your TTS service does not support multiple sample rates, make sure the sample rate of the returned audio matches this value.

  • Optional response_format?: string
    Default value
    pcm

    The output audio format. Currently only pcm is supported.

  • Optional instructions?: string

    Instructions for voice style, emotion, or other playback directives.

  • Optional enable_request_id?: boolean
    Default value
    false

    Whether to include request_id, request_seq_id, and request_end in the TTS request body.

The table extends beyond the current width. Scroll horizontally to view more columns.

Caution

Your TTS service must implement the OpenAI TTS protocol. The parameters listed on this page are validated for use with the Conversational AI Platform. Any additional fields in params are passed through directly to your TTS service without validation.

This integration is fully supported for use with the Conversational AI Platform. While it has completed functional validation, it is newer to the platform, and additional provider-specific edge cases may be identified as usage scales across a broader range of applications and workloads.