OpenAI

Updated

Integrate OpenAI TTS into Conversational AI Engine.

OpenAI provides natural-sounding text-to-speech with customizable voice instructions and multiple voice options.

Info

This integration is fully supported for use with Conversational AI Engine. While it has completed functional validation, it is newer to the platform, and additional provider-specific edge cases may be identified as usage scales across a broader range of applications and workloads.

Sample configuration

The following examples show how to configure OpenAI TTS when starting a conversational AI agent.

from agora_agent import Agent
from agora_agent.agentkit.vendors import OpenAITTS

# client is your configured Agora client
agent = (
    Agent(client)
    .with_stt(...)  # configure your STT vendor
    .with_llm(...)  # configure your LLM vendor
    .with_tts(OpenAITTS(
        api_key='your-openai-key',  # omit api_key and base_url to use Agora-managed mode
        base_url='https://api.openai.com/v1',
        model='gpt-4o-mini-tts',
        voice='coral',
        instructions='Please use standard American English, natural tone, moderate pace, and steady intonation',
        speed=1,
    ))
)
import { Agent, OpenAITTS } from 'agora-agents';

// client is your configured Agora client
const agent = new Agent({ client })
  .withStt(/* configure your STT vendor */)
  .withLlm(/* configure your LLM vendor */)
  .withTts(new OpenAITTS({
    apiKey: 'your-openai-key',  // omit apiKey and baseUrl to use Agora-managed mode
    baseUrl: 'https://api.openai.com/v1',
    model: 'gpt-4o-mini-tts',
    voice: 'coral',
    instructions: 'Please use standard American English, natural tone, moderate pace, and steady intonation',
    speed: 1,
  }));
import "github.com/AgoraIO/agora-agents-go/v2/agentkit/vendors"

// client is your configured Agora client
agent := agentkit.NewAgent(client).WithStt(/* configure your STT vendor */).
  WithLlm(/* configure your LLM vendor */).
  WithTts(
    vendors.NewOpenAITTS(vendors.OpenAITTSOptions{
        APIKey:       "your-openai-key", // omit APIKey and BaseURL to use Agora-managed mode
        BaseURL:      "https://api.openai.com/v1",
        Model:        "gpt-4o-mini-tts",
        Voice:        "coral",
        Instructions: "Please use standard American English, natural tone, moderate pace, and steady intonation",
        Speed:        agora.Float64(1),
    }),
)
  • Managed mode

    To use OpenAI TTS with Agora-managed credentials, set credential_mode to "managed" in the tts block. When using managed mode, params.api_key is not required. You can still use the tts field to configure additional settings such as voice and instructions. For more information, see Use managed mode.

    "tts": {
      "credential_mode": "managed",
      "vendor": "openai",
      "params": {
        "base_url": "https://api.openai.com/v1",
        "model": "tts-1",
        "voice": "coral",
        "instructions": "Please use standard American English, natural tone, moderate pace, and steady intonation",
        "speed": 1
      }
    }
  • BYOK

    "tts": {
      "vendor": "openai",
      "params": {
        "base_url": "https://api.openai.com/v1",
        "api_key": "<your_openai_key>",
        "model": "gpt-4o-mini-tts",
        "voice": "coral",
        "instructions": "Please use standard American English, natural tone, moderate pace, and steady intonation",
        "speed": 1
      }
    }

Caution

The parameters listed on this page are validated for use with Conversational AI Engine. Required parameters must be provided as documented. Any additional parameters are passed through directly to the underlying vendor without validation. For advanced configuration options, available voices, and detailed parameter descriptions, see the OpenAI TTS documentation.

Key parameters

The following parameters are required when using your own API key (BYOK). When using managed mode, api_key is not required. For more information, see Use managed mode.

params
base_urlstring
required

The endpoint URL for the OpenAI TTS service. Required when using your own API key (BYOK). See How to use data residency.

api_keystring
optional

The API key used for authentication. Required when using your own API key (BYOK). Get your API key from the OpenAI Console.

modelstring
required

Identifier of the model to be used. Required when using your own API key (BYOK).

voicestring
required

The voice identifier for speech synthesis.

instructionsstring
optional

Custom instructions for voice style, accent, pace, and tone. Helps fine-tune the speech characteristics.

speednumber
optional
Default value
1.0

Speaking rate multiplier. Values between 0.25 and 4.0, where 1.0 is normal speed.

Info

OpenAI TTS models do not support changing the sample rate. The audio output is fixed at 24,000 Hz.