# Direct LLM audio output (/en/ai/build/custom-model-integration/audio-output)
> For AI agents: see the complete documentation index at [llms.txt](/llms.txt).
In addition to text output, the Agora Conversational AI Engine can deliver responses in audio format, allowing for more natural and immersive user interactions. This page describes how to configure the agent for audio output and modify the interface to support features such as context management, subtitle alignment, and agent broadcasting.
## Prerequisites [#prerequisites]
* A reference implementation of a conversational AI agent that includes the [basic logic](../../get-started/quickstart) for interacting with an AI agent.
* If you plan to implement subtitles for audio output, read the [Display live subtitles](../transcripts) documentation and complete the required configuration.
## Implement audio output [#implement-audio-output]
Take the following steps to set up and configure the audio output mode.
### Select the output mode [#select-the-output-mode]
To configure the output mode, set the `llm.output_modalities` field when you [Start a conversational AI agent](/en/api-reference/api-ref/conversational-ai/join), as follows:
* `["audio"]`: Sets the output to audio-only mode. In this configuration, you don't need to set up a text-to-speech (TTS) module. The agent directly plays the audio returned by the custom LLM. This page focuses on how to use this audio-only mode.
* `["text", "audio"]`: Sets the output to both text and audio modes. In this configuration, two types of audio are returned: one generated by the TTS module and one provided by the custom LLM.
### Modify the LLM interface [#modify-the-llm-interface]
To use the Agora Conversational AI Engine with the [OpenAI Chat Completions API](https://platform.openai.com/docs/api-reference/chat), ensure that your LLM service is compatible with the expected request and response formats. Agora uses an extended API that supports multiple response types, including text, audio, subtitles, and verbatim subtitle timestamps. It also introduces an additional `words` field for real-time subtitle alignment.
To adapt your LLM service, refer to [Custom LLM](custom-llm) for guidance on transforming your API to meet these requirements.
### Request format [#request-format]
Compared with `text` requests, `audio` requests include two additional optional fields:
* `modalities`: Specifies the output mode.
* `audio`: Specifies the output timbre (`voice`) and `format`.
Include these fields in the `llm.params` object of the [Start a conversational AI agent](/en/api-reference/api-ref/conversational-ai/join) request.
The following example shows the format of an audio request:
```json
{
"model": "gpt-4o-audio-preview",
"modalities": ["audio"],
"audio": { "voice": "alloy", "format": "wav" },
"messages": [
{
"role": "user",
"content": "Is a golden retriever a good family dog?"
}
]
}
```
### Response format [#response-format]
The `audio` response includes three types of content. You can send each type to the agent independently for processing:
| Audio type | Description | Source | Agent processing |
| :-------------------------------------- | :--------------------------------------------------- | :-------------------------------------------------------------------------------------- | :--------------------------------------------- |
| Audio data
`data` | Base64-encoded PCM byte stream array | LLM generation with audio processing capabilities, or a custom audio processing service | Plays the audio directly |
| Transcription content
`transcript` | The complete text content corresponding to the audio | LLM generation | Stores the text in short-term memory (context) |
| Verbatim subtitles
`words` | Subtitle content with word-by-word timestamps | Supports LLM generation with verbatim output | Processes into verbatim real-time subtitles |
The specific data structure of a streaming response is as follows:
```json
{"choices":[{"index":0,"delta":{"role":"assistant","audio":{"data": ""}},"logprobs":null,"finish_reason":null}]}
// Audio data
{"choices":[{"index":0,"delta":{"audio":{"data": "base64 encoded pcm data"}},"logprobs":null,"finish_reason":null}]}
// Transcribe subtitles
{"choices":[{"index":0,"delta":{"audio":{"transcript": "Hello world!", "words":[{"text":"Hello", "start_ts":100, "end_ts":140, "duration":40}]}},"logprobs":null,"finish_reason":null}]}
// Word-by-word subtitles
{"choices":[{"index":0,"delta":{"audio":{"words":[{"text":"world", "start_ts":150, "end_ts":190, "duration":40}, {"text":"!", "start_ts":190, "end_ts":200, "duration":10}]}},"logprobs":null,"finish_reason":null}]}
....
{"choices":[{"index":0,"delta":{},"logprobs":null,"finish_reason":"stop"}]}
```
The conversational AI agent summarizes different types of responses in the following format:
```json
{
"choices": [
{
"delta": {
"audio": {
"data": "base64 encoded pcm data",
"transcript": "Hello world!",
"words": [
{
"text": "Hello",
"start_ts": 100,
"end_ts": 140,
"duration": 40
},
{
"text": "world",
"start_ts": 150,
"end_ts": 190,
"duration": 40
},
{
"text": "!",
"start_ts": 190,
"end_ts": 200,
"duration": 10
}
]
}
}
}
]
}
```
The `audio` object contains the following fields:
Audio data as a Base64-encoded PCM byte stream.
Subtitle content corresponding to the audio.
An array of word-level subtitle objects. The LLM must support word-level output.
The spoken word.
Start time in milliseconds relative to the beginning of the PCM audio data.
End time in milliseconds relative to the beginning of the PCM audio data.
Duration in milliseconds that the word is played.
Depending on your application use-case, configure your custom large model to selectively process and return the relevant fields:
* **Audio-only output**: Only the `data` field is required.
* **Subtitle-only output**: Only the `transcript` field is required. The agent displays the subtitle but does not play it using the TTS module.
* **Audio with verbatim subtitles**: The `data` and words fields are required. Each item in the words array must include the `text`, `start_ts`, `end_ts`, and `duration` fields.
## Advanced features [#advanced-features]
Conversational AI Engine supports the following advanced features:
### Context management [#context-management]
When the response contains the `audio.transcript` field, the agent automatically stores the subtitle content in its context manager for use in subsequent interactions. If the `audio.transcript` field is not included, the content is not stored.
To ensure the agent retains the audio modality output in short-term memory, include the `audio.transcript` field in the response.
### Subtitle alignment [#subtitle-alignment]
When you [Display live subtitles](../transcripts), the Conversational AI Engine can use the `audio.words` field to segment the audio. The engine aligns subtitles based on the `start_ts`, `end_ts`, and `duration` fields within `audio.words`.
To enable the agent to segment the audio based on subtitle content during playback, make sure that the LLM sends both the `audio.data` and `audio.words` fields in the response.
### Agent message broadcasts [#agent-message-broadcasts]
If the agent isn't configured with a TTS module but has `output_modalities` set to `["audio"]`, you can enable it to broadcast custom messages by adapting your LLM as follows:
In the `messages` list received by the agent and passed to the LLM, the model should handle the last message based on its `role`:
* `assistant`: The model treats the message as a directive that doesn't require reasoning. It converts the message to audio and returns it to the agent. The agent plays the audio directly.
* `user`: The model treats the message as a prompt that requires reasoning. The agent decides whether to broadcast the message based on whether the model’s response includes audio. This corresponds to a typical user–agent conversation.
Info
The following agent broadcasts also adopt this protocol:
* When the agent announces a greeting `greeting_message` or a processing failure prompt `failure_message`.
* When you [Broadcast a custom message using the TTS module](/en/api-reference/api-ref/conversational-ai/speak).
## Reference [#reference]
This section contains content that completes the information on this page, or points you to documentation that explains other aspects to this product.
### Sample project [#sample-project]
Agora provides an open-source [Conversational AI sample server project](https://github.com/AgoraIO-Community/Conversational-AI-Server-Sample) for your reference. Download the project or view the source code for a complete example.