Talking while waiting

In conversational AI, delays in LLM responses may cause users to wonder if the agent is still processing, repeat their question, or disengage entirely. Filler words address this by playing short phrases while the agent waits for the LLM to generate a response. This keeps the conversation flowing, reduces user anxiety, and creates a more human-like interaction.

Common use cases for filler words include:

• MCP tool calls: When the agent invokes tools through an MCP server, response times can increase significantly. Filler words bridge this gap while the agent waits for tool results.
• Complex queries: Queries that require more LLM processing time benefit from a brief acknowledgment to signal that the agent is working on a response.
• Customer service scenarios: In support interactions, filler words such as "Let me look into that for you" reassure users that their request is being handled.

Understand the tech

You configure filler words as part of the agent's configuration. Their behavior depends on the content mode you choose and how the engine times playback against the main LLM response.

Filler word modes

Specify where filler word content comes from:

• Static (default): Selects a phrase from a predefined list. This is also the behavior when no mode is set.
• Generated: Dynamically generates a filler phrase based on the current conversation. If the generated result isn't ready in time, generation fails, or the content is invalid, the agent falls back to a phrase from the static list.

Filler word selection and playback

For qualifying speech input or a custom instruction, the engine starts the main LLM request and dynamic filler word generation at the same time:

1. If the main LLM produces playable content before the response wait threshold elapses, the agent plays the main reply directly, without playing a filler word.
2. If the main LLM is still waiting when the threshold elapses and a dynamic filler word is already ready, the agent plays the dynamic filler word.
3. If the dynamic filler word isn't ready, generation failed, or the content is invalid, the agent plays a static fallback filler word instead.
4. Playing a filler word doesn't cancel, discard, or restart the main LLM request. The agent continues into the main reply once it's ready.

The agent plays at most one filler word per turn. A dynamic filler word that arrives late doesn't replace content that's already been selected, and it doesn't trigger a second playback. Prefetch turns don't trigger filler words.

When you use static mode, or don't set a mode, the engine doesn't start dynamic filler word generation at all. Once the main LLM's wait time reaches the response wait threshold, the agent selects and plays a static phrase directly.

Prerequisites

• Use the Agora CLI to create or select an Agora project, and verify project readiness:

  agora login
  agora project create conv-ai-tutorial
  agora project use conv-ai-tutorial
  agora project doctor

• Implemented the basic logic for interacting with a conversational AI agent by following the quickstart.

Implementation

Configure static or generated filler words, depending on the mode you want to use.

Configure static filler words

The following example configures filler words to play a random phrase when the LLM takes longer than 1.5 seconds to respond. Add the following filler_words object to properties in your Start a conversational AI agent request body:

"filler_words": {
  "enable": true,
  "trigger": {
    "mode": "fixed_time",
    "fixed_time_config": {
      "response_wait_ms": 1500
    }
  },
  "content": {
    "mode": "static",
    "static_config": {
      "phrases": [
        "Let me look into that.",
        "One moment, please.",
        "Sure, give me a second.",
        "Hmmm, let me check."
      ],
      "selection_rule": "shuffle"
    }
  }
}

Field reference

The static phrase list supports 1 to 100 non-empty strings. A phrase containing only Latin characters supports up to 50 space-separated words; a phrase containing any non-Latin character supports up to 50 Unicode code points.

Choose a response wait threshold based on your use case:

• Lower values (500-1000 ms): Better for fast-paced interactions where silence is more noticeable.
• Higher values (1500-3000 ms): Suitable for scenarios where users expect some processing time, such as complex queries or data lookups.

Configure generated filler words

Set the content mode to generated, and keep a static fallback configured. Add the following filler_words object to properties in your Start a conversational AI agent request body:

"filler_words": {
  "enable": true,
  "trigger": {
    "mode": "fixed_time",
    "fixed_time_config": {
      "response_wait_ms": 1500
    }
  },
  "content": {
    "mode": "generated",
    "static_config": {
      "phrases": [
        "Let me look into that.",
        "I'm on it."
      ],
      "selection_rule": "shuffle"
    },
    "generated_config": {
      "prompt": "Generate a short, natural filler phrase indicating you are processing the request. Do not answer the question directly.",
      "fallback_strategy": "static"
    }
  }
}

Field reference

Filler-word generation is Agora-hosted, so there's no model provider field to configure. Generation runs in parallel with the main LLM request, and falls back to the static phrase list if it isn't ready by the response wait threshold.

Best practices

Keep the following tips in mind when configuring filler words:

• Keep phrases short and natural: Use brief, conversational phrases that sound like something a person would say.
• Match phrasing to your use case: For customer support agents, use reassuring phrases like "Let me look into that for you." For casual assistants, use informal phrases like "Hmm, one sec."
• Tune the trigger threshold: Start with a response wait threshold of 1500 ms and adjust based on your LLM's typical response time. If your agent frequently invokes tools, consider a lower threshold to cover longer processing times.
• Provide enough variety: Include at least 4-6 phrases to avoid sounding repetitive. Use shuffle selection to maximize variety across conversation turns.
• Keep a solid static fallback: When using generated mode, still provide a well-chosen static phrase list, since it plays whenever generation isn't ready in time.

References

• Start a conversational AI agent