Interrupt the agent mid-response

When interacting with an agent, you may need to interrupt the agent to begin a new round of conversation. The Agora Conversational AI Engine supports agent interruption in the following ways:

Voice interruption: The engine detects user voice input and automatically stops the agent’s response.
Manual interruption: Your app can explicitly stop the agent by calling a REST API or client SDK method—typically triggered by a button tap or custom command.

This page describes how to implement agent interruption in your app.

Voice interruption

Conversational AI Engine supports an intelligent interruption feature that allows a user's voice input to automatically interrupt the speaking agent. This enables quicker response times and more natural, fluid interactions.

To customize interruption behavior, configure turn detection when starting a conversational AI agent. To enable AIVAD-based interruption, set the end of speech mode to "semantic". The following example shows how to configure turn detection:

# ... other agent configuration ...
.with_turn_detection({
    'mode': 'default',
    'config': {
        'speech_threshold': 0.5,
        'start_of_speech': {
            'mode': 'vad',
            'vad_config': {
                'interrupt_duration_ms': 160,
                'speaking_interrupt_duration_ms': 320,
                'prefix_padding_ms': 800,
            },
        },
        'end_of_speech': {
            'mode': 'semantic',
            'semantic_config': {
                'silence_duration_ms': 320,
                'max_wait_ms': 3000,
            },
        },
    },
})
# ... continue with .create_session() ...

// ... other agent configuration ...
.withTurnDetection({
  mode: 'default',
  config: {
    speech_threshold: 0.5,
    start_of_speech: {
      mode: 'vad',
      vad_config: {
        interrupt_duration_ms: 160,
        speaking_interrupt_duration_ms: 320,
        prefix_padding_ms: 800,
      },
    },
    end_of_speech: {
      mode: 'semantic',
      semantic_config: {
        silence_duration_ms: 320,
        max_wait_ms: 3000,
      },
    },
  },
})
// ... continue with .createSession() ...

// ... other agent configuration ...
.WithTurnDetection(&agentkit.TurnDetectionConfig{
    Mode: agora.String("default"),
    Config: &agentkit.StartAgentsRequestPropertiesTurnDetectionConfig{
        SpeechThreshold: agora.Float64(0.5),
        StartOfSpeech: &agentkit.StartAgentsRequestPropertiesTurnDetectionConfigStartOfSpeech{
            Mode: agora.String("vad"),
            VadConfig: &agentkit.StartAgentsRequestPropertiesTurnDetectionConfigStartOfSpeechVadConfig{
                InterruptDurationMs:         agora.Float64(160),
                SpeakingInterruptDurationMs: agora.Float64(320),
                PrefixPaddingMs:             agora.Float64(800),
            },
        },
        EndOfSpeech: &agentkit.StartAgentsRequestPropertiesTurnDetectionConfigEndOfSpeech{
            Mode: agora.String("semantic"),
            SemanticConfig: &agentkit.StartAgentsRequestPropertiesTurnDetectionConfigEndOfSpeechSemanticConfig{
                SilenceDurationMs: agora.Float64(320),
                MaxWaitMs:         agora.Float64(3000),
            },
        },
    },
})
// ... continue with .CreateSession() ...

curl --request post \
--url https://api.agora.io/api/conversational-ai-agent/v2/projects/:appid/join \
--header 'Authorization: Basic <your_base64_encoded_credentials>' \
--data '
{
  "name": "unique_name",
  "properties": {
    "channel": "channel_name",
    "token": "token",
    "agent_rtc_uid": "1001",
    "remote_rtc_uids": ["1002"],
    "turn_detection": {
      "mode": "default",
      "config": {
        "speech_threshold": 0.5,
        "start_of_speech": {
          "mode": "vad",
          "vad_config": {
            "interrupt_duration_ms": 160,
            "speaking_interrupt_duration_ms": 320,
            "prefix_padding_ms": 800
          }
        },
        "end_of_speech": {
          "mode": "semantic",
          "semantic_config": {
            "silence_duration_ms": 320,
            "max_wait_ms": 3000
          }
        }
      }
    },
    "llm": {
      "url": "https://api.openai.com/v1/chat/completions",
      "api_key": "",
      "system_messages": [
        {
          "role": "system",
          "content": "You are a helpful assistant."
        }
      ],
      "params": {
        "model": "gpt-4o-mini"
      }
    },
    "tts": {
      "vendor": "microsoft",
      "params": {
        "key": "",
        "region": "eastus",
        "voice_name": "en-US-AndrewMultilingualNeural"
      }
    },
    "asr": {
      "language": "en-US"
    }
  }
}'

If the request succeeds, the API returns a 200 status code and a response body like the following:

{
  "agent_id": "1NT29X10YHxxxxxWJOXLYHNYB",
  "create_ts": 1737111452,
  "status": "RUNNING"
}

Manual interruption

Conversational AI Engine supports actively triggering an interruption by calling RESTful APIs or client component APIs. This allows users to interrupt the agent through a button click or a specific command.

Server-side interruption

Use the Interrupt agent API to manually initiate an interruption request.

session.interrupt()

await session.interrupt();

if err := session.Interrupt(ctx); err != nil {
    log.Fatal(err)
}

curl --request post \
--url https://api.agora.io/api/conversational-ai-agent/v2/projects/:appid/agents/:agentId/interrupt \
--header 'Authorization: Basic <credentials>' \
--data '{}'

If the request is successful, the API returns a 200 status code and the following response body:

{
  "agent_id": "1NT29XxxxxxxxxELWEHC8OS",
  "channel": "test_channel",
  "start_ts": 1744877089
}

Call the client toolkit API

Agora provides a set of flexible, scalable and standardized client components for its conversational AI engine. These components support iOS, Android, and Web platforms and encapsulate scenario-based APIs. You can use them to integrate Agora Real-Time Communication (RTC) and Signaling capabilities, enabling the following features:

Interrupt the agent
Display real-time transcript
Receive event notifications
Optimize audio (Android and iOS only)

Before you begin, make sure you:

Integrate Video SDK v4.5.1 or later and follow the Quickstart guide to implement basic real-time audio and video features.
Enable Signaling for your project in the Agora Console and follow the Signaling Quickstart to implement real-time messaging.
Implement the basic logic to communicate with a conversational AI agent.
Ensure that the RTC engine instance is initialized and the app is logged in to Signaling. The toolkit does not handle initialization, lifecycle management, authentication, or login for Video SDK or Signaling.

Integrate the toolkit

Copy the convoaiApi folder to your project and import it before calling API methods. Refer to the component structure to understand the role of each file.

Create a configuration object for the RTC engine and Signaling client instances, then use it to initialize the component instance.

// Create a configuration object for the RTC and RTM instances
val config = ConversationalAIAPIConfig(
    rtcEngine = rtcEngineInstance,
    rtmClient = rtmClientInstance,
    enableLog = true
)

// Create the component instance
val api = ConversationalAIAPIImpl(config)

Call Start a conversational AI agent using the following parameter settings:
- advanced_features.enable_rtm: true: Start Signaling (Required)
- parameters.data_channel: "rtm": Enable the RTM data transmission channel (Required)
- parameters.enable_metrics: true: Receive agent performance data (Enabled on demand)
- parameters.enable_error_message: true: Receive agent error events (Enable on demand)
After the call is successful, the agent joins the specified RTC channel and the user can start interacting with the agent.

Call the interrupt method to interrupt the agent.

api.interrupt("agentId") { error -> /* ... */ }

When the agent interaction ends, destroy the component instance to release all resources.
```
api.destroy()
```

Copy the ConversationalAIAPI folder to your project and import it before calling API methods. Refer to the component structure to understand the role of each file.

Create a configuration object for the RTC engine and Signaling client instances, then use it to initialize the component instance.

// Create a configuration object for the RTC and RTM instances
let config = ConversationalAIAPIConfig(
    rtcEngine: rtcEngine,
    rtmEngine: rtmEngine,
    enableLog: true
)

// Create the component instance
convoAIAPI = ConversationalAIAPIImpl(config: config)

Call Start a conversational AI agent using the following parameter settings:
- advanced_features.enable_rtm: true: Start Signaling (Required)
- parameters.data_channel: "rtm": Enable the RTM data transmission channel (Required)
- parameters.enable_metrics: true: Receive agent performance data (Enabled on demand)
- parameters.enable_error_message: true: Receive agent error events (Enable on demand)
After the call is successful, the agent joins the specified RTC channel and the user can start interacting with the agent.

Call the interrupt method to interrupt the agent.

convoAIAPI.interrupt(agentUserId: "\(agentUid)") { error in
    if let error = error {
        print("Interruption failed: \(error.message)")
    } else {
        print("Interruption succeeded")
    }
}

When the agent interaction ends, destroy the component instance to release all resources.
```
convoAIAPI.destroy()
```

Copy the conversational-ai-api file to your project and import it before calling API methods. Refer to the component structure to understand the role of each file.

Create a configuration object for the RTC engine and Signaling client instances, then use it to initialize the component instance.

// Create a configuration object for the RTC and RTM instances
ConversationalAIAPI.init({
    rtcEngine,
    rtmEngine,
})

// Get the API instance (singleton)
const conversationalAIAPI = ConversationalAIAPI.getInstance()

Call Start a conversational AI agent using the following parameter settings:
- advanced_features.enable_rtm: true: Start Signaling (Required)
- parameters.data_channel: "rtm": Enable the RTM data transmission channel (Required)
- parameters.enable_metrics: true: Receive agent performance data (Enabled on demand)
- parameters.enable_error_message: true: Receive agent error events (Enable on demand)
After the call is successful, the agent joins the specified RTC channel and the user can start interacting with the agent.

Call the interrupt method to interrupt the agent.

await conversationalAIAPI.interrupt(`${agent_rtc_uid}`)

When the agent interaction ends, destroy the component instance to release all resources.
```
conversationalAIAPI.destroy()
```

Copy only the following files and folders to integrate the client component. You do not need to copy other files.

IConversationalAIAPI.kt: API interface and related data structures and enumerations
ConversationalAIAPIImpl.kt: ConversationalAI API main implementation logic
ConversationalAIUtils.kt: Tool functions and event callback management
subRender/
- v3/: Transcript module
  - TranscriptionController.kt: Transcript controller
  - MessageParser.kt: Message parser

ConversationalAIAPI.swift: API interface and related data structures and enumerations
ConversationalAIAPIImpl.swift: ConversationalAI API main implementation logic
Transcription/
- TranscriptionController.swift: Transcript controller

index.ts: API Class
type.ts: API interface and related data structures and enumerations
utils/index.ts: API utility functions
utils/events.ts: Event management class, which can be extended to easily implement event monitoring and broadcasting
utils/sub-render.ts: Transcript module

Interrupt the agent mid-response

Voice interruption

Manual interruption

Server-side interruption

Call the client toolkit API

Integrate the toolkit

Reference

Sample project

Component structure

API reference

RESTful API

Toolkit

Contact Us

Contact Us