Looking for the previous docs site? Visit the legacy docs homepage. (opens in a new tab)
Agora Docs

Manually control start and end of speech

Updated

Explicitly control when a user's speech starts and ends using manual SoS and EoS.

When your business scenario isn't a good fit for letting the server automatically determine when a user starts or stops speaking, use manual Start of Speech (SoS) and End of Speech (EoS) to have the client explicitly declare the user's turn boundaries. This is useful for AI interviews, interactive quizzes, walkie-talkie style push-to-talk, and other "press to start, tap to submit" interactions. This guide explains how to implement manual control over user speech boundaries by configuring the agent and using the client toolkit.

Understand the tech

Manual turn control consists of two independent capabilities:

  • Manual SoS: The client explicitly declares that the user has started speaking.
  • Manual EoS: The client explicitly declares that the user has finished speaking.

start_of_speech.mode and end_of_speech.mode are independent settings. Choose which boundary needs manual control based on your use case:

Business scenariostart_of_speech.modeend_of_speech.modeClient call
The user can start speaking freely, but must tap "submit answer"vadmanualCall only manualEOS
The user taps or holds a button to start speaking, and the server determines when speech endsmanualvadCall only manualSOS
Press-and-hold to talk, release to send (walkie-talkie mode)manualmanualCall manualSOS first, then manualEOS
  • If only end_of_speech.mode is manual, the server still uses VAD to automatically detect when the user starts speaking. Your app only needs to call manualEOS when the user taps submit.
  • If only start_of_speech.mode is manual, the server starts accepting audio for the current turn only after your app calls manualSOS. End of speech is still detected automatically by VAD, so your app does not need to call manualEOS.
  • If both start_of_speech.mode and end_of_speech.mode are manual, the server starts accepting audio for the current turn only after your app calls manualSOS, and submits the turn only after your app calls manualEOS.

Prerequisites

Before you begin, ensure that you have:

  • Completed the basic steps for interacting with an agent. See Quickstart.
  • Integrated Video SDK v4.5.1 or later and followed the Quickstart guide to implement basic real-time audio and video features.
  • Enabled Signaling for your project in the Agora Console and followed the Signaling Quickstart to implement real-time messaging.
  • Made sure the RTC engine instance is initialized and the app is logged in to Signaling. The client toolkit does not handle initialization, lifecycle management, authentication, or login for Video SDK or Signaling.

Implement manual control

Manual turn control requires configuring both the agent and the client toolkit:

  1. Configure manual mode when you start the agent.
  2. Initialize the client toolkit and subscribe to channel messages.
  3. Register handlers for the manual SoS/EoS result events.
  4. Call manualSOS/manualEOS when the user taps start or submit.

Enable manual turn control when starting the agent

When calling Start a conversational AI agent, use the following parameter settings:

  • advanced_features.enable_rtm: true: Start Signaling (Required)
  • parameters.data_channel: "rtm": Enable the RTM data transmission channel (Required)
  • parameters.enable_metrics: true: Receive agent performance data (Enable on demand)
  • parameters.enable_error_message: true: Receive agent error events (Enable on demand)

Then set the boundary or boundaries you want to control manually to "manual".

The following curl examples show three common configurations:

If you only need to manually control end of speech, keep automatic SoS and set only end_of_speech.mode to manual.

curl --request POST \
  --url https://api.agora.io/api/conversational-ai-agent/v2/projects/<your_app_id>/join \
  --header 'Authorization: agora token="007abcxxxxxxx123"' \
  --data '
{
  "name": "manual-eos-agent",
  "properties": {
    "channel": "channel_name",
    "token": "token",
    "agent_rtc_uid": "0",
    "remote_rtc_uids": [
      "123"
    ],
    "advanced_features": {
      "enable_rtm": true
    },
    "asr": {
      "language": "en-US"
    },
    "llm": {
      "url": "https://api.xxxx/v1/xxxx",
      "api_key": "xxx",
      "system_messages": [
        {
          "role": "system",
          "content": "You are a helpful chatbot."
        }
      ],
      "greeting_message": "Hello, how can I help you?",
      "failure_message": "Sorry, I am unable to answer that question.",
      "max_history": 10,
      "params": {
        "model": "xxxx"
      }
    },
    "tts": {
      "vendor": "minimax",
      "params": {
        "key": "your-minimax-key",
        "model": "speech-01-turbo",
        "voice_setting": {
          "voice_id": "female-shaonv",
          "speed": 1,
          "vol": 1,
          "pitch": 0,
          "emotion": "happy"
        },
        "audio_setting": {
          "sample_rate": 16000
        }
      }
    },
    "turn_detection": {
      "mode": "default",
      "config": {
        "start_of_speech": {
          "mode": "vad",
          "vad_config": {
            "interrupt_duration_ms": 160,
            "speaking_interrupt_duration_ms": 320,
            "prefix_padding_ms": 800
          }
        },
        "end_of_speech": {
          "mode": "manual"
        }
      }
    },
    "parameters": {
      "data_channel": "rtm"
    }
  }
}
'

If you only need to manually control start of speech, set start_of_speech.mode to manual and keep automatic EoS.

curl --request POST \
  --url https://api.agora.io/api/conversational-ai-agent/v2/projects/<your_app_id>/join \
  --header 'Authorization: agora token="007abcxxxxxxx123"' \
  --data '
{
  "name": "manual-sos-agent",
  "properties": {
    "channel": "channel_name",
    "token": "token",
    "agent_rtc_uid": "0",
    "remote_rtc_uids": [
      "123"
    ],
    "advanced_features": {
      "enable_rtm": true
    },
    "asr": {
      "language": "en-US"
    },
    "llm": {
      "url": "https://api.xxxx/v1/xxxx",
      "api_key": "xxx",
      "system_messages": [
        {
          "role": "system",
          "content": "You are a helpful chatbot."
        }
      ],
      "greeting_message": "Hello, how can I help you?",
      "failure_message": "Sorry, I am unable to answer that question.",
      "max_history": 10,
      "params": {
        "model": "xxxx"
      }
    },
    "tts": {
      "vendor": "minimax",
      "params": {
        "key": "your-minimax-key",
        "model": "speech-01-turbo",
        "voice_setting": {
          "voice_id": "female-shaonv",
          "speed": 1,
          "vol": 1,
          "pitch": 0,
          "emotion": "happy"
        },
        "audio_setting": {
          "sample_rate": 16000
        }
      }
    },
    "turn_detection": {
      "mode": "default",
      "config": {
        "start_of_speech": {
          "mode": "manual"
        },
        "end_of_speech": {
          "mode": "vad",
          "vad_config": {
            "silence_duration_ms": 480
          }
        }
      }
    },
    "parameters": {
      "data_channel": "rtm"
    }
  }
}
'

If you need a complete push-to-talk experience, set both start_of_speech.mode and end_of_speech.mode to manual.

curl --request POST \
  --url https://api.agora.io/api/conversational-ai-agent/v2/projects/<your_app_id>/join \
  --header 'Authorization: agora token="007abcxxxxxxx123"' \
  --data '
{
  "name": "manual-sos-eos-agent",
  "properties": {
    "channel": "channel_name",
    "token": "token",
    "agent_rtc_uid": "0",
    "remote_rtc_uids": [
      "123"
    ],
    "advanced_features": {
      "enable_rtm": true
    },
    "asr": {
      "language": "en-US"
    },
    "llm": {
      "url": "https://api.xxxx/v1/xxxx",
      "api_key": "xxx",
      "system_messages": [
        {
          "role": "system",
          "content": "You are a helpful chatbot."
        }
      ],
      "greeting_message": "Hello, how can I help you?",
      "failure_message": "Sorry, I am unable to answer that question.",
      "max_history": 10,
      "params": {
        "model": "xxxx"
      }
    },
    "tts": {
      "vendor": "minimax",
      "params": {
        "key": "your-minimax-key",
        "model": "speech-01-turbo",
        "voice_setting": {
          "voice_id": "female-shaonv",
          "speed": 1,
          "vol": 1,
          "pitch": 0,
          "emotion": "happy"
        },
        "audio_setting": {
          "sample_rate": 16000
        }
      }
    },
    "turn_detection": {
      "mode": "default",
      "config": {
        "start_of_speech": {
          "mode": "manual"
        },
        "end_of_speech": {
          "mode": "manual"
        }
      }
    },
    "parameters": {
      "data_channel": "rtm"
    }
  }
}
'

After a successful response, the agent joins the specified RTC channel and is ready to interact with the user.

Integrate and initialize the client toolkit

  1. Add the toolkit to your project using Maven or source code. See Install the Android toolkit. Refer to the component structure to understand the role of each file.

  2. Create a configuration object for the RTC engine and Signaling client instances, then use it to initialize the component instance.

    val config = ConversationalAIAPIConfig(
    rtcEngine = rtcEngine,
    rtmClient = rtmClient,
    renderMode = TranscriptRenderMode.Word,
    enableLog = true,
    enableRenderModeFallback = true
)
val api = ConversationalAIAPIImpl(config)

  3. Register a handler for the manual SoS/EoS result events, then subscribe to the agent's channel.

    api.addHandler(object : IConversationalAIAPIEventHandler {
    override fun onUserManualSosEvent(agentUserId: String, event: UserManualSosEvent) {
        // Handle the manual SoS result. event.payload.success indicates whether the server accepted it.
    }

    override fun onUserManualEosEvent(agentUserId: String, event: UserManualEosEvent) {
        // Handle the manual EoS result.
    }

    override fun onAgentManualEosEvent(agentUserId: String, event: AgentManualEosEvent) {
        // Handle the server-triggered automatic EoS notification.
    }
})

api.subscribeMessage("channelName") { error ->
    if (error != null) {
        // Handle subscription failure.
    }
}

  1. Add the toolkit to your project using CocoaPods, Swift Package Manager, or source code. See Install the iOS toolkit. Refer to the component structure to understand the role of each file.

  2. Create a configuration object for the RTC engine and Signaling client instances, then use it to initialize the component instance.

    let config = ConversationalAIAPIConfig(
    rtcEngine: rtcEngine,
    rtmEngine: rtmEngine,
    renderMode: .words,
    enableLog: true,
    enableRenderModeFallback: true
)
convoAIAPI = ConversationalAIAPIImpl(config: config)

  3. Register a handler for the manual SoS/EoS result events, then subscribe to the agent's channel.

    convoAIAPI.addHandler(handler: self)

func onUserManualSosEvent(agentUserId: String, event: UserManualSosEvent) {
    // Handle the manual SoS result. event.payload.success indicates whether the server accepted it.
}

func onUserManualEosEvent(agentUserId: String, event: UserManualEosEvent) {
    // Handle the manual EoS result.
}

func onAgentManualEosEvent(agentUserId: String, event: AgentManualEosEvent) {
    // Handle the server-triggered automatic EoS notification.
}

convoAIAPI.subscribeMessage(channelName: channelName) { error in
    if let error = error {
        // Handle subscription failure.
    }
}

  1. Add the toolkit to your project using a package manager or source code. See Install the Web toolkit. Refer to the component structure to understand the role of each file.

  2. Create a configuration object for the RTC engine and Signaling client instances, then use it to initialize the component instance.

    const config: IConversationalAIAPIConfig = {
  rtcEngine,
  rtmEngine,
  renderMode: ETranscriptHelperMode.WORD,
  enableLog: true,
  enableRenderModeFallback: true,
}

const conversationalAIAPI = await ConversationalAIAPI.init(config)

  3. Register a handler for the manual SoS/EoS result events, then subscribe to the agent's channel.

    conversationalAIAPI.on(EConversationalAIAPIEvents.USER_MANUAL_SOS_RESULT, (agentUserId, event) => {
  // Handle the manual SoS result. event.payload.success indicates whether the server accepted it.
})

conversationalAIAPI.on(EConversationalAIAPIEvents.USER_MANUAL_EOS_RESULT, (agentUserId, event) => {
  // Handle the manual EoS result.
})

conversationalAIAPI.on(EConversationalAIAPIEvents.AGENT_MANUAL_EOS_RESULT, (agentUserId, event) => {
  // Handle the server-triggered automatic EoS notification.
})

conversationalAIAPI.subscribeMessage(channelName)

Info

These handlers receive the server's processing result for your manual SoS/EoS requests. Use the event, requestId, and error information to implement your own business logic. See the client toolkit API reference for details.

Send manual SoS when the user starts speaking

Call manualSOS only when start_of_speech.mode is "manual". If you use automatic SoS through VAD, skip this step.

api.manualSOS(agentUserId = "agentUserId") { requestId, error ->
    if (error != null) {
        Log.e("ManualSOS", "Send failed: ${error.errorMessage}, requestId=$requestId")
    } else {
        Log.i("ManualSOS", "SoS sent, requestId=$requestId")
    }
}
convoAIAPI.manualSOS(agentUserId: agentUid) { requestId, error in
    if let error = error {
        print("Send failed: \(error.message), requestId=\(requestId)")
    } else {
        print("SoS sent, requestId=\(requestId)")
    }
}
const requestId = await conversationalAIAPI.manualSOS(agentUserId)

Once manual SoS takes effect, the server starts counting subsequent audio toward the current user turn. Audio that arrives before this point is not counted.

Send manual EoS when the user finishes speaking

Call manualEOS only when end_of_speech.mode is "manual". If you use automatic EoS through VAD, skip this step.

api.manualEOS(agentUserId = "agentUserId") { requestId, error ->
    if (error != null) {
        Log.e("ManualEOS", "Send failed: ${error.errorMessage}, requestId=$requestId")
    } else {
        Log.i("ManualEOS", "EoS sent, requestId=$requestId")
    }
}
convoAIAPI.manualEOS(agentUserId: agentUid) { requestId, error in
    if let error = error {
        print("Send failed: \(error.message), requestId=\(requestId)")
    } else {
        print("EoS sent, requestId=\(requestId)")
    }
}
const requestId = await conversationalAIAPI.manualEOS(agentUserId)

Once manual EoS takes effect, the agent submits the turn for ASR and LLM inference as usual. This signal only marks the end of the user's current speech segment — it does not immediately end the entire turn.

Destroy the component instance

When the agent interaction ends, destroy the component instance to release all resources.

api.destroy()
convoAIAPI.destroy()
conversationalAIAPI.destroy()

Reference

API reference

RESTful API

Toolkit

Contact Us

+1 (408) 879-5885

2804 Mission College Blvd.Santa Clara, CA, USA 95054

AgoraCopyright © 2026 AgoraAll rights reserved