Voice Activity Detection (Beta)

Understand the tech

The Agora Web SDK processes media through the following pipeline:

The VAD plugin operates during the preprocessing stage, analyzing audio data in real-time to detect voice activity and background music.

Prerequisites

Before you begin, make sure:

• Your project integrates the SDK quickstart using Web SDK v4.15.0 or later.
• You are using a supported browser:
• Desktop: Latest Chrome (recommended), Edge, or Firefox
• Safari: 14.1+ (supported but not recommended due to performance issues)
• Your system meets performance requirements:
• Single plugin: Standard hardware is sufficient
• Multiple plugins: Use an Intel Core i5 (4-core) or equivalent for best performance
• High system load: Running multiple plugins with heavy CPU usage may cause audio/video to freeze

Implement the logic

This section shows you how to integrate and use the VAD extension in your project.

Follow these steps to install, configure, and use the VAD plugin in your project:

1. Install the VAD plugin from npm.

2. Run the following command to install the VAD plugin:

   npm install agora-extension-vad

3. Import the VAD module in your JavaScript or TypeScript file:

   import { VADExtension } from "agora-extension-vad";

4. Deploy Wasm files

The VAD plugin requires WebAssembly files to function. Make these files accessible to your web application:

1. Copy the Wasm files from node_modules/agora-extension-vad/wasm/ to your application's public assets folder, such as public/wasm/ or static/wasm/. When you create the VADExtension object, specify this path so the plugin can load these files.

Sample file structure:

your-project/
├── public/
│  └── wasm/      // Copy Wasm files here
│    ├── file1.wasm
│    └── file2.wasm
└── src/
  └── app.js

1. Configure Content Security Policy (if applicable)

If your application uses CSP headers, add the following permissions:

• For Chrome 97+, Edge 97+, and modern browsers:

  <meta http-equiv="Content-Security-Policy"
     content="script-src 'self' 'wasm-unsafe-eval' blob:">

• For older browsers (Chrome versions earlier than 97, Edge versions earlier than 97):

  <meta http-equiv="Content-Security-Policy"
     content="script-src 'self' 'unsafe-eval' blob:">

1. Register the plugin

Call the AgoraRTC.registerExtensions method and pass the created VADExtension instance to register the VAD plugin. You can also listen for events when the Wasm file fails to load.

import AgoraRTC from "agora-rtc-sdk-ng";
import { VADExtension } from "agora-extension-vad";

// Create the VADExtension instance with the public path where the Wasm files are located
// Don't include a trailing slash in the path
const extension = new VADExtension({ assetsPath: "./wasm" });

// Check compatibility
if (!extension.checkCompatibility()) {
 console.warn("VAD plugin is not supported");
 return;
}

// Listen for Wasm file loading failures
extension.onloaderror = (error) => {
 // Handle VAD loading errors
 console.error("VAD plugin failed to load:", error);
};

// Register the plugin
AgoraRTC.registerExtensions([extension]);

1. Enable the plugin

To enable the VAD plugin:

1. Call the createProcessor method to create an instance of IVADProcessor. You can set the plugin to be enabled or disabled by default. If you don't specify a state, the plugin uses the last known state.

   // Create an IVADProcessor instance
   const processor = extension.createProcessor();
   
   // Enable the plugin by default
   // await processor.enable();
   
   // Disable the plugin by default
   // await processor.disable();
   
   // Optional: Listen for processor overload events
   processor.on("overload", async () => {
    console.log("Processor overload detected");
    // Choose to disable the plugin
    // await processor.disable();
    // Or continue processing audio despite the overload
   });

2. Inject the plugin into the audio processing pipeline by calling the SDK's pipe method and specify the processorDestination property.

   // Create an audio track
   const audioTrack = await AgoraRTC.createMicrophoneAudioTrack();
   // Inject the plugin into the audio processing pipeline
   audioTrack.pipe(processor).pipe(audioTrack.processorDestination);

3. If the plugin is not enabled by default, call the enable method to enable the plugin.

   await processor.enable();

4. Get detection results

After you enable the plugin, listen to the result event to get the detection results and process them as needed.

// Listen to the detection results of the VAD plugin
processor.on("result", handleResult);

function handleResult(result) {
 // Output voiceProb, which indicates the probability that someone is speaking
 console.info("voiceProb: ", result.voiceProb);
 // Output musicProb, which indicates the probability that music is playing
 console.info("musicProb: ", result.musicProb);
}

1. Disable the plugin

When you no longer need the VAD plugin, call the disable method to disable the plugin.

await processor.disable();

Reference

This section contains content that completes the information in this page, or points you to documentation that explains other aspects to this product.

API reference

This section provides the API reference for the Voice Activity Detection extension.

IVADExtension

checkCompatibility

Checks whether the current browser supports the VAD plugin. The result returned by this method is for reference only.

checkCompatibility(): boolean;

createProcessor

Creates an IVADProcessor instance.

createProcessor(): IVADProcessor;

onloaderror

Callback for Wasm dependency file loading failures.

onloaderror?: (error: Error) => void;

IVADProcessor

kind

Processor type. Returns either video or audio.

get kind(): 'video' | 'audio';

enabled

Indicates whether the plugin is enabled.

get enabled(): boolean;

enable

Enables the VAD function.

enable(): void | Promise<void>;

disable

Disables the VAD function.

disable(): void | Promise<void>;

on

Listens for processor overload events.

on("overload", callback: () => void): void;

Type definitions

VADExtensionOptions

export interface VADExtensionOptions {
  assetsPath: string;
}

Parameters:

• assetsPath: The URL path of the Wasm files that the VAD plugin depends on. Don't end the path with a slash, for example "./external".

The plugin assembles the full path to the Wasm files from the assetsPath. If the path is incorrect and the Wasm files fail to load, you receive an onloaderror callback.