Voice Activity Detection (Beta)

Understand the tech

The Agora Web SDK processes media through the following pipeline:

The VAD plugin operates during the preprocessing stage, analyzing audio data in real-time to detect voice activity and background music.

Prerequisites

Before you begin, make sure:

• Your project integrates the SDK quickstart using Web SDK v4.15.0 or later.
• You are using a supported browser:
  • Desktop: Latest Chrome (recommended), Edge, or Firefox
  • Safari: 14.1+ (supported but not recommended due to performance issues)
• Your system meets performance requirements:
  • Single plugin: Standard hardware is sufficient
  • Multiple plugins: Use an Intel Core i5 (4-core) or equivalent for best performance
  • High system load: Running multiple plugins with heavy CPU usage may cause audio/video to freeze

Implement the logic

This section shows you how to integrate and use the VAD extension in your project.

Follow these steps to install, configure, and use the VAD plugin in your project:

1. Install the VAD plugin from npm.

   Run the following command to install the VAD plugin:

   npm install agora-extension-vad

   Import the VAD module in your JavaScript or TypeScript file:

   import { VADExtension } from "agora-extension-vad";

2. Deploy Wasm files.

   The VAD plugin requires WebAssembly files to function. Make these files accessible to your web application:

   1. Copy the Wasm files from node_modules/agora-extension-vad/wasm/ to your application's public assets folder, such as public/wasm/ or static/wasm/. When you create the VADExtension object, specify this path so the plugin can load these files.

      Sample file structure:

      your-project/
      ├── public/
      │  └── wasm/      // Copy Wasm files here
      │    ├── file1.wasm
      │    └── file2.wasm
      └── src/
        └── app.js

      • HTTPS required: Serve Wasm files over HTTPS in production.
      • CORS configuration: If you host Wasm files on a different domain than your app, enable CORS for that domain.
      • Same-origin recommended: For simplicity, host Wasm files on the same domain as your application.

   2. Configure Content Security Policy (if applicable).

      If your application uses CSP headers, add the following permissions:

      • For Chrome 97+, Edge 97+, and modern browsers:

        <meta http-equiv="Content-Security-Policy"
           content="script-src 'self' 'wasm-unsafe-eval' blob:">

      • For older browsers (Chrome versions earlier than 97, Edge versions earlier than 97):

        <meta http-equiv="Content-Security-Policy"
           content="script-src 'self' 'unsafe-eval' blob:">

3. Register the plugin.

   Call the AgoraRTC.registerExtensions method and pass the created VADExtension instance to register the VAD plugin. You can also listen for events when the Wasm file fails to load.

   Agora recommends that you create only one instance of VADExtension.

   import AgoraRTC from "agora-rtc-sdk-ng";
   import { VADExtension } from "agora-extension-vad";
   
   // Create the VADExtension instance with the public path where the Wasm files are located
   // Don't include a trailing slash in the path
   const extension = new VADExtension({ assetsPath: "./wasm" });
   
   // Check compatibility
   if (!extension.checkCompatibility()) {
    console.warn("VAD plugin is not supported");
    return;
   }
   
   // Listen for Wasm file loading failures
   extension.onloaderror = (error) => {
    // Handle VAD loading errors
    console.error("VAD plugin failed to load:", error);
   };
   
   // Register the plugin
   AgoraRTC.registerExtensions([extension]);

4. Enable the plugin.

   To enable the VAD plugin:

   1. Call the createProcessor method to create an instance of IVADProcessor. You can set the plugin to be enabled or disabled by default. If you don't specify a state, the plugin uses the last known state.

      // Create an IVADProcessor instance
      const processor = extension.createProcessor();
      
      // Enable the plugin by default
      // await processor.enable();
      
      // Disable the plugin by default
      // await processor.disable();
      
      // Optional: Listen for processor overload events
      processor.on("overload", async () => {
       console.log("Processor overload detected");
       // Choose to disable the plugin
       // await processor.disable();
       // Or continue processing audio despite the overload
      });

   2. Inject the plugin into the audio processing pipeline by calling the SDK's pipe method and specify the processorDestination property.

      // Create an audio track
      const audioTrack = await AgoraRTC.createMicrophoneAudioTrack();
      // Inject the plugin into the audio processing pipeline
      audioTrack.pipe(processor).pipe(audioTrack.processorDestination);

   3. If the plugin is not enabled by default, call the enable method to enable the plugin.

      await processor.enable();

5. Get detection results.

   After you enable the plugin, listen to the result event to get the detection results and process them as needed.

   // Listen to the detection results of the VAD plugin
   processor.on("result", handleResult);
   
   function handleResult(result) {
    // Output voiceProb, which indicates the probability that someone is speaking
    console.info("voiceProb: ", result.voiceProb);
    // Output musicProb, which indicates the probability that music is playing
    console.info("musicProb: ", result.musicProb);
   }

   voiceProb and musicProb are floating-point numbers from 0 to 1 that represent the probability that someone is speaking and that music is playing.

6. Disable the plugin.

   When you no longer need the VAD plugin, call the disable method to disable the plugin.

   await processor.disable();

Reference

This section contains content that completes the information in this page, or points you to documentation that explains other aspects to this product.

API reference

This section provides the API reference for the Voice Activity Detection extension.

IVADExtension

checkCompatibility

Checks whether the current browser supports the VAD plugin. The result returned by this method is for reference only.

checkCompatibility(): boolean;

createProcessor

Creates an IVADProcessor instance.

createProcessor(): IVADProcessor;

onloaderror

Callback for Wasm dependency file loading failures.

onloaderror?: (error: Error) => void;

IVADProcessor

kind

Processor type. Returns either video or audio.

get kind(): 'video' | 'audio';

enabled

Indicates whether the plugin is enabled.

get enabled(): boolean;

enable

Enables the VAD function.

enable(): void | Promise<void>;

disable

Disables the VAD function.

disable(): void | Promise<void>;

on

Listens for processor overload events.

on("overload", callback: () => void): void;

Type definitions

VADExtensionOptions

export interface VADExtensionOptions {
  assetsPath: string;
}

Parameters:

• assetsPath: The URL path of the Wasm files that the VAD plugin depends on. Don't end the path with a slash, for example "./external".

The plugin assembles the full path to the Wasm files from the assetsPath. If the path is incorrect and the Wasm files fail to load, you receive an onloaderror callback.