Custom audio source

Updated

Integrate a custom video or audio capture into your client

The default audio module of RTC SDK meets the need of using basic audio functions in your app. For adding advanced audio functions, RTC SDK supports using custom audio sources and custom audio rendering modules.

RTC SDK uses the basic audio module on the device your app runs on by default. However, there are certain use-cases where you want to integrate a custom audio source into your app, such as:

  • Your app has its own audio module.
  • You need to process the captured audio with a pre-processing library for audio enhancement.
  • You need flexible device resource allocation to avoid conflicts with other services.

This page shows you how to capture and render audio from custom sources.

Understand the tech

To set an external audio source, you configure the Agora Engine before joining a channel. To manage the capture and processing of audio frames, you use methods from outside the RTC SDK that are specific to your custom source. RTC SDK enables you to push processed audio data to the subscribers in a channel.

Prerequisites

Ensure that you have implemented the SDK quickstart in your project.

Implementation

Custom audio capture

The SDK provides the createCustomAudioTrack method, which enables you to create a local audio track by passing in a browser-native MediaStreamTrack object. This method allows you to achieve custom audio capture. To use multiple audio sources, you can call the createCustomAudioTrack method multiple times to create separate local audio track objects.

The following example uses getUserMedia to obtain a MediaStreamTrack object and passes it to createCustomAudioTrack to create a local audio track object for use with the SDK.

async function createAndPublishCustomAudioTrack(client) {
  try {
    // Get the audio media stream.
    const mediaStream = await navigator.mediaDevices.getUserMedia({
      video: false,
      audio: true,
    });

    // Extract the audio track from the media stream.
    const audioMediaStreamTrack = mediaStream.getAudioTracks()[0];

    // Create a custom audio track.
    const customAudioTrack = await AgoraRTC.createCustomAudioTrack({
      mediaStreamTrack: audioMediaStreamTrack,
    });

    // Store the custom audio track for later use in a shared object.
    rtc.localAudioTrack = customAudioTrack;

    // Publish the custom audio track to the RTC channel.
    await rtc.client.publish([rtc.localAudioTrack]);

    console.log('Custom audio track published successfully!');
  } catch (error) {
    console.error('Failed to create or publish custom audio track:', error);
  }
}

To implement custom audio processing, use the Web Audio API to obtain the MediaStreamTrack.

Multiopus multichannel audio

Web SDK v4.24.10 or later supports sending and receiving audio with 4, 6, or 8 channels through Multiopus. Set multiopus to true when creating both the sending and receiving clients. Multichannel audio processing requires browser support for MediaStreamTrackProcessor and MediaStreamTrackGenerator.

Note

Multiopus is supported on Chrome 94 or later. Firefox and Safari are not supported.

Pass a MediaStreamTrack that already contains the required channels to createCustomAudioTrack, and set channelCount to match. The option does not add channels to the input audio.

const client = AgoraRTC.createClient({
  mode: "rtc",
  codec: "vp8",
  multiopus: true,
});
// multichannelMediaStreamTrack is supplied by your custom audio source.
const track = AgoraRTC.createCustomAudioTrack({
  mediaStreamTrack: multichannelMediaStreamTrack,
  channelCount: 4,
});
// Join the channel before publishing the track.
await client.join(appId, channel, token, uid);
await client.publish(track);

On local and subscribed remote audio tracks, use selectChannels to select channels by their zero-based indices, and set one of the following rendering modes:

  • "surround": Preserve the original channel count and positions, and silence unselected channels.
  • "stereo": Mix selected channels into stereo, with even indices on the left and odd indices on the right.
  • "mono": Mix selected channels into a single channel.
track.selectChannels([0, 1], "stereo");
const levels = track.getVolumeLevels();

getVolumeLevels measures each original channel before selection and mixing. If you pass a list of channel indices, the returned array keeps those original indices and leaves unqueried positions empty. Before audio data is available, the array or its positions can be empty. Without multichannel processing, selectChannels leaves the track unchanged and getVolumeLevels returns an array containing the single getVolumeLevel() result.

Reference

This section explains how to implement different sound effects and audio mixing in your app, covering essential steps and code snippets.

API reference