Audio extension

Understand the tech

An audio filter accesses voice data when it is captured from the user's local device, modifies it, then plays the updated data to local and remote video channels.

A typical transmission pipeline consists of a chain of procedures, including capture, pre-processing, encoding, transmitting, decoding, post-processing, and playback. To modify the voice or video data in the transmission pipeline, audio extensions are inserted into either the pre-processing or post-processing stage.

Prerequisites

• Android Studio 4.1 or higher.
• Android SDK API Level 24 or higher.
• A mobile device that runs Android 4.1 or higher.
• A project to develop in.

Project setup

In order to integrate an extension into your project:

• Follow the SDK setup flow in Quickstart - integrate an extension.
• Prepare the Android project structure and Gradle dependencies required for the extension package.

You are now ready to develop your extension.

Create an audio extension

After encapsulating the filter into an extension, you need to register and package it into a .aar or .so file, and submit it together with a file that contains the extension name, vendor name, and filter name to Agora.

Package the extension

1. Register the extension.

   Register the extension with the REGISTER_AGORA_EXTENSION_PROVIDER macro in AgoraExtensionProviderEntry.h. Use this macro at the entrance of the extension implementation. When the SDK loads the extension, this macro automatically registers it to the SDK. For example:

   REGISTER_AGORA_EXTENSION_PROVIDER(ByteDance, agora::extension::ExtensionProvider);

2. Link the libagora-rtc-sdk-jni.so file.

   In CMakeLists.txt, specify the path to save the libagora-rtc-sdk-jni.so file in the downloaded SDK package according to the following table:

   File	Path
   64-bit libagora-rtc-sdk-jni.so	AgoraWithByteDanceAndroid/agora-bytedance/src/main/agoraLibs/arm64-v8a
   32-bit libagora-rtc-sdk-jni.so	AgoraWithByteDanceAndroid/agora-bytedance/src/main/agoraLibs/arm64-v7a

3. Provide extension information.

   Create a .java or .md file to provide the following information:

   • EXTENSION_NAME: The name of the target link library used in CMakeLists.txt. For example, for a .so file named libagora-bytedance.so, the EXTENSION_NAME should be agora-bytedance.
   • EXTENSION_VENDOR_NAME: The name of the extension provider, which is used for registering in the agora-bytedance.cpp file.
   • EXTENSION_FILTER_NAME: The name of the filter, which is defined in ExtensionProvider.h.

Test your implementation

Once you have developed your extension and API endpoints, the next step is to test whether they work properly.

• Run functional and performance tests. Your test report must contain:
  • Proof that the extension is enabled and loaded in the SDK normally.
  • Verification that all key-value pairs in setExtensionProperty work properly.
  • Verification that all event callbacks of your extension work properly through IMediaExtensionObserver.
  • Performance data such as average per-frame processing time, peak memory usage, and peak CPU/GPU usage.
• Run an extension listing test. Verify the extension listing in the Marketplace test environment, including activation, deactivation, and the linked documentation.
• Write the integration document for your extension. Follow Integration guide for your extension to prepare the public integration document.
• Apply for testing. Contact Agora and provide your extension package, extension listing assets, company name, public email, Provisioning API endpoints, Usage and Billing API endpoints, pricing model, support page URL, website URL, and implementation guide URL.
• Test your extension listing after Agora enables the test environment.

You are then ready to Publish your extension.

Reference

• RtcEngineConfig.addExtension
• enableExtension
• setExtensionProperty
• IMediaExtensionObserver

Understand the tech

An audio filter accesses voice data when it is captured from the user's local device, modifies it, then plays the updated data to local and remote video channels.

A typical transmission pipeline consists of a chain of procedures, including capture, pre-processing, encoding, transmitting, decoding, post-processing, and playback. To modify the voice or video data in the transmission pipeline, audio extensions are inserted into either the pre-processing or post-processing stage.

Prerequisites

• Xcode 12.0 or higher.
• A device running iOS 9.0 or higher.
• A project to develop in.

Project setup

In order to integrate an extension into your project:

• Follow the iOS SDK setup flow in Quickstart - integrate an extension.
• Prepare the project structure, frameworks, and signing settings needed for your extension package.

You are now ready to develop your extension.

Create an audio extension

Package the extension as an iOS framework or xcframework, register it with the SDK, and provide the metadata Agora needs for Marketplace integration.

Test your implementation

• Run functional and performance tests.
• Validate the extension listing in Agora's test environment.
• Prepare the public integration document and update it after end-to-end verification.

You are then ready to Publish your extension.

Reference

• RTC iOS API reference
• RTC iOS API reference

Understand the tech

An audio filter accesses voice data when it is captured from the user's local device, modifies it, then plays the updated data to local and remote video channels.

A typical transmission pipeline consists of a chain of procedures, including capture, pre-processing, encoding, transmitting, decoding, post-processing, and playback. To modify the voice or video data in the transmission pipeline, audio extensions are inserted into either the pre-processing or post-processing stage.

Prerequisites

• Xcode 12.0 or higher.
• A device running macOS 10.11 or higher.
• An Apple developer account.
• A project to develop in.

Project setup

• Follow the macOS SDK setup flow in Quickstart - integrate an extension.
• Prepare the framework packaging, code signing, and runtime loading flow for your extension.

Create an audio extension

Implement the extension and package it as a macOS framework or xcframework for testing and Marketplace submission.

Reference

• Quickstart - integrate an extension

Understand the tech

An audio filter accesses voice data when it is captured from the user's local device, modifies it, then plays the updated data to local and remote video channels.

A typical transmission pipeline consists of a chain of procedures, including capture, pre-processing, encoding, transmitting, decoding, post-processing, and playback. To modify the voice or video data in the transmission pipeline, audio extensions are inserted into either the pre-processing or post-processing stage.

Prerequisites

• A supported browser.
• Physical media input devices such as a camera and a microphone.
• A JavaScript package manager such as npm.
• A project to develop in.

Project setup

• Start from the Web flow in Quickstart - integrate an extension.
• Prepare your extension package, bundler integration, and runtime registration flow.

Create an audio extension

Implement the extension as a Web audio processing plug-in that can be installed and registered from your JavaScript application.

Reference

• Quickstart - integrate an extension

Understand the tech

An audio filter accesses voice data when it is captured from the user's local device, modifies it, then plays the updated data to local and remote video channels.

A typical transmission pipeline consists of a chain of procedures, including capture, pre-processing, encoding, transmitting, decoding, post-processing, and playback. To modify the voice or video data in the transmission pipeline, audio extensions are inserted into either the pre-processing or post-processing stage.

Prerequisites

• A device running Windows 7 or higher.
• Microsoft Visual Studio 2017 or higher.
• C#.
• A project to develop in.

Project setup

• Follow the Windows SDK setup flow in Quickstart - integrate an extension.
• Prepare the extension DLL packaging and loading flow.

Create an audio extension

Package the extension as a Windows DLL and wire it into your RTC SDK project.

Understand the tech

An audio filter accesses voice data when it is captured from the user's local device, modifies it, then plays the updated data to local and remote video channels.

Prerequisites

• Physical media input devices such as a camera and a microphone.
• A JavaScript package manager such as npm.
• A project to develop in.

Project setup

• Follow the Electron SDK setup flow in Quickstart - integrate an extension.
• Prepare the native extension packaging and runtime loading flow.

Create an audio extension

Implement and package the extension so that it can be loaded from your Electron application.

We are still developing extensions for this platform. The docs will magically appear here as soon as it is released.

We are still developing extensions for this platform. The docs will magically appear here as soon as it is released.

We are still developing extensions for this platform. The docs will magically appear here as soon as it is released.