Dubbing and Voice Changer

Using the setExtensionProperty method provided in Agora SDK v4.x and passing in the specified key and value parameters, you can quickly integrate real-time AI voice conversion ability into your app.

The key parameter you specify in the setExtensionProperty method corresponds to the name of the API, and the value parameter wraps some or all of the parameters of the API in JSON format. By passing in the specified key and value parameters, you can call the corresponding API to realize the functions related to real-time AI sound conversion.

Using the setExtensionPropertyWithVendor method provided in Agora SDK v4.x and passing in the specified key and value parameters, you can quickly integrate real-time AI voice conversion ability into your app.

The key parameter you specify in the setExtensionPropertyWithVendor method corresponds to the name of the API, and the value parameter wraps some or all of the parameters of the API in JSON format. By passing in the specified key and value parameters, you can call the corresponding API to realize the functions related to real-time AI sound conversion.

• Android Studio 4.1 or later.
• A physical device (not an emulator) running Android 5.0 or later.

• Xcode 9.0 or later.
• A physical device (not an emulator) running iOS 8.0 or later.

1. Integrate the Dubbing AI Voice Changer extension

   To integrate the extension in your project:

   1. Enter the Extensions Marketplace and download the Dubbing AI Voice Changer extension package. After unzipping, save all .aar files to the project folder /app/libs.

   2. Get the following resource files and save them in the same directory of the project folder. For example, a new vc_model directory:

      • License file and tone files: Contact Agora to obtain these files. The suffix of the tone file is .dat, and the tone file is issued according to the license.
      • Model file: Download the required resources.

   3. Open the app/build.gradle file and add the following lines under dependencies:

      implementation fileTree(dir: "libs", include: ["*.jar", "*.aar"])

2. Enable the plugin

   After creating and initializing RtcEngine, call enableExtension to enable the plug-in, before you call other APIs such as enableVideo and joinChannel.

   // Declare parameters
   private val EXTENSION_NAME = "dubbing_vc"
   private val EXTENSION_VENDOR_NAME = "Dubbing"
   private val EXTENSION_AUDIO_FILTER = "DubbingVC"
   
   private val changeSpeaker_ = "changeSpeaker"
   private val startRealTimeTranscribe_ = "startRealTimeTranscribe"
   private val stopRealTimeTranscribe_ = "stopRealTimeTranscribe"
   private val getSpeakersInfo_ = "getSpeakersInfo"
   
   val config = RtcEngineConfig()
   config.mContext = baseContext
   config.mAppId = appId
   config.mEventHandler = mRtcEventHandler
   // Load plugin
   config.addExtension(EXTENSION_NAME)
   config.mExtensionObserver = extensionObserver
   // Create and initialize RtcEngine
   mRtcEngine = RtcEngine.create(config)
   mRtcEngine.enableAudio()
   // Enable plugin
   mRtcEngine.enableExtension(EXTENSION_VENDOR_NAME, EXTENSION_AUDIO_FILTER, enable)

3. Set the resource file path

   When you download and integrate the plug-in, the license, sound, and model files are saved to the specified directory. In this step, you specify the path where these resource files are located.

   val modelPath: String = "${context.filesDir}${File.separator}vc_model"

4. Get the tone list

   After receiving the onExtensionStarted callback from Agora SDK, call getExtensionProperty passing in the getSpeakersInfo key to get the timbre list:

   val speakerList = mRtcEngine.getExtensionProperty(
       EXTENSION_VENDOR_NAME,
       EXTENSION_AUDIO_FILTER,
       getSpeakersInfo_
   )
   // Convert JSON to array
   val arr = JSONArray(speakerList)

   The timbre list is returned as JSON data that you parse yourself.

5. Start changing

   Call setExtensionProperty and pass in the corresponding key and value:

   mRtcEngine.setExtensionProperty(
       EXTENSION_VENDOR_NAME,
       EXTENSION_AUDIO_FILTER,
       startRealTimeTranscribe_,
       "true"
   )

6. Select timbre

   Pass in the timbre ID from the obtained timbre list to set the corresponding timbre:

   mRtcEngine.setExtensionProperty(
       EXTENSION_VENDOR_NAME,
       EXTENSION_AUDIO_FILTER,
       changeSpeaker_,
       id
   )

7. Stop changing

   Call the API to stop the voice changer:

   mRtcEngine.setExtensionProperty(
       EXTENSION_VENDOR_NAME,
       EXTENSION_AUDIO_FILTER,
       stopRealTimeTranscribe_,
       "true"
   )

8. Release resources

   Close the plug-in and release the resources used by the plug-in:

   mRtcEngine.enableExtension(EXTENSION_VENDOR_NAME, EXTENSION_AUDIO_FILTER, false)

1. Integrate the Dubbing AI Voice Changer extension

   To integrate the extension in your project:

   1. Enter the Extensions Marketplace page and download the plug-in package for Extensions plug-in.
   2. Unzip the folder, import all .framework files into the project, and modify Embed to Embed & Sign.
   3. Get the following resource files and save them in the same directory of the project folder (For example, Resource path):
      • License file and tone files: Contact Agora to obtain these files. The suffix of the tone file is named .dat, and the tone file is issued according to the license.
      • Model file: Download the required resources.
   4. Import the libc++ system library and third-party dynamic libraries into the project MJExtension.

2. Enable the plugin

   After creating and initializing RtcEngine, call enableExtensionWithVendor to enable the plug-in, before you call other APIs such as enableVideo, and joinChannelByToken

   AgoraRtcEngineConfig *cfg = [[AgoraRtcEngineConfig alloc] init];
   cfg.appId = appId;
   cfg.eventDelegate = self;
   cfg.channelProfile = AgoraChannelProfileLiveBroadcasting;
   cfg.audioScenario = AgoraAudioScenarioGameStreaming;
   self.agoraKit = [AgoraRtcEngineKit sharedEngineWithConfig:cfg delegate:self];
   [self.agoraKit setClientRole:AgoraClientRoleBroadcaster];
   
   [self.agoraKit enableExtensionWithVendor:[EngineFilterManager vendorName] extension:@"RealTimeTranscribe" enabled:YES];

3. Set the resource file path

   When you download and integrate the plug-in, the license, sound, and model files are saved to the specified directory. In this step, you specify the path where these resource files are located.

   NSString *sourcePath = [[NSBundle mainBundle] resourcePath];
   [self.agoraKit setExtensionPropertyWithVendor:[EngineFilterManager vendorName] extension:@"RealTimeTranscribe" key:@"setResourcesFilePath" value:sourcePath];

4. Get the tone list

   After receiving the onExtensionStarted callback from Agora SDK, call getExtensionPropertyWithVendor passing in the getSpeakersInfo key to get the timbre list:

   NSString *voices = [self.agoraKit getExtensionPropertyWithVendor:[EngineFilterManager vendorName] extension:@"RealTimeTranscribe" key:@"getSpeakersInfo"];
   // Convert JSON to array
   NSData *nsData = [voices dataUsingEncoding:NSUTF8StringEncoding];
   _speakerArray = [NSJSONSerialization JSONObjectWithData:nsData options:kNilOptions error:nil];

   The timbre list is returned as JSON data that you parse yourself.

5. Start changing

   Call setExtensionPropertyWithVendor and pass in the corresponding key and value:

   [self.agoraKit setExtensionPropertyWithVendor:[EngineFilterManager vendorName] extension:@"RealTimeTranscribe" key:@"startRealTimeTranscribe" value:@"startRealTimeTranscribe"];

6. Select timbre

   Pass in the timbre ID from the obtained timbre list to set the corresponding timbre:

   [self.agoraKit setExtensionPropertyWithVendor:[EngineFilterManager vendorName] extension:@"RealTimeTranscribe" key:@"changeSpeaker" value:@“”];

7. Stop changing

   Call the API to stop the voice changer:

   [self.agoraKit setExtensionPropertyWithVendor:[EngineFilterManager vendorName] extension:@"RealTimeTranscribe" key:@"stopRealTimeTranscribe" value:@"stopRealTimeTranscribe"];

8. Release resources

   Close the plug-in and release the resources used by the plug-in:

   [self.agoraKit enableExtensionWithVendor:[EngineFilterManager vendorName] extension:@"RealTimeTranscribe" enabled:NO];

Sample project

The Dubbing AI Voice Changer extension provides a GitHub sample project for testing. To clone and run the project:

1. Clone the repository

   git clone https://github.com/Dubbing-AI-Voice-Changer/DubbingAgoraDemo.git

2. Refer to the README.md file in the repository to run the demo.

API reference

• addExtension in the RtcEngineConfig class
• enableExtension in the RtcEngine class
• setExtensionProperty in the RtcEngine class

Sample project

The Dubbing AI Voice Changer extension provides a GitHub sample project for testing. To clone and run the project:

1. Clone the repository

   git clone https://github.com/Dubbing-AI-Voice-Changer/DubbingAgoraDemo.git

2. Refer to the README.md file in the repository to run the demo.

API reference

• enableExtensionWithVendor in the AgoraRtcEngineKit class
• setExtensionPropertyWithVendor in the AgoraRtcEngineKit class