Banuba AR Face Filter

Prerequisites

• The latest Banuba SDK archive
• Banuba Extension files for Android
• Banuba trial client token.

To receive a trial token or a full commercial licence from Banuba - please fill in our form on form on banuba.com website, or contact us via info@banuba.com.

Integrate Banuba Face AR extension

Get Started

1. Make sure that you have Android NDK and CMake are installed! Project contains C++ sources, so this is required! Recommended NDK versions is 21.1.6352462, required CMake version is 3.9, 3.14 or above is recommended (Android SDK manager provides 3.18.1, it should be installed).
2. Get the latest Banuba SDK archive, Banuba trial client token. To receive full commercial license from Banuba - please fill in our form on form on banuba.com website, or contact us via info@banuba.com.
3. Copy .aar file from the Banuba SDK archive into libs dir:

   banuba_effect_player-release.aar => libs/

4. Copy include/bnb directory into libs/bnb_sdk/:

   include/bnb => libs/bnb_sdk/bnb

5. Get the latest Agora Video SDK archive.
6. Copy jar files from the Agora Video SDK archive into libs dir:

   agora-rtc-sdk.jar => libs/

7. Copy architecture folders from the Agora Video SDK archive into jniLibs dir:

   arm64-v8a, armeabi-v7a and x86_64 => libs/jniLibs/

8. Copy Agora SDK headers (*.h files) into libs/agora_rtc_sdk/include:

   low_level_api/include => libs/agora_rtc_sdk/include

9. Copy and Paste your Banuba client token into appropriate section of app/src/main/java/com/banuba/sdk/agorapluginexample/ClientToken.kt with " " symbols. For example: BANUBA_CLIENT_TOKEN = "place_your_token_here"
10. Visit agora.io to sign up and get token, app ID and channel ID.
11. Copy and Paste your Agora token, app ID and channel ID into appropriate section of app/src/main/java/com/banuba/sdk/agorapluginexample/ClientToken.kt with " " symbols. For example: AGORA_APP_ID = "place_your_token_here"
12. Open the project in Android Studio and run the necessary target using the usual steps.

How to use BanubaFiltersAgoraExtension

Add following imports:

import com.banuba.agora.plugin.BanubaEffectsLoader
import com.banuba.agora.plugin.BanubaExtensionManager
import com.banuba.agora.plugin.BanubaResourceManager
import com.banuba.agora.plugin.model.ArEffect
import com.banuba.sdk.utils.ContextProvider

Add extension to RtcEngineConfig:

RtcEngineConfig().apply {
    ...
    System.loadLibrary("banuba")
    addExtension(BanubaExtensionManager.EXTENSION_NAME)
    ContextProvider.setContext(mContext)
    ...
}

Create and initialize BanubaResourceManager:

private val banubaResourceManager by lazy(LazyThreadSafetyMode.NONE) {
    BanubaResourceManager(this)
}

override fun onCreate(savedInstanceState: Bundle?) {
    ...
    banubaResourceManager.prepare()
}

After those steps enable and initialize extension:

agoraRtc.enableExtension(
    BanubaExtensionManager.VENDOR_NAME,
    BanubaExtensionManager.VIDEO_FILTER_NAME,
    true
)

private fun initBanubaPlugin() {
    agoraRtc.setExtensionProperty(
        BanubaExtensionManager.VENDOR_NAME,
        BanubaExtensionManager.VIDEO_FILTER_NAME,
        BanubaExtensionManager.KEY_SET_RESOURCES_PATH,
        banubaResourceManager.resourcesPath
    )
    agoraRtc.setExtensionProperty(
        BanubaExtensionManager.VENDOR_NAME,
        BanubaExtensionManager.VIDEO_FILTER_NAME,
        BanubaExtensionManager.KEY_SET_EFFECTS_PATH,
        banubaResourceManager.effectsPath
    )
    agoraRtc.setExtensionProperty(
        BanubaExtensionManager.VENDOR_NAME,
        BanubaExtensionManager.VIDEO_FILTER_NAME,
        BanubaExtensionManager.KEY_SET_TOKEN,
        BANUBA_CLIENT_TOKEN
    )
}

To enable/disable effects use the following code:

private val onEffectPrepared = object : BanubaResourceManager.EffectPreparedCallback {
    override fun onPrepared(effectName: String) {
        sendEffectToFilter(effectName)
    }
}

private fun sendEffectToFilter(effect: String) {
    agoraRtc.setExtensionProperty(
        BanubaExtensionManager.VENDOR_NAME,
        BanubaExtensionManager.VIDEO_FILTER_NAME,
        BanubaExtensionManager.KEY_LOAD_EFFECT,
        effect
    )
}

banubaResourceManager.prepareEffect(Effect name, onEffectPrepared)

Effects managing

To retrieve current effects use the following code:

val effects: List<ArEffect> = BanubaEffectsLoader(this).loadEffects()

ArEffect contains following information:
name: String - pass to banubaResourceManager.prepareEffect(Effect name, onEffectPrepared). Also can be used to display label on the UI
preview: Bitmap - can be used as preview image

To modify effects, add or remove effect folder in app/src/main/assets/effects directory. By default sample contains the following effects:

1. ElvisUnleashed
2. EnglandEightPieceCap
3. FashionHistory1940_male
4. MorphingHatBow
5. Nerd
6. SnapBubblesGlasses
7. Space
8. StarGlow
9. TitanicJack

Run the demo

• Demo app (public)
• Github repository

Prerequisites

• The latest Banuba SDK archive
• Banuba Extension files for iOS
• Banuba trial client token.

To receive a trial token or a full commercial licence from Banuba - please fill in our form on form on banuba.com website, or contact us via info@banuba.com.

Integrate Banuba Face AR extension

Get Started

1. Get the following Banuba trial client token. To receive a trial token or a full commercial licence from Banuba - please fill in our form on form on banuba.com website, or contact us via info@banuba.com.

2. Execute pod install to get the Banuba SDK.

3. Open the BanubaAgoraFilters.xcworkspace file in Xcode.

4. Copy and Paste your Banuba client token into the appropriate section of /BanubaAgoraFilters/Token.swift with " " symbols. For example:

   let banubaClientToken = "Banuba Token"

5. Visit agora.io to sign up and get the token, app and channel ID.

6. Copy and Paste your agora token, app and chanel ID into appropriate section of /BanubaAgoraFilters/Token.swift with " " symbols. For example:

   internal let agoraAppID = "Agora App ID"
   internal let agoraClientToken = "Agora Token"
   internal let agoraChannelId = "Agora Channel ID"

7. Donwload the needed effects from here. This guarantees, that you will use the up-to-date version of the effects. The effects must be copied to the agora-plugin-filters-ios -> BanubaAgoraFilters -> effects folder.

8. Open the BanubaAgoraFilters.xcodeproj project in Xcode and run the BanubaAgoraFilters target.

If you have any problems with installing Agora frameworks with Swift Package Manager refer to this page

Connecting Banuba SDK and AgoraRtcKit to your own project

Connecting Banuba SDK to your project is similar to the steps in the Getting Started section. As for AgoraRtcKit, we advise to use Swift Package Manager. You should use the following settings: URL: https://github.com/AgoraIO/AgoraRtcEngine_iOS.git Version Rule: Up to next minor Version: 4.0.0

How to use BanubaFiltersAgoraExtension

To control BanubaFiltersAgoraExtension with Agora libs look available keys listed below:

public struct BanubaPluginKeys {
    public static let vendorName = "Banuba"
    public static let extensionName = "BanubaFilter"
    public static let loadEffect = "load_effect"
    public static let unloadEffect = "unload_effect"
    public static let setEffectsPath = "set_effects_path"
    public static let setToken = "set_token"
}

To enable/disable BanubaFiltersAgoraExtension use the following method:

import BanubaFiltersAgoraExtension

agoraKit?.enableExtension(
    withVendor: BanubaPluginKeys.vendorName,
    extension: BanubaPluginKeys.extensionName,
    enabled: true
)

Before applying an effect on your video you have to initialize BanubaFiltersAgoraExtension with the path to effects and banuba client token. Look how it can be achieved:

agoraKit?.setExtensionPropertyWithVendor(
    BanubaPluginKeys.vendorName,
    extension: BanubaPluginKeys.extensionName,
    key: BanubaPluginKeys.setEffectsPath,
    value: "place_path_to_effects_folder_here"
)

agoraKit?.setExtensionPropertyWithVendor(
    BanubaPluginKeys.vendorName,
    extension: BanubaPluginKeys.extensionName,
    key: BanubaPluginKeys.setToken,
    value: "place_your_banuba_token_here".trimmingCharacters(in: .whitespacesAndNewlines)
)

After those steps you can tell BanubaFiltersAgoraExtension to enable or disable the mask:

agoraKit?.setExtensionPropertyWithVendor(
    BanubaPluginKeys.vendorName,
    extension: BanubaPluginKeys.extensionName,
    key: BanubaPluginKeys.loadEffect,
    value: "put_effect_name_here"
)

agoraKit?.setExtensionPropertyWithVendor(
    BanubaPluginKeys.vendorName,
    extension: BanubaPluginKeys.extensionName,
    key: BanubaPluginKeys.unloadEffect,
    value: " "
)

If the mask has parameters and you want to change them, you can do it the next way:

agoraKit?.setExtensionPropertyWithVendor(
    BanubaPluginKeys.vendorName,
    extension: BanubaPluginKeys.extensionName,
    key: BanubaPluginKeys.evalJSMethod,
    value: string
)

string must be a string with method's name and parameters. You can find an example in our documentation.

How to build BanubaFiltersAgoraExtension

To build the BanubaFiltersAgoraExtension manually, please follow the steps bellow:

1. Execute pod install to get the Banuba SDK.

2. Open the BanubaAgoraFilters.xcworkspace file in Xcode.

3. Choose File->Packages->Reset Package Cashes from Xcode menu.

4. Build the target BanubaFiltersAgoraExtension. It will be built with your Swift version. After this you should open the section Products in the Project Navigator (the left part of the Xcode screen). Click on the BanubaFiltersAgoraExtension with the right click of the mouse and choose Show in Finder. Copy the BanubaFiltersAgoraExtension.framework from the folder.

5. Then put the framework to the /Frameworks folder of the BanubaAgoraFilters.xcodeproj (or of your project). Then you can build BanubaAgoraFilters or your project.

The reconnection of the BanubaFiltersAgoraExtension.framework to the example project may be required. To do it, you should remove the BanubaFiltersAgoraExtension.framework from the Project Settings: General-> Frameworks, Libraries and Embedded Content. Then you should drag&drop the BanubaFiltersAgoraExtension.framework to this section. You should choose Embed&Sign for this framework.

Effects managing

To retrieve effects list use the following code:

let effectsPath = BanubaEffectsManager.effectsURL.path
let effectsService = EffectsService(effectsPath: effectsPath)
let effectViewModels =
    effectsService
    .getEffectNames()
    .sorted()
    .compactMap { effectName in
        guard let effectPreviewImage = effectsService.getEffectPreview(effectName) else {
            return nil
        }

        let effectViewModel = EffectViewModel(image: effectPreviewImage, effectName: effectName)
        return effectViewModel
    }

EffectViewModel has the following properties:

class EffectViewModel {
    let image: UIImage
    let effectName: String?
    var cancelEffectModel: Bool {
        return effectName == nil
    }
}

Run the demo

• Demo app (distribution by request via support channels at the bottom of the page)
• Github repository