FaceUnity

The FaceUnity AR Filter extension uses 3D vision, 3D graphics, and deep learning technologies to provide industry’s leading AR portrait video effects for Video SDK. It enables you to add diverse special effects that provide full coverage of the human face, including:

• Basic beauty: Skin beautification, whitening, rosy.
• Advanced beauty: Video beauty effects for shape, face and skin.
• Number detection: Detect the number of faces, humans, or gestures.

This page shows you how to integrate and use the FaceUnity AR Filter extension in your app.

To quickly integrate FaceUnity's AR filter capabilities in your app, you set extension properties using the key and value parameters in Video SDK. When you call setExtensionProperty or setExtensionPropertyWithVendor method of Video SDK and pass in a pair of key and value parameters, it is equivalent to calling the corresponding FaceUnity API. The key is named after the FaceUnity API method, and value wraps the parameters required for that method in JSON. Currently, the extension encapsulates part of the APIs of the FaceUnity Nama SDK. For details, see the FaceUnity key-value overview.

The development environment requirements are as follows:

• Android Studio 4.1 or higher.
• Android SDK API Level 24 or higher.
• A mobile device that runs Android 4.1 or higher.

• Implemented SDK quickstart for Video Calling.

Open the SDK quickstart for Video Calling project you created earlier.

To get the FaceUnity extension, take the following steps:

1. Download the FaceUnity AR Filter package for your platform from the Extensions Marketplace.
2. Contact Agora for activation and provide your package name. For example, yourCompany.yourProject.faceunity. You then receive the authpack certificate file that corresponds to your package name for subsequent integrations.
3. Download the FaceUnity resource package.

1. Unzip the FaceUnity AR Filter package, and save all .aar files to the /app/libs folder in your Android project.

2. Save the certificate file authpack.java to the path corresponding to the package name you provided when obtaining the certificate. For example, if the package name is io.agora.rte.extension.faceunity.example, the certificate file should be saved to /app/src/main/java/io/agora/rte/extension/faceunity/example).

3. Save the model and prop files you need from the resource package to the /app/src/main/assets folder in your project. For details of files provided in the resource pack, see Resource package structure.

4. In the app/build.gradle file, add the following line under dependencies:

   
   

   _1

   implementation fileTree(dir: "libs", include: ["*.jar", "*.aar"])

   
   

5. To use the required classes, ensure that the following libraries are included in the list of import statements in your Android activity:

   
   

   _8

   import io.agora.rtc2.Constants;

   _8

   import io.agora.rtc2.IMediaExtensionObserver;

   _8

   import io.agora.rtc2.IRtcEngineEventHandler;

   _8

   import io.agora.rtc2.RtcEngine;

   _8

   import io.agora.rtc2.RtcEngineConfig;

   _8

   import io.agora.rtc2.video.VideoCanvas;

   _8

   _8

   import io.agora.rte.extension.faceunity.ExtensionManager;

   
   

This section describes the call sequence you implement to use FaceUnity features in your app.

1. Enable the extension

   When initializing RtcEngine, call enableExtension before other APIs (including enableVideo and joinChannel) to enable the extension.

   
   

   _5

   private void enableExtension(boolean enabled) {

   _5

   // Initialize ExtensionManager before calling enableExtension

   _5

   ExtensionManager.getInstance(mRtcEngine).initialize(this);

   _5

   mRtcEngine.enableExtension("FaceUnity", "Effect", enabled);

   _5

   }

   
   

2. Initialize the extension

   After receiving the onExtensionStarted callback, call setExtensionProperty, and pass in the corresponding key and value pair:

   1. Initialize the extension: Set the key as fuSetup, and the value as the pointer to the certificate file.
   2. Load the AI model: Set the key as fuLoadAIModelFromPackage, and the value contains the path of the AI capability model file ai_xxx.bundle and the AI capability type.
   
   

   _31

   private void initExtension() {

   _31

   // Initialization

   _31

   try {

   _31

   JSONObject jsonObject = new JSONObject();

   _31

   JSONArray jsonArray = new JSONArray();

   _31

   for (byte it : authpack.A()) {

   _31

   jsonArray.put(it);

   _31

   }

   _31

   jsonObject.put("authdata", jsonArray);

   _31

   setExtensionProperty("fuSetup", jsonObject.toString());

   _31

   } catch (JSONException e) {

   _31

   Log.e(TAG, e.toString());

   _31

   }

   _31

   _31

   // Load the AI model

   _31

   File modelDir = new File(getExternalFilesDir("assets"),

   _31

   "face_unity/model/ai_face_processor.bundle");

   _31

   try {

   _31

   JSONObject jsonObject = new JSONObject();

   _31

   jsonObject.put("data", modelDir.getAbsolutePath());

   _31

   jsonObject.put("type", 1 << 8);

   _31

   setExtensionProperty("fuLoadAIModelFromPackage", jsonObject.toString());

   _31

   } catch (JSONException e) {

   _31

   Log.e(TAG, e.toString());

   _31

   }

   _31

   }

   _31

   _31

   // Only updates the key and value when calling setExtensionProperty

   _31

   private void setExtensionProperty(String key, String property) {

   _31

   mRtcEngine.setExtensionProperty("FaceUnity", "Effect", key, property);

   _31

   }

   
   

3. Configure beauty effects and body recognition

   Call setExtensionProperty and pass in the corresponding keys and values.

   You can implement the following functions:

   • Load props and adjust beautification intensity
   • Recognize and track human faces, gestures, and bodies

   You can call the method as needed. For a full list of keys and values, see the FaceUnity key-value overview.

4. Release the resources

   When you do not need the extension, follow these steps to release the resources:

   1. Call setExtensionProperty and pass in the fuDestroyLibData key.
   2. After receiving the fuDestroyLibData callback, call destroy to destroy the Agora Engine.

The complete Android FaceUnity sample code and project is available on GitHub.

1. Clone the repository:

   
   

   _1

   git clone https://github.com/AgoraIO-Community/AgoraMarketPlace.git

   
   

2. On the Extensions Marketplace Downloads page, download the Android package of FaceUnity AR Filter. Unzip the package, and save all .aar files to the FaceUnity/android/app/libs path.

3. Contact Agora for activation and provide your package name. For example, yourCompany.yourProject.faceunity. You receive the authpack certificate file that corresponds to your package name for subsequent integrations.

4. Download the FaceUnity resource package. Unzip the package and save all files to FaceUnity/android/app/src/main/assets/Resource path.

5. Open the sample project FaceUnity/android in Android Studio.

6. Sync the project with Gradle files.

7. Change io.agora.rte.extension.faceunity.example to the package name you provided when obtaining the certificate file, and save the certificate file authpack.java in the path corresponding to the package name.

8. Open the FaceUnity/android/app/src/main/java/io/Agora/rte/extension/FaceUnity/example/Config.java file, and replace <YOUR_APP_ID> with your App ID. To get an App ID, see Getting Started with Agora.

   
   

   _4

   public interface Config {

   _4

   String mAppId = "<YOUR_APP_ID>";

   _4

   String mToken = null;

   _4

   }

   
   

9. Connect an Android device (not an emulator) and run the project.

This section contains information that completes the information in this page, or points you to documentation that explains other aspects to this product.

This section lists the FaceUnity APIs you can use with Agora SDK.

The key corresponds to the name of the FaceUnity API, and the value corresponds to the parameters of the API. In this section, if the value is the same as the parameters of the FaceUnity API, the link leads to the FaceUnity documentation. If the value is different from the parameters of the FaceUnity API, the link leads to a section on this page.

Method keys refer to the keys you pass in when calling the setExtensionProperty or setExtensionPropertyWithVendor method of the Video SDK.

Method keys	Description
fuSetup	Initialize the extension and authenticate the user. Must be executed before other keys.
fuLoadAIModelFromPackage	Preload AI capabilities.
fuReleaseAIModel	Free up resources occupied by AI capabilities.

Method keys	Description
fuCreateItemFromPackage	Loads the prop package.
fuLoadTongueModel	Loads tongue detection data.
fuItemSetParam	Modifies or sets the value of a variable in the prop package.

Method keys	Description
fuDestroyItem	Destroys a specified item.
fuDestroyAllItems	Destroys all loaded items and releases all occupied resources.
fuOnDeviceLost	Resets the system's GL state. Use this key when the OpenGL context is released/destroyed by external resources.
fuDestroyLibData	Frees up the memory allocated to the face tracking module after calling fuSetup.

Method keys	Description
fuBindItems	Binds resource items to a target item.
fuUnbindItems	Unbinds the resource items from a target item.
fuIsTracking	Sets whether to get the number of faces being tracked.
fuSetMaxFaces	Sets the maximum number of tarcked faces.
fuSetDefaultRotationMode	Sets the default human face orientation.

Method keys	Description
fuFaceProcessorSetMinFaceRatio	Sets the distance of face detection.
fuSetTrackFaceAIType	Sets the fuTrackFace algorithm type.
fuSetFaceProcessorFov	Sets the fov (equivalent to focal length) of the FaceProcessor algorithm module.
fuHumanProcessorReset	Resets the state of the HumanProcessor algorithm module.
fuHumanProcessorSetMaxHumans	Sets the number of bodies tracked by the HumanProcessor algorithm module.
fuHumanProcessorGetNumResults	Sets whether to get the number of bodies tracked by the HumanProcessor algorithm module.
fuHumanProcessorSetFov	Sets the fov (equivalent to focal length) used by the HumanProcessor algorithm module to track 3D key points on human bodies.
fuHandDetectorGetResultNumHands	Sets whether to get the number of gestures tracked by the HandGesture algorithm module. Note that ai_gesture.bundle needs to be loaded.

Value parameters	Description
authdata	The path to the certificate file.

Value parameters	Description
data	String. The path of the AI capability model file ai_xxx.bundle. Such model files are located in the assets/AI_Model directory of the resource package.
type	Int. The AI capability type corresponding to the bundle file. Possible values are listed in enum FUAITYPE.

Value parameters	Description
data	String. The path to the prop package you want to load. A prop package usually has a suffix *.bundle .

Value parameters	Description
data	String. The path of tongue model data tongue.bundle.

Value parameters	Description
obj_handle	String. The path of the prop package passed in when calling fuCreateItemFromPackage.
name	String. The name of the variable to set in the prop package.
value	Object. The variable value to be set.

For details on the variable names and values in the prop package, refer to the FaceUnity documentation.

Value parameters	Description
item	String. The path of the prop package passed in when calling fuCreateItemFromPackage.

Value parameters	Description
obj_handle	String. The path of the target item.
p_items	String Array. The paths to the resource items you want to bind.

Value parameters	Description
obj_handle	String. The path of the target item.
p_items	String Array, the paths to the resource items you want to unbind.

Value parameters	Description
enable	Bool. Whether to get the number of faces being tracked. If set to true, you can receive the fuIsTracking callback.

Value parameters	Description
enable	Bool. Whether to get the number of human bodies tracked by the HumanProcessor algorithm module. If set to true, you can receive the fuHumanProcessorGetNumResults callback.

Value parameters	Description
enable	Bool. Whether to get the number of gestures tracked by the HandGesture algorithm module. If set to true, you can receive the fuHandDetectorGetResultNumHands callback.

​

Callback keys refer to the keys returned in the onEvent callback of the Agora SDK.

Callback keys	Description
fuIsTracking	Returns the number of faces being tracked.
fuHumanProcessorGetNumResults	Returns the number of human bodies tracked by the HumanProcessor algorithm module.
fuHandDetectorGetResultNumHands	Returns the number of gestures tracked by the HandGesture algorithm module.
fuDestroyLibData	Reports that the memory allocated to the face tracking module after calling fuSetup is released.

​

Value parameters	Description
faces	Int. The number of faces being tracked.

​

Value parameters	Description
people	Int. The number of bodies being tracked.

​

Value parameters	Description
hands	Int. The number of gestures being tracked.

​

Its value contains no parameters.

• setExtensionProperty

• addExtension in the RtcEngineConfig class

• enableExtension in the RtcEngineConfig class

• onEvent

This section describes the error codes returned when API calls fail, including the following two types:

• The error code at the SDK layer: Check the return value of the corresponding API.
• Error code at the extension layer: View it through the agorasdk.log file. For example, fuSetup ret: -1 means that an error occurred when calling setExtensionProperty/setExtensionPropertyWithVendor and passing in the key as fuSetup; the error code is -1.

API	Error Code	Description
enableExtension	-3 (SDK layer)	The SDK cannot find the corresponding extension dynamic library. The possible reasons are the following: The extension dynamic library is not correctly imported into the project or the passed extension name is incorrect.
setExtensionProperty	-1 (extension layer)	When the key is fuSetup: Repeated calls, fuSetup only needs to be called once. When the key is fuLoadAIModelFromPackage or fuCreateItemFromPackage: The passed value is wrong.
	-2 (SDK layer)	The timing of calling enableExtension is wrong.
	-2 (extension layer)	Parameter error. Possible reasons are: The input key or value is empty or the value cannot be parsed according to the JSON rules. Please refer to the specification in the document to pass the value.
	-20 (extension layer)	The incoming key is not supported yet. When the key is fuLoadAIModelFromPackage: The AI capability has already been loaded, and there is no need to load it again. When the key is fuDestroyItem: The item has been destroyed, and there is no need to destroy it again.