MetaKit

The MetaKit extension is an innovative product designed to enhance interactive video experiences. By integrating multiple advanced AI technologies, it provides users with creative and personalized video enhancement functions.

MetaKit can add rich video interactive effects, allowing you to choose flexibly, according to your specific requirements:

• Social entertainment: Enhance social entertainment and live broadcasts with features like Animoji and portrait edge flames, providing more creativity and personalization for hosts.
• Online education: Create a more vivid and engaging teaching environment with 360-degree backgrounds to enhance students' interest in learning.
• Online conferences: Use 3D lighting to create a presentation environment comparable to professional effects, enhancing the visual impact of your presentations.

The MetaKit extension includes the following key functions:

Function	Description
Virtual human	Easily generate virtual characters and create unique virtual images with custom options like face pinching and fashion dressup. Capture user expressions in real time and render them back on the virtual image to enhance interaction.
Animoji	Apply various Animoji effects to portraits in real time using AR and face capture technology. Show real-time changes in head dynamics and expressions to display a unique personality.
Lighting	Provide users with precise and efficient light and shadow effects, including 3D light (one light with customizable motion trajectory), atmosphere light (simulating multiple real light effects with fixed motion trajectory), advertising light, and other modes. Intelligent light and shadow control allows users to experience more realistic effects in a virtual environment.
Atmospheric effects	Create an artistic atmosphere using lighting effects, including portrait edge flames, aurora, ripples, and other modes.
360 Background	Provide users with customized panoramic virtual background effects.

info

The MetaKit extension offers an open art ecosystem, supporting one-click import of Animoji and avatar images created according to Agora art standards. This provides users with more flexible creation and integration options.

To use this feature, contact technical support.

The effects of some functions are as follows:

Virtual human (girl)

Animoji (dog)

Atmospheric effects (portrait edge flames)

Lighting (ambient light)

Lighting (3D light)

This page explains how to integrate MetaKit extension into your project to utilize the virtual human, Animoji, lighting effects, and 360 background functions.

To follow this procedure, you must have:

• Integrated the v4.2.x or v4.3.x of the Video SDK and implemented basic real-time audio and video functions in your app. See SDK quickstart.
  info

  • When integrating through Maven Central, specify io.agora.rtc:full-sdk:x.y.z and replace x.y.z with the specific SDK version number.
  • The MetaKit extension uses the Face Capture extension ( libagora_face_capture_extension.so) and the Virtual Background extension (libagora_segmentation_extension.so). You can delete unnecessary extensions as needed to reduce the size of the app.

• Android Studio v4.2 or above.

• An Android device model produced in 2019 or later, to ensure that the front camera and microphone are functioning properly.

• A computer that can access the Internet. If your network environment has a firewall deployed, refer to Firewall requirements to use the Agora services normally.

To implement MetaKit effects in your app, open the SDK quickstart for Video Calling project and take the steps described below.

To integrate the MetaKit extension, take the following steps:

1. Download and unzip the MetaKit Android extension.

2. Open the folder and copy the /sdk files in the path to the corresponding project path.

   Library	Function	Integration path
   AgoraMetaKit.aar	Rendering runtime layer	/app/libs
   metakit.jar	Wrapper layer Java package	/app/libs
   libagora_metakit_extension.so	Wrapper layer	/app/src/main/jniLibs/arm64-v8a or /app/src/main/jniLibs/armeabi-v7a

3. In the project's /app directory, add dependencies for all .jar and .aar files located under the libs path in the dependencies section of the build.gradle file.

   
   

   _1

   implementation fileTree(dir: 'libs', include: ['*.jar', '*.aar'])

   
   

To configure the extension, take the following steps:

1. Open the folder of the MetaKit extension for Android. The /assets/DefaultPackage path contains the Bundle file resources required for different scenes and functions. The table below lists the resource name, purpose, and size:

   Name	Required/Optional	Usage	Size
   Base	Required	Basic scene resources. Each functional module is built on this scene resource and includes related resources that support the hot update function.	2.38 MB
   Avatar	Function-specific	Virtual human model subpackage resources, including virtual human images such as girl and huamulan. Supports face capture, face pinching, and dress-up capabilities.	girl: 14.8 MB huamulan: 3.2 MB (does not support face pinching and dress-up)
   AvatarAnimoji	Function-specific	Animoji model subpackage resources, including Animoji images such as dog, girlhead, and arkit. Supports face capture.	dog: 1.4 MB girlhead: 954 KB arkit: 44 KB
   AREffect	Function-specific	Lighting effects and 360 background subpackage resources, including 3D lighting, atmosphere lighting, advertising lighting, screen ripples, aurora effects, portrait edge flames, and other effects.	3.97 MB

2. Combine the basic resources (Base) and the subpackage resources (Avatar, AvatarAnimoji, and AREffect) of specific functional modules into a complete resource package to experience the corresponding functional module. The functional modules and their corresponding resource package combinations are shown in the following table:

   Functional module	Resource package combination
   Virtual human	Base + Avatar
   Animoji	Base + AvatarAnimoji
   Lighting effects	Base + AREffect
   360 Background	Base + AREffect

3. To experience the virtual human and 360 background features, combine the Base, Avatar, and AREffect resources into a single directory, as shown below. After preparing the resource directory, place it in the SD card directory of the mobile device, such as /sdcard/metaAssets/15. When loading scene resources, set the absolute path of the resource directory to MetaKit.

   

To request the required permissions, take the following steps:

1. Navigate to the project's /app/src/main directory and add the following permissions to the AndroidManifest.xml file:

   
   

   _15

   <!-- Required Permissions -->

   _15

   <uses-permission android:name="android.permission.INTERNET"/>

   _15

   _15

   <!-- Optional Permissions -->

   _15

   <uses-permission android:name="android.permission.CAMERA"/>

   _15

   <uses-permission android:name="android.permission.RECORD_AUDIO"/>

   _15

   <uses-permission android:name="android.permission.MODIFY_AUDIO_SETTINGS"/>

   _15

   <uses-permission android:name="android.permission.ACCESS_WIFI_STATE"/>

   _15

   <uses-permission android:name="android.permission.ACCESS_NETWORK_STATE"/>

   _15

   <uses-permission android:name="android.permission.BLUETOOTH"/>

   _15

   <uses-permission android:name="android.permission.WRITE_EXTERNAL_STORAGE"/>

   _15

   <uses-permission android:name="android.permission.READ_EXTERNAL_STORAGE"/>

   _15

   <!-- For Android 12.0 and above, also add the following permissions -->

   _15

   <uses-permission android:name="android.permission.READ_PHONE_STATE"/>

   _15

   <uses-permission android:name="android.permission.BLUETOOTH_SCAN"/>

   
   

   The MetaKit extension primarily uses the following Android system permissions:

   Permissions	Function	Description
   CAMERA	Access your phone's camera.	Functions such as expression driving and background segmentation require access to the camera for AI reasoning.
   INTERNET	Access the network.	Authorize the AI module when the extension is enabled.
   READ_EXTERNAL_STORAGE	Read external storage.	Read the Bundle resource file from the SD card.
   WRITE_EXTERNAL_STORAGE	Write to external storage.	Record SDK-related log files.

2. Android 6.0 and later versions enforce stricter permission management. Besides declaring permissions statically in AndroidManifest.xml, certain permissions must also be requested dynamically within the application's business logic. Here's an example of how this can be implemented:

   Java

   // Obtain the necessary permissions for real-time audio-video interactionprivate String[] getRequiredPermissions() {     if (android.os.Build.VERSION.SDK_INT >= android.os.Build.VERSION_CODES.S) {         // Permissions required for Android 12 (S) and above         return new String[]{                 Manifest.permission.RECORD_AUDIO, // Audio recording permission                 Manifest.permission.CAMERA, // Camera permission                 Manifest.permission.READ_PHONE_STATE, // Read phone state permission                 Manifest.permission.READ_EXTERNAL_STORAGE, // Read external storage permission                 Manifest.permission.WRITE_EXTERNAL_STORAGE // Write external storage permission         };     } else {         // Permissions required for Android 11 (R) and below         return new String[]{                 Manifest.permission.RECORD_AUDIO,                 Manifest.permission.CAMERA,                 Manifest.permission.READ_EXTERNAL_STORAGE,                 Manifest.permission.WRITE_EXTERNAL_STORAGE         };     }}// Check if the app has obtained all required permissionsprivate boolean checkPermissions() {     for (String permission : getRequiredPermissions()) {         int permissionCheck = ContextCompat.checkSelfPermission(this, permission);         if (permissionCheck != PackageManager.PERMISSION_GRANTED) {             return false;         }     }     return true;}

   Kotlin

   // Obtain the necessary permissions for real-time audio-video interactionprivate fun getRequiredPermissions(): Array<String> {     return if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.S) {         // Permissions required for Android 12 (S) and above         arrayOf(             Manifest.permission.RECORD_AUDIO, // Audio recording permission             Manifest.permission.CAMERA, // Camera permission             Manifest.permission.READ_PHONE_STATE, // Read phone state permission             Manifest.permission.READ_EXTERNAL_STORAGE, // Read external storage permission             Manifest.permission.WRITE_EXTERNAL_STORAGE // Write external storage permission         )     } else {         // Permissions required for Android 11 (R) and below         arrayOf(             Manifest.permission.RECORD_AUDIO,             Manifest.permission.CAMERA,             Manifest.permission.READ_EXTERNAL_STORAGE,             Manifest.permission.WRITE_EXTERNAL_STORAGE         )     }}// Check if the app has obtained all required permissionsprivate fun checkPermissions(): Boolean {     return getRequiredPermissions().all { permission ->         ContextCompat.checkSelfPermission(this, permission) == PackageManager.PERMISSION_GRANTED     }}

The MetaKit extension currently supports the arm64-v8a and armeabi-v7a architectures. To optimize the app size, it's advisable to select only the necessary architecture during integration. Here's an example of how this can be implemented:

Java

ndk {    abiFilters "arm64-v8a"}

Kotlin

ndk {    abiFilters("arm64-v8a")}

Once the project configuration is complete, follow these steps to explore the various functional modules of the MetaKit extension:

When calling createInitialize on RtcEngine, ensure the following configurations are performed in RtcEngineConfig:

1. Call addExtension with AgoraFaceCapturePlugin (agora_face_capture_extension) and MetaKitPlugin (agora_metakit_extension). Then, implement the event callback interface IMediaExtensionObserver for extensions and register it for onEvent extension event callbacks.

   Java

   // Configure RtcEngineConfigRtcEngineConfig config = new RtcEngineConfig();config.mContext = getBaseContext();config.mAppId = appId;config.mEventHandler = mRtcEventHandler;// Add Face Capture extensionconfig.addExtension("agora_face_capture_extension");// Add MetaKit extensionconfig.addExtension("agora_metakit_extension");// Create the event callback interface class for extensions and register callbacks for extension events such as onEventconfig.mExtensionObserver = new IMediaExtensionObserver() {     public void onEvent(String provider, String extension, String key, String value) {     // Implementation of onEvent callback     }     public void onStarted(String provider, String extension) {     // Implementation of onStarted callback     }     public void onStopped(String provider, String extension) {     // Implementation of onStopped callback     }     public void onError(String provider, String extension, int error, String message) {     // Implementation of onError callback     }};// Create and initialize RtcEnginemRtcEngine = RtcEngine.create(config);

   Kotlin

   // Configure RtcEngineConfigval config = RtcEngineConfig().apply {     mContext = baseContext     mAppId = appId     mEventHandler = mRtcEventHandler     // Add Face Capture extension     addExtension("agora_face_capture_extension")     // Add MetaKit extension     addExtension("agora_metakit_extension")     // Create the event callback interface class for extensions and register callbacks for extension events such as onEvent     mExtensionObserver = object : IMediaExtensionObserver {         override fun onEvent(provider: String, extension: String, key: String, value: String) {             // Implementation of onEvent callback         }         override fun onStarted(provider: String, extension: String) {             // Implementation of onStarted callback         }         override fun onStopped(provider: String, extension: String) {             // Implementation of onStopped callback         }         override fun onError(provider: String, extension: String, error: Int, message: String) {             // Implementation of onError callback         }     } }// Create and initialize RtcEnginemRtcEngine = RtcEngine.create(config)

2. In the callback, specify provider as agora_video_filters_metakit and extension as metakit to filter events from the MetaKit extension. The onEvent event transmits engine status events transparently, such as unityLoadFinish (Unity environment loading completed) and loadSceneResp (scene resource loading completed).

   Java

   public void onEvent(String provider, String ext, String key, String msg) {     // Filter events from the MetaKit extension     if (!provider.equals("agora_video_filters_metakit") || !ext.equals("metakit")) return;     // Log event details     Log.i(TAG, "metakitx onEvent: " + key + ", msg: " + msg);     // Handle different event keys     switch(key) {         case "initializeFinish":             runningState = IMetaRunningState.initialized;             break;         // Unity environment loaded         case "unityLoadFinish":             runningState = IMetaRunningState.unityLoaded;             Log.d(TAG, "metakitx to enter scene");             enterScene();             break;         // Scene resource loaded         case "loadSceneResp":             Log.d(TAG,"metakitx receive loadSceneResp");             runningState = IMetaRunningState.sceneLoaded;             setMetaFeatureMode(curFeatrueType);             break;         case "addSceneViewResp":             runningState = IMetaRunningState.sceneViewLoaded;             // If special effects are set, configure background and effects             if (setSpecialEffect) {                 setMetaBGMode(BackgroundType.BGTypePano);                 configMetaBackgroundEffectMode(curSpecialEffectType, true);             }             break;         case "unloadSceneResp":             runningState = IMetaRunningState.sceneUnloaded;             // Perform scene cleanup if necessary             //destroyScene();             break;     }     isSyncing = false; } public void onError(String provider, String ext, int key, String msg) { // Filter errors from the MetaKit extension if (!provider.equals("agora_video_filters_metakit") || !ext.equals("metakit")) return;     // Log error details     Log.i("[MetaKit]", "onError: " + key + ", msg: " + msg); } public void onStart(String provider, String ext) { // Filter start events from the MetaKit extension if (!provider.equals("agora_video_filters_metakit") || !ext.equals("metakit")) return;     // Log start event     Log.i("[MetaKit]", "onStart"); } public void onStop(String provider, String ext) { // Filter stop events from the MetaKit extension if (!provider.equals("agora_video_filters_metakit") || !ext.equals("metakit")) return;     // Log stop event     Log.i("[MetaKit]", "onStop"); }

   Kotlin

   fun onEvent(provider: String, ext: String, key: String, msg: String) {     // Filter events from the MetaKit extension     if (provider != "agora_video_filters_metakit" || ext != "metakit") return     // Log event details     Log.i(TAG, "metakitx onEvent: $key, msg: $msg")     // Handle different event keys     when (key) {         "initializeFinish" -> runningState = IMetaRunningState.initialized         // Unity environment loaded         "unityLoadFinish" -> {             runningState = IMetaRunningState.unityLoaded             Log.d(TAG, "metakitx to enter scene")             enterScene()         }         // Scene resource loaded         "loadSceneResp" -> {             Log.d(TAG, "metakitx receive loadSceneResp")             runningState = IMetaRunningState.sceneLoaded             setMetaFeatureMode(curFeatrueType)         }         "addSceneViewResp" -> {             runningState = IMetaRunningState.sceneViewLoaded             // If special effects are set, configure background and effects             if (setSpecialEffect) {                 setMetaBGMode(BackgroundType.BGTypePano)                 configMetaBackgroundEffectMode(curSpecialEffectType, true)             }         }         "unloadSceneResp" -> runningState = IMetaRunningState.sceneUnloaded     }     isSyncing = false } fun onError(provider: String, ext: String, key: Int, msg: String) { // Filter errors from the MetaKit extension if (provider != "agora_video_filters_metakit" || ext != "metakit") return     // Log error details     Log.i("[MetaKit]", "onError: $key, msg: $msg") } fun onStart(provider: String, ext: String) { // Filter start events from the MetaKit extension if (provider != "agora_video_filters_metakit" || ext != "metakit") return     // Log start event     Log.i("[MetaKit]", "onStart") } fun onStop(provider: String, ext: String) { // Filter stop events from the MetaKit extension if (provider != "agora_video_filters_metakit" || ext != "metakit") return     // Log stop event     Log.i("[MetaKit]", "onStop") }

Before enabling the MetaKit extension, ensure that both the Facial Capture extension and the Virtual Background extension are enabled.

To enable the Face Capture extension, follow these steps:

1. Call registerExtension and enableExtension with the provider name agora_video_filters_face_capture and the extension name face_capture.

   Java

   // Register the facial capture extension mRtcEngine.registerExtension("agora_video_filters_face_capture", "face_capture", Constants.MediaSourceType.PRIMARY_CAMERA_SOURCE); // Enable the facial capture extension mRtcEngine.enableExtension("agora_video_filters_face_capture", "face_capture", true);

   Kotlin

   // Register the facial capture extension mRtcEngine.registerExtension("agora_video_filters_face_capture", "face_capture", Constants.MediaSourceType.PRIMARY_CAMERA_SOURCE) // Enable the facial capture extension mRtcEngine.enableExtension("agora_video_filters_face_capture", "face_capture", true)

2. Call setExtensionProperty to authenticate and authorize the extension. Use authentication_information as the key, and a value containing the company name (company_id) and the face capture certificate (license).

   Java

   mRtcEngine.setExtensionProperty("agora_video_filters_face_capture", "face_capture", "authentication_information",     "{"company_id":"agoraDemo"," +         ""license":"" +         "xxxxxxxxxx"}", Constants.MediaSourceType.PRIMARY_CAMERA_SOURCE);

   Kotlin

   mRtcEngine.setExtensionProperty("agora_video_filters_face_capture", "face_capture", "authentication_information",     "{"company_id":"agoraDemo"," +         ""license":"" +         "xxxxxxxxxx"}", Constants.MediaSourceType.PRIMARY_CAMERA_SOURCE)
   Info

   Contact Agora to obtain the company name and certificate.

To enable the Virtual Background extension, take the following steps:

1. Call setParameters to set "rtc.video.seg_before_exts" to true:

   Java

   mRtcEngine.setParameters("{"rtc.video.seg_before_exts":true}");

   Kotlin

   mRtcEngine.setParameters("{"rtc.video.seg_before_exts":true}")

2. Call enableVirtualBackground with the following configurations:

   • Set backgroundSourceType to 0 to process the background into alpha information, separating the portrait from the background.

   • Set modelType to 1 to select background processing suitable for all scenes.

     Java

     VirtualBackgroundSource source = new VirtualBackgroundSource();  // Set backgroundSourceType to 0 to process the background into alpha information, separating the portrait from the background  source.backgroundSourceType = 0;  source.color = 0xFFFFFF;  source.source = "";  source.blurDegree = 1;  SegmentationProperty param = new SegmentationProperty();  // Set modelType to 1 to select background processing suitable for all scenes  param.modelType = 1;  param.greenCapacity = 0.5f;  // Enable the Virtual Background extension  mRtcEngine.enableVirtualBackground(true, source, param, Constants.MediaSourceType.PRIMARY_CAMERA_SOURCE);

     Kotlin

     val source = VirtualBackgroundSource().apply {      // Set backgroundSourceType to 0 to process the background into alpha information, separating the portrait from the background      backgroundSourceType = 0      color = 0xFFFFFF      source = ""      blurDegree = 1  }  val param = SegmentationProperty().apply {  // Set modelType to 1 to select background processing suitable for all scenes  modelType = 1  greenCapacity = 0.5f  }  // Enable the Virtual Background extension  mRtcEngine.enableVirtualBackground(true, source, param, Constants.MediaSourceType.PRIMARY_CAMERA_SOURCE)

To enable the MetaKit extension, follow these steps:

1. Call registerExtension with the service provider name agora_video_filters_metakit and the extension name metakit.

   Java

   mRtcEngine.registerExtension("agora_video_filters_metakit", "metakit", Constants.MediaSourceType.PRIMARY_CAMERA_SOURCE);

   Kotlin

   mRtcEngine.registerExtension("agora_video_filters_metakit", "metakit", Constants.MediaSourceType.PRIMARY_CAMERA_SOURCE)

2. Call enableExtension with the same service provider name and extension name.

   Java

   mRtcEngine.enableExtension("agora_video_filters_metakit", "metakit", true);

   Kotlin

   mRtcEngine.enableExtension("agora_video_filters_metakit", "metakit", true)

1. To set the Android activity context for starting the rendering engine, call setExtensionProperty with the following parameters:

   • key: setActivityContext
   • value: The activity context address
   Java

   Activity mActivity; JSONObject valueObj = new JSONObject(); try {     long address = getContextHandler(mActivity);     valueObj.put("activityContext", String.valueOf(address)); } catch (JSONException e) {     e.printStackTrace(); } mRtcEngine.setExtensionProperty("agora_video_filters_metakit", "metakit", "setActivityContext", valueObj.toString());

   Kotlin

   var mActivity: Activity? = null val valueObj = JSONObject() try {     val address = getContextHandler(mActivity)     valueObj.put("activityContext", address.toString()) } catch (e: JSONException) {     e.printStackTrace() } mRtcEngine.setExtensionProperty("agora_video_filters_metakit", "metakit", "setActivityContext", valueObj.toString())

2. To initialize the MetaKit extension, call setExtensionProperty with the following parameters:

   • key: initialize
   • value: an empty string
   Java

   mRtcEngine.setExtensionProperty("agora_video_filters_metakit", "metakit", "initialize","{}");

   Kotlin

   mRtcEngine.setExtensionProperty("agora_video_filters_metakit", "metakit", "initialize","{}")

1. When the onEvent callback captures the unityLoadFinish event, it indicates that the environment has been loaded. At this point, you can call setExtensionProperty to load the MetaKit scene resources. Use the following parameters:

   • key: loadScene
   • value: A string containing relevant information about the scene resources
   Java

   JSONObject valueObj = new JSONObject(); try {     JSONObject sceneObj = new JSONObject();     // Set the path of the scene resources on the phone     // Assume the resources are stored at /first/second/DefaultPackage/ on the phone; only /first/second needs to be specified in scenePath     sceneObj.put("scenePath", "/sdcard/metaAssets/15");     JSONObject customObj = new JSONObject();     // Set the scene index to 0     customObj.put("sceneIndex", 0);     valueObj.put("sceneInfo", sceneObj);     valueObj.put("assetManifest", "");     valueObj.put("userId", "123456");     valueObj.put("extraCustomInfo", customObj.toString()); } catch (JSONException e) {     e.printStackTrace(); } // Load scene resources based on the JSON configuration mRtcEngine.setExtensionProperty("agora_video_filters_metakit", "metakit", "loadScene", valueObj.toString());

   Kotlin

   val valueObj = JSONObject() try {     val sceneObj = JSONObject()     // Set the path of the scene resources on the phone     // Assume the resources are stored at /first/second/DefaultPackage/ on the phone; only /first/second needs to be specified in scenePath     sceneObj.put("scenePath", "/sdcard/metaAssets/15")     val customObj = JSONObject()     // Set the scene index to 0     customObj.put("sceneIndex", 0)     valueObj.put("sceneInfo", sceneObj)     valueObj.put("assetManifest", "")     valueObj.put("userId", "123456")     valueObj.put("extraCustomInfo", customObj.toString()) } catch (e: JSONException) {     e.printStackTrace() } // Load scene resources based on the JSON configuration mRtcEngine.setExtensionProperty("agora_video_filters_metakit", "metakit", "loadScene", valueObj.toString())

   1. When the onEvent callback captures the loadSceneResp event, it indicates that the scene resources have been loaded. You can then follow these steps to experience the virtual human, Animoji, lighting effects, and 360 background modules.

      Java

      JSONObject valueObj = new JSONObject(); try {     JSONObject configObj = new JSONObject();     configObj.put("key", "bsname"); // The key is the resource ID of the face pinching part     configObj.put("value", 30); // The value is the corresponding intensity of the face pinching, ranging from [0,100], with a default of 50     valueObj.put("value", configObj); } catch (JSONException e) {     e.printStackTrace(); } // Perform face pinching operation based on JSON configuration mRtcEngine.setExtensionProperty("agora_video_filters_metakit", "metakit", "updateFace", valueObj.toString());

      Kotlin

      val valueObj = JSONObject() try {     val configObj = JSONObject()     configObj.put("key", "bsname") // The key is the resource ID of the face pinching part     configObj.put("value", 30) // The value is the corresponding intensity of the face pinching, ranging from [0,100], with a default of 50     valueObj.put("value", configObj) } catch (e: JSONException) {     e.printStackTrace() } // Perform face pinching operation based on JSON configuration mRtcEngine.setExtensionProperty("agora_video_filters_metakit", "metakit", "updateFace", valueObj.toString())

1. Call setExtensionProperty to request texture and render the virtual human scene. Set key to requestTexture and value to include the scene configuration information. To experience the virtual human feature, set avatarMode to 0 for the virtual human scene mode and specify the avatar as your desired virtual human image, such as girl or huamulan.

   Note

   In addition to the default avatars, girl and huamulan, the Agora MetaKit extension offers an open artistic ecosystem. It supports one-click import of virtual human models created according to Agora's art standards, providing users with more flexible creation and integration options. Contact Agora technical support to use this feature.

   Java

   JSONObject valueObj = new JSONObject(); try {     valueObj.put("index", 0); // Texture index, currently only supports 0     valueObj.put("enable", true); // Enable texture request     JSONObject configObj = new JSONObject();     configObj.put("width", 640);     configObj.put("height", 480);     JSONObject extraObj = new JSONObject();     extraObj.put("sceneIndex", 0); // Scene index, currently only supports 0     extraObj.put("avatarMode", 0); // Set scene mode to 0, which is virtual human mode     extraObj.put("avatar", "huamulan"); // Set the virtual human image to "huamulan"     extraObj.put("userId", "123");     configObj.put("extraInfo", extraObj.toString());     valueObj.put("config", configObj); } catch (JSONException e) {     e.printStackTrace(); } // Render the virtual human scene based on the JSON configuration mRtcEngine.setExtensionProperty("agora_video_filters_metakit", "metakit", "requestTexture", valueObj.toString());

   Kotlin

   val valueObj = JSONObject() try {     valueObj.put("index", 0) // Texture index, currently only supports 0     valueObj.put("enable", true) // Enable texture request     val configObj = JSONObject()     configObj.put("width", 640)     configObj.put("height", 480)     val extraObj = JSONObject()     extraObj.put("sceneIndex", 0) // Scene index, currently only supports 0     extraObj.put("avatarMode", 0) // Set scene mode to 0, which is virtual human mode     extraObj.put("avatar", "huamulan") // Set the virtual human image to "huamulan"     extraObj.put("userId", "123")     configObj.put("extraInfo", extraObj.toString())     valueObj.put("config", configObj) } catch (e: JSONException) {     e.printStackTrace() } // Render the virtual human scene based on the JSON configuration mRtcEngine.setExtensionProperty("agora_video_filters_metakit", "metakit", "requestTexture", valueObj.toString())

   After the scene rendering is complete, a Blendshape-driven virtual human image will be displayed, capturing your facial expressions and making corresponding facial changes, following your head movements.

2. Call setExtensionProperty to perform face pinching operations on the virtual human. Set key to updateFace and value to support passing multiple sets of resource IDs for face pinching parts and their corresponding adjustment ranges. See face pinching for details.

   Java

   JSONObject valueObj = new JSONObject(); try {     JSONObject configObj = new JSONObject();     configObj.put("key", "bsname"); // The key is the resource ID of the face pinching part     configObj.put("value", 30); // The value is the corresponding intensity of the face pinching, ranging from [0,100], with a default of 50     valueObj.put("value", configObj); } catch (JSONException e) {     e.printStackTrace(); } // Perform face pinching operations based on the JSON configuration mRtcEngine.setExtensionProperty("agora_video_filters_metakit", "metakit", "updateFace", valueObj.toString());

   Kotlin

   val valueObj = JSONObject() try {     val configObj = JSONObject()     configObj.put("key", "bsname") // The key is the resource ID of the face pinching part     configObj.put("value", 30) // The value is the corresponding intensity of the face pinching, ranging from [0,100], with a default of 50     valueObj.put("value", configObj) } catch (e: JSONException) {     e.printStackTrace() } // Perform face pinching operations based on the JSON configuration mRtcEngine.setExtensionProperty("agora_video_filters_metakit", "metakit", "updateFace", valueObj.toString())

3. Call setExtensionProperty to perform dress-up operations on the virtual human. Set key to updateDress and value to support passing an array of integers containing multiple resource IDs for dressing parts. See dressing resources for details.

   Java

   JSONObject valueObj = new JSONObject(); try {     valueObj.put("id", "[10002]"); // Set the ID to an array of integers containing multiple resource IDs } catch (JSONException e) {     e.printStackTrace(); } // Perform dressing operations based on the JSON configuration mRtcEngine.setExtensionProperty("agora_video_filters_metakit", "metakit", "updateDress", valueObj.toString());

   Kotlin

   val valueObj = JSONObject() try {     valueObj.put("id", "[10002]") // Set the ID to an array of integers containing multiple resource IDs } catch (e: JSONException) {     e.printStackTrace() } // Perform dressing operations based on the JSON configuration mRtcEngine.setExtensionProperty("agora_video_filters_metakit", "metakit", "updateDress", valueObj.toString())

Call setExtensionProperty to request the texture and render the Animoji scene. Set key to requestTexture, which includes the scene configuration information. To experience the Animoji function, set avatarMode to 1 for Animoji scene mode. Specify avatar to the Animoji image you want to use, such as dog, girl, or headarkit.

Info

In addition to the already available Animoji images (dog, girl, headarkit), the Agora MetaKit extension provides an open art ecosystem. It supports one-click import of Animoji images created according to Agora's art standards, offering users more flexible creation and integration options. Contact Agora technical support to use this feature.

Java

JSONObject valueObj = new JSONObject(); try {    valueObj.put("index", 0); // Texture index, currently only supports 0    valueObj.put("enable", true); // Enable texture request    JSONObject configObj = new JSONObject();     configObj.put("width", 640);    configObj.put("height", 480);    JSONObject extraObj = new JSONObject();     extraObj.put("sceneIndex", 0); // Scene index, currently only supports 0    extraObj.put("avatarMode", 1); // Set scene mode to 1, which is Animoji mode    extraObj.put("avatar", "dog"); // Set Animoji image to "dog"    extraObj.put("userId", "123");     configObj.put("extraInfo", extraObj.toString());    valueObj.put("config", configObj);} catch (JSONException e) {    e.printStackTrace();}// Render the Animoji scene based on the JSON configurationmRtcEngine.setExtensionProperty("agora_video_filters_metakit", "metakit", "requestTexture", valueObj.toString());

Kotlin

val valueObj = JSONObject() try {    valueObj.put("index", 0) // Texture index, currently only supports 0    valueObj.put("enable", true) // Enable texture request    val configObj = JSONObject()     configObj.put("width", 640)     configObj.put("height", 480)    val extraObj = JSONObject()     extraObj.put("sceneIndex", 0) // Scene index, currently only supports 0    extraObj.put("avatarMode", 1) // Set scene mode to 1, which is Animoji mode    extraObj.put("avatar", "dog") // Set Animoji image to "dog"    extraObj.put("userId", "123")     configObj.put("extraInfo", extraObj.toString())    valueObj.put("config", configObj)} catch (e: JSONException) {    e.printStackTrace()}// Render the Animoji scene based on the JSON configurationmRtcEngine.setExtensionProperty("agora_video_filters_metakit", "metakit", "requestTexture", valueObj.toString())

Call setExtensionProperty to request the texture and render the sticker scene. Set key to loadMaterial and value to the material configuration. Specify the corresponding resource name depending on the sticker that you want to use. For example, material_sticker_glass for glasses.

Info

In addition to the already available stickers veil, glass, facemask, and dragonhat, the Agora MetaKit extension provides an open art ecosystem and supports one-click import of sticker images created according to Agora's art standards. This offers users more flexible creation and integration options. Contact Agora technical support to use this feature.

Java

long addressHandle = 0;JSONObject valueObj = new JSONObject();try {    valueObj.put("view", String.valueOf(addressHandle));    valueObj.put("path", path_to_material_sticker_glass);} catch (JSONException e) {    e.printStackTrace();}mRtcEngine.setExtensionProperty("agora_video_filters_metakit", "metakit", "loadMaterial", valueObj.toString());

Kotlin

var addressHandle: Long = 0val valueObj = JSONObject()try {    valueObj.put("view", addressHandle.toString())    valueObj.put("path", path_to_material_sticker_glass)} catch (e: JSONException) {    e.printStackTrace()}mRtcEngine.setExtensionProperty("agora_video_filters_metakit", "metakit", "loadMaterial", valueObj.toString())

When the onEvent callback captures the materialLoaded event, it means that the scene view has been added. At this time, a glasses sticker covering the eyes will be displayed in the view, following your head movements.

1. Call setExtensionProperty to request the texture and render a scene with lighting effects and 360 background features. The key is requestTexture, and the value contains the configuration information of the scene. To experience lighting effects and the 360 background feature, set avatarMode to 2, which corresponds to lighting effects and 360 background mode.

   Java

   JSONObject valueObj = new JSONObject();  try {     valueObj.put("index", 0); // Texture index, currently only supports 0     valueObj.put("enable", true); // Enable texture request     JSONObject configObj = new JSONObject();      configObj.put("width", 640);      configObj.put("height", 480);     JSONObject extraObj = new JSONObject();      extraObj.put("sceneIndex", 0); // Scene index, currently only supports 0     extraObj.put("avatarMode", 2); // Set scene mode to 2, which is lighting effects and 360 background mode     extraObj.put("userId", "123");      configObj.put("extraInfo", extraObj.toString());           valueObj.put("config", configObj); } catch (JSONException e) {     e.printStackTrace(); } // Render the scene with lighting effects and 360 background based on the JSON configuration mRtcEngine.setExtensionProperty("agora_video_filters_metakit", "metakit", "requestTexture", valueObj.toString());

   Kotlin

   val valueObj = JSONObject()  try {     valueObj.put("index", 0) // Texture index, currently only supports 0     valueObj.put("enable", true) // Enable texture request     val configObj = JSONObject()      configObj.put("width", 640)      configObj.put("height", 480)     val extraObj = JSONObject()      extraObj.put("sceneIndex", 0) // Scene index, currently only supports 0     extraObj.put("avatarMode", 2) // Set scene mode to 2, which is lighting effects and 360 background mode     extraObj.put("userId", "123")      configObj.put("extraInfo", extraObj.toString())           valueObj.put("config", configObj) } catch (e: JSONException) {     e.printStackTrace() } // Render the scene with lighting effects and 360 background based on the JSON configuration mRtcEngine.setExtensionProperty("agora_video_filters_metakit", "metakit", "requestTexture", valueObj.toString())

2. Experience lighting effects and 360 background.

   1. Lighting effects:

      Call setExtensionProperty to set up lighting effects. The key is setEffectVideo, and the value contains a series of lighting materials and their corresponding parameter configurations. The MetaKit extension provides lighting effects such as 3D lighting, screen ripples, aurora effects, and portrait edge flames, and supports fine-tuning of parameters such as color, intensity, and range. See the Lighting effects key-value documentation for more details. The example code below demonstrates how to overlay advertising lights on a live video.

      Java

      JSONObject configObj = new JSONObject(); try {     configObj.put("id", 3002); // Specify the effect material ID as 3002, which is advertising lights     configObj.put("enable", true); // Enable lighting effect } catch (JSONException e) {     e.printStackTrace(); } // Add advertising light effect based on the JSON configuration mRtcEngine.setExtensionProperty("agora_video_filters_metakit", "metakit", "setEffectVideo", configObj.toString());

      Kotlin

      val configObj = JSONObject() try {     configObj.put("id", 3002) // Specify the effect material ID as 3002, which is advertising lights     configObj.put("enable", true) // Enable lighting effect } catch (e: JSONException) {     e.printStackTrace() } // Add advertising light effect based on the JSON configuration mRtcEngine.setExtensionProperty("agora_video_filters_metakit", "metakit", "setEffectVideo", configObj.toString())

   2. 360 background:

      Call setExtensionProperty to set up a 360 panoramic background. The key is setBGVideo, and the value sets the background mode, resource path, and rotation angle.

      Java

      JSONObject picObj = new JSONObject(); try {     picObj.put("mode", "tex360"); // Set background mode to 360 panoramic background mode     JSONObject configObj = new JSONObject();     configObj.put("path", "/sdcard/metaFiles/bg_pano.jpg"); // Specify the file path of the background resource     picObj.put("param", configObj); } catch (JSONException e) {     e.printStackTrace(); } // Add 360 background based on the JSON configuration mRtcEngine.setExtensionProperty("agora_video_filters_metakit", "metakit", "setBGVideo", picObj.toString());

      Kotlin

      val picObj = JSONObject() try {     picObj.put("mode", "tex360") // Set background mode to 360 panoramic background mode     val configObj = JSONObject()     configObj.put("path", "/sdcard/metaFiles/bg_pano.jpg") // Specify the file path of the background resource     picObj.put("param", configObj) } catch (e: JSONException) {     e.printStackTrace() } // Add 360 background based on the JSON configuration mRtcEngine.setExtensionProperty("agora_video_filters_metakit", "metakit", "setBGVideo", picObj.toString())

      You can also call setExtensionProperty to enable the gyroscope, specify key as setCameraGyro, and enable the gyroscope function in the value to further enhance the interactivity and immersion of the background.

      Java

      JSONObject gyroObj = new JSONObject(); try {     gyroObj.put("state", "on"); // Enable gyroscope function } catch (JSONException e) {     e.printStackTrace(); } // Enable gyroscope function based on the JSON configuration mRtcEngine.setExtensionProperty("agora_video_filters_metakit", "metakit", "setCameraGyro", gyroObj.toString());

      Kotlin

      val gyroObj = JSONObject() try {     gyroObj.put("state", "on") // Enable gyroscope function } catch (e: JSONException) {     e.printStackTrace() } // Enable gyroscope function based on the JSON configuration mRtcEngine.setExtensionProperty("agora_video_filters_metakit", "metakit", "setCameraGyro", gyroObj.toString())

      After successfully setting this effect, you can see that the video background is replaced with the specified resource, and you can experience the panoramic effect by rotating the phone. For more configurations, see the 360 Background key-value documentation.

When you are done using the extension, you can follow the sample code below to stop texture requests, unload scene resources, and destroy the engine.

Java

// 1. Stop texture requestsJSONObject valueObj = new JSONObject();try {    valueObj.put("index", 0); // Texture index, currently only supports setting to 0    valueObj.put("enable", false); // Set enable to false to stop the texture request feature} catch (JSONException e) {    e.printStackTrace();}mRtcEngine.setExtensionProperty("agora_video_filters_metakit", "metakit", "requestTexture", valueObj.toString());// 2. Unload scene resourcesmRtcEngine.setExtensionProperty("agora_video_filters_metakit", "metakit", "unloadScene", "{}");// 3. Destroy the enginemRtcEngine.setExtensionProperty("agora_video_filters_metakit", "metakit", "destroy", "{}");

Kotlin

// 1. Stop texture requestsval valueObj = JSONObject()try {    valueObj.put("index", 0) // Texture index, currently only supports setting to 0    valueObj.put("enable", false) // Set enable to false to stop the texture request feature} catch (e: JSONException) {    e.printStackTrace()}mRtcEngine.setExtensionProperty("agora_video_filters_metakit", "metakit", "requestTexture", valueObj.toString())// 2. Unload scene resourcesmRtcEngine.setExtensionProperty("agora_video_filters_metakit", "metakit", "unloadScene", "{}")// 3. Destroy the enginemRtcEngine.setExtensionProperty("agora_video_filters_metakit", "metakit", "destroy", "{}")

This section completes the information on this page, or points you to documentation that explains other aspects about this product.

To implement the capabilities of the MetaKit extension, use the setExtensionProperty method provided by the Agora Video SDK v4.x. Pass in the specified key and value as follows:

• key: Corresponds to different interfaces of the MetaKit extension.
• value: Encapsulates some or all of the interface parameters in the JSON format.

This guide explains how to use different key-value pairs to implement the MetaKit extension's virtual human, Animoji, lighting effects, and 360 background function modules.

This section covers how to implement the basic functions of the MetaKit extension, such as initialization, loading scene resources, enabling texture requests, switching texture scene modes, and avatars. Once you have implemented the basic functions, you can explore the specific functional modules.

• key: setActivityContext
• value: Object. Contains the following field:
  • activityContext: String. The address of the activity context.

• key: initialize
• value: {}

• key: loadScene
• value: Object. Contains the following fields:
  • sceneInfo: Object. Contains the following field:
    • scenePath: String. The path of the scene asset package, for example, "/sdcard/metaAssets/15".
  • extraCustomInfo: Object. Contains the following field:
    • sceneIndex: Int. The index of the scene, currently only supports 0.

Request a texture and render the specified scene content on the texture. This includes virtual humans, Animoji, lighting effects, and 360 backgrounds.

• key: requestTexture
• value: Object. Contains the following fields:
  • index: Int. Texture index, currently only supports 0.
  • enable: Boolean. Whether to enable the texture request. true: Enable; false: Disable (default).
  • config: Object. Contains the following fields:
    • width: Int. The width of the view (px). Set this to the current camera acquisition resolution, the width and height of the screen layout, or a common resolution like 720 × 1280.
    • height: Int. The height of the view (px). Set this to the current camera acquisition resolution, the width and height of the screen layout, or a common resolution like 720 × 1280.
    • extraInfo: Object. Contains the following fields:
      • sceneIndex: (optional) Int. Scene index, currently only supports 0.
      • avatarMode: (optional) Int. Scene mode. 0: Avatar (default); 1: Animoji; 2: Light or background.
      • avatar: (optional) String. Avatar or Animoji image. If avatarMode is 0 (avatar), set to girl or huamulan (default is girl); if avatarMode is 1 (Animoji), set to dog, girlhead, or arkit (default is dog).

Note

The requestTexture and addSceneView methods can both be used to render a specified scene on TextureView. Agora recommends using requestTexture for better rendering performance and lower latency. The differences are as follows:

• requestTexture does not require passing the render target TextureView to the MetaKit extension; it automatically generates and sends back texture data. addSceneView requires manual creation and management of TextureView.
• With requestTexture, the obtained texture is directly rendered, previewed, encoded by the SDK, and transmitted to the remote end. addSceneView requires an additional call to enableSceneVideo to enable scene screen capture.
• requestTexture supports a single view; addSceneView supports multiple views.
• For scene mode or avatar switching, use requestTexture to request textures and switchTextureAvatarMode for scene switching. Use addSceneView to add scene views and switchAvatarMode to complete scene switching.
• To release scene resources, use requestTexture and set enable to false to stop texture requests. If you added a scene view using addSceneView, use removeSceneView to remove it.

After enabling texture requests, switch the scene mode of the texture view, or the virtual human or Animoji image in the scene.

• key: switchTextureAvatarMode
• value: Object. Contains the following fields:
  • index: Int. Texture index, currently only supports 0.
  • mode: (optional) Int. Scene mode to switch to. 0: Avatar; 1: Animoji; 2: Video capture screen.
  • avatar: (optional) String. Avatar or Animoji to switch to. If avatarMode is 0 (avatar), set to girl or huamulan; if avatarMode is 1 (Animoji), set to dog, girlhead, or arkit.

Add a MetaKit scene to a native view and render the specified scene content. This includes virtual human, Animoji, lighting effects, and 360 background.

Note

• Supports adding up to 8 scene views.
• Currently, only lighting and background effects for video capture are supported. To enable backgroundEffect, avatarMode must be set to 2.

• key: addSceneView
• value: Object. Contains the following fields:
  • view: Int64. The address handle of the view.
  • config: Object. Contains the following fields:
    • width: (optional) Int. The width of the view (px). Defaults to full screen if not specified.
    • height: (optional) Int. The height of the view (px). Defaults to full screen if not specified.
    • extraInfo: Object. Contains the following fields:
      • sceneIndex: Int. Scene index, currently only supports 0.
      • avatarMode: (optional) Int. Scene mode. 0: (default) Avatar; 1: Animoji; 2: Video capture screen.
      • avatar: (optional) String. Avatar or Animoji image. If avatarMode is 0 (avatar), set to girl or huamulan (default is girl). If avatarMode is 1 (Animoji), set to dog, girlhead, or arkit (default is dog).
      • backgroundEffect: (optional) Boolean. Enables lighting effects and 360 background functions. true: Enable; false: (default) Disable.

After adding a scene view, you can switch the scene mode, or the virtual human or Animoji image in the scene.

• key: switchAvatarMode
• value: Object. Contains the following fields:
  • viewAddress: Int64. The address handle of the view.
  • mode: (optional) Int. Specifies the scene mode to switch to. 0: avatar; 1: Animoji; 2: Video capture screen.
  • avatar: (optional) String. Specifies the avatar or Animoji to switch to. If avatarMode is 0 (avatar), set to girl or huamulan. If avatarMode is 1 (Animoji), set to dog, girlhead, or arkit.

After enabling scene view capture, call joinChannel to join the channel and publish the video stream of the scene view.

• key: enableSceneVideo
• value: Object. Contains the following fields:
  • view: Int64. The address handle of the view.
  • enable: (optional) Boolean. Enables scene view capture. true: Enable; false: (default) Disable.

Remove the MetaKit scene view from view.

• key: removeSceneView
• value: Object. Contains the following field:
  • view: Int64. The address handle of the view.

• key: unloadScene
• value: {}

• key: destroy
• value: {}

The MetaKit extension allows you to switch the image, viewpoint, face, and outfit of the avatar. To experience the avatar-related functions, set avatarMode to 0 when enabling texture request or adding scene view.

Note

In addition to the existing girl and huamulan avatars, the Agora MetaKit extension provides an open art ecosystem and supports one-click import of avatar models made according to Agora's art standards, providing users with more flexible creation and integration options. Contact Agora technical support to use this feature.

• key: setCamera
• value: Object. Contains the following field:
  • viewMode: Int. The avatar camera view. 0: Show the avatar's full body; 1: (default) Focus on the avatar's upper body; 2: Focus on the avatar's face.

The MetaKit extension provides a set of face-pinching resources for virtual images.

Note

Currently only the girl avatar supports face pinching.

• key: updateFace

• value: Object. Contains the following fields:

  • key: String. Resource ID, such as MC_updown_1 (upward bend of the mouth corner) and MC_updown_2 (downward bend of the mouth corner). See Face pinching resources for details.
  • value: Int. Adjustment range, range is [0, 100], default value is 50. Supports passing in multiple sets of face-pinching resource IDs (key) and corresponding adjustment ranges (value) to achieve the final face-pinching effect. The example of setting MC_updown_1 and MC_updown_2 to 100 respectively is as follows:

  

The MetaKit extension provides a set of dress-up resources for avatars.

Note

Currently only the girl avatar supports dressup.

• key: updateDress

• value: Object. Contains the following field:

  • id: Int[]. An Int array consisting of resource IDs of multiple clothing items or body parts. Supports dressing operations on multiple items or parts, such as hair, tops, jackets, pants, and so on. Each part provides multiple dressing resources to choose from, that is, each part corresponds to multiple dressing resource IDs. Only one resource can be specified for each part at a time. See Dress-up resources for details.

  The recommended set combinations are as follows:

  1. Set 1

     
     

     _2

     // The following resource IDs correspond to the following clothing items/body parts [hair, eyebrows, blush, headdress, top coat, pants, shoes]

     _2

     "id": [10001, 10101, 10401, 10801, 12101, 14101, 15001]

     
     
     

  2. Set 2

     
     

     _2

     // The following resource IDs correspond to the following clothing items/body parts [hair, eyebrows, blush, coat, gloves, pants, shoes]

     _2

     "id": [10002, 10102, 10402, 12102, 12501, 14102, 15002]

     
     
     

The MetaKit extension allows you to switch the image of Animoji. To experience Animoji-related functions, set avatarMode to 1 when enabling texture request or adding scene view.

Note

In addition to the existing dog, girlhead and arkit Animoji, the Agora MetaKit extension provides an open art ecosystem and supports one-click import of Animoji images made according to Agora's art standards, providing users with more flexible creation and integration options. Contact Agora technical support to use this feature.

The MetaKit extension provides three rendering levels: Low, medium, and high. You can choose the corresponding rendering level according to the device performance to achieve the best match between device performance and rendering effect.

Note

Currently, only the dog Animoji image supports adjusting the rendering level.

• key: setRenderQuality
• value: Object. Contains the following field:
  • general: Int. 0: Low configuration; 1: (default) Medium configuration; 2: High configuration.

The MetaKit extension provides lighting effects such as 3D lighting, screen ripples, aurora effects, and portrait edge flames, and supports fine-tuning the color, intensity, range, and other parameters of the lighting effects. To experience the lighting effects-related functions, set avatarMode to 2 when enabling texture requests or set backgroundEffect to true when adding a scene view.

• key: setEffectVideo
• value: Object. Contains the following fields:
  • id: Int. Special effect material ID.
  • enable: Boolean. Whether to enable the special effect. true: Enable; false: Disable.
  • param: (optional) Object. Each special effect material ID corresponds to a set of configuration parameters, which allows you to fine-tune the color, intensity, range, and so on of the lighting effect. If you do not fill in the parameters, the default parameter configuration will be used.

The mapping relationship between special effect material ID and configuration parameters is as follows:

ID	Effect	Parameters
1001	3D Lighting	- color (Int64): Lighting color. When passing the parameter, the hexadecimal color needs to be converted to an Int64 value. For example, for red, the hexadecimal color is #FF0000, and the converted Int64 value is 16711680. - intensity (Float): Light intensity. The recommended value range is [1.0, 2.0]. The default value is 1.6. - scale (Float): Lighting scale. The recommended range is [0.3, 0.6]. The default value is 0.4.
1002	Screen ripples	- color (Int64): Ripple color. When passing parameters, the hexadecimal color needs to be converted to an Int64 value. For example, for red, the hexadecimal color is #FF0000, and the converted Int64 value is 16711680. - speed (Float): Fluctuation speed. The recommended value range is [-0.2, 0.2]. The default value is -0.12. - scale (Float): Ripple size. The recommended value range is [3.0, 6.0]. The default value is 4.0.
1003	Aurora	- color (Int64): Aurora color. When passing parameters, the hexadecimal color needs to be converted to an Int64 value. For example, for red, the hexadecimal color is #FF0000, and the converted Int64 value is 16711680. - intensity (Float): Aurora intensity. The recommended value range is [0.8, 1.5]. The default value is 1.0.
2001	Portrait edge flame	- color (Int64): Flame color. When passing parameters, the hexadecimal color needs to be converted to an Int64 value. For example, for red, the hexadecimal color is #FF0000, and the converted Int64 value is 16711680. - intensity (Float): Flame intensity. The recommended value range is [0.2, 1.5]. The default value is 0.2.
3001	Ambient lighting set	N/A
3002	Advertising lights	- startColor (Int64): The initial color of the advertising light. When passing parameters, the hexadecimal color needs to be converted to an Int64 value. For example, for red, the hexadecimal color is #FF0000, and the converted Int64 value is 16711680. - endColor (Int64): The end color of the advertising light. When passing parameters, you need to convert the hexadecimal color into an Int64 value. After configuring the starting color, a gradient effect from the initial color to the ending color will be created. - size (Float): The size of the advertisement light texture. The recommended value range is [8, 15]. The default value is 10. - intensity (Float): Advertising light intensity. The recommended value range is [100, 1000], and the default value is 1000. - range (Float): The distance of the advertising light. The recommended range is [10, 40]. The default value is 15.

The MetaKit extension allows you to enable 360-degree panoramic background mode, customize background replacement resources, and enable the gyroscope function to enhance the interactivity and immersion of the scene background. To experience 360-degree background-related functions, set avatarMode to 2 when enabling texture request or set backgroundEffect to true when adding a scene view.

After successful setting, you can observe that the video background is replaced with the specified resource, and you can experience the panoramic effect by rotating the phone.

• key: setBGVideo
• value: Object. Contains the following fields:
  • mode: String. Set to tex360, which means 360-degree panoramic background.
  • param:
    • path: String. Specifies the URL or local path of the background resource.
    • rotation: (optional) Int. Rotation angle, default value is 0.

The gyroscope function is only supported after successfully setting up a 360-degree panoramic background. Enabling the gyroscope function can further enhance the interactivity and immersion of the background.

• key: setCameraGyro
• value: Object. Contains the following field:
  • state: Boolean. Background gyroscope function status. on: Enabled; off: (default) Disabled.

This section introduces the virtual human face-pinching resources provided by the MetaKit extension.

This section introduces the face-shaping resources for girl.

An example of lifting (CK_raise_1) and lowering (CK_raise_2) the cheeks is shown in the following video:

The girl resource supports face-pinching operations on the following parts of the face:

Resource ID	Location
FE_raise_1	Forehead protrusion
FE_raise_2	Forehead collapse
TP_raise_1	Temple protrusion
TP_raise_2	Temple collapse
CK_raise_1	Cheek raise
CK_raise_2	Cheek collapse
MD_width_1	Mandible outward
MD_width_2	Mandible inward
MD_updown_1	Mandible up and down
MD_updown_2	Mandible up
C_width_1	Chin stretch (left and right)
C_width_2	Chin tightening (left and right)
C_updown_1	Chin stretch
C_updown_2	Chin stretch

The following are examples of adjusting the eyebrows to be longer (EB_length_1) and shorter (EB_length_2):

The girl resource supports face-pinching operations on the following parts of the eyebrows:

Resource ID	Location
EB_width_1	Eyebrows moved inwards
EB_width_2	Eyebrows moved outwards
EB_updown_1	Eyebrows moved down
EB_updown_2	Eyebrows moved up
EB_thickness	Adjust the thickness of eyebrows
EBIN_updown_1	Inner eyebrow moved up
EBIN_updown_2	Inner eyebrow moved down
EBMID_updown_1	Middle eyebrow moved up
EBMID_updown_2	Middle eyebrow moved down
EB_length_1	Adjust eyebrow length
EB_length_2	Adjust eyebrow length
EBOUT_updown_1	Outer eyebrow high position
EBOUT_updown_2	Outer eyebrow low position

The following are examples of adjusting the overall enlargement (E_size_1) and shrinking of the eyes (E_size_2):

The girl resource supports face-pinching operations on the following parts of the eyes:

Resource ID	Location
E_width_1	Eyes inward
E_width_2	Eyes outward
E_updown_1	Eyes up adjustment
E_updown_2	Eyes down adjustment
IC_width_1	Inner corner of eye facing inward
IC_width_2	Inner corner of eye facing outward
IC_updown_1	Inner corner of eye upward
IC_updown_2	Inner corner of eye downward
UEIN_updown_1	Upper eyelid tip up
UEIN_updown_2	Upper eyelid tip down
UE_updown_1	Upper eyelid upwards
UE_updown_2	Upper eyelid downwards
UEOUT_updown_1	Upper eyelid ends upward
UEOUT_updown_2	Upper eyelid ends downward
LE_updown_1	Lower eyelid downwards
LE_updown_2	Lower eyelid upwards
OC_width_1	Outer corner of eye inward
OC_width_2	Outer corner of eye outward
OC_updown_1	Outer corner of eye upward
OC_updown_2	Outer corner of eye downward
E_rotate_1	Eye rotation 1
E_rotate_2	Eye rotation 2
E_size_1	Enlarge the entire eye
E_size_2	Reduce the entire eye size
EL_updown_1	Eyelids wider
EL_updown_2	Eyelids narrower

The following are examples of adjusting the overall enlargement (NT_size_1) and shrinking (NT_size_2) of the nose tip:

The girl resource supports face-pinching operations on the following parts of the nose:

Resource ID	Location
N_width_1	Enlarge the nose (left and right)
N_width_2	Shrink the nose (left and right)
N_updown_1	Nose up
N_updown_2	Nose down
NB_raise_1	Nose raised
NB_raise_2	Nose bridge concave
NT_size_1	Enlarge nose tip
NT_size_2	Shrink nose tip
NW_width_1	Nose wings outward
NW_width_2	Nose wings inward
NW_updown_1	Nose wings upward
NW_updown_2	Nose wings downward

The following are examples of adjusting the mouth to move down (M_updown_1) and up (M_updown_2):

The girl resource supports face-pinching operations on the following parts of the mouth:

Resource ID	Location
UL_width_1	Wider upper lip
UL_width_2	Narrower upper lip
LL_width_1	Wider lower lip
LL_width_2	Narrower lower lip
MC_updown_1	Mouth corners curved upward
MC_updown_2	Mouth corners curved downward
M_size_1	Enlarge the mouth (left and right)
M_size_2	Shrink the mouth (left and right)
M_updown_1	Mouth downward
M_updown_2	Mouth upward

The complete face-shaping JSON is as follows:

_338

{

_338

"faceParameters": [

_338

{

_338

"avatar": "girl",

_338

"blendshape": [

_338

{

_338

"type": "Face",

_338

"shapes": [

_338

{

_338

"key": "FE_raise_1",

_338

"ch": "prominence of forehead"

_338

},

_338

{

_338

"key": "FE_raise_2",

_338

"ch": "forehead collapse"

_338

},

_338

{

_338

"key": "TP_raise_1",

_338

"ch": "prominence of the temple"

_338

},

_338

{

_338

"key": "TP_raise_2",

_338

"ch": "collapse of the temple"

_338

},

_338

{

_338

"key": "CK_raise_1",

_338

"ch": "prominence of cheek"

_338

},

_338

{

_338

"key": "CK_raise_2",

_338

"ch": "sunken cheek"

_338

},

_338

{

_338

"key": "MD_width_1",

_338

"ch": "mandible outward"

_338

},

_338

{

_338

"key": "MD_width_2",

_338

"ch": "mandible inward"

_338

},

_338

{

_338

"key": "MD_updown_1",

_338

"ch": "mandible down"

_338

},

_338

{

_338

"key": "MD_updown_2",

_338

"ch": "mandible up"

_338

},

_338

{

_338

"key": "C_width_1",

_338

"ch": "Stretch your jaw left and right"

_338

},

_338

{

_338

"key": "C_width_2",

_338

"ch": "chin tightening left and right"

_338

},

_338

{

_338

"key": "C_updown_1",

_338

"ch": "chin stretch"

_338

},

_338

{

_338

"key": "C_updown_2",

_338

"ch": "chin stretch"

_338

}

_338

]

_338

},

_338

{

_338

"type": "Eyebrow",

_338

"shapes": [

_338

{

_338

"key": "EB_width_1",

_338

"ch": "Eyebrows move inward"

_338

},

_338

{

_338

"key": "EB_width_2",

_338

"ch": "Eyebrows move outward"

_338

},

_338

{

_338

"key": "EB_updown_1",

_338

"ch": "Eyebrows move downward"

_338

},

_338

{

_338

"key": "EB_updown_2",

_338

"ch": "Eyebrows move upward"

_338

},

_338

{

_338

"key": "EB_thickness",

_338

"ch": "Adjust the thickness of eyebrows"

_338

},

_338

{

_338

"key": "EBIN_updown_1",

_338

"ch": "Inner eyebrow moves upward"

_338

},

_338

{

_338

"key": "EBIN_updown_2",

_338

"ch": "Inner eyebrow moves downward"

_338

},

_338

{

_338

"key": "EBMID_updown_1",

_338

"ch": "Middle eyebrow curved upward"

_338

},

_338

{

_338

"key": "EBMID_updown_2",

_338

"ch": "Middle eyebrow concave"

_338

},

_338

{

_338

"key": "EB_length_1",

_338

"ch": "Adjust the length of eyebrows"

_338

},

_338

{

_338

"key": "EB_length_2",

_338

"ch": "Adjust eyebrows to short"

_338

},

_338

{

_338

"key": "EBOUT_updown_1",

_338

"ch": "high position of outer eyebrows"

_338

},

_338

{

_338

"key": "EBOUT_updown_2",

_338

"ch": "low position of outer eyebrow"

_338

}

_338

]

_338

},

_338

{

_338

"type": "Eye",

_338

"shapes": [

_338

{

_338

"key": "E_width_1",

_338

"ch": "Eyes inward"

_338

},

_338

{

_338

"key": "E_width_2",

_338

"ch": "eyes outward"

_338

},

_338

{

_338

"key": "E_updown_1",

_338

"ch": "Eye adjustment"

_338

},

_338

{

_338

"key": "E_updown_2",

_338

"ch": "Under-eye adjustment"

_338

},

_338

{

_338

"key": "IC_width_1",

_338

"ch": "Inner corner of eye facing inward"

_338

},

_338

{

_338

"key": "IC_width_2",

_338

"ch": "Inner corner of eye facing outward"

_338

},

_338

{

_338

"key": "IC_updown_1",

_338

"ch": "Inner corner of eye upward"

_338

},

_338

{

_338

"key": "IC_updown_2",

_338

"ch": "Inner corner of eye down"

_338

},

_338

{

_338

"key": "UEIN_updown_1",

_338

"ch": "The front of the upper eyelid is pointing upward"

_338

},

_338

{

_338

"key": "UEIN_updown_2",

_338

"ch": "The front of the upper eyelid is facing downward"

_338

},

_338

{

_338

"key": "UE_updown_1",

_338

"ch": "Upper eyelid upward"

_338

},

_338

{

_338

"key": "UE_updown_2",

_338

"ch": "Upper eyelids move downwards as a whole"

_338

},

_338

{

_338

"key": "UEOUT_updown_1",

_338

"ch": "The upper eyelid ends upward"

_338

},

_338

{

_338

"key": "UEOUT_updown_2",

_338

"ch": "Upper eyelid ends downward"

_338

},

_338

{

_338

"key": "LE_updown_1",

_338

"ch": "Lower eyelid downward"

_338

},

_338

{

_338

"key": "LE_updown_2",

_338

"ch": "Lower eyelid upward"

_338

},

_338

{

_338

"key": "OC_width_1",

_338

"ch": "Outer corner of eye moves inward"

_338

},

_338

{

_338

"key": "OC_width_2",

_338

"ch": "Outer corners of the eyes turn outward"

_338

},

_338

{

_338

"key": "OC_updown_1",

_338

"ch": "Outer corner of eye up"

_338

},

_338

{

_338

"key": "OC_updown_2",

_338

"ch": "Outer corner of eye down"

_338

},

_338

{

_338

"key": "E_rotate_1",

_338

"ch": "Eye rotation 1"

_338

},

_338

{

_338

"key": "E_rotate_2",

_338

"ch": "Eye rotation 2"

_338

},

_338

{

_338

"key": "E_size_1",

_338

"ch": "Enlarge the eyes as a whole"

_338

},

_338

{

_338

"key": "E_size_2",

_338

"ch": "The eyes shrink overall"

_338

},

_338

{

_338

"key": "EL_updown_1",

_338

"ch": "Eyelids become wider"

_338

},

_338

{

_338

"key": "EL_updown_2",

_338

"ch": "eyelid distance narrows"

_338

}

_338

]

_338

},

_338

{

_338

"type": "Nose",

_338

"shapes": [

_338

{

_338

"key": "N_width_1",

_338

"ch": "Enlarge the nose left and right"

_338

},

_338

{

_338

"key": "N_width_2",

_338

"ch": "The nose shrinks left and right"

_338

},

_338

{

_338

"key": "N_updown_1",

_338

"ch": "nose up"

_338

},

_338

{

_338

"key": "N_updown_2",

_338

"ch": "nose down"

_338

},

_338

{

_338

"key": "NB_raise_1",

_338

"ch": "convex nose"

_338

},

_338

{

_338

"key": "NB_raise_2",

_338

"ch": "concave nose"

_338

},

_338

{

_338

"key": "NT_size_1",

_338

"ch": "Enlarge the nose tip as a whole"

_338

},

_338

{

_338

"key": "NT_size_2",

_338

"ch": "Nose tip overall reduction"

_338

},

_338

{

_338

"key": "NW_width_1",

_338

"ch": "The nose wings are stretched outward"

_338

},

_338

{

_338

"key": "NW_width_2",

_338

"ch": "The nose wings are stretched inwards"

_338

},

_338

{

_338

"key": "NW_updown_1",

_338

"ch": "Stretch on nose wing"

_338

},

_338

{

_338

"key": "NW_updown_2",

_338

"ch": "Stretch under nose"

_338

}

_338

]

_338

},

_338

{

_338

"type": "Mouth",

_338

"shapes": [

_338

{

_338

"key": "UL_width_1",

_338

"ch": "Upper lip widens"

_338

},

_338

{

_338

"key": "UL_width_2",

_338

"ch": "upper lip narrowing"

_338

},

_338

{

_338

"key": "LL_width_1",

_338

"ch": "Lower lip widens"

_338

},

_338

{

_338

"key": "LL_width_2",

_338

"ch": "lower lip narrowing"

_338

},

_338

{

_338

"key": "MC_updown_1",

_338

"ch": "upward curve of the mouth corner"

_338

},

_338

{

_338

"key": "MC_updown_2",

_338

"ch": "corner of mouth curved downward"

_338

},

_338

{

_338

"key": "M_size_1",

_338

"ch": "Enlarge the mouth left and right"

_338

},

_338

{

_338

"key": "M_size_2",

_338

"ch": "The mouth shrinks left and right"

_338

},

_338

{

_338

"key": "M_updown_1",

_338

"ch": "The mouth moves downward"

_338

},

_338

{

_338

"key": "M_updown_2",

_338

"ch": "The mouth moves upward"

_338

}

_338

]

_338

}

_338

]

_338

},

_338

{

_338

"avatar": "huamulan",

_338

"blendshape": []

_338

}

_338

]

_338

}

This section introduces the virtual human dress-up resources provided by the MetaKit extension.

The parts of the girl's outfit and their corresponding resource IDs are as follows:

Clothing item/Body part	Resource ID
Hair	10000, 10001, 10002
Eyebrows	10100, 10101, 10102
Blush	10401, 10402
Headdress	10801
Tops and jackets	12100, 12101, 12102
Gloves	12501
Pants	14100, 14101, 14102
Socks	14301
Shoes	15000, 15001, 15002

The complete JSON for the replacement is as follows:

_87

{

_87

"dressResources": [

_87

{

_87

"avatar": "girl",

_87

"resources": [

_87

{

_87

"id": 100,

_87

"name": "Hair",

_87

"assets": [

_87

10000,

_87

10001,

_87

10002

_87

]

_87

},

_87

{

_87

"id": 101,

_87

"name": "Eyebrows",

_87

"assets": [

_87

10100,

_87

10101,

_87

10102

_87

]

_87

},

_87

{

_87

"id": 104,

_87

"name": "Blush",

_87

"assets": [

_87

10401,

_87

10402

_87

]

_87

},

_87

{

_87

"id": 108,

_87

"name": "Headdress",

_87

"assets": [

_87

10801

_87

]

_87

},

_87

{

_87

"id": 121,

_87

"name": "Tops and Jackets",

_87

"assets": [

_87

12100,

_87

12101,

_87

12102

_87

]

_87

},

_87

{

_87

"id": 125,

_87

"name": "Gloves",

_87

"assets": [

_87

12501

_87

]

_87

},

_87

{

_87

"id": 141,

_87

"name": "Pants",

_87

"assets": [

_87

14100,

_87

14101,

_87

14102

_87

]

_87

},

_87

{

_87

"id": 143,

_87

"name": "Socks",

_87

"assets": [

_87

14301

_87

]

_87

},

_87

{

_87

"id": 150,

_87

"name": "Shoes",

_87

"assets": [

_87

15000,

_87

15001,

_87

15002

_87

]

_87

}

_87

]

_87

},

_87

{

_87

"avatar": "huamulan",

_87

"resources": []

_87

}

_87

]

_87

}