DeepAR

DeepAR Extension is a wrapper around DeepAR Web that simplifies integration with the Agora RTC platform.

For more information, see DeepAR.

Prerequisites

In order to use the DeepAR Web extension you need to set up a license key for your web app on developer.deepar.ai.

Create an account: https://developer.deepar.ai/signup.
Create a project: https://developer.deepar.ai/projects.
Add a web app to the project. Note that you need to specify the domain name which you plan to use for hosting the app.

Download the extension

You need both DeepAR Web extension and Agora RTC packages.

Using npm:

npm install deepar-agora-extension agora-rtc-sdk-ng

Using yarn:

yarn add deepar-agora-extension agora-rtc-sdk-ng

Add the imports

Import VideoExtension and AgoraRTC:

import { VideoExtension } from 'deepar-agora-extension';
import AgoraRTC from 'agora-rtc-sdk-ng'

You are also going to need import WebAssembly file that DeepAR runs and machine learning models that DeepAR uses for face tracking, background removal and such.

We recommend using a bundler to correctly include assets like models, effects and WebAssembly files.

For example, if using Webpack, you can use Asset Modules. Add this to your webpack.config.js:

module.exports = {
  // ...
  module: {
    rules: [
      {
        test: /\.(wasm)|(bin)|(obj)$/i,
        include: [
          path.resolve(__dirname, 'node_modules/deepar/'),
        ],
        type: 'asset/resource',
      },
      {
        include: [
          path.resolve(__dirname, 'effects/'),
        ],
        type: 'asset/resource',
      },
    ],
  },
  // ...

Then you can import paths for these files like this:

import deeparWasm from 'deepar/wasm/deepar.wasm'
import faceModel from 'deepar/models/face/models-68-extreme.bin'
import segmentationModel from 'deepar/models/segmentation/segmentation-160x160-opt.bin'

Enable the extension

Create a div tag that will be used as a container for camera preview:

<div class="video-container"></div>

Be sure to set the width and height of the div!

<style>
    .video-container{
        width: 640px;
        height: 480px;
    }
</style>

Initialize the extension:

const videoExtension = new VideoExtension({
    licenseKey: 'your_license_key_here', // create the license key here https://developer.deepar.ai/projects
    deeparWasmPath: deeparWasm,
    segmentationConfig: { // don't need to define this if you don't use background removal effects
        modelPath: segmentationModel
    },
    onInitialize: function (deepAR) {
      // at this point DeepAR is initialized and can be used normally
    },
});

Set up video processing

To add video processing to the previously created div container:

//register extension
AgoraRTC.registerExtensions([videoExtension]);

//create DeepAR extension processor
const processor = videoExtension.createProcessor();

//create CameraVideoTrack
const videoTrack = await AgoraRTC.createCameraVideoTrack();

//piping processor
videoTrack.pipe(processor).pipe(videoTrack.processorDestination);

await videoTrack.play(document.querySelector('.video-container'), {mirror: false});

You can use the deepAR object normally as in the standard DeepAR Web SDK. See API reference here.

Access deepAR

You do this using the VideoExtension's onInitialize callback:

const videoExtension = new VideoExtension({
  ...
  onInitialize: function (deepAR) {
    // at this point DeepAR is initialized and can be used normally
  }
  ...
});

Load the face tracking model

deepAR.downloadFaceTrackingModel('path/to/models-68-extreme.bin');

All masks, filters, background removal, etc. are represented by effect files in DeepAR. You can load them to preview the effect. You can download a free filter pack here: https://docs.deepar.ai/deep-ar-studio/free-filter-pack or visit DeepAR effect store.

Load an effect using the switchEffect method:

deepAR.switchEffect(0, 'slot', './effects/alien');

Load different effects on different persons' faces:

deepAR.switchEffect(0, 'slot', './effects/alien');
deepAR.switchEffect(1, 'slot', './effects/lion');

Load a background removal effect:

deepAR.switchEffect(0, 'slot', './effects/background_segmentation');

Background removal or blur

To use background segmentation DeepAR needs to initialize the segmentation model.

import segmentationModelPath from 'deepar/models/segmentation/segmentation-160x160-opt.bin';

// ...

const videoExtension = new VideoExtension({
  segmentationConfig: {
    modelPath: segmentationModelPath,
  },
  // other params ...
});

Shoe try-on

To use shoe try-on feature DeepAR needs to initialize foot tracking. All the footTrackingConfig parameters are required.

import poseEstimationWasmPath from 'deepar/wasm/libxzimgPoseEstimation.wasm';
import footDetectorPath from 'deepar/models/foot/foot-detector-android.bin'; // or ...-ios.bin
import footTrackerPath from 'deepar/models/foot/foot-tracker-android.bin'; // or ...-ios.bin
import footObjPath from 'deepar/models/foot/foot-model.obj';

// ...

const videoExtension = new VideoExtension({
  footTrackingConfig: {
    poseEstimationWasmPath,
    detectorPath: footDetectorPath,
    trackerPath: footTrackerPath,
    objPath: footObjPath,
  },
  // other params ...
});