Using Image Enhancement Beta

Introduction

Agora Native SDK users can integrate some special visual effects for image enhancement if required in one of the following methods:

Note

It is recommended to turn off the image enhancement function if the overall resource consumption is huge.

Supported Platform

  • Android
  • iOS

Obtaining the SDK

Download the SDK from downloads or contact sales-us@agora.io.

Integrating the Agora Preprocessor

Android

Agora Preprocessor is an optional component, which you can integrate into your application for image enhancement and other visual effects (Now only supports image enhancement) if necessary. The preprocessor module is between the camera module and encoder module as shown below:

../../_images/agora_preprocessor.png

API

The following is the complete API description:

public class AgoraYuvEnhancer {
 // ctor and dtor
 public AgoraYuvEnhancer(Context ctxt);

 /** startPreProcess
  *
  * @return
  */
 public native int StartPreProcess();
 /** stopPreProcess
  *
  * @return
  */
 public native int StopPreProcess();
 /** SetLighteningFactor
  *
  * @param factor
  * @return
  */
 public native int SetLighteningFactor(float factor); // [0.0, 1.0], default 0.7
 /** SetSmoothnessFactor
  *
  * @param level
  * @return
  */
 public native int SetSmoothnessFactor(float level); // [0.0, 1.0], default 0.57
}

Procedure

The SDK binary is videoprp.aar, which also includes equivalent.so and .jar files.

Take the IDE as Android Studio for example:

  1. Add the binary of the preprocessor component videoprp.aar to your project. For example, here assuming the library path is app/src/main/PREBUILT.
cp videoprp.aar app/src/main/PREBUILT
app/build.gradle.

repositories {
    flatDir {
        dirs 'src/main/PREBUILT'
    }
}
dependencies {
    compile(name:'videoprp', ext:'aar'))
}
  1. Add the following codes and open the preprocessor.
import io.agora.videoprp.AgoraYuvEnhancer;

private AgoraYuvEnhancer yuvEnhancer = null;
yuvEnhancer = new AgoraYuvEnhancer(context);
yuvEnhancer.StartPreProcess();
  1. Close the preprocessor.
if(yuvEnhancer != null)
   yuvEnhancer.StopPreProcess();

Note

  • The users can open or close the preprocessor any time after creating the Agora engine.
  • For eclipse users, unzip .jar and .so files from videoprp.aar.

iOS

API

The following is the complete API description:

@interface AgoraYuvEnhancerObjc : NSObject
@property (assign, nonatomic) CGFloat lighteningFactor;// in the range [0.0, 1.0]; default 0.7
@property (assign, nonatomic) CGFloat smoothness;//in the range [0.0, 1.0]; default 0.7

- (void)turnOn;
- (void)turnOff;
@end

Procedure

The SDK binary is videoprp.framework:

  1. Add the binary of the preprocessor component videoprp.framework to your project.
  2. Add the following codes. Start the preprocessor:
#import <videoprp/AgoraYuvEnhancerObjc.h>

var agoraEnhancer: AgoraYuvEnhancerObjc?

Open:
let enhancer = AgoraYuvEnhancerObjc()
enhancer.turnOn()
self.agoraEnhancer = enhancer

Close:
agoraEnhancer?.turnOff()

Note

  • The users can open or close the preprocessor any time after creating the Agora engine.

  • The preprocessor works fine is if the Xcode prints some strings like the following every several seconds:

    fps: 126.2 (size 640 x 360)
    

Overriding the Camera and Preprocessor

This section includes different solutions for:

Live Broadcast

You can develop your own image enhancement function based on the Agora SDK by overriding the Agora camera and preprocessor modules.

Communication

Android

API

The following is the complete API description:

   public class AgoraVideoSource {
    // ctor
    public AgoraVideoSource();
    // connect this preprocessor to the pipeline
    public synchronized void Attach();
    // disconnect this preprocessor from the pipeline
    public synchronized void Detach();

    /** Input a frame to engine
     *
     * width, height: size of 'buf' in pixels
     * cropLeft: how many pixels to crop on the left boundary
     * cropTop: how many pixels to crop on the top boundary
     * cropRight: how many pixels to crop on the right boundary
     * cropBottom: how many pixels to crop on the bottom boundary
     * rotation: 0, 90, 180, 270. See document for rotation calculation
     * ts: timestamp for this frame. in milli-second
     * format: 1: I420 2: ARGB 3: NV21 4: RGBA
     *
     * width/height/cropLeft/cropTop/cropRight/cropBottom: specifying the rotated buffer,
     * not pre-rotate buffer
     */
    public synchronized void DeliverFrame(byte[] buf,
                    int width, int height,
                    int cropLeft, int cropTop, int cropRight, int cropBottom,
                    int rotation, long ts, int format);
}
Procedure
  1. Import AgoraVideoSource.
import io.agora.extvideo.AgoraVideoSource;
  1. Define a frame buffer with a proper size. For example,
private byte[] mTransBuf = new byte [640 * 480 * 3 / 2 + 1024 * 16];
  1. Set videoSource as null.
private AgoraVideoSource videoSource = null;
  1. After creating RtcEngineEx , create the AgoraVideoSource instance before calling enableVideo() or startPreview().
mRtcEngine = RtcEngineEx.create(mContext,
                vendorKey,
                Constants.APP_CATEGORY_LIVE_BROADCASTING,
                mEngineEventHandler.mRtcEventHandler);
videoSource = new AgoraVideoSource();
videoSource.Attach();
  1. Send the frame to the encoder using AgoraVideoSource.
videoSource.DeliverFrame(mTransBuf,
     prevWidth, prevHeight,
     cropLeft, cropTop, cropRight, cropBottom,
     rotate, System.currentTimeMillis(), format);

Note

  • The timestamp is in milliseconds.
  • Rotation using the following calculation:
public int getDisplayRotation() {
  int rotation = ((WindowManager) (getContext().getSystemService(Context.WINDOW_SERVICE))).getDefaultDisplay().getRotation();
  switch (rotation) {
  case Surface.ROTATION_0: return 0;
  case Surface.ROTATION_90: return 90;
  case Surface.ROTATION_180: return 180;
  case Surface.ROTATION_270: return 270;
      }
      return 0;
  }
  // calculate rotated degrees (camera to view)
  int degrees = getDisplayRotation();
  int rotation;
  Log.d(TAG, "Camera orientation " + mCameraRotation);
  if(mCameraIndex == Camera.CameraInfo.CAMERA_FACING_FRONT) {
      rotation = (mCameraRotation - degrees + 360) % 360;
  }
  else {
      rotation = (mCameraRotation + degrees + 360) % 360;
  }

iOS

API

The following is the complete API description:

@interface AgoraVideoSource : NSObject
-(void) Attach;
-(void) Detach;

/** Input a frame to engine
 *
 * width, height: size of 'buf' in pixels
 * cropLeft: how many pixels to crop on the left boundary
 * cropTop: how many pixels to crop on the top boundary
 * cropRight: how many pixels to crop on the right boundary
 * cropBottom: how many pixels to crop on the bottom boundary
 * rotation: 0, 90, 180, 270. See document for rotation calculation
 * ts: timestamp for this frame. in milli-second
 * format: 1: I420 2: ARGB 3: NV21 4: RGBA
 *
 * width/height/cropLeft/cropTop/cropRight/cropBottom: specifying the rotated buffer,
 * not pre-rotate buffer
 */
- (void)DeliverFrame:(void *)buf width:(int)width height:(int)height
  cropLeft:(int)cropLeft cropTop:(int)cropTop cropRight:(int)cropRight cropBottom:(int)cropBottom
  rotation:(int)rotation timeStamp:(long long)ts format:(int)format;

@end
Procedure
let exVideo: AgoraVideoSource = {
 let ext = AgoraVideoSource()
 return ext
 }()

 exVideo.Attach()
 agoraKit.enableVideo()

You can send the frame to the encoder using AgoraVideoSource:

exVideo.DeliverFrame (/* … */)

Note

  • The timestamp is in milliseconds.
  • Rotation using the following calculation:
int degrees = [self getDisplayRotation];
int rotation;
if(isFrontCamera) {
    rotation = (cameraOrientation + degrees + 360) % 360;
}
else {
    rotation = (cameraOrientation - degrees + 360) % 360;
}

where cameraOrientation == 90

- (int) getDisplayRotation
{
    UIInterfaceOrientation orientation = [UIapplication sharedapplication].statusBarOrientation;
    switch(orientation) {
    case UIDeviceOrientationPortrait:
    default:
        return 0;
    case UIDeviceOrientationLandscapeLeft:
        return 90;
    case UIDeviceOrientationPortraitUpsideDown:
        return 180;
    case UIDeviceOrientationLandscapeRight:
        return 270;
    }
}