Raw video processing

In certain use-cases, it may be necessary to process raw video captured through the camera and microphone to achieve desired functionality or enhance the user experience. Video SDK provides the capability to pre-process and post-process the captured video data, allowing for the implementation of custom playback effects.

Video SDK enables you to pre-process the captured video frames before sending the data to the encoder or perform post-processing on the received video frames after sending the data to the decoder.

The following figure shows the video data processing flow in the SDK video module.

Process raw video

• Position (2) corresponds to the onCaptureVideoFrame callback.
• Position (3) corresponds to the onPreEncodeVideoFrame callback.
• Position (4) corresponds to theonRenderVideoFrame callback.

Ensure that you have implemented the SDK quickstart in your project.

To implement raw video data functionality in your project, refer to the following steps:

1. Before joining the channel, create an IVideoFrameObserver object and register the video observer by calling the registerVideoFrameObserver method.

Java

    int ret = engine.registerVideoFrameObserver(iVideoFrameObserver);

Kotlin

   val ret = engine.registerVideoFrameObserver(iVideoFrameObserver)

1. Implement the onCaptureVideoFrame and onRenderVideoFrame callbacks. After obtaining the video data, process it according to your specific use-case.

Java

private final IVideoFrameObserver iVideoFrameObserver = new IVideoFrameObserver() {    @Override    public boolean onCaptureVideoFrame(VideoFrame videoFrame) {        Log.i(TAG, "OnEncodedVideoImageReceived" + Thread.currentThread().getName());        if (isSnapshot) {            isSnapshot = false;                        // Get the image bitmap            VideoFrame.Buffer buffer = videoFrame.getBuffer();            VideoFrame.I420Buffer i420Buffer = buffer.toI420();            int width = i420Buffer.getWidth();            int height = i420Buffer.getHeight();                        ByteBuffer bufferY = i420Buffer.getDataY();            ByteBuffer bufferU = i420Buffer.getDataU();            ByteBuffer bufferV = i420Buffer.getDataV();                        byte[] i420 = YUVUtils.toWrappedI420(bufferY, bufferU, bufferV, width, height);                        Bitmap bitmap = YUVUtils.NV21ToBitmap(getContext(),                YUVUtils.I420ToNV21(i420, width, height),                width,                height);                        Matrix matrix = new Matrix();            matrix.setRotate(270);            // Rotate around the center            Bitmap newBitmap = Bitmap.createBitmap(bitmap, 0, 0, width, height, matrix, false);            // Save to file            saveBitmap2Gallery(newBitmap);                        bitmap.recycle();            i420Buffer.release();        }        return false;    }        @Override    public boolean onScreenCaptureVideoFrame(VideoFrame videoFrame) {        return false;    }        @Override    public boolean onMediaPlayerVideoFrame(VideoFrame videoFrame, int i) {        return false;    }        @Override    public boolean onRenderVideoFrame(String s, int i, VideoFrame videoFrame) {        return false;    }        @Override    public int getVideoFrameProcessMode() {        return 0;    }        @Override    public int getVideoFormatPreference() {        return 1;    }        @Override    public int getRotationApplied() {        return 0;    }        @Override    public boolean getMirrorApplied() {        return false;    }};

Kotlin

private val iVideoFrameObserver = object : IVideoFrameObserver {   override fun onCaptureVideoFrame(videoFrame: VideoFrame): Boolean {       Log.i(TAG, "OnEncodedVideoImageReceived${Thread.currentThread().name}")       if (isSnapshot) {           isSnapshot = false           // Get the image bitmap           val buffer = videoFrame.buffer           val i420Buffer = buffer.toI420()           val width = i420Buffer.width           val height = i420Buffer.height           val bufferY = i420Buffer.dataY           val bufferU = i420Buffer.dataU           val bufferV = i420Buffer.dataV           val i420 = YUVUtils.toWrappedI420(bufferY, bufferU, bufferV, width, height)           val bitmap = YUVUtils.NV21ToBitmap(               context = context,               nv21Data = YUVUtils.I420ToNV21(i420, width, height),               width = width,               height = height           )           val matrix = Matrix().apply { setRotate(270f) }           // Rotate around the center           val newBitmap = Bitmap.createBitmap(bitmap, 0, 0, width, height, matrix, false)           // Save to file           saveBitmap2Gallery(newBitmap)           bitmap.recycle()           i420Buffer.release()       }       return false   }   override fun onScreenCaptureVideoFrame(videoFrame: VideoFrame): Boolean {       return false   }   override fun onMediaPlayerVideoFrame(videoFrame: VideoFrame, i: Int): Boolean {       return false   }   override fun onRenderVideoFrame(s: String, i: Int, videoFrame: VideoFrame): Boolean {       return false   }   override fun getVideoFrameProcessMode(): Int {       return 0   }   override fun getVideoFormatPreference(): Int {       return 1   }   override fun getRotationApplied(): Int {       return 0   }   override fun getMirrorApplied(): Boolean {       return false   }}

Caution

When modifying parameters in a VideoFrame, ensure that the updated parameters match the actual video frame in the buffer. Mismatches may cause issues like unexpected rotation, distortion, or other visual problems in the local preview and the remote video.

This section contains content that completes the information on this page, or points you to documentation that explains other aspects to this product.

Agora provides an open source sample project ProcessRawData on GitHub. Download it or view the source code for a more detailed example.

• registerVideoFrameObserver
• registerVideoEncodedFrameObserver
• onCaptureVideoFrame
• onRenderVideoFrame
• getVideoFrameProcessMode
• onEncodedVideoFrameReceived