Raw video processing

Understand the tech

RTC SDK enables you to pre-process the captured video frames before sending the data to the encoder or perform post-processing on the received video frames after sending the data to the decoder.

The following figure shows the video data processing flow in the SDK video module.

Process raw video

• Position (2) corresponds to the onCaptureVideoFrame callback.
• Position (3) corresponds to the onPreEncodeVideoFrame callback.
• Position (4) corresponds to theonRenderVideoFrame callback.

Prerequisites

Ensure that you have implemented the SDK quickstart in your project.

Implement raw video processing

To implement raw video data functionality in your project, refer to the following steps:

1. Before joining the channel, create an IVideoFrameObserver object and register the video observer by calling the RegisterVideoFrameObserver method. To apply pre-processed video data, set the formatPreference to FRAME_TYPE_YUV420 and the mode to INTPTR.

   // Register the video observer
   RtcEngine.RegisterVideoFrameObserver(
     new VideoFrameObserver(this),
     VIDEO_OBSERVER_FRAME_TYPE.FRAME_TYPE_YUV420,
     VIDEO_MODULE_POSITION.POSITION_POST_CAPTURER |
     VIDEO_MODULE_POSITION.POSITION_PRE_RENDERER |
     VIDEO_MODULE_POSITION.POSITION_PRE_ENCODER,
     OBSERVER_MODE.INTPTR
   );

2. After obtaining video data via the OnCaptureVideoFrame and OnRenderVideoFrame callbacks, process it according to your requirements.

   class VideoFrr : IVideoFrameObserver {
     // Get the original video data captured by the local camera
     public override bool OnCaptureVideoFrame(VIDEO_SOURCE_TYPE sourceType, VideoFrame videoFrame) {
       ProcessVideoFrame(videoFrame);
       return true;
     }
   
     // Get the video data sent by the remote user
     public override bool OnRenderVideoFrame(string channelId, uint remoteUid, VideoFrame videoFrame) {
       ProcessVideoFrame(videoFrame);
       return true;
     }
   
     public void ProcessVideoFrame(VideoFrame videoFrame) {
       int yBufferLength = videoFrame.yStride * videoFrame.height;
       int uBufferLength = videoFrame.uStride * videoFrame.height / 2;
       int vBufferLength = videoFrame.vStride * videoFrame.height / 2;
   
       byte[] bytes = new byte[uBufferLength];
       for (int i = 0; i < uBufferLength; i++) {
         bytes[i] = 128;
       }
   
       System.Runtime.InteropServices.Marshal.Copy(bytes, 0, videoFrame.uBufferPtr, uBufferLength);
       System.Runtime.InteropServices.Marshal.Copy(bytes, 0, videoFrame.vBufferPtr, vBufferLength);
     }
   }

Reference

This section contains content that completes the information on this page, or points you to documentation that explains other aspects to this product.

Sample project

Agora provides an open source sample project ProcessRawData on GitHub. Download it or view the source code for a more detailed example.

API reference

• RegisterVideoFrameObserver
• RegisterVideoEncodedFrameObserver
• OnCaptureVideoFrame
• OnRenderVideoFrame