Raw video processing

Understand the tech

RTC SDK enables you to pre-process the captured video frames before sending the data to the encoder or perform post-processing on the received video frames after sending the data to the decoder.

The following figure shows the video data processing flow in the SDK video module.

Process raw video

• Position (2) corresponds to the onCaptureVideoFrame callback.
• Position (3) corresponds to the onPreEncodeVideoFrame callback.
• Position (4) corresponds to theonRenderVideoFrame callback.

Prerequisites

Ensure that you have implemented the SDK quickstart in your project.

Implement raw video processing

To implement raw video data functionality in your project, refer to the following steps:

1. Before joining the channel, call setVideoFrameDelegate to register the video observer object.

2. Implement the onCaptureVideoFrame and onRenderVideoFrame callbacks to handle the capture and rendering of video frames.

   When modifying parameters in a videoFrame, ensure that the updated parameters match the actual video frame in the buffer. Failure to do so may result in unexpected rotation or distortion in both the local preview and the remote video.

   class RawVideoDataMain: BaseViewController {
     var localVideo = Bundle.loadVideoView(type: .local, audioOnly: false)
     var remoteVideo = Bundle.loadVideoView(type: .remote, audioOnly: false)
   
     @IBOutlet weak var container: AGEVideoContainer!
     // Define agoraKit variable
     var agoraKit: AgoraRtcEngineKit!
   
     // ...
     // Initialize agoraKit and register the corresponding callbacks
     agoraKit = AgoraRtcEngineKit.sharedEngine(with: config, delegate: self)
   
     // Call setVideoFrameDelegate to register the video observer object
     agoraKit.setVideoFrameDelegate (self)
   
     // ...
   
     // Implement the AgoraVideoFrameDelegate protocol extension in the current class
     extension RawVideoDataMain: AgoraVideoFrameDelegate {
   
       // Implement the onCaptureVideoFrame callback
       func onCaptureVideoFrame(_ videoFrame: AgoraOutputVideoFrame) -> Bool {
         return true
       }
   
       // Implement the onScreenCaptureVideoFrame callback
       func onScreenCaptureVideoFrame(_ videoFrame: AgoraOutputVideoFrame) -> Bool {
         return true
       }
   
       // Implement the onRenderVideoFrame callback
       func onRenderVideoFrame(_ videoFrame: AgoraOutputVideoFrame, uid: UInt, channelId: String) -> Bool {
         return true
       }
     }
   }

Reference

This section contains content that completes the information on this page, or points you to documentation that explains other aspects to this product.

Sample project

Agora provides an open source sample project RawVideoData on GitHub. Download it or view the source code for a more detailed example.

API reference

• setVideoFrameDelegate(_:)
• onRenderVideoFrame(_:uid:channelId:)
• onCapture(_:sourceType:)