Skip to main content
  • Core Products
  • Add-Ons
  • Extensions Marketplace
Android
iOS
macOS
Web
Windows
Electron
Flutter
React Native
React JS
Unity
Unreal Engine
Unreal (Blueprint)

Raw video processing

In certain use-cases, it is necessary to process raw video captured through the camera to achieve desired functionality or enhance the user experience. Video SDK provides the capability to pre-process and post-process the captured video data, allowing you to implement custom playback effects.

Understand the tech​

Video SDK enables you to pre-process the captured video frames before sending the data to the encoder or perform post-processing on the received video frames after sending the data to the decoder.

The following figure shows the video data processing flow in the SDK video module.

Process raw video

  • Position (2) corresponds to the onCaptureVideoFrame callback.
  • Position (3) corresponds to the onPreEncodeVideoFrame callback.
  • Position (4) corresponds to theonRenderVideoFrame callback.

Prerequisites​

Ensure that you have implemented the SDK quickstart in your project.

Implement raw video processing​

To implement raw video data functionality in your project, refer to the following steps:

  1. Before joining the channel, call setVideoFrameDelegate to register the video observer object.

  2. Implement the onCaptureVideoFrame and onRenderVideoFrame callbacks to handle the capture and rendering of video frames.

    Caution

    When modifying parameters in a videoFrame, ensure that the updated parameters match the actual video frame in the buffer. Failure to do so may result in unexpected rotation or distortion in both the local preview and the remote video.

    class RawVideoDataMain: BaseViewController {
    var localVideo = Bundle.loadVideoView(type: .local, audioOnly: false)
    var remoteVideo = Bundle.loadVideoView(type: .remote, audioOnly: false)

    @IBOutlet weak var container: AGEVideoContainer!
    // Define agoraKit variable
    var agoraKit: AgoraRtcEngineKit!

    // ...
    // Initialize agoraKit and register the corresponding callbacks
    agoraKit = AgoraRtcEngineKit.sharedEngine(with: config, delegate: self)

    // Call setVideoFrameDelegate to register the video observer object
    agoraKit.setVideoFrameDelegate (self)

    // ...

    // Implement the AgoraVideoFrameDelegate protocol extension in the current class
    extension RawVideoDataMain: AgoraVideoFrameDelegate {

    // Implement the onCaptureVideoFrame callback
    func onCaptureVideoFrame(_ videoFrame: AgoraOutputVideoFrame) -> Bool {
    return true
    }

    // Implement the onScreenCaptureVideoFrame callback
    func onScreenCaptureVideoFrame(_ videoFrame: AgoraOutputVideoFrame) -> Bool {
    return true
    }

    // Implement the onRenderVideoFrame callback
    func onRenderVideoFrame(_ videoFrame: AgoraOutputVideoFrame, uid: UInt, channelId: String) -> Bool {
    return true
    }
    }
    }
    Copy

Reference​

This section contains content that completes the information on this page, or points you to documentation that explains other aspects to this product.

Sample project​

Agora provides an open source sample project RawVideoData on GitHub. Download it or view the source code for a more detailed example.

API reference​

Page Content

Interactive Live Streaming