Raw video processing

Understand the tech

Video SDK enables you to pre-process the captured video frames before sending the data to the encoder or perform post-processing on the received video frames after sending the data to the decoder.

The following figure shows the video data processing flow in the SDK video module.

Process raw video

• Position (2) corresponds to the onCaptureVideoFrame callback.
• Position (3) corresponds to the onPreEncodeVideoFrame callback.
• Position (4) corresponds to theonRenderVideoFrame callback.

Prerequisites

Ensure that you have implemented the SDK quickstart in your project.

Implement raw video processing

To implement raw video data functionality in your project, refer to the following steps:

1. Before joining the channel, create an IVideoFrameObserver object and register the video observer by calling the registerVideoFrameObserver method.

  int ret = engine.registerVideoFrameObserver(iVideoFrameObserver);

  val ret = engine.registerVideoFrameObserver(iVideoFrameObserver)

1. Implement the onCaptureVideoFrame and onRenderVideoFrame callbacks. After obtaining the video data, process it according to your specific use-case.

private final IVideoFrameObserver iVideoFrameObserver = new IVideoFrameObserver() {
  @Override
  public boolean onCaptureVideoFrame(VideoFrame videoFrame) {
    Log.i(TAG, "OnEncodedVideoImageReceived" + Thread.currentThread().getName());
    if (isSnapshot) {
      isSnapshot = false;

      // Get the image bitmap
      VideoFrame.Buffer buffer = videoFrame.getBuffer();
      VideoFrame.I420Buffer i420Buffer = buffer.toI420();
      int width = i420Buffer.getWidth();
      int height = i420Buffer.getHeight();

      ByteBuffer bufferY = i420Buffer.getDataY();
      ByteBuffer bufferU = i420Buffer.getDataU();
      ByteBuffer bufferV = i420Buffer.getDataV();

      byte[] i420 = YUVUtils.toWrappedI420(bufferY, bufferU, bufferV, width, height);

      Bitmap bitmap = YUVUtils.NV21ToBitmap(getContext(),
        YUVUtils.I420ToNV21(i420, width, height),
        width,
        height);

      Matrix matrix = new Matrix();
      matrix.setRotate(270);
      // Rotate around the center
      Bitmap newBitmap = Bitmap.createBitmap(bitmap, 0, 0, width, height, matrix, false);
      // Save to file
      saveBitmap2Gallery(newBitmap);

      bitmap.recycle();
      i420Buffer.release();
    }
    return false;
  }

  @Override
  public boolean onScreenCaptureVideoFrame(VideoFrame videoFrame) {
    return false;
  }

  @Override
  public boolean onMediaPlayerVideoFrame(VideoFrame videoFrame, int i) {
    return false;
  }

  @Override
  public boolean onRenderVideoFrame(String s, int i, VideoFrame videoFrame) {
    return false;
  }

  @Override
  public int getVideoFrameProcessMode() {
    return 0;
  }

  @Override
  public int getVideoFormatPreference() {
    return 1;
  }

  @Override
  public int getRotationApplied() {
    return 0;
  }

  @Override
  public boolean getMirrorApplied() {
    return false;
  }
};

private val iVideoFrameObserver = object : IVideoFrameObserver {
  override fun onCaptureVideoFrame(videoFrame: VideoFrame): Boolean {
    Log.i(TAG, "OnEncodedVideoImageReceived${Thread.currentThread().name}")
    if (isSnapshot) {
      isSnapshot = false

      // Get the image bitmap
      val buffer = videoFrame.buffer
      val i420Buffer = buffer.toI420()
      val width = i420Buffer.width
      val height = i420Buffer.height

      val bufferY = i420Buffer.dataY
      val bufferU = i420Buffer.dataU
      val bufferV = i420Buffer.dataV

      val i420 = YUVUtils.toWrappedI420(bufferY, bufferU, bufferV, width, height)

      val bitmap = YUVUtils.NV21ToBitmap(
        context = context,
        nv21Data = YUVUtils.I420ToNV21(i420, width, height),
        width = width,
        height = height
      )

      val matrix = Matrix().apply { setRotate(270f) }
      // Rotate around the center
      val newBitmap = Bitmap.createBitmap(bitmap, 0, 0, width, height, matrix, false)

      // Save to file
      saveBitmap2Gallery(newBitmap)

      bitmap.recycle()
      i420Buffer.release()
    }
    return false
  }

  override fun onScreenCaptureVideoFrame(videoFrame: VideoFrame): Boolean {
    return false
  }

  override fun onMediaPlayerVideoFrame(videoFrame: VideoFrame, i: Int): Boolean {
    return false
  }

  override fun onRenderVideoFrame(s: String, i: Int, videoFrame: VideoFrame): Boolean {
    return false
  }

  override fun getVideoFrameProcessMode(): Int {
    return 0
  }

  override fun getVideoFormatPreference(): Int {
    return 1
  }

  override fun getRotationApplied(): Int {
    return 0
  }

  override fun getMirrorApplied(): Boolean {
    return false
  }
}

Reference

This section contains content that completes the information on this page, or points you to documentation that explains other aspects to this product.

Sample project

Agora provides an open source sample project ProcessRawData on GitHub. Download it or view the source code for a more detailed example.

API reference

• registerVideoFrameObserver
• registerVideoEncodedFrameObserver
• onCaptureVideoFrame
• onRenderVideoFrame
• getVideoFrameProcessMode
• onEncodedVideoFrameReceived

Understand the tech

Video SDK enables you to pre-process the captured video frames before sending the data to the encoder or perform post-processing on the received video frames after sending the data to the decoder.

The following figure shows the video data processing flow in the SDK video module.

Process raw video

• Position (2) corresponds to the onCaptureVideoFrame callback.
• Position (3) corresponds to the onPreEncodeVideoFrame callback.
• Position (4) corresponds to theonRenderVideoFrame callback.

Prerequisites

Ensure that you have implemented the SDK quickstart in your project.

Implement raw video processing

To implement raw video data functionality in your project, refer to the following steps:

1. Before joining the channel, call setVideoFrameDelegate to register the video observer object.

2. Implement the onCaptureVideoFrame and onRenderVideoFrame callbacks to handle the capture and rendering of video frames.

   When modifying parameters in a videoFrame, ensure that the updated parameters match the actual video frame in the buffer. Failure to do so may result in unexpected rotation or distortion in both the local preview and the remote video.

class RawVideoDataMain: BaseViewController {
  var localVideo = Bundle.loadVideoView(type: .local, audioOnly: false)
  var remoteVideo = Bundle.loadVideoView(type: .remote, audioOnly: false)

  @IBOutlet weak var container: AGEVideoContainer!
  // Define agoraKit variable
  var agoraKit: AgoraRtcEngineKit!

  // ...
  // Initialize agoraKit and register the corresponding callbacks
  agoraKit = AgoraRtcEngineKit.sharedEngine(with: config, delegate: self)

  // Call setVideoFrameDelegate to register the video observer object
  agoraKit.setVideoFrameDelegate (self)

  // ...

  // Implement the AgoraVideoFrameDelegate protocol extension in the current class
  extension RawVideoDataMain: AgoraVideoFrameDelegate {

    // Implement the onCaptureVideoFrame callback
    func onCaptureVideoFrame(_ videoFrame: AgoraOutputVideoFrame) -> Bool {
      return true
    }

    // Implement the onScreenCaptureVideoFrame callback
    func onScreenCaptureVideoFrame(_ videoFrame: AgoraOutputVideoFrame) -> Bool {
      return true
    }

    // Implement the onRenderVideoFrame callback
    func onRenderVideoFrame(_ videoFrame: AgoraOutputVideoFrame, uid: UInt, channelId: String) -> Bool {
      return true
    }
  }
}

Reference

This section contains content that completes the information on this page, or points you to documentation that explains other aspects to this product.

Sample project

Agora provides an open source sample project RawVideoData on GitHub. Download it or view the source code for a more detailed example.

API reference

• setVideoFrameDelegate(_:)
• onRenderVideoFrame(_:uid:channelId:)
• onCapture(_:sourceType:)

Understand the tech

Video SDK enables you to pre-process the captured video frames before sending the data to the encoder or perform post-processing on the received video frames after sending the data to the decoder.

The following figure shows the video data processing flow in the SDK video module.

Process raw video

• Position (2) corresponds to the onCaptureVideoFrame callback.
• Position (3) corresponds to the onPreEncodeVideoFrame callback.
• Position (4) corresponds to theonRenderVideoFrame callback.

Prerequisites

Ensure that you have implemented the SDK quickstart in your project.

Implement raw video processing

To implement raw video data functionality in your project, refer to the following steps:

1. Before joining the channel, call setVideoFrameDelegate to register the video observer object.

2. Implement the onCaptureVideoFrame and onRenderVideoFrame callbacks to handle the capture and rendering of video frames.

   When modifying parameters in a videoFrame, ensure that the updated parameters match the actual video frame in the buffer. Failure to do so may result in unexpected rotation or distortion in both the local preview and the remote video.

class RawVideoDataMain: BaseViewController {
  var localVideo = Bundle.loadVideoView(type: .local, audioOnly: false)
  var remoteVideo = Bundle.loadVideoView(type: .remote, audioOnly: false)

  @IBOutlet weak var container: AGEVideoContainer!
  // Define agoraKit variable
  var agoraKit: AgoraRtcEngineKit!

  // ...
  // Initialize agoraKit and register the corresponding callbacks
  agoraKit = AgoraRtcEngineKit.sharedEngine(with: config, delegate: self)

  // Call setVideoFrameDelegate to register the video observer object
  agoraKit.setVideoFrameDelegate (self)

  // ...

  // Implement the AgoraVideoFrameDelegate protocol extension in the current class
  extension RawVideoDataMain: AgoraVideoFrameDelegate {

    // Implement the onCaptureVideoFrame callback
    func onCaptureVideoFrame(_ videoFrame: AgoraOutputVideoFrame) -> Bool {
      return true
    }

    // Implement the onScreenCaptureVideoFrame callback
    func onScreenCaptureVideoFrame(_ videoFrame: AgoraOutputVideoFrame) -> Bool {
      return true
    }

    // Implement the onRenderVideoFrame callback
    func onRenderVideoFrame(_ videoFrame: AgoraOutputVideoFrame, uid: UInt, channelId: String) -> Bool {
      return true
    }
  }
}

Reference

This section contains content that completes the information on this page, or points you to documentation that explains other aspects to this product.

Sample project

Agora provides an open source sample project RawVideoData on GitHub. Download it or view the source code for a more detailed example.

API reference

• setVideoFrameDelegate(_:)
• onRenderVideoFrame(_:uid:channelId:)
• onCapture(_:sourceType:)

Understand the tech

Video SDK enables you to pre-process the captured video frames before sending the data to the encoder or perform post-processing on the received video frames after sending the data to the decoder.

The following figure shows the video data processing flow in the SDK video module.

Process raw video

• Position (2) corresponds to the onCaptureVideoFrame callback.
• Position (3) corresponds to the onPreEncodeVideoFrame callback.
• Position (4) corresponds to theonRenderVideoFrame callback.

Prerequisites

Ensure that you have implemented the SDK quickstart in your project.

Implement raw video processing

To implement raw video data functionality in your project, refer to the following steps:

1. Before joining the channel, create an instance of IVideoFrameObserver and call registerVideoFrameObserver to register the video observer.

   When modifying parameters in a VideoFrame, ensure that the updated parameters match the actual video frame in the buffer. Failure to do so may result in unexpected rotation or distortion in both the local preview and the remote video.

// Register or unregister the video observer
BOOL CGrayVideoProcFrameObserver::RegisterVideoFrameObserver(
  BOOL bEnable, IVideoFrameObserver *videoFrameObserver) {

  // Create an AutoPtr instance using IMediaEngine as a template for IMediaEngine
  agora::util::AutoPtr<agora::media::IMediaEngine> mediaEngine;
  // Use the queryInterface method through AutoPtr instance to get a pointer to IMediaEngine instance
  // Access the IMediaEngine instance pointer through the arrow operator of AutoPtr instance
  // Call registerVideoFrameObserver through IMediaEngine instance using AutoPtr instance
  mediaEngine.queryInterface(m_rtcEngine, AGORA_IID_MEDIA_ENGINE);
  int nRet = 0;

  agora::base::AParameter apm(*m_rtcEngine);

  if (mediaEngine.get() == NULL) return FALSE;

  if (bEnable) {
    // Register the video frame observer
    nRet = mediaEngine->registerVideoFrameObserver(videoFrameObserver);
  } else {
    // Unregister the video observer
    nRet = mediaEngine->registerVideoFrameObserver(nullptr);
  }

  return nRet == 0 ? TRUE : FALSE;
}

2. Obtain video data through the onCaptureVideoFrame and onRenderVideoFrame callbacks and process the video frames according to your requirements.

   // Process locally captured raw video data from the camera, perform grayscale processing, and then send it back to the SDK
   bool CGrayVideoProcFrameObserver::onCaptureVideoFrame(VideoFrame &videoFrame) {
     // Calculate the size of the video frame
     int nSize = videoFrame.height * videoFrame.width;
   
     // Apply grayscale processing to the U and V components of the video frame
     memset(videoFrame.uBuffer, 128, nSize / 4);
     memset(videoFrame.vBuffer, 128, nSize / 4);
   
     // Return true to indicate successful processing
     return true;
   }
   
   // Process raw video data received from a remote user
   bool CGrayVideoProcFrameObserver::onRenderVideoFrame(const char *channelId,
                             rtc::uid_t remoteUid,
                             VideoFrame &videoFrame) {
     // Return true to indicate that the received video frame is processed successfully
     return true;
   }
   
   // Apply an average filter to the locally captured raw video data
   bool CAverageFilterVideoProcFrameObserver::onCaptureVideoFrame(
     VideoFrame &videoFrame) {
     // Static variables to control the step size and direction of the average filter
     static int step = 1;
     static bool flag = true;
   
     // Update the step size based on the flag
     if (flag) {
       step += 2;
     } else {
       step -= 2;
     }
   
     // Adjust step size and direction based on certain conditions
     if (step >= 151) {
       flag = false;
       step -= 4;
     } else if (step <= 0) {
       flag = true;
       step += 4;
     }
   
     // Apply average filtering to the Y, U, and V components of the video frame
     AverageFiltering((unsigned char *)videoFrame.yBuffer, videoFrame.width,
             videoFrame.height, step);
     AverageFiltering((unsigned char *)videoFrame.uBuffer, videoFrame.width / 2,
             videoFrame.height / 2, step);
     AverageFiltering((unsigned char *)videoFrame.vBuffer, videoFrame.width / 2,
             videoFrame.height / 2, step);
   
     // Return true to indicate successful processing
     return true;
   }
   
   // Process the locally captured raw video data from screen capture
   bool CGrayVideoProcFrameObserver::onScreenCaptureVideoFrame(
     VideoFrame &videoFrame) {
     // Return true to indicate that the screen-captured video frame is processed successfully
     return true;
   }

Reference

This section contains content that completes the information on this page, or points you to documentation that explains other aspects to this product.

Sample project

Agora provides an open source sample project OriginalVideo on GitHub. Download it or view the source code for a more detailed example.

API reference

• registerVideoFrameObserver
• registerVideoEncodedFrameObserver
• onCaptureVideoFrame
• onRenderVideoFrame
• getVideoFrameProcessMode
• onEncodedVideoFrameReceived

Understand the tech

Video SDK enables you to pre-process the captured video frames before sending the data to the encoder or perform post-processing on the received video frames after sending the data to the decoder.

The following figure shows the video data processing flow in the SDK video module.

Process raw video

• Position (2) corresponds to the onCaptureVideoFrame callback.
• Position (3) corresponds to the onPreEncodeVideoFrame callback.
• Position (4) corresponds to theonRenderVideoFrame callback.

Prerequisites

Ensure that you have implemented the SDK quickstart in your project.

Implement raw video processing

To implement raw video data functionality in your project, refer to the following steps:

1. Before joining the channel, create an IVideoFrameObserver object and register the video observer by calling the RegisterVideoFrameObserver method. To apply pre-processed video data, set the formatPreference to FRAME_TYPE_YUV420 and the mode to INTPTR.

   // Register the video observer
   RtcEngine.RegisterVideoFrameObserver(
     new VideoFrameObserver(this),
     VIDEO_OBSERVER_FRAME_TYPE.FRAME_TYPE_YUV420,
     VIDEO_MODULE_POSITION.POSITION_POST_CAPTURER |
     VIDEO_MODULE_POSITION.POSITION_PRE_RENDERER |
     VIDEO_MODULE_POSITION.POSITION_PRE_ENCODER,
     OBSERVER_MODE.INTPTR
   );

2. After obtaining video data via the OnCaptureVideoFrame and OnRenderVideoFrame callbacks, process it according to your requirements.

   class VideoFrr : IVideoFrameObserver {
     // Get the original video data captured by the local camera
     public override bool OnCaptureVideoFrame(VIDEO_SOURCE_TYPE sourceType, VideoFrame videoFrame) {
       ProcessVideoFrame(videoFrame);
       return true;
     }
   
     // Get the video data sent by the remote user
     public override bool OnRenderVideoFrame(string channelId, uint remoteUid, VideoFrame videoFrame) {
       ProcessVideoFrame(videoFrame);
       return true;
     }
   
     public void ProcessVideoFrame(VideoFrame videoFrame) {
       int yBufferLength = videoFrame.yStride * videoFrame.height;
       int uBufferLength = videoFrame.uStride * videoFrame.height / 2;
       int vBufferLength = videoFrame.vStride * videoFrame.height / 2;
   
       byte[] bytes = new byte[uBufferLength];
       for (int i = 0; i < uBufferLength; i++) {
         bytes[i] = 128;
       }
   
       System.Runtime.InteropServices.Marshal.Copy(bytes, 0, videoFrame.uBufferPtr, uBufferLength);
       System.Runtime.InteropServices.Marshal.Copy(bytes, 0, videoFrame.vBufferPtr, vBufferLength);
     }
   }

Reference

This section contains content that completes the information on this page, or points you to documentation that explains other aspects to this product.

Sample project

Agora provides an open source sample project ProcessRawData on GitHub. Download it or view the source code for a more detailed example.

API reference

• RegisterVideoFrameObserver
• RegisterVideoEncodedFrameObserver
• OnCaptureVideoFrame
• OnRenderVideoFrame

Understand the tech

Video SDK allows you to process video frames using the Canvas API. Use the API to modify video frames before publishing to the channel.

This guide explains how to extract video frames from the local video track, modify them, and publish the processed frames to the channel.

The following figure shows the video data processing flow in the SDK video module.

Process raw video

Prerequisites

Ensure that you have implemented the SDK quickstart in your project.

Implement raw video processing

To implement raw video data functionality in your project, refer to the following steps:

1. Extract video frames:

   1. Create a local video track to capture the raw video stream from the camera.

   2. Assign the local video stream to a hidden <video> element to enable frame extraction.

   3. Call processFrame on onplay to modify frames before rendering or publishing.

   4. Create a canvas to process the extracted video frames.

   5. Obtain a 2D drawing context from the canvas to manipulate pixel data.

      // Declare global variables for video processing
      let cameraTrack, canvas, ctx;
      let isProcessing = false;
      
      async function extractVideoFrames() {
        // Create a video track from the camera
        cameraTrack = await AgoraRTC.createCameraVideoTrack();
      
        // Create a hidden video element to play the camera feed
        const videoElement = document.createElement('video');
        videoElement.srcObject = new MediaStream([cameraTrack.getMediaStreamTrack()]);
        videoElement.autoplay = true;
        videoElement.playsInline = true;
        videoElement.style.display = 'none';
        document.body.appendChild(videoElement);
      
        isProcessing = true;
      
        videoElement.onloadedmetadata = () => {
          canvas.width = videoElement.videoWidth;
          canvas.height = videoElement.videoHeight;
        };
      
        videoElement.onplay = () => processFrame(videoElement);
      
        // Create a canvas element to manipulate video frames
        canvas = document.createElement('canvas');
        ctx = canvas.getContext('2d', { willReadFrequently: true });
        document.body.appendChild(canvas);
      }

2. Process video frames:

   To modify video frames in real time, follow these steps:

   1. Draw the video frame onto the canvas to capture pixel data.

   2. Retrieve the pixel data from the canvas and modify it as needed.

   3. Apply the modified pixel data back to the canvas.

   4. Call requestAnimationFrame to continuously process frames in a loop.

      function processFrame(videoElement) {
        // Exit if processing is stopped or required elements are missing
        if (!isProcessing || !canvas || !ctx) return;
      
        // Ensure the video has loaded and has valid dimensions
        if (!videoElement.videoWidth || !videoElement.videoHeight) return;
      
        // Update canvas size to match video dimensions if necessary
        if (canvas.width !== videoElement.videoWidth || canvas.height !== videoElement.videoHeight) {
          canvas.width = videoElement.videoWidth;
          canvas.height = videoElement.videoHeight;
        }
      
        // Draw the current video frame onto the canvas
        ctx.drawImage(videoElement, 0, 0, canvas.width, canvas.height);
      
        // Get pixel data from the canvas
        const frame = ctx.getImageData(0, 0, canvas.width, canvas.height);
      
        // Iterate through each pixel and remove the green channel
        for (let i = 0; i < frame.data.length; i += 4) {
          frame.data[i + 1] = 0;
        }
      
        // Apply the modified pixel data back to the canvas
        ctx.putImageData(frame, 0, 0);
      
        // Request the next frame to continue processing in a loop
        requestAnimationFrame(() => processFrame(videoElement));
      }

3. Publish processed frames:

   To publish the processed frame to the channel:

   1. Create a custom video track from the modified frames using createCustomVideoTrack.

   2. Publish the custom track.

      async function extractVideoFrames() {
        // Create a custom video track from the canvas stream
        const processedTrack = await AgoraRTC.createCustomVideoTrack({
          // Capture video frames from the canvas at 30 FPS and extract the video track
          mediaStreamTrack: canvas.captureStream(30).getVideoTracks()[0],
        });
      
        // Publish the processed video track to the RTC channel
        await rtc.client.publish([processedTrack]);
      }

Reference

This section contains content that completes the information on this page, or points you to documentation that explains other aspects to this product.

API reference

• createCameraVideoTrack
• createCustomVideoTrack
• publish

Understand the tech

Video SDK enables you to pre-process the captured video frames before sending the data to the encoder or perform post-processing on the received video frames after sending the data to the decoder.

The following figure shows the video data processing flow in the SDK video module.

Process raw video

• Position (2) corresponds to the onCaptureVideoFrame callback.
• Position (3) corresponds to the onPreEncodeVideoFrame callback.
• Position (4) corresponds to theonRenderVideoFrame callback.

Prerequisites

Ensure that you have implemented the SDK quickstart in your project.

Implement raw video processing

To implement raw video data functionality in your project, refer to the following steps:

To implement raw video data functionality in your project, refer to the following steps:

1. Initialize MediaEngine and register video observer

Before joining the channel, obtain the MediaEngine interface and create an instance of IVideoFrameObserver, then call registerVideoFrameObserver to register it:

void InitAgoraEngine(FString APP_ID, FString TOKEN, FString CHANNEL_NAME)
{
  // Initialize RtcEngine context and event handler
  agora::rtc::RtcEngineContext RtcEngineContext;
  UserRtcEventHandlerEx = MakeShared<FUserRtcEventHandlerEx>(this);
  std::string StdStrAppId = TCHAR_TO_UTF8(*APP_ID);
  RtcEngineContext.appId = StdStrAppId.c_str();
  RtcEngineContext.eventHandler = UserRtcEventHandlerEx.Get();
  RtcEngineContext.channelProfile = agora::CHANNEL_PROFILE_TYPE::CHANNEL_PROFILE_LIVE_BROADCASTING;

  // Initialize the RtcEngine
  int ret = AgoraUERtcEngine::Get()->initialize(RtcEngineContext);

  // Query for the MediaEngine interface
  AgoraUERtcEngine::Get()->queryInterface(AGORA_IID_MEDIA_ENGINE, (void**)&MediaEngine);

  // Create and register video frame observer
  UserVideoFrameObserver = MakeShared<FUserVideoFrameObserver>(this);
  MediaEngine->registerVideoFrameObserver(UserVideoFrameObserver.Get());
}

2. Join channel and configure video

Enable audio and video, set the client role, and join the channel:

void OnBtnJoinChannelClicked()
{
  AgoraUERtcEngine::Get()->enableAudio();
  AgoraUERtcEngine::Get()->enableVideo();
  AgoraUERtcEngine::Get()->setClientRole(CLIENT_ROLE_BROADCASTER);
  int ret = AgoraUERtcEngine::Get()->joinChannel(TCHAR_TO_UTF8(*Token), TCHAR_TO_UTF8(*ChannelName), "", 0);

  // Set up UI for raw video rendering
  MakeVideoViewForRawData();
}

3. Implement video frame observer class

Create a class that inherits from IVideoFrameObserver and implement the required callbacks to receive and process video frames:

class FUserVideoFrameObserver : public agora::media::IVideoFrameObserver
{
public:
  FUserVideoFrameObserver(UProcessVideoRawDataWidget* InWidgetPtr) : WidgetPtr(InWidgetPtr) {}

  bool onCaptureVideoFrame(agora::rtc::VIDEO_SOURCE_TYPE sourceType, agora::media::base::VideoFrame& videoFrame) override
  {
    if (!IsWidgetValid())
      return false;

    // Process captured video frames
    WidgetPtr->RenderRawData(videoFrame);
    return true;
  }

  bool onPreEncodeVideoFrame(agora::rtc::VIDEO_SOURCE_TYPE sourceType, agora::media::base::VideoFrame& videoFrame) override
  {
    return false;
  }

  bool onMediaPlayerVideoFrame(agora::media::base::VideoFrame& videoFrame, int mediaPlayerId) override
  {
    return false;
  }

  bool onRenderVideoFrame(const char* channelId, agora::rtc::uid_t remoteUid, agora::media::base::VideoFrame& videoFrame) override
  {
    // Process remote user video frames
    return false;
  }

  bool onTranscodedVideoFrame(agora::media::base::VideoFrame& videoFrame) override
  {
    return false;
  }

  // Required methods to specify video processing mode and format preference
  agora::media::IVideoFrameObserver::VIDEO_FRAME_PROCESS_MODE getVideoFrameProcessMode() override
  {
    return agora::media::IVideoFrameObserver::PROCESS_MODE_READ_ONLY;
  }

  agora::media::base::VIDEO_PIXEL_FORMAT getVideoFormatPreference() override
  {
    return agora::media::base::VIDEO_PIXEL_RGBA;
  }

private:
  UProcessVideoRawDataWidget* WidgetPtr;

  bool IsWidgetValid() const
  {
    return WidgetPtr && IsValid(WidgetPtr);
  }
};

4. Implement raw video data rendering

Process the received video frames and render them using Unreal Engine's texture system:

void RenderRawData(agora::media::base::VideoFrame& videoFrame)
{
  TWeakObjectPtr<UProcessVideoRawDataWidget> SelfWeakPtr(this);
  if (!SelfWeakPtr.IsValid())
    return;

  int Width = videoFrame.width;
  int Height = videoFrame.height;
  uint8* rawdata = new uint8[Width * Height * 4];
  memcpy(rawdata, videoFrame.yBuffer, Width * Height * 4);

  AsyncTask(ENamedThreads::GameThread, [=, this]()
  {
    if (!SelfWeakPtr.IsValid())
      return;

    TWeakObjectPtr<UDraggableVideoViewWidget> VideoRenderViewWeakPtr(VideoRenderView);
    if (!VideoRenderViewWeakPtr.IsValid())
      return;

    if (RenderTexture == nullptr || !RenderTexture->IsValidLowLevel() ||
      RenderTexture->GetSizeX() != Width || RenderTexture->GetSizeY() != Height)
    {
      RenderTexture = UTexture2D::CreateTransient(Width, Height, PF_R8G8B8A8);
    }

    // Copy raw data to texture
    uint8* raw = (uint8*)RenderTexture->GetPlatformData()->Mips[0].BulkData.Lock(LOCK_READ_WRITE);
    memcpy(raw, rawdata, Width * Height * 4);
    delete[] rawdata;
    RenderTexture->GetPlatformData()->Mips[0].BulkData.Unlock();
    RenderTexture->UpdateResource();

    // Update UI brush and size
    RenderBrush.SetResourceObject(RenderTexture);
    RenderBrush.SetImageSize(FVector2D(Width, Height));
    VideoRenderViewWeakPtr->View->SetBrush(RenderBrush);

    UCanvasPanelSlot* CanvasPanelSlot = UWidgetLayoutLibrary::SlotAsCanvasSlot(VideoRenderViewWeakPtr.Get());
    CanvasPanelSlot->SetSize(FVector2D(Width, Height));
  });
}

5. Set up UI for raw video display

Create and manage the UI widget for displaying the raw video data:

void MakeVideoViewForRawData()
{
  ReleaseVideoViewForRawData();

  UWorld* world = GEngine->GameViewport->GetWorld();
  VideoRenderView = CreateWidget<UDraggableVideoViewWidget>(world, DraggableVideoViewTemplate);

  FText ShowedText = FText::FromString(FString("RawDataRenderView"));
  VideoRenderView->Text->SetText(ShowedText);

  UPanelSlot* PanelSlot = CanvasPanel_VideoView->AddChild(VideoRenderView);
  UCanvasPanelSlot* CanvasPanelSlot = UWidgetLayoutLibrary::SlotAsCanvasSlot(VideoRenderView);
}

void ReleaseVideoViewForRawData()
{
  if (VideoRenderView != nullptr)
  {
    VideoRenderView->RemoveFromParent();
    VideoRenderView = nullptr;
  }
}

6. Unregister the video frame observer

Call registerVideoFrameObserver with nullptr to unregister the video frame observer before leaving the channel:

void UnInitAgoraEngine()
{
  if (AgoraUERtcEngine::Get() != nullptr)
  {
    AgoraUERtcEngine::Get()->leaveChannel();
    ReleaseVideoViewForRawData();

    if (MediaEngine != nullptr)
    {
      // Unregister the video frame observer
      MediaEngine->registerVideoFrameObserver(nullptr);
    }

    AgoraUERtcEngine::Get()->unregisterEventHandler(UserRtcEventHandlerEx.Get());
    AgoraUERtcEngine::Release();
  }
}

Reference

This section contains content that completes the information on this page, or points you to documentation that explains other aspects to this product.

Sample project

Agora provides an open source sample project ProcessVideoRawData on GitHub. Download it or view the source code for a more detailed example.

API reference

• registerVideoFrameObserver
• registerVideoEncodedFrameObserver
• onCaptureVideoFrame
• onRenderVideoFrame
• getVideoFrameProcessMode
• onEncodedVideoFrameReceived