Raw Video Data

During real-time communications, you can pre- and post-process the video data and modify it for desired playback effects.

The Native SDK uses the IVideoFrameObserver class to provide raw video data functions. You can pre-process the data before sending it to the encoder and modify the captured video frames. You can also post-process the data after sending it to the decoder and modify the received video frames.

Agora provides the following open-source sample projects on GitHub that implement the raw video data function:

• iOS: RawMediaData
• macOS: RawMediaData

You can download them to try out this function and view the source code.

Follow these steps to implement the raw video data function in your project:

1. Before joining the channel, call registerVideoFrameObserver to register the video frame observer, and implement the IVideoFrameObserver class.
2. After registering the observer, the SDK returns the raw video data in the onCaptureVideoFrame, onPreEncodeVideoFrame, or onRenderVideoFrame callbacks for each video frame.
3. Process the raw video data, and then send the processed data back to the SDK through the callbacks mentioned in step 2.

The Agora SDK provides only C++ methods and callbacks for implementing the raw video data function. Therefore, to implement the preceding steps, you need to call C++ APIs on iOS or macOS and pay attention to the following:

Mix the C++ and Objective-C code in a .mm file.

Ensure that you import the C++ header files at the beginning of the .mm file:

#import <AgoraRtcKit/IAgoraMediaEngine.h>
#import <AgoraRtcKit/IAgoraRtcEngine.h>

When you use the SDK v3.0.1 or later, note that the SDK no longer guarantees that callback functions in IVideoFrameObserver are reported in the same thread. The SDK only guarantees the sequence of these callbacks. If you are using OpenGL to perform image enhancement on the raw video data, you need to actively switch the OpenGL context in the callback function in IVideoFrameObserver to adapt to a multi-threaded scenario; otherwise, the image enhancement cannot work.

The following diagram shows how to implement the raw data function in your project:

In addition to the API call sequence diagram, you can refer to the following code samples to implement the raw video data function in your project.

1. Initialize AgoraRtcEngineKit, and enable the video module.

   // Swift
   

   // Initialize AgoraRtcEngineKit
   

   let config = AgoraRtcEngineConfig()
   

   agoraKit = AgoraRtcEngineKit.sharedEngine(with: config, delegate: self)
   

   
   

   // Enable the vide module
   

   agoraKit.enableVideo()
   

2. Register the video frame observer.

   // Swift
   

   let videoType:ObserverVideoType = ObserverVideoType(rawValue: ObserverVideoType.captureVideo.rawValue | ObserverVideoType.renderVideo.rawValue | ObserverVideoType.preEncodeVideo.rawValue)
   

   agoraMediaDataPlugin?.registerVideoRawDataObserver(videoType)
   

   agoraMediaDataPlugin?.videoDelegate = self;
   

   In the .mm file, call the C++ APIs to implement registerVideoRawDataObserver.

   - (void)registerVideoRawDataObserver:(ObserverVideoType)observerType {
   

   // Get the C++ handle of the Native SDK
   

   agora::rtc::IRtcEngine* rtc_engine = (agora::rtc::IRtcEngine*)self.agoraKit.getNativeHandle;
   

   // Create IMediaEngine
   

   agora::util::AutoPtr<agora::media::IMediaEngine> mediaEngine;
   

   mediaEngine.queryInterface(rtc_engine, agora::AGORA_IID_MEDIA_ENGINE);
   

   // Call registerVideoFrameObserver to register the video frame observer
   

   mediaEngine->registerVideoFrameObserver(&s_videoFrameObserver);
   

   s_videoFrameObserver.mediaDataPlugin = self;
   

   }
   

3. Join a channel.

   // Swift
   

   let result = agoraKit.joinChannel(byToken: KeyCenter.Token, channelId: channelName, info: nil, uid: 0) {[unowned self] (channel, uid, elapsed) -> Void in
   

   self.isJoined = true
   

   LogUtils.log(message: "Join \(channel) with uid \(uid) elapsed \(elapsed)ms", level: .info)
   

   }
   

4. Get the raw video data, and process the data.

   // Swift
   

   // Get the video frame captured by the local camera.
   

   func mediaDataPlugin(_ mediaDataPlugin: AgoraMediaDataPlugin, didCapturedVideoRawData videoRawData: AgoraVideoRawData) -> AgoraVideoRawData {
   

   return videoRawData
   

   }
   

   
   

   // Get the video frame sent by the remote user.
   

   func mediaDataPlugin(_ mediaDataPlugin: AgoraMediaDataPlugin, willRenderVideoRawData videoRawData: AgoraVideoRawData, ofUid uid: uint) -> AgoraVideoRawData {
   

   return videoRawData
   

   }
   

   In the .mm file, call the C++ APIs to implement the callbacks that get the raw data.

   // Get the video frame captured by the local camera through the onCaptureVideoFrame callback
   

   virtual bool onCaptureVideoFrame(VideoFrame& videoFrame) override
   

   {
   

   if (!mediaDataPlugin && ((mediaDataPlugin.observerVideoType >> 0) == 0)) return true;
   

   @autoreleasepool {
   

   AgoraVideoRawData *newData = nil;
   

   if ([mediaDataPlugin.videoDelegate respondsToSelector:@selector(mediaDataPlugin:didCapturedVideoRawData:)]) {
   

   AgoraVideoRawData *data = getVideoRawDataWithVideoFrame(videoFrame);
   

   newData = [mediaDataPlugin.videoDelegate mediaDataPlugin:mediaDataPlugin didCapturedVideoRawData:data];
   

   modifiedVideoFrameWithNewVideoRawData(videoFrame, newData);
   

   }
   

   }
   

   return true;
   

   }
   

   
   

   // Get the video frame sent by the remote user through the onRenderVideoFrame callback
   

   virtual bool onRenderVideoFrame(unsigned int uid, VideoFrame& videoFrame) override
   

   {
   

   if (!mediaDataPlugin && ((mediaDataPlugin.observerVideoType >> 1) == 0)) return true;
   

   @autoreleasepool {
   

   AgoraVideoRawData *newData = nil;
   

   if ([mediaDataPlugin.videoDelegate respondsToSelector:@selector(mediaDataPlugin:willRenderVideoRawData:ofUid:)]) {
   

   AgoraVideoRawData *data = getVideoRawDataWithVideoFrame(videoFrame);
   

   newData = [mediaDataPlugin.videoDelegate mediaDataPlugin:mediaDataPlugin willRenderVideoRawData:data ofUid:uid];
   

   modifiedVideoFrameWithNewVideoRawData(videoFrame, newData);
   

   }
   

   }
   

   return true;
   

   }
   

5. Unregister the video frame observer.

   // Swift
   

   agoraMediaDataPlugin?.deregisterVideoRawDataObserver(ObserverVideoType(rawValue: 0))
   

   Call the C++ APIs in the .mm file to implement deregisterVideoRawDataObserver.

   - (void)deregisterVideoRawDataObserver:(ObserverVideoType)observerType {
   

   // Get the C++ handle of the Native SDK
   

   agora::rtc::IRtcEngine* rtc_engine = (agora::rtc::IRtcEngine*)self.agoraKit.getNativeHandle;
   

   // Create IMediaEngine
   

   agora::util::AutoPtr<agora::media::IMediaEngine> mediaEngine;
   

   mediaEngine.queryInterface(rtc_engine, agora::AGORA_IID_MEDIA_ENGINE);
   

   // Call registerVideoFrameObserver to unregister the video frame observer
   

   mediaEngine->registerVideoFrameObserver(NULL);
   

   s_videoFrameObserver.mediaDataPlugin = nil;
   

   }
   

• getNativeHandle
• registerVideoFrameObserver
• onCaptureVideoFrame
• onRenderVideoFrame
• onPreEncodeVideoFrame

• The raw video data function is implemented by the C++ APIs of the SDK, and you need to mix the Objective-C and C++ in a .mm file. See AgoraMediaDataPlugin.mm for reference.
• Ensure that you call getNativeHandle to get the C++ handle each time before calling registerVideoFrameObserver.
• The raw video data function is resource intensive and may affect performance.

Refer to Raw Audio Data if you want to implement the raw audio data function in your project.