Modify Raw Data

Introduction

Agora raw data interface is an advanced feature provided in the SDK library for users to obtain the audio/video raw data of the SDK engine. You, as a developer, can modify the audio or video data and create special effects to meet special needs of their applications.

You can insert a pre-processing stage before sending the data to the encoder, modifying the captured of video frames or audio signals. You can also insert a post-processing stage before sending the data to the decoder, modifying the received video frames or audio signals.

Agora raw data interface is a C++ interface.

Note

  • If you use Agora Native SDK for Android, you need to use the JNI and plug-in manager included in the SDK library.
  • For all the APIs used in this document, refer to Raw Data API.

Prerequisites

Ensure that you read this section before implementing the functions mentioned in this document. The following lists the steps for each platform SDK:

Procedure Platform
Modifying audio raw data Android/iOS/Mac/Windows
Modifying video raw data Android/iOS/Mac/Windows
Creating Agora Native SDK Plugin Android/Windows

Modifying Audio Raw Data

  1. Define AgoraAudioFrameObserver by inheriting IAudioFrameObserver(The IAudioFrameObserver Class is defined in IAgoraMediaEngine.h). You need to realize the following virtual interfaces:

For example,

class AgoraAudioFrameObserver : public agora::media::IAudioFrameObserver
{
  public:
    virtual bool onRecordAudioFrame(AudioFrame& audioFrame) override
    {
      return true;
    }
    virtual bool onPlaybackAudioFrame(AudioFrame& audioFrame) override
    {
      return true;
     }
    virtual bool onPlaybackAudioFrameBeforeMixing(unsigned int uid, AudioFrame& audioFrame) override
     {
      return true;
     }
    virtual bool onMixedAudioFrame(AudioFrame& audioFrame) override
     {
     return true;
     }

};

The above example returns true for audio pre-processing or post-processing interfaces. Users can modify the data if necessary:

class IAudioFrameObserver
{
  public:
      enum AUDIO_FRAME_TYPE {
      FRAME_TYPE_PCM16 = 0, //PCM 16bit little endian
      };
  struct AudioFrame {
      AUDIO_FRAME_TYPE type;
      int samples;  //number of samples in this frame
      int bytesPerSample; //number of bytes per sample: 2 for PCM 16
      int channels; // number of channels (data are interleaved if stereo)
      int samplesPerSec; //sampling rate
      void* buffer; //data buffer
      int64_t renderTimeMs;
     };
  public:
      virtual bool onRecordAudioFrame(AudioFrame& audioFrame) = 0;
      virtual bool onPlaybackAudioFrame(AudioFrame& audioFrame) = 0;
      virtual bool onPlaybackAudioFrameBeforeMixing(unsigned int uid, AudioFrame& audioFrame) = 0;
      virtual bool onMixedAudioFrame(AudioFrame& audioFrame) = 0;
};
  1. Register the audio frame observer to the SDK engine. After creating the IRtcEngine object, and before joining a channel, you can register the object of audio observer.
AgoraAudioFrameObserver s_audioFrameObserver;

agora::util::AutoPtr<agora::media::IMediaEngine> mediaEngine;
mediaEngine.queryInterface(*engine, agora::rtc::AGORA_IID_MEDIA_ENGINE);
if (mediaEngine)
{
  mediaEngine->registerAudioFrameObserver(&s_audioFrameObserver);
}

Note

  • For Android and Widnows: Here *engine can be passed in by loadAgoraRtcEnginePlugin in Creating Agora Native SDK Plugin.

  • For iOS and Mac: Here *engine can be obtained using the following method, and kit means AgoraRtcEngineKit*.

    agora::rtc::IRtcEngine* rtc_engine = (agora::rtc::IRtcEngine*)kit.getNativeHandle;
    agora::util::AutoPtr<agora::media::IMediaEngine> mediaEngine;
    mediaEngine.queryInterface(*rtc_engine, agora::rtc::AGORA_IID_MEDIA_ENGINE);
    

Modifying Video Raw Data

  1. Define AgoraVideoFrameObserver by inheriting IVideoFrameObserver(The IVideoFrameObserver Class is defined in IAgoraMediaEngine.h). You need to realize the following virtual interfaces:

    For example,

    class AgoraVideoFrameObserver : public agora::media::IVideoFrameObserver
    {
      public:
        virtual bool onCaptureVideoFrame(VideoFrame& videoFrame) override
        {
          return true;
        }
        virtual bool onRenderVideoFrame(unsigned int uid, VideoFrame& videoFrame) override
        {
          return true;
        }
    };
    

The above example returns true for audio pre-processing or post-processing interfaces. Users can modify the data if necessary:

class IVideoFrameObserver
{
  public:
    enum VIDEO_FRAME_TYPE {
    FRAME_TYPE_YUV420 = 0,  //YUV 420 format
    };
  struct VideoFrame {
    VIDEO_FRAME_TYPE type;
    int width;  //width of video frame
    int height;  //height of video frame
    int yStride;  //stride of Y data buffer
    int uStride;  //stride of U data buffer
    int vStride;  // stride of V data buffer
    void* yBuffer;  //Y data buffer
    void* uBuffer;  //U data buffer
    void* vBuffer;  //V data buffer
    int rotation; // rotation of this frame (0, 90, 180, 270)
    int64_t renderTimeMs;
    };
  public:
    virtual bool onCaptureVideoFrame(VideoFrame& videoFrame) = 0;
    virtual bool onRenderVideoFrame(unsigned int uid, VideoFrame& videoFrame) = 0;
};
  1. Register the video frame observer to the SDK engine. After creating the IRtcEngine object and enabling the video mode, and before joining a channel, you can register the object of video observer.
AgoraVideoFrameObserver s_videoFrameObserver;

agora::util::AutoPtr<agora::media::IMediaEngine> mediaEngine;
  mediaEngine.queryInterface(*engine,agora::rtc::AGORA_IID_MEDIA_ENGINE);
  if (mediaEngine)
  {
     mediaEngine->registerVideoFrameObserver(&s_videoFrameObserver);
  }

Note

  • For Android and Widnows: Here *engine can be passed in by loadAgoraRtcEnginePlugin in Creating Agora Native SDK Plugin.

  • For iOS and Mac: Here *engine can be obtained using the following method, and kit means AgoraRtcEngineKit*.

    agora::rtc::IRtcEngine* rtc_engine = (agora::rtc::IRtcEngine*)kit.getNativeHandle;
    agora::util::AutoPtr<agora::media::IMediaEngine> mediaEngine;
    mediaEngine.queryInterface(*rtc_engine, agora::rtc::AGORA_IID_MEDIA_ENGINE);
    

Creating Agora Native SDK Plugin

Android

Agora Native SDK supports loading the third-party plugin by using dynamic link libraries. The usage is as follows:

  1. Create a shared library project, and the SO file must be created with libapm- as prefix and so as suffix. For example, libapm-encryption.so.
  2. Implement and export the related interface functions.
  • The following plugin interface function is mandatory which is to be called when the Agora SDK loads the plugin. It returns 0 upon a successful registration, and the registration fails when it does not return 0.

    extern "C" __attribute__((visibility("default"))) int loadAgoraRtcEnginePlugin(agora::rtc::IRtcEngine* engine);
    

    Sample Code:

    extern "C" __attribute__((visibility("default"))) int loadAgoraRtcEnginePlugin(agora::rtc::IRtcEngine* engine)
    {
        agora::util::AutoPtr<agora::media::IMediaEngine> mediaEngine;
        mediaEngine.queryInterface(*engine,agora::rtc::AGORA_IID_MEDIA_ENGINE);
        if (mediaEngine)
        {
            mediaEngine->registerVideoFrameObserver(&s_videoFrameObserver);
        }
      return 0;
    }
    
  • The following plugin interface function is optional which is to be called when the Agora SDK removes the plugin.

    extern "C" __attribute__((visibility("default"))) void unloadAgoraRtcEnginePlugin(agora::rtc::IRtcEngine* engine);
    

    Sample Code:

    extern "C" __attribute__((visibility("default"))) void unloadAgoraRtcEnginePlugin(agora::rtc::IRtcEngine* engine)
    {
        agora::util::AutoPtr<agora::media::IMediaEngine> mediaEngine;
        mediaEngine.queryInterface(*engine,agora::rtc::AGORA_IID_MEDIA_ENGINE);
        if (mediaEngine)
        {
            mediaEngine->registerVideoFrameObserver(NULL);
        }
    }
    

The process of Agora Native SDK loading the plugin is listed as follows:

  1. The Agora SDK engine searches all matched dynamic link libraries in the format of libapm-xxx.so in the directory that contains the native libraries during the initialization. For example, libapm-encryption.so.
  2. The Agora SDK loads the plugin using dlopen.
  3. The user must implement the plugin and export the loadAgoraRtcEnginePlugin interface according to the above instructions. The SDK obtains and calls loadAgoraRtcEnginePlugin for each plugin found. If it returns 0 after calling the loadAgoraRtcEnginePlugin function, then the execution succeeds. Otherwise the SDK will remove the plugin (FreeLibrary). The input parameter of the interface is agora:rtc:: IRtcEngine interface object, which can be used by the plugin in the loadAgoraRtcEnginePlug function to register IPacketObserver, IAudioFrameObserver, IVideoFrameObserver and etc.
  4. When the SDK engine is destroyed, call the optional entry function unloadAgoraRtcEnginePlugin of the plugin.

Windows

Agora Native SDK supports loading the third-party plugin by using dynamic link libraries. The usage is as follows:

  1. Create a Windows DLL project, and the DLL file must be created with libapm- as prefix and dll as suffix. For example, libapm-encryption.dll.
  2. Implement and export the related interface functions.
  • The following plugin interface function is mandatory which is to be called when the Agora SDK loads the plugin. It returns 0 upon a successful registration, and the registration fails when it does not return 0.

    extern "C" __declspec(dllexport) __cdecl int loadAgoraRtcEnginePlugin(agora::rtc::IRtcEngine* engine);
    

Sample Code:

extern "C" __declspec(dllexport) __cdecl int loadAgoraRtcEnginePlugin(agora::rtc::IRtcEngine* engine)
{
    agora::util::AutoPtr<agora::media::IMediaEngine> mediaEngine;
    mediaEngine.queryInterface(*engine,agora::rtc::AGORA_IID_MEDIA_ENGINE);
    if (mediaEngine)
    {
        mediaEngine->registerVideoFrameObserver(&s_videoFrameObserver);
    }
  return 0;
}


* The following plugin interface function is optional which is to be called when the Agora SDK removes the plugin.
extern "C" __declspec(dllexport) __cdecl void unloadAgoraRtcEnginePlugin(agora::rtc::IRtcEngine* engine);

Sample Code:

extern "C" __declspec(dllexport) __cdecl void unloadAgoraRtcEnginePlugin(agora::rtc::IRtcEngine* engine)
{
    agora::util::AutoPtr<agora::media::IMediaEngine> mediaEngine;
    mediaEngine.queryInterface(*engine,agora::rtc::AGORA_IID_MEDIA_ENGINE);
    if (mediaEngine)
    {
        mediaEngine->registerVideoFrameObserver(NULL);
    }
}

The process of Agora Native SDK loading the plugin is listed as follows:

  1. The Agora SDK engine searches all matched dynamic link libraries in the format of libapm-xxx.dll in the directory that contains the native libraries during the initialization. For example, libapm-encryption.dll.
  2. The Agora SDK loads the plugin using LoadLibrary.
  3. The user must implement the plugin and export the loadAgoraRtcEnginePlugin interface according to the above instructions. The SDK obtains and calls loadAgoraRtcEnginePlugin for each plugin found. If it returns 0 after calling the loadAgoraRtcEnginePlugin function, then the execution succeeds. Otherwise the SDK will remove the plugin (FreeLibrary). The input parameter of the interface is agora:rtc:: IRtcEngine interface object, which can be used by the plugin in the loadAgoraRtcEnginePlug function to register IPacketObserver, IAudioFrameObserver, IVideoFrameObserver and etc.
  4. When the SDK engine is destroyed, call the optional entry function unloadAgoraRtcEnginePlugin of the plugin.