The Agora Raw Data interface is an advanced feature provided in the SDK library for users to obtain the raw voice or video data of the SDK engine. Developers can modify the voice or video data and create special effects to meet their needs.
You can implement a preprocessing stage before sending the data to the encoder and modifying the captured video frames or voice signals. You can also implement a postprocessing stage after sending the data to the decoder and modifying the received video frames or voice signals.
The Agora Raw Data interface is a C++ interface. If you use the Agora Native SDK for Android, you need to use the JNI and plug-in manager included in the SDK library.
Modify the Raw Voice Data
Define
AgoraAudioFrameObserver
by inheritingIAudioFrameObserver
(theIAudioFrameObserver
class is defined inIAgoraMediaEngine.h
). You need the following virtual interface:class AgoraAudioFrameObserver : public agora::media::IAudioFrameObserver { public: virtual bool onRecordAudioFrame(AudioFrame& audioFrame) override { return true; } virtual bool onPlaybackAudioFrame(AudioFrame& audioFrame) override { return true; } virtual bool onPlaybackAudioFrameBeforeMixing(unsigned int uid, AudioFrame& audioFrame) override { return true; } virtual bool onMixedAudioFrame(AudioFrame& audioFrame) override { return true; } };
This example returns true for voice preprocessing or postprocessing interfaces. Users can modify the data, if necessary:
class IvoiceFrameObserver { public: enum AUDIO_FRAME_TYPE { FRAME_TYPE_PCM16 = 0, //PCM 16bit little endian }; struct AudioFrame { AUDIO_FRAME_TYPE type; int samples; //number of samples in this frame int bytesPerSample; //number of bytes per sample: 2 for PCM 16 int channels; // number of channels (data are interleaved if stereo) int samplesPerSec; //sampling rate void* buffer; //data buffer int64_t renderTimeMs; }; public: virtual bool onRecordAudioFrame(AudioFrame& audioFrame) = 0; virtual bool onPlaybackAudioFrame(AudioFrame& audioFrame) = 0; virtual bool onPlaybackAudioFrameBeforeMixing(unsigned int uid, AudioFrame& audioFrame) = 0; virtual bool onMixedAudioFrame(AudioFrame& audioFrame) = 0; };
Register the voice frame observer to the SDK engine. After creating the IRtcEngine object, and before joining a channel, you can register the voice frame observer object.
AgoraAudioFrameObserver s_audioFrameObserver; agora::util::AutoPtr<agora::media::IMediaEngine> mediaEngine; mediaEngine.queryInterface(engine, agora::AGORA_IID_MEDIA_ENGINE); if (mediaEngine) { mediaEngine->registerAudioFrameObserver(&s_audioFrameObserver); }
engine
can be passed in byloadAgoraRtcEnginePlugin
in Creating the Agora Native SDK Plugin.
Modify the Raw Video Data
Define
AgoraVideoFrameObserver
by inheritingIVideoFrameObserver
(theIVideoFrameObserver
class is defined inIAgoraMediaEngine.h
). You need the following virtual interfaces:class AgoraVideoFrameObserver : public agora::media::IVideoFrameObserver { public: virtual bool onCaptureVideoFrame(VideoFrame& videoFrame) override { return true; } virtual bool onRenderVideoFrame(unsigned int uid, VideoFrame& videoFrame) override { return true; } };
This example returns true for voice preprocessing or postprocessing interfaces. Users can modify the data, if necessary:
class IVideoFrameObserver { public: enum VIDEO_FRAME_TYPE { FRAME_TYPE_YUV420 = 0, //YUV 420 format }; struct VideoFrame { VIDEO_FRAME_TYPE type; int width; //width of the video frame int height; //height of the video frame int yStride; //stride of the Y data buffer int uStride; //stride of the U data buffer int vStride; // stride of the V data buffer void* yBuffer; //Y data buffer void* uBuffer; //U data buffer void* vBuffer; //V data buffer int rotation; // rotation of this frame (0, 90, 180, 270) int64_t renderTimeMs; }; public: virtual bool onCaptureVideoFrame(VideoFrame& videoFrame) = 0; virtual bool onRenderVideoFrame(unsigned int uid, VideoFrame& videoFrame) = 0; };
Register the video frame observer to the SDK engine. After creating the
IRtcEngine
object and enabling the video mode, you can register the video frame observer object before joining a channel, .AgoraVideoFrameObserver s_videoFrameObserver; agora::util::AutoPtr<agora::media::IMediaEngine> mediaEngine; mediaEngine.queryInterface(engine,agora::AGORA_IID_MEDIA_ENGINE); if (mediaEngine) { mediaEngine->registerVideoFrameObserver(&s_videoFrameObserver); }
engine
can be passed in byloadAgoraRtcEnginePlugin
in Creating the Agora Native SDK Plugin.
Creating the Agora Native SDK Plugin
The Agora Native SDK supports loading the third-party plugin by using dynamically linked libraries. The usage is as follows:
Create a shared library project, and the SO file must be created with
libapm-
as the prefix andso
as the suffix. For example,libapm-encryption.so
.Implement and export the related interface functions.
- The following plugin interface function is mandatory and called when the Agora SDK loads the plugin. It returns 0 upon a successful registration, and the registration fails when it does not return 0.
extern "C" __attribute__((visibility("default"))) int loadAgoraRtcEnginePlugin(agora::IRtcEngine* engine);
Sample Code:
extern "C" __attribute__((visibility("default"))) int loadAgoraRtcEnginePlugin(agora::IRtcEngine* engine) { agora::util::AutoPtr<agora::media::IMediaEngine> mediaEngine; mediaEngine.queryInterface(engine,agora::AGORA_IID_MEDIA_ENGINE); if (mediaEngine) { mediaEngine->registerVideoFrameObserver(&s_videoFrameObserver); } return 0; }
- The following plugin interface function is optional and can be called when the Agora SDK removes the plugin.
extern "C" __attribute__((visibility("default"))) void unloadAgoraRtcEnginePlugin(agora::IRtcEngine* engine);
Sample Code:
extern "C" __attribute__((visibility("default"))) void unloadAgoraRtcEnginePlugin(agora::IRtcEngine* engine) { agora::util::AutoPtr<agora::media::IMediaEngine> mediaEngine; mediaEngine.queryInterface(engine,agora::AGORA_IID_MEDIA_ENGINE); if (mediaEngine) { mediaEngine->registerVideoFrameObserver(NULL); } }
Loading the Agora Native SDK Plugin
The process of the Agora Native SDK loading the plugin is as follows:
The Agora SDK engine searches all matched dynamically linked libraries in the
libapm-xxx.so
format in the directory that contains the native libraries during initialization. For example,libapm-encryption.so
.The Agora SDK loads the plugin using
dlopen
.The user must implement the plugin and export the
loadAgoraRtcEnginePlugin
interface according to the instructions above. The SDK obtains and callsloadAgoraRtcEnginePlugin
for each plugin found. If it returns 0 after calling theloadAgoraRtcEnginePlugin
function, then the execution succeeds. Otherwise, the SDK will remove the plugin (FreeLibrary). The input parameter of the interface isagora::
. TheIRtcEngine
interface object can be used by the plugin in theloadAgoraRtcEnginePlug
function to registerIPacketObserver
,IAudioFrameObserver
, andIVideoFrameObserver
.When the SDK engine is destroyed, call the optional
unloadAgoraRtcEnginePlugin
function of the plugin.