Agora 原始数据接口是 SDK 库提供的高级功能，便于你（开发者）获取媒体引擎的原始语音或视频数据。开发者可以修改语音或视频数据，创建特效来更好地满足自己应用程序的特殊需求。

你可以在将数据发送给编码器前插入一个前处理阶段，对捕捉到的视频帧或语音信号进行修改。也可以在将数据发送给解码器后插入一个后处理阶段，对接收到的视频帧和语音信号进行修改。

Agora 原始数据接口是一个 C++ 接口。

修改语音数据

1. 定义 AgoraAudioFrameObserver 继承 IAudioFrameObserver (接口类 IAudioFrameObserver 在 IAgoraMediaEngine.h 定义) 。需要实现以下虚拟接口:

    class AgoraAudioFrameObserver : public agora::media::IAudioFrameObserver
    {
        public:
            // 10 毫秒自动回调：获取录制的声音
            virtual bool onRecordAudioFrame(AudioFrame& audioFrame) override
            {
                return true;
            }
            // 10 毫秒自动回调：获取播放的声音
            virtual bool onPlaybackAudioFrame(AudioFrame& audioFrame) override
            {
                return true;
             }
            // 10 毫秒自动回调：获取混音前的指定用户的声音
            virtual bool onPlaybackAudioFrameBeforeMixing(unsigned int uid, AudioFrame& audioFrame) override
             {
                return true;
             }
            // 10 毫秒自动回调：获取录制和播放语音混音后的数据
            virtual bool onMixedAudioFrame(AudioFrame& audioFrame) override
             {
             return true;
             }
   
    };

   上述例子为语音前处理或后处理接口返回 true。如有需要，用户可以修改数据:

    class IAudioFrameObserver
    {
        public:
                enum AUDIO_FRAME_TYPE {
                FRAME_TYPE_PCM16 = 0, //PCM 16bit little endian
                };
        struct AudioFrame {
                AUDIO_FRAME_TYPE type;
                int samples;  //number of samples in this frame
                int bytesPerSample; //number of bytes per sample: 2 for PCM 16
                int channels; // number of channels (data are interleaved if stereo)
                int samplesPerSec; //sampling rate
                void* buffer; //data buffer
                int64_t renderTimeMs; // timestamp of the audio frame
             };
        public:
                virtual bool onRecordAudioFrame(AudioFrame& audioFrame) = 0;
                virtual bool onPlaybackAudioFrame(AudioFrame& audioFrame) = 0;
                virtual bool onPlaybackAudioFrameBeforeMixing(unsigned int uid, AudioFrame& audioFrame) = 0;
                virtual bool onMixedAudioFrame(AudioFrame& audioFrame) = 0;
    };

2. 向媒体引擎注册视频帧观测器。在创建了 IRtcEngine 对象后、加入频道前，你可以注册语音观测器对象。

    AgoraAudioFrameObserver s_audioFrameObserver;
   
    agora::util::AutoPtr<agora::media::IMediaEngine> mediaEngine;
    mediaEngine.queryInterface(*engine, agora::AGORA_IID_MEDIA_ENGINE);
    if (mediaEngine)
    {
        mediaEngine->registerAudioFrameObserver(&s_audioFrameObserver);
    }

可以通过以下方法获得 engine。其中 kit 指的是 AgoraRtcEngineKit。

agora::IRtcEngine* rtc_engine = (agora::IRtcEngine*)kit.getNativeHandle;
agora::util::AutoPtr<agora::media::IMediaEngine> mediaEngine;
mediaEngine.queryInterface(rtc_engine, agora::AGORA_IID_MEDIA_ENGINE);

API 参考

• registerAudioFrameObserver
• setRecordingAudioFrameParameters
• setPlaybackAudioFrameParameters
• setMixedAudioFrameParameters
• onRecordAudioFrame
• onPlaybackAudioFrame
• onPlaybackAudioFrameBeforeMixing
• onMixedAudioFrame

修改视频数据

1. 定义 AgoraVideoFrameObserver 继承 IVideoFrameObserver(接口类 IVideoFrameObserver 在 IAgoraMediaEngine.h 定义) 。需要实现以下虚拟接口:

   class AgoraVideoFrameObserver : public agora::media::IVideoFrameObserver
   {
     public:
       // 获取采集的视频
       virtual bool onCaptureVideoFrame(VideoFrame& videoFrame) override
       {
         return true;
       }
       // 获取其他用户的视频
       virtual bool onRenderVideoFrame(unsigned int uid, VideoFrame& videoFrame) override
       {
         return true;
       }
   };

   上述例子仅需保证语音前后处理的返回值为 true。如有需要，可参考 API 修改语音帧的样本数量、频道数量、采样率等:

    class IVideoFrameObserver
    {
        public:
            enum VIDEO_FRAME_TYPE {
            FRAME_TYPE_YUV420 = 0,  //YUV 420 format
            };
        struct VideoFrame {
            VIDEO_FRAME_TYPE type;
            int width;  //width of video frame
            int height;  //height of video frame
            int yStride;  //stride of Y data buffer
            int uStride;  //stride of U data buffer
            int vStride;  // stride of V data buffer
            void* yBuffer;  //Y data buffer
            void* uBuffer;  //U data buffer
            void* vBuffer;  //V data buffer
            int rotation; // rotation of this frame (0, 90, 180, 270)
            int64_t renderTimeMs; //timestamp of the audio frame
            };
        public:
            virtual bool onCaptureVideoFrame(VideoFrame& videoFrame) = 0;
            virtual bool onRenderVideoFrame(unsigned int uid, VideoFrame& videoFrame) = 0;
    };

2. 上述例子仅需保证视频前后处理的返回值为返回为 true。如有需要，可参考 API 修改视频帧的视频像素、行跨度等:

    AgoraVideoFrameObserver s_videoFrameObserver;
   
    agora::util::AutoPtr<agora::media::IMediaEngine> mediaEngine;
        mediaEngine.queryInterface(*engine,agora::AGORA_IID_MEDIA_ENGINE);
        if (mediaEngine)
        {
             mediaEngine->registerVideoFrameObserver(&s_videoFrameObserver);
        }

可以通过以下方法获得 engine。其中 kit 指的是 AgoraRtcEngineKit。

agora::IRtcEngine* rtc_engine = (agora::IRtcEngine*)kit.getNativeHandle;
agora::util::AutoPtr<agora::media::IMediaEngine> mediaEngine;
mediaEngine.queryInterface(*rtc_engine, agora::AGORA_IID_MEDIA_ENGINE);