Introduction

During real-time communications, you can pre- and post-process the audio and video data and modify them for desired playback effects

The Native SDK uses the IAudioFrameObserver and IVideoFrameObserver class to provide raw data functions. You can pre-process the data before sending it to the encoder and modify the captured video frames and voice signals. You can also post-process the data after sending it to the decoder and modify the received video frames and voice signals.

This article tells how to use raw audio data with the IAudioFrameObserver class.

Implementation

Before using the raw data functions, ensure that you have implemented the basic real-time communication functions in your project. For details, see the following documents:

Follow these steps to implement the raw data functions in your project:

  1. Call the registerAudioFrameObserver method to register a audio observer object before joining the channel. You need to implement an IAudioFrameObserver class in this method.
  2. After you successfully register the observer object, the SDK triggers the onRecordAudioFrame, onPlaybackAudioFrame, onPlaybackAudioFrameBeforeMixing, or onMixedAudioFrame callback to send the raw audio data at the set time interval.
  3. Process the captured raw data according to your needs. Send the processed data back to the SDK through the onRecordAudioFrame, onPlaybackAudioFrame, onPlaybackAudioFrameBeforeMixing, or onMixedAudioFrame callback.

API call sequence

The following diagram shows how to implement the raw data functions in your project:

Sample code

Refer to the following sample code to implement the raw data functions in your project:

class AgoraAudioFrameObserver : public agora::media::IAudioFrameObserver
{
    public:
        // Occurs when the recorded audio frame is received.
        virtual bool onRecordAudioFrame(AudioFrame& audioFrame) override
        {
            return true;
        }
        // Occurs when the audio playback frame is received
        virtual bool onPlaybackAudioFrame(AudioFrame& audioFrame) override
        {
            return true;
         }
        // Occurs when the audio playback frame of a specified user is received.
        virtual bool onPlaybackAudioFrameBeforeMixing(unsigned int uid, AudioFrame& audioFrame) override
         {
            return true;
         }
        // Occurs when the mixed recorded and playback audio frame is received.
        virtual bool onMixedAudioFrame(AudioFrame& audioFrame) override
         {
         return true;
         }

};

class IAudioFrameObserver
{
    public:
            enum AUDIO_FRAME_TYPE {
            FRAME_TYPE_PCM16 = 0, // The audio frame type: PCM16
            };
    struct AudioFrame {
            AUDIO_FRAME_TYPE type;
            int samples;  // The number of samples in this frame
            int bytesPerSample; // The number of bytes per sample: 2 for PCM 16
            int channels; // The number of channels (data are interleaved if stereo)
            int samplesPerSec; // The aampling rate
            void* buffer; // The audio data buffer
            int64_t renderTimeMs; // Timestamp of the current audio frame
         };
    public:
            virtual bool onRecordAudioFrame(AudioFrame& audioFrame) = 0;
            virtual bool onPlaybackAudioFrame(AudioFrame& audioFrame) = 0;
            virtual bool onPlaybackAudioFrameBeforeMixing(unsigned int uid, AudioFrame& audioFrame) = 0;
            virtual bool onMixedAudioFrame(AudioFrame& audioFrame) = 0;
};

We also provide an open-source AgoraAudioIO-Objective-C demo project on GitHub. You can try the demo or view the source code.

API reference

Call the following methods to modify the audio sample rate in the above callbacks:

Considerations

The methods that we use in this article are in C++. For iOS or macOS, refer to the following sample code to register the audio observer object.

static AgoraAudioFrameObserver* s_audioFrameObserver;
- (void)addRegiset:(AgoraRtcEngineKit *)agoraKit {

    // Agora Engine of C++
    agora::rtc::IRtcEngine* rtc_engine = (agora::rtc::IRtcEngine*)agoraKit.getNativeHandle;
    agora::util::AutoPtr<agora::media::IMediaEngine> mediaEngine;
    mediaEngine.queryInterface(rtc_engine, agora::AGORA_IID_MEDIA_ENGINE);

    if (mediaEngine) {
        s_audioFrameObserver = new AgoraAudioFrameObserver();
        mediaEngine->registerAudioFrameObserver(s_audioFrameObserver);
    }
}

- (void)cancelRegiset {
    agora::rtc::IRtcEngine* rtc_engine = (agora::rtc::IRtcEngine*)self.agoraKit.getNativeHandle;
    agora::util::AutoPtr<agora::media::IMediaEngine> mediaEngine;
    mediaEngine.queryInterface(rtc_engine, agora::AGORA_IID_MEDIA_ENGINE);
    mediaEngine->registerAudioFrameObserver(NULL);
}