During real-time communications, you can pre- and post-process the audio and video data and modify them for desired playback effects
The Native SDK uses the IAudioFrameObserver
and IVideoFrameObserver
class to provide raw data functions. You can pre-process the data before sending it to the encoder and modify the captured video frames and voice signals. You can also post-process the data after sending it to the decoder and modify the received video frames and voice signals.
This article tells how to use raw audio data with the IAudioFrameObserver
class.
Agora provides the open-source sample project that implements the raw audio data function on GitHub:
You can download them to try out this function and view the source code.
Before proceeding, ensure that you have implemented the basic real-time communication functions in your project.
Follow these steps to implement the raw data functions in your project:
registerAudioFrameObserver
method to register a audio observer object before joining the channel. You need to implement an IAudioFrameObserver
class in this method.onRecordAudioFrame
, onPlaybackAudioFrame
, onPlaybackAudioFrameBeforeMixing
, or onMixedAudioFrame
callback to send the raw audio data at the set time interval..mm
file..mm
file.registerAudioFrameObserver
, you need to call getNativeHandler
in order to receive C++ callbacks.The following diagram shows the data transfer with the IAudioFrameObserver
class.
With onRecordAudioFrame
, onPlaybackAudioFrame
, onPlaybackAudioFrameBeforeMixing
, or onMixedAudioFrame
, you can:
The following diagram shows how to implement the raw data functions in your project:
In addition to the API call sequence diagram, you can refer to the following code samples.
1. Initialize AgoraRtcEngineKit
Call sharedEngineWithConfig
to initialize AgoraRtcEngineKit.
// Swift
// Initialize AgoraRtcEngineKit
let config = AgoraRtcEngineConfig()
agoraKit = AgoraRtcEngineKit.sharedEngine(with: config, delegate: self)
2. Register an audio frame observer
// Swift
let audioType:ObserverAudioType = ObserverAudioType(rawValue: ObserverAudioType.recordAudio.rawValue | ObserverAudioType.playbackAudioFrameBeforeMixing.rawValue | ObserverAudioType.mixedAudio.rawValue | ObserverAudioType.playbackAudio.rawValue);
agoraMediaDataPlugin?.registerAudioRawDataObserver(audioType)
agoraMediaDataPlugin?.audioDelegate = self
The Agora SDK provides only C++ methods and callbacks for implementing the raw audio data function. Therefore, you need to call C++ methods on iOS or macOS to register an audio frame observer.
.mm
file.// Import the C++ header file
#import <AgoraRtcKit/IAgoraMediaEngine.h>
#import <AgoraRtcKit/IAgoraRtcEngine.h>
- (void)registerAudioRawDataObserver:(ObserverAudioType)observerType {
// Gets the C++ event handler of the RTC Native SDK
agora::rtc::IRtcEngine* rtc_engine = (agora::rtc::IRtcEngine*)self.agoraKit.getNativeHandle;
// Creates an IMediaEngine instance
agora::util::AutoPtr<agora::media::IMediaEngine> mediaEngine;
// Ensure that you call queryInterface in IMediaEngine to set the agora::AGORA_IID_MEDIA_ENGINE interface, or you cannot implement registerAudioFrameObserver
mediaEngine.queryInterface(rtc_engine, agora::AGORA_IID_MEDIA_ENGINE);
NSInteger oldValue = self.observerAudioType;
self.observerAudioType |= observerType;
if (mediaEngine && oldValue == 0)
{
// Register an audio frame observer
mediaEngine->registerAudioFrameObserver(&s_audioFrameObserver);
s_audioFrameObserver.mediaDataPlugin = self;
}
}
3. Set the audio frame capture parameters
If you want to modify the audio sampling rate, use mode, number of channels, or sampling interval of the captured audio data, you can call the following methods:
// Swift
// Sets the audio data format returned in onRecordAudioFrame
agoraKit.setRecordingAudioFrameParametersWithSampleRate(44100, channel: 1, mode: .readWrite, samplesPerCall: 4410)
// Sets the audio data format returned in onMixedAudioFrame
agoraKit.setMixedAudioFrameParametersWithSampleRate(44100, samplesPerCall: 4410)
// Sets the audio data format returned in onPlaybackAudioFrame
agoraKit.setPlaybackAudioFrameParametersWithSampleRate(44100, channel: 1, mode: .readWrite, samplesPerCall: 4410)
4. Join a channel
Call joinChannelByToken
to join an RTC channel.
// Swift
agoraKit.joinChannel(byToken: KeyCenter.Token, channelId: channelName, info: nil, uid: 0) {[unowned self] (channel, uid, elapsed)}
5. Receive the captured audio data
After joining a channel, you can receive the captured audio data from the callbacks in the IAudioFrameObserver
class. You can also use these callbacks to send the processed audio data back to the SDK.
// Swift
// Gets the raw audio data of the local user, and sends the data back to the SDK after processing
func mediaDataPlugin(_mediaDataPlugin: AgoraMediaDataPlugin, didRecord audioRawData: AgoraAudioRawDate) -> AgoraAudioRawData {
return audioRawData
}
// Gets the raw audio data of all remote users, and sends the data back to the SDK after processing
func mediaDataPlugin(_mediaDataPlugin: AgoraMediaDataPlugin, willPlaybackAudioRawData audioRawData: AgoraRawData) -> AgoraAudioRawData {
return audioRawData
}
// Gets the raw audio data of a specified remote user, and sends the data back to the SDK after processing
func mediaDataPlugin(_mediaDataPlugin: AgoraMediaDataPlugin, willPlaybackBeforeMixing audioRawData: AgoraAudioRawData, ofUid uid: uint) -> AgoraAudioRawData {
return audioRawData
}
// Gets the raw audio data of the local user and all remote users, and sends the data back to the SDK after processing
func mediaDataPlugin(_mediaDataPlugin: AgoraMediaDataPlugin, didMixedAudioRawData audioRawData: AgoraAudioRawData) -> AgoraAudioRawData {
return audioRawData
}
Call the C++ methods in the .mm
file to implement the callbacks.
// Swift
class AgoraMediaDataPluginAudioFrameObserver : public agora::media::IAudioFrameObserver
{
public:
AgoraMediaDataPlugin *mediaDataPlugin;
// Defines the format of the raw audio data
AgoraAudioRawData* getAudioRawDataWithAudioFrame(AudioFrame& audioFrame)
{
AgoraAudioRawData *data = [[AgoraAudioRawData alloc] init];
data.samples = audioFrame.samples;
data.bytesPerSample = audioFrame.bytesPerSample;
data.channels = audioFrame.channels;
data.samplesPerSec = audioFrame.samplesPerSec;
data.renderTimeMs = audioFrame.renderTimeMs;
data.buffer = (char *)audioFrame.buffer;
data.bufferSize = audioFrame.samples * audioFrame.bytesPerSample;
return data;
}
// Defines the format of the processed audio data
void modifiedAudioFrameWithNewAudioRawData(AudioFrame& audioFrame, AgoraAudioRawData *audioRawData)
{
audioFrame.samples = audioRawData.samples;
audioFrame.bytesPerSample = audioRawData.bytesPerSample;
audioFrame.channels = audioRawData.channels;
audioFrame.samplesPerSec = audioRawData.samplesPerSec;
audioFrame.renderTimeMs = audioRawData.renderTimeMs;
}
// Receives the raw audio data of the local user from onRecordAudioFrame
virtual bool onRecordAudioFrame(AudioFrame& audioFrame) override
{
if (!mediaDataPlugin && ((mediaDataPlugin.observerAudioType >> 0) == 0)) return true;
@autoreleasepool {
if ([mediaDataPlugin.audioDelegate respondsToSelector:@selector(mediaDataPlugin:didRecordAudioRawData:)]) {
AgoraAudioRawData *data = getAudioRawDataWithAudioFrame(audioFrame);
AgoraAudioRawData *newData = [mediaDataPlugin.audioDelegate mediaDataPlugin:mediaDataPlugin didRecordAudioRawData:data];
modifiedAudioFrameWithNewAudioRawData(audioFrame, newData);
}
}
// Sets the return value as true, meaning to send the data back to the SDK
return true;
}
// Receives the raw audio data of all remote users from onPlaybackAudioFrame
virtual bool onPlaybackAudioFrame(AudioFrame& audioFrame) override
{
if (!mediaDataPlugin && ((mediaDataPlugin.observerAudioType >> 1) == 0)) return true;
@autoreleasepool {
if ([mediaDataPlugin.audioDelegate respondsToSelector:@selector(mediaDataPlugin:willPlaybackAudioRawData:)]) {
AgoraAudioRawData *data = getAudioRawDataWithAudioFrame(audioFrame);
AgoraAudioRawData *newData = [mediaDataPlugin.audioDelegate mediaDataPlugin:mediaDataPlugin willPlaybackAudioRawData:data];
modifiedAudioFrameWithNewAudioRawData(audioFrame, newData);
}
}
// Sets the return value as true, meaning to send the data back to the SDK
return true;
}
// Receives the raw audio data of all remote users from onPlaybackAudioFrameBeforeMixing
virtual bool onPlaybackAudioFrameBeforeMixing(unsigned int uid, AudioFrame& audioFrame) override
{
if (!mediaDataPlugin && ((mediaDataPlugin.observerAudioType >> 2) == 0)) return true;
@autoreleasepool {
if ([mediaDataPlugin.audioDelegate respondsToSelector:@selector(mediaDataPlugin:willPlaybackBeforeMixingAudioRawData:ofUid:)]) {
AgoraAudioRawData *data = getAudioRawDataWithAudioFrame(audioFrame);
AgoraAudioRawData *newData = [mediaDataPlugin.audioDelegate mediaDataPlugin:mediaDataPlugin willPlaybackBeforeMixingAudioRawData:data ofUid:uid];
modifiedAudioFrameWithNewAudioRawData(audioFrame, newData);
}
}
// Sets the return value as true, meaning to send the data back to the SDK
return true;
}
// Receives the raw audio data of the local user and all remote users from onMixedAudioFrame
virtual bool onMixedAudioFrame(AudioFrame& audioFrame) override
{
if (!mediaDataPlugin && ((mediaDataPlugin.observerAudioType >> 3) == 0)) return true;
@autoreleasepool {
if ([mediaDataPlugin.audioDelegate respondsToSelector:@selector(mediaDataPlugin:didMixedAudioRawData:)]) {
AgoraAudioRawData *data = getAudioRawDataWithAudioFrame(audioFrame);
AgoraAudioRawData *newData = [mediaDataPlugin.audioDelegate mediaDataPlugin:mediaDataPlugin didMixedAudioRawData:data];
modifiedAudioFrameWithNewAudioRawData(audioFrame, newData);
}
}
// Sets the return value as true, meaning to send the data back to the SDK
return true;
}
};
6. Stop registering the audio frame observer
Call registerAudioFrameObserver(NULL)
to stop registering the audio frame observer.
.mm
file.- (void)deregisterAudioRawDataObserver:(ObserverAudioType)observerType {
agora::rtc::IRtcEngine* rtc_engine = (agora::rtc::IRtcEngine*)self.agoraKit.getNativeHandle;
agora::util::AutoPtr<agora::media::IMediaEngine> mediaEngine;
mediaEngine.queryInterface(rtc_engine, agora::AGORA_IID_MEDIA_ENGINE);
self.observerAudioType ^= observerType;
if (mediaEngine && self.observerAudioType == 0)
{
mediaEngine->registerAudioFrameObserver(NULL);
s_audioFrameObserver.mediaDataPlugin = nil;
}
}
.mm
file. See AgoraMediaDataPlugin.mm for reference.getNativeHandler
to get the C++ handle before calling the C++ method.