Raw audio processing

In some use-cases, raw audio captured through the microphone must be processed to enhance the user experience or achieve the desired functionality Video SDK enables you to pre-process and post-process the captured audio for implementation of custom playback effects.

This article shows you how to pre-process and post-process collected raw audio data.

Understand the tech

For use-cases that require self-processing of audio data, Agora Video SDK provides raw data processing functionality. You can perform pre-processing to modify the captured audio signal before sending the data to the encoder, or post-process data to modify the received audio signal after sending the data to the decoder.

To implement processing of raw audio data in your app, take the following steps.

Register an instance of the audio frame observer before joining a channel.
Set the format of audio frames captured by each callback.
Implement callbacks in the frame observers to process raw audio data.
Unregister the frame observers before you leave a channel.

The following figure shows the basic processing of raw audio data:

Process raw audio

Prerequisites

Ensure that you have implemented the SDK quickstart in your project.

Implement raw audio processing

Follow these steps to implement raw audio data processing functionality in your app:

Before joining a channel, create an IAudioFrameObserver instance and call the registerAudioFrameObserver method to register the audio observer.
Call setRecordingAudioFrameParameters, setPlaybackAudioFrameParameters, and setMixedAudioFrameParameters to configure the audio frame format.
Implement onRecordAudioFrame, onPlaybackAudioFrame, onPlaybackAudioFrameBeforeMixing, and onMixedAudioFrame callbacks. These callbacks receive and process audio frames. If the return value of these callbacks is false, it indicates that the processing of the audio frames is invalid.

Refer to the following sample code to implement this logic:

// Call registerAudioFrameObserver to register an audio observer and pass in an IAudioFrameObserver instance
engine.registerAudioFrameObserver(new IAudioFrameObserver() {
  // Implement the onRecordAudioFrame callback
  @Override
  public boolean onRecordAudioFrame(byte[] samples, int numOfSamples, int bytesPerSample, int channels, int samplesPerSec) {
    if(isEnableLoopBack){
      mAudioPlayer.play(samples, 0, numOfSamples * bytesPerSample);
    }

    return false;
  }

  // Implement the onPlaybackAudioFrame callback
  @Override
  public boolean onPlaybackAudioFrame(byte[] samples, int numOfSamples, int bytesPerSample, int channels, int samplesPerSec) {
    return false;
  }

  // Implement the onPlaybackAudioFrameBeforeMixing callback
  @Override
  public boolean onPlaybackAudioFrameBeforeMixing(byte[] samples, int numOfSamples, int bytesPerSample, int channels, int samplesPerSec, int uid) {
    return false;
  }

  // Implement the onMixedAudioFrame callback
  @Override
  public boolean onMixedAudioFrame(byte[] samples, int numOfSamples, int bytesPerSample, int channels, int samplesPerSec) {
    return false;
  }

  // Call methods with 'set' prefix to configure the audio frames captured by each callback
  engine.setRecordingAudioFrameParameters(SAMPLE_RATE, SAMPLE_NUM_OF_CHANNEL, Constants.RAW_AUDIO_FRAME_OP_MODE_READ_WRITE, SAMPLES_PER_CALL);
  engine.setMixedAudioFrameParameters(SAMPLE_RATE, SAMPLES_PER_CALL);
  engine.setPlaybackAudioFrameParameters(SAMPLE_RATE, SAMPLE_NUM_OF_CHANNEL, Constants.RAW_AUDIO_FRAME_OP_MODE_READ_WRITE, SAMPLES_PER_CALL);
});

// Register an audio frame observer using registerAudioFrameObserver with an IAudioFrameObserver instance
engine.registerAudioFrameObserver(object : IAudioFrameObserver {

  // Implement the onRecordAudioFrame callback
  override fun onRecordAudioFrame(
    samples: ByteArray,
    numOfSamples: Int,
    bytesPerSample: Int,
    channels: Int,
    samplesPerSec: Int
  ): Boolean {
    if (isEnableLoopBack) {
      mAudioPlayer.play(samples, 0, numOfSamples * bytesPerSample)
    }
    return false
  }

  // Implement the onPlaybackAudioFrame callback
  override fun onPlaybackAudioFrame(
    samples: ByteArray,
    numOfSamples: Int,
    bytesPerSample: Int,
    channels: Int,
    samplesPerSec: Int
  ): Boolean {
    return false
  }

  // Implement the onPlaybackAudioFrameBeforeMixing callback
  override fun onPlaybackAudioFrameBeforeMixing(
    samples: ByteArray,
    numOfSamples: Int,
    bytesPerSample: Int,
    channels: Int,
    samplesPerSec: Int,
    uid: Int
  ): Boolean {
    return false
  }

  // Implement the onMixedAudioFrame callback
  override fun onMixedAudioFrame(
    samples: ByteArray,
    numOfSamples: Int,
    bytesPerSample: Int,
    channels: Int,
    samplesPerSec: Int
  ): Boolean {
    return false
  }
})

// Call methods with 'set' prefix to configure the audio frames captured by each callback
engine.setRecordingAudioFrameParameters(SAMPLE_RATE, SAMPLE_NUM_OF_CHANNEL, Constants.RAW_AUDIO_FRAME_OP_MODE_READ_WRITE, SAMPLES_PER_CALL)
engine.setMixedAudioFrameParameters(SAMPLE_RATE, SAMPLES_PER_CALL)
engine.setPlaybackAudioFrameParameters(SAMPLE_RATE, SAMPLE_NUM_OF_CHANNEL, Constants.RAW_AUDIO_FRAME_OP_MODE_READ_WRITE, SAMPLES_PER_CALL)

Video SDK uses a synchronous callback mechanism for processing raw audio data. When you save or rewrite data using the callbacks, consider the following best practices:

To ensure continuity of the audio stream, do not block the SDK thread by processing data directly in the callback function. Instead, make a deep copy of the received audio data and transfer the copied data to another thread for processing.
If you choose to process the audio data synchronously within the callback function, you must strictly control the processing time. For example, if the callback function is triggered every 10 milliseconds, then the processing time within the callback must be less than 10 milliseconds to prevent delays or interruptions in the audio stream.

Reference

This section contains content that completes the information on this page, or points you to documentation that explains other aspects to this product.

Audio module

Sample project

Agora provides an open-source example project ProcessAudioRawData for your reference. Download or view the project for a more detailed example.

API reference

This article shows you how to pre-process and post-process collected raw audio data.

Understand the tech

To implement processing of raw audio data in your app, take the following steps.

Register an instance of the audio frame observer before joining a channel.
Set the format of audio frames captured by each callback.
Implement callbacks in the frame observers to process raw audio data.
Unregister the frame observers before you leave a channel.

The following figure shows the basic processing of raw audio data:

Process raw audio

Prerequisites

Ensure that you have implemented the SDK quickstart in your project.

Implement raw audio processing

Follow these steps to implement raw audio data processing functionality in your app:

Before joining a channel, call setAudioFrameDelegate to set the audio frame delegate.
Call setRecordingAudioFrameParameters, setPlaybackAudioFrameParameters, and setMixedAudioFrameParameters to configure the audio frame format.
Implement onRecordAudioFrame, onPlaybackAudioFrame, onPlaybackAudioFrameBeforeMixing, and onMixedAudioFrame callbacks. These callbacks receive and process audio frames. If the return value of these callbacks is false, it indicates that the processing of the audio frames is invalid.

Refer to the following sample code to implement this logic:

class RawAudioDataMain: BaseViewController {
  var localVideo = Bundle.loadVideoView(type: .local, audioOnly: true)
  var remoteVideo = Bundle.loadVideoView(type: .remote, audioOnly: true)

  @IBOutlet weak var container: AGEVideoContainer!
  // Define agoraKit variable
  var agoraKit: AgoraRtcEngineKit!

  // ...
  // Initialize agoraKit and register the corresponding callbacks
  agoraKit = AgoraRtcEngineKit.sharedEngine(with: config, delegate: self)

  // Set the audio frame delegate.
  // You need to implement the AgoraAudioFrameDelegate protocol in this method
  agoraKit.setAudioFrameDelegate(self)

  // Configure the audio frames captured by each callback
  agoraKit.setRecordingAudioFrameParametersWithSampleRate(44100, channel: 1, mode: .readWrite, samplesPerCall: 4410)
  agoraKit.setMixedAudioFrameParametersWithSampleRate(44100, samplesPerCall: 4410)
  agoraKit.setPlaybackAudioFrameParametersWithSampleRate(44100, channel: 1, mode: .readWrite, samplesPerCall: 4410)

  // ...

  // Under the current class, implement the extension of the AgoraAudioFrameDelegat protocol
  extension RawAudioDataMain: AgoraAudioFrameDelegate {

    // Implement the onRecordAudioFrame callback
    func onRecordAudioFrame(_ frame: AgoraAudioFrame, channelId: String) -> Bool {
      return true
    }
    // Implement the onPlaybackAudioFrame callback
    func onPlaybackAudioFrame(_ frame: AgoraAudioFrame, channelId: String) -> Bool {
      return true
    }
    // Implement the onMixedAudioFrame callback
    func onMixedAudioFrame(_ frame: AgoraAudioFrame, channelId: String) -> Bool {
      return true
    }
    // Implement the onPlaybackAudioFrameBeforeMixing callback
    func onPlaybackAudioFrame(beforeMixing frame: AgoraAudioFrame, channelId: String, uid: UInt) -> Bool {
      return true
    }
  }
}

Video SDK uses a synchronous callback mechanism for processing raw audio data. When you save or rewrite data using the callbacks, consider the following best practices:

To ensure continuity of the audio stream, do not block the SDK thread by processing data directly in the callback function. Instead, make a deep copy of the received audio data and transfer the copied data to another thread for processing.
If you choose to process the audio data synchronously within the callback function, you must strictly control the processing time. For example, if the callback function is triggered every 10 milliseconds, then the processing time within the callback must be less than 10 milliseconds to prevent delays or interruptions in the audio stream.

Reference

This section contains content that completes the information on this page, or points you to documentation that explains other aspects to this product.

Audio module

Sample project

Agora provides an open-source example project RawAudioData for your reference. Download or view the project for a more detailed example.

API reference

This article shows you how to pre-process and post-process collected raw audio data.

Understand the tech

To implement processing of raw audio data in your app, take the following steps.

Register an instance of the audio frame observer before joining a channel.
Set the format of audio frames captured by each callback.
Implement callbacks in the frame observers to process raw audio data.
Unregister the frame observers before you leave a channel.

The following figure shows the basic processing of raw audio data:

Process raw audio

Prerequisites

Ensure that you have implemented the SDK quickstart in your project.

Implement raw audio processing

Follow these steps to implement raw audio data processing functionality in your app:

Before joining a channel, call setAudioFrameDelegate to set the audio frame delegate.
Call setRecordingAudioFrameParameters, setPlaybackAudioFrameParameters, and setMixedAudioFrameParameters to configure the audio frame format.
Implement onRecordAudioFrame, onPlaybackAudioFrame, onPlaybackAudioFrameBeforeMixing, and onMixedAudioFrame callbacks. These callbacks receive and process audio frames. If the return value of these callbacks is false, it indicates that the processing of the audio frames is invalid.

Refer to the following sample code to implement this logic:


class RawAudioDataMain: BaseViewController
{
  var localVideo = Bundle.loadVideoView(type: .local, audioOnly: true)
  var remoteVideo = Bundle.loadVideoView(type: .remote, audioOnly: true)

  @IBOutlet weak var container: AGEVideoContainer!
  // Define the agoraKit variable
  var agoraKit: AgoraRtcEngineKit!

  // ...
  // Initialize agoraKit and register the corresponding callbacks
  agoraKit = AgoraRtcEngineKit.sharedEngine(with: config, delegate: self)

  // Set the audio frame delegate.
  // You need to implement the AgoraAudioFrameDelegate protocol in this method
  agoraKit.setAudioFrameDelegate(self)

  // Configure the audio frames captured by each callback
  agoraKit.setRecordingAudioFrameParametersWithSampleRate(44100, channel: 1, mode: .readWrite, samplesPerCall: 4410)
  agoraKit.setMixedAudioFrameParametersWithSampleRate(44100, samplesPerCall: 4410)
  agoraKit.setPlaybackAudioFrameParametersWithSampleRate(44100, channel: 1, mode: .readWrite, samplesPerCall: 4410)

  // ...

  // In the current class, implement an extension for the AgoraAudioFrameDelegate protocol
  extension RawAudioDataMain: AgoraAudioFrameDelegate {

    // Implement the onRecordAudioFrame callback
    func onRecordAudioFrame(_ frame: AgoraAudioFrame, channelId: String) -> Bool {
      return true
    }
    // Implement the onPlaybackAudioFrame callback
    func onPlaybackAudioFrame(_ frame: AgoraAudioFrame, channelId: String) -> Bool {
      return true
    }
    // Implement the onMixedAudioFrame callback
    func onMixedAudioFrame(_ frame: AgoraAudioFrame, channelId: String) -> Bool {
      return true
    }
    // Implement the onPlaybackAudioFrameBeforeMixing callback
  func onPlaybackAudioFrame(beforeMixing frame: AgoraAudioFrame, channelId: String, uid: UInt) -> Bool {
      return true
    }
  }
}

Video SDK uses a synchronous callback mechanism for processing raw audio data. When you save or rewrite data using the callbacks, consider the following best practices:

To ensure continuity of the audio stream, do not block the SDK thread by processing data directly in the callback function. Instead, make a deep copy of the received audio data and transfer the copied data to another thread for processing.
If you choose to process the audio data synchronously within the callback function, you must strictly control the processing time. For example, if the callback function is triggered every 10 milliseconds, then the processing time within the callback must be less than 10 milliseconds to prevent delays or interruptions in the audio stream.

Reference

This section contains content that completes the information on this page, or points you to documentation that explains other aspects to this product.

Audio module

Sample project

Agora provides an open-source example project RawAudioData for your reference. Download or view the project for a more detailed example.

API reference

This article shows you how to pre-process and post-process collected raw audio data.

Understand the tech

To implement processing of raw audio data in your app, take the following steps.

Register an instance of the audio frame observer before joining a channel.
Set the format of audio frames captured by each callback.
Implement callbacks in the frame observers to process raw audio data.
Unregister the frame observers before you leave a channel.

The following figure shows the basic processing of raw audio data:

Process raw audio

Prerequisites

Ensure that you have implemented the SDK quickstart in your project.

Implement raw audio processing

Follow these steps to implement raw audio data processing functionality in your app:

Before joining a channel, create an IAudioFrameObserver instance and call the registerAudioFrameObserver method to register the audio frame observer.
Call setRecordingAudioFrameParameters, setPlaybackAudioFrameParameters, and setMixedAudioFrameParameters methods to configure the audio frame format.
Implement onRecordAudioFrame, onPlaybackAudioFrame, onPlaybackAudioFrameBeforeMixing, and onMixedAudioFrame callbacks. These callbacks receive and process audio frames. If the return value of these callbacks is false, it indicates that the processing of the audio frames is invalid.

Refer to the following sample code to implement this logic:

BOOL CAgoraOriginalAudioDlg::RegisterAudioFrameObserver(
  BOOL bEnable, IAudioFrameObserver *audioFrameObserver) {

  agora::util::AutoPtr<agora::media::IMediaEngine> mediaEngine;
  // Query AGORA_IID_MEDIA_ENGINE interface
  mediaEngine.queryInterface(m_rtcEngine, agora::rtc::AGORA_IID_MEDIA_ENGINE);
  int nRet = 0;

  if (mediaEngine.get() == NULL) return FALSE;

  if (bEnable) {
    // Register the audio observer
    nRet = mediaEngine->registerAudioFrameObserver(audioFrameObserver);
  } else {
    // Unregister the audio observer
    nRet = mediaEngine->registerAudioFrameObserver(NULL);
  }

  return nRet == 0 ? TRUE : FALSE;
}

// Implement the onRecordAudioFrame callback
bool COriginalAudioProcFrameObserver::onRecordAudioFrame(const char* channelId,
                             AudioFrame& audioFrame) {
  SIZE_T nSize = audioFrame.channels * audioFrame.samplesPerChannel * 2;
  unsigned int readByte = 0;
  int timestamp = GetTickCount();
  short *pBuffer = (short *)audioFrame.buffer;

  for (SIZE_T i = 0; i < nSize / 2; i++) {
    if (pBuffer[i] * 2 > 32767) {
      pBuffer[i] = 32767;
    } else if (pBuffer[i] * 2 < -32768) {
      pBuffer[i] = -32768;
    } else {
      pBuffer[i] *= 2;
    }
  }

#ifdef _DEBUG
  CString strInfo;
  strInfo.Format(_T("audio Frame buffer size:%d, timestamp:%d \n"), nSize, timestamp);
  OutputDebugString(strInfo);
  audioFrame.renderTimeMs = timestamp;
#endif

  return true;
}

// Implement the onPlaybackAudioFrame callback
bool COriginalAudioProcFrameObserver::onPlaybackAudioFrame(
  const char* channelId, AudioFrame& audioFrame) {
  return true;
}

// Implement the onMixedAudioFrame callback
bool COriginalAudioProcFrameObserver::onMixedAudioFrame(const char* channelId, AudioFrame& audioFrame) {
  return true;
}

// Implement the onPlaybackAudioFrameBeforeMixing callback
bool COriginalAudioProcFrameObserver::onPlaybackAudioFrameBeforeMixing(const char* channelId, rtc::uid_t uid, AudioFrame& audioFrame) {
  return true;
}

// Configure the audio frames captured by each callback
m_rtcEngine->setRecordingAudioFrameParameters(44100, 2, RAW_AUDIO_FRAME_OP_MODE_READ_WRITE, 1024);
m_rtcEngine->setPlaybackAudioFrameParameters(44100, 2, RAW_AUDIO_FRAME_OP_MODE_READ_WRITE, 1024);
m_rtcEngine->setPlaybackAudioFrameBeforeMixingParameters(44100, 2);
m_rtcEngine->setMixedAudioFrameParameters(44100, 2, 1024);

Video SDK uses a synchronous callback mechanism for processing raw audio data. When you save or rewrite data using the callbacks, consider the following best practices:

To ensure continuity of the audio stream, do not block the SDK thread by processing data directly in the callback function. Instead, make a deep copy of the received audio data and transfer the copied data to another thread for processing.
If you choose to process the audio data synchronously within the callback function, you must strictly control the processing time. For example, if the callback function is triggered every 10 milliseconds, then the processing time within the callback must be less than 10 milliseconds to prevent delays or interruptions in the audio stream.

Reference

This section contains content that completes the information on this page, or points you to documentation that explains other aspects to this product.

Audio module

Sample project

Agora provides an open-source example project ProcessRawAudioData for your reference. Download or view the project for a more detailed example.

API reference

This article shows you how to pre-process and post-process collected raw audio data.

Understand the tech

For use-cases that require self-processing of audio data, use the Web Audio API with Agora Video SDK to enable raw audio data processing. The Web Audio API allows you to modify the captured audio signal before transmitting it to the channel.

The following figure shows the basic processing of raw audio data:

Process raw audio

Prerequisites

Ensure that you have implemented the SDK quickstart in your project.

Implement raw audio processing

To modify raw audio frames before publishing, follow these steps:

Create a local audio track and publish it to the channel.
Use the Web Audio API to create an AudioContext and connect the audio track to an AnalyserNode.
Extract audio data using an AnalyserNode and process it.

The following example extracts the audio data and logs it to the console.

async function createAndProcessAudioTrack() {
  try {
    // Create a microphone audio track
    localAudioTrack = await AgoraRTC.createMicrophoneAudioTrack();

    // Publish the audio track to the RTC client
    await client.publish([localAudioTrack]);

    // Initialize the Web Audio API context for audio processing
    audioContext = new AudioContext();

    // Create a media stream source from the microphone track
    const source = audioContext.createMediaStreamSource(
      new MediaStream([localAudioTrack.getMediaStreamTrack()])
    );

    // Create an AnalyserNode to extract real-time audio frequency data
    analyser = audioContext.createAnalyser();
    analyser.fftSize = 2048;

    // Create a buffer to store the frequency data
    dataArray = new Uint8Array(analyser.frequencyBinCount);

    // Connect the audio source to the analyser
    source.connect(analyser);

    // Start continuously processing audio frames
    processAudioFrames();
  } catch (error) {
    console.error('Error creating or processing audio track:', error);
  }
}

function processAudioFrames() {
  if (!analyser) {
    console.warn('Analyser not initialized.');
    return;
  }

  // Retrieve frequency data from the analyser node
  analyser.getByteFrequencyData(dataArray);
  console.log('Audio Data:', dataArray);

  // Schedule the next frame for continuous audio analysis
  animationFrameId = requestAnimationFrame(processAudioFrames);
}

Reference

This section contains content that completes the information on this page, or points you to documentation that explains other aspects to this product.

API reference

This article shows you how to pre-process and post-process collected raw audio data.

Understand the tech

To implement processing of raw audio data in your game, take the following steps.

Register an instance of the audio frame observer before joining a channel.
Set the format of audio frames captured by each callback.
Implement callbacks in the frame observers to process raw audio data.
Unregister the frame observers before you leave a channel.

The following figure shows the basic processing of raw audio data:

Process raw audio

Prerequisites

Ensure that you have implemented the SDK quickstart in your project.

Implement raw audio processing

Follow these steps to implement raw audio data processing functionality in your game:

Before joining a channel, create an IAudioFrameObserver instance and call RegisterAudioFrameObserver to register the audio observer. To process and write back the audio data, set mode to INTPTR.
Call SetRecordingAudioFrameParameters, SetPlaybackAudioFrameParameters, and SetMixedAudioFrameParameters to configure the desired audio frame format.
Implement the following callbacks to collect and process audio frames:
- OnRecordAudioFrame
- OnPlaybackAudioFrame
- OnPlaybackAudioFrameBeforeMixing
- OnMixedAudioFrame

Refer to the following sample code to implement this logic:

// Register the audio observer
// Set the mode to INTPTR to process and write back audio data
RtcEngine.RegisterAudioFrameObserver(new AudioFrameObserver(this),
  AUDIO_FRAME_POSITION.AUDIO_FRAME_POSITION_PLAYBACK |
  AUDIO_FRAME_POSITION.AUDIO_FRAME_POSITION_RECORD |
  AUDIO_FRAME_POSITION.AUDIO_FRAME_POSITION_MIXED |
  AUDIO_FRAME_POSITION.AUDIO_FRAME_POSITION_BEFORE_MIXING |
  AUDIO_FRAME_POSITION.AUDIO_FRAME_POSITION_EAR_MONITORING,
  OBSERVER_MODE.INTPTR);

int CHANNEL = 2;
int SAMPLE_RATE = 48000;

// Configure the playback audio frame format
RtcEngine.SetPlaybackAudioFrameParameters(SAMPLE_RATE, CHANNEL, RAW_AUDIO_FRAME_OP_MODE_TYPE.RAW_AUDIO_FRAME_OP_MODE_READ_WRITE, 1024);

// Configure the recording audio frame format
RtcEngine.SetRecordingAudioFrameParameters(SAMPLE_RATE, CHANNEL, RAW_AUDIO_FRAME_OP_MODE_TYPE.RAW_AUDIO_FRAME_OP_MODE_READ_WRITE, 1024);

// Configure the mixed audio frame format
RtcEngine.SetMixedAudioFrameParameters(SAMPLE_RATE, CHANNEL, 1024);

internal class AudioFrameObserver : IAudioFrameObserver
{
  // Callback for recording audio frames
  public override bool OnRecordAudioFrame(string channelId, AudioFrame audioFrame)
  {
    ProcessAudioData(audioFrame);
    return true;
  }

  // Callback for playback audio frames
  public override bool OnPlaybackAudioFrame(string channelId, AudioFrame audioFrame)
  {
    ProcessAudioData(audioFrame);
    return true;
  }

  // Callback for playback audio frames before mixing
  public override bool OnPlaybackAudioFrameBeforeMixing(string channelId, uint uid, AudioFrame audioFrame)
  {
    ProcessAudioData(audioFrame);
    return true;
  }

  // Callback for mixed audio frames
  public override bool OnMixedAudioFrame(string channelId, AudioFrame audioFrame)
  {
    ProcessAudioData(audioFrame);
    return true;
  }

  // Process and modify the audio data from audioFrame.buffer
  public void ProcessAudioData(AudioFrame audioFrame)
  {
    long length = (long)audioFrame.bytesPerSample * audioFrame.channels * audioFrame.samplesPerChannel;
    for (long i = 0; i < length; i += (long)audioFrame.bytesPerSample)
    {
      IntPtr pos = (IntPtr)(i + (long)audioFrame.buffer);
      System.Runtime.InteropServices.Marshal.WriteByte(pos, 0);
    }
  }
}

Video SDK uses a synchronous callback mechanism for processing raw audio data. When you save or rewrite data using the callbacks, consider the following best practices:

To ensure continuity of the audio stream, do not block the SDK thread by processing data directly in the callback function. Instead, make a deep copy of the received audio data and transfer the copied data to another thread for processing.
If you choose to process the audio data synchronously within the callback function, you must strictly control the processing time. For example, if the callback function is triggered every 10 milliseconds, then the processing time within the callback must be less than 10 milliseconds to prevent delays or interruptions in the audio stream.

Reference

This section contains content that completes the information on this page, or points you to documentation that explains other aspects to this product.

Audio module

Sample project

Agora provides an open-source example project ProcessAudioRawData for your reference. Download or view the project for a more detailed example.

API reference

This article shows you how to pre-process and post-process collected raw audio data.

Understand the tech

To implement processing of raw audio data in your game, take the following steps.

Register an instance of the audio frame observer before joining a channel.
Set the format of audio frames captured by each callback.
Implement callbacks in the frame observers to process raw audio data.
Unregister the frame observers before you leave a channel.

The following figure shows the basic processing of raw audio data:

Process raw audio

Prerequisites

Ensure that you have implemented the SDK quickstart in your project.

Implement raw audio processing

Follow these steps to implement raw audio data processing functionality in your game:

Initialize MediaEngine and register the audio observer

Before joining a channel, obtain the MediaEngine interface and create an instance of IAudioFrameObserver, then call registerAudioFrameObserver to register it:

void InitAgoraEngine(FString APP_ID, FString TOKEN, FString CHANNEL_NAME)
{
    // Initialize RtcEngine context and event handler
    agora::rtc::RtcEngineContext RtcEngineContext;
    UserRtcEventHandlerEx = MakeShared<FUserRtcEventHandlerEx>(this);
    std::string StdStrAppId = TCHAR_TO_UTF8(*APP_ID);
    RtcEngineContext.appId = StdStrAppId.c_str();
    RtcEngineContext.eventHandler = UserRtcEventHandlerEx.Get();
    RtcEngineContext.channelProfile = agora::CHANNEL_PROFILE_TYPE::CHANNEL_PROFILE_LIVE_BROADCASTING;

    // Initialize the RtcEngine
    int ret = AgoraUERtcEngine::Get()->initialize(RtcEngineContext);

    // Query for the MediaEngine interface
    AgoraUERtcEngine::Get()->queryInterface(AGORA_IID_MEDIA_ENGINE, (void**)&MediaEngine);

    // Create and register audio frame observer
    UserAudioFrameObserver = MakeShared<FUserAudioFrameObserver>(this);
    MediaEngine->registerAudioFrameObserver(UserAudioFrameObserver.Get());
}

Configure audio parameters

Set the audio frame format by initializing the AudioParams structure. This determines the sample rate, number of channels, and other audio processing configurations.

void InitConfig()
{
    // Configure AudioParam
    audioParams.channels = 2;
    audioParams.sample_rate = 44100;
    audioParams.mode = agora::rtc::RAW_AUDIO_FRAME_OP_MODE_TYPE::RAW_AUDIO_FRAME_OP_MODE_READ_WRITE;
    audioParams.samples_per_call = 1024;
}

Implement audio frame observer class

Create a class that inherits from IAudioFrameObserver and implement the required callbacks to receive and process audio frames. Returning false from any of these callbacks indicates that the processing of the audio frame was invalid.

class FUserAudioFrameObserver : public agora::media::IAudioFrameObserver
{
public:
    FUserAudioFrameObserver(UProcessAudioRawDataWidget* InWidgetPtr) : WidgetPtr(InWidgetPtr) {}

    bool onPlaybackAudioFrameBeforeMixing(const char* channelId, agora::rtc::uid_t uid, AudioFrame& audioFrame) override
    {
        if (!IsWidgetValid())
            return false;

        if (WidgetPtr->GetFirstRemoteUID() == uid) {
            // Process audio frames for specific remote user
            WidgetPtr->GetAgoraSoundWaveProcedural()->AddToFrames(audioFrame);
        }
        return true;
    }

    bool onRecordAudioFrame(const char* channelId, AudioFrame& audioFrame) override
    {
        if (!IsWidgetValid())
            return false;

        return true;
    }

    bool onPlaybackAudioFrame(const char* channelId, AudioFrame& audioFrame) override
    {
        if (!IsWidgetValid())
            return false;

        return true;
    }

    bool onMixedAudioFrame(const char* channelId, AudioFrame& audioFrame) override
    {
        return true;
    }

    bool onEarMonitoringAudioFrame(AudioFrame& audioFrame) override
    {
        return true;
    }

    int getObservedAudioFramePosition() override
    {
        return (int)(AUDIO_FRAME_POSITION::AUDIO_FRAME_POSITION_PLAYBACK |
            AUDIO_FRAME_POSITION::AUDIO_FRAME_POSITION_RECORD |
            AUDIO_FRAME_POSITION::AUDIO_FRAME_POSITION_BEFORE_MIXING |
            AUDIO_FRAME_POSITION::AUDIO_FRAME_POSITION_MIXED |
            AUDIO_FRAME_POSITION::AUDIO_FRAME_POSITION_EAR_MONITORING);
    }

    agora::media::IAudioFrameObserverBase::AudioParams getPlaybackAudioParams() override
    {
        if (!IsWidgetValid())
            return agora::media::IAudioFrameObserverBase::AudioParams();
        return WidgetPtr->GetAudioParams();
    }

    agora::media::IAudioFrameObserverBase::AudioParams getRecordAudioParams() override
    {
        if (!IsWidgetValid())
            return agora::media::IAudioFrameObserverBase::AudioParams();
        return WidgetPtr->GetAudioParams();
    }

    agora::media::IAudioFrameObserverBase::AudioParams getMixedAudioParams() override
    {
        if (!IsWidgetValid())
            return agora::media::IAudioFrameObserverBase::AudioParams();
        return WidgetPtr->GetAudioParams();
    }

    agora::media::IAudioFrameObserverBase::AudioParams getEarMonitoringAudioParams() override
    {
        if (!IsWidgetValid())
            return agora::media::IAudioFrameObserverBase::AudioParams();
        return WidgetPtr->GetAudioParams();
    }

private:
    UProcessAudioRawDataWidget* WidgetPtr;

    bool IsWidgetValid() const
    {
        return WidgetPtr && IsValid(WidgetPtr);
    }
};

Join channel and configure audio

Join the channel and optionally configure audio settings. You can mute the original playback to handle audio independently.

void OnBtnJoinChannelClicked()
{
    // Mute the playback volume of SDK (not UE) to handle audio independently
    AgoraUERtcEngine::Get()->adjustPlaybackSignalVolume(0);

    AgoraUERtcEngine::Get()->enableAudio();
    AgoraUERtcEngine::Get()->setClientRole(CLIENT_ROLE_BROADCASTER);
    int ret = AgoraUERtcEngine::Get()->joinChannel(TCHAR_TO_UTF8(*Token), TCHAR_TO_UTF8(*ChannelName), "", 0);
}

Unregister the audio frame observer

Call registerAudioFrameObserver with nullptr to unregister the audio frame observer before leaving the channel:

void UnInitAgoraEngine()
{
    if (AgoraUERtcEngine::Get() != nullptr)
    {
        AgoraUERtcEngine::Get()->leaveChannel();
        if (MediaEngine != nullptr)
        {
            // Unregister the audio frame observer
            MediaEngine->registerAudioFrameObserver(nullptr);
        }
        AgoraUERtcEngine::Get()->unregisterEventHandler(UserRtcEventHandlerEx.Get());
        AgoraUERtcEngine::Release();
    }
}

Video SDK uses a synchronous callback mechanism for processing raw audio data. When you save or rewrite data using the callbacks, consider the following best practices:

To ensure continuity of the audio stream, do not block the SDK thread by processing data directly in the callback function. Instead, make a deep copy of the received audio data and transfer the copied data to another thread for processing.
If you choose to process the audio data synchronously within the callback function, you must strictly control the processing time. For example, if the callback function is triggered every 10 milliseconds, then the processing time within the callback must be less than 10 milliseconds to prevent delays or interruptions in the audio stream.

Reference

This section contains content that completes the information on this page, or points you to documentation that explains other aspects to this product.

Audio module

Sample project

Agora provides an open-source example project ProcessAudioRawData for your reference. Download or view the project for a more detailed example.

Contact Us

Contact Us