Raw Voice Data API

Raw Voice Data API

Methods

Set the Audio Recording Format (setRecordingAudioFrameParameters)

int setRecordingAudioFrameParameters(int sampleRate,
                                     int channel,
                                     AgoraAudioRawFrameOperationModeTYPE mode,
                                     int samplesPerCall);

This method sets the format of the callback data in onRecordAudioFrame.

Name Description
sampleRate Sampling rate in the callback data returned by onRecordAudioFrame, which can set be as 8000, 16000, 32000, 44100, or 48000.
channel

Number of channels in the callback data returned by onRecordAudioFrame, which can be set as 1 or 2:

  • 1: Mono
  • 2: Dual-track
mode

Use mode of the onRecordAudioFrame callback:

  • RAW_AUDIO_FRAME_OP_MODE_READ_ONLY = 0: Read-only mode, users only read the AudioFrame data without modifying anything. For example,

    when users acquire data with the Agora SDK, and push RTMP streams by themselves.

  • RAW_AUDIO_FRAME_OP_MODE_WRITE_ONLY = 1: Write-only mode, users replace the AudioFrame data with their own data for the Agora SDK to do code transmission. For example,

    when users acquire data by themselves.

  • RAW_AUDIO_FRAME_OP_MODE_READ_WRITE = 2: Read and write mode, users read the data from AudioFrame, modify it and then play it. For example,

    when users have their own sound-effect processing module, and want to do voice pre-processing, such as a voice change.

samplesPerCall Sampling points in the called data returned in onRecordAudioFrame. For example, it is usually set as 1024 for stream pushing.

Set the Audio Playback Format (setPlaybackAudioFrameParameters)

int setPlaybackAudioFrameParameters(int sampleRate,
                                    int channel,
                                    AgoraAudioRawFrameOperationModeTYPE mode,
                                    int samplesPerCall);

This method sets the format of the callback data in onPlaybackAudioFrame.

Name Description
sampleRate Sampling rate in the callback data returned by onPlaybackAudioFrame, which can set be as 8000, 16000, 32000, 44100, or 48000.
channel

Number of channels in the callback data returned by onPlaybackAudioFrame, which can be set as 1 or 2:

  • 1: Mono
  • 2: Dual-track
mode

Use mode of the onPlaybackAudioFrame callback:

  • RAW_AUDIO_FRAME_OP_MODE_READ_ONLY = 0: Read-only mode, users only read the AudioFrame data without modifying anything. For example,

    when users acquire data with the Agora SDK, then push RTMP streams by themselves.

  • RAW_AUDIO_FRAME_OP_MODE_WRITE_ONLY = 1: Write-only mode, users replace the AudioFrame data with their own data for the Agora SDK to do code transmission. For example,

    when users acquire data by themselves.

  • RAW_AUDIO_FRAME_OP_MODE_READ_WRITE = 2: Read and write mode, users read the data from AudioFrame, modify it and then play it. For example,

    when users have their own sound-effect processing module, and want to do voice post-processing based, such as a voice change.

samplesPerCall Sampling points in the called data returned in onPlaybackAudioFrame. For example, it is usually set as 1024 for stream pushing.

Set the Recording and Playback Mixed Data Format (setMixedAudioFrameParameters)

int setMixedAudioFrameParameters(int sampleRate,
                                 int samplesPerCall);

This method sets the format of the callback data in onMixedAudioFrame.

Name Description
sampleRate Sampling rate in the callback data returned by onMixedAudioFrame, which can set be as 8000, 16000, 32000, 44100, or 48000.
samplesPerCall Ssampling points in the called data returned in onMixedAudioFrame. For example, it is usually set as 1024 for stream pushing.

Register the Audio Observer Object (registerAudioFrameObserver)

int IMediaEngine::registerAudioFrameObserver(IAudioFrameObserver* observer);

This method registers the audio observer object. When you need the engine to return the callback of onRecordAudioFrame, onPlaybackAudioFrame, or onPlaybackAudioFrameObserver, call this method to register the callback.

Name Description
observer

Interface object instance.

Set the value to NULL to cancel registering, if necessary.

Register the Video Observer Object (registerVideoFrameObserver)

int registerVideoFrameObserver(agora::media::IVideoFrameObserver *observer);

This method registers the video observer object. When you need the engine to return the callback of onCaptureVideoFrame or onRenderVideoFrame, call this method to register the callback.

Name Description
observer

Interface object instance.

Set the value to NULL to cancel registering, if necessary.

For the Android platform, registerVideoFrameObserver is defined in libHDACEngine.so, which you need to load by yourself.

Callback Events

Get the Recorded Audio Frame (onRecordAudioFrame)

virtual  bool  onRecordAudioFrame(AudioFrame&audioFrame)  override {
return  true;
}

This method gets the recorded audio frame.

Name Description
AudioFrame
  • samples: Number of samples in the frame
  • bytesPerSample: Number of bytes per sample: 2 for PCM 16
  • channels: Number of channels (data is interleaved, if stereo)
  • samplesPerSec: Sampling rate
  • buffer: Data buffer
  • renderTimeMs: Timestamp to render the audio stream. It instructs the users to use this timestamp to synchronize the audio stream render while rendering the audio streams. [1]

Footnotes

[1](1, 2) This timestamp is for audio stream rendering, and is not the timestamp for audio stream capturing.
struct  AudioFrame  {
AUDIO_FRAME_TYPE  type;
int  samples;
int  bytesPerSample;
int  channels;
int  samplesPerSec;
void*  buffer;
int64_t renderTimeMs;
}

Get the Audio Playback Frame (onPlaybackAudioFrame)

virtual  bool  onPlaybackAudioFrame(AudioFrame&  audioFrame)  override {
return  true;
}

This method gets the audio playback frame. The parameter description is the same as onRecordAudioFrame。

Get the Audio Playback Frame of a Specified User (onPlaybackAudioFrameBeforeMixing)

virtual  bool  onPlaybackAudioFrameBeforeMixing(unsigned  int  uid,  AudioFrame&  audioFrame)  override
{
return  true;
}

This method gets the audio playback frame of a specified user. The parameter description is the same as onRecordAudioFrame.

Name Description
uid UID of the specified user

Get the Recording and Playback Mixed Audio Data (onMixedAudioFrame)

virtual bool onMixedAudioFrame(AudioFrame& audioFrame) override
{
return true;
}

This method gets the recording and playback mixed audio data. It only returns the single-channel data.

Name Description
AudioFrame
  • samples: Number of samples in the frame
  • bytesPerSample: Number of bytes per sample: 2 for PCM 16
  • channels: Number of channels (data is interleaved, if stereo)
  • samplesPerSec: Sampling rate
  • buffer: Data buffer
  • renderTimeMs: Timestamp to render the audio stream. It instructs the users to use this timestamp to synchronize the audio stream render while rendering the audio streams. [1]

Get the Captured Video Frame (onCaptureVideoFrame)

virtual bool onCaptureVideoFrame(VideoFrame&videoFrame)

This method gets the camera’s captured image.

Name Description
VideoFrame
  • yBuffer: Pointer to the Y buffer pointer in the YUV data
  • uBuffer: Pointer to the U buffer pointer in the YUV data
  • vBuffer: Pointer to the V buffer pointer in the YUV data
  • width: Video pixel width
  • height: Video pixel height
  • yStride: Line span of Y buffer in YUV data
  • uStride: Line span of U buffer in YUV data
  • vStride: Line span of V buffer in YUV data
  • rotation: Set the rotation of this frame before rendering the video, and it supports 0, 90, 180, 270 degrees.
  • renderTimeMs: Timestamp to render the video stream. It instructs the users to use this timestamp to synchronize the video stream render while rendering the video streams. [2]
Return Value None

Footnotes

[2]This timestamp is for video stream rendering, and is not the timestamp of video stream capturing.

The video data format is YUV420. The buffer provides a pointer to a pointer. However, the interface cannot modify the pointer of the buffer, but can only modify the content of the buffer.

struct  VideoFrame  {
VIDEO_FRAME_TYPE  type;
int  width;
int  height;
int  yStride;
int  uStride;
int  vStride;
void*  yBuffer;
void*  uBuffer;
void*  vBuffer;
int rotation; // rotation of this frame (0, 90, 180, 270)
int64_t renderTimeMs;
};

Get the Video Frame of the Other User (onRenderVideoFrame)

virtual bool onRenderVideoFrame(unsigned int uid, VideoFrame& videoFrame)

This method processes the received image of the other user (post-processing).

Name Description
uid UID of the specified user
VideoFrame
  • yBuffer: Pointer to the Y buffer pointer in the YUV data
  • uBuffer: Pointer to the U buffer pointer in the YUV data
  • vBuffer: Pointer to the V buffer pointer in the YUV data
  • width: Video pixel width
  • height: Video pixel height
  • yStride: Line span of Y buffer in YUV data
  • uStride: Line span of U buffer in YUV data
  • vStride: Line span of V buffer in YUV data
Return Value None
struct  VideoFrame  {
VIDEO_FRAME_TYPE  type;
int  width;
int  height;
int  yStride;
int  uStride;
int  vStride;
void*  yBuffer;
void*  uBuffer;
void*  vBuffer;
};
Is this page helpful?