Custom Media Input and Output API

Custom Media Input and Output API

To enable developers to use the custom media input and output sources, Agora has developed three sets of APIs: The Customized Media Source and Sink API, Configuring the External Data API, and Raw Data API.

API Description
Customized Media Source and Sink API To create the custom media source and sink, and pass them to the underlying media engine to replace the default media source and sink. This API will replace the Configuring the External Data API and Raw Data API in the future.
Configuring the External Data API To configure the external media data. Agora will maintain this class but no new functions will be added.
Raw Data API To modify the media data. Agora recommends new customers not to use this API.

Note

Currently, the Customized Media Source & Sink API supports customizing the video device only. Agora will add APIs to enable the audio device customization in the future. For now, Agora recommends using the Configuring the External Data API to customize the audio device.

Customized Media Source and Sink API

The Customized Media Source and Sink API consists of the following two sets of interfaces:

The IVideoSource Interface Defines a set of protocols to implement the custom video source and pass it to the underlying media engine to replace the default video source.
The IVideoSink Interface Defines a set of protocols to implement the custom video sink and pass it to the underlying media engine to replace the default video sink.

The IVideoSource Interface

The IVideoSource interface allows you to take ownership of the video source and manipulate it.

By default, when enabling real-time communications, the Agora SDK enables the default video input device (built-in camera) to start video streaming.

By calling Set the Video Source (setVideoSource), you can change the default video input device, control it, and send the video source from the specified input device to the Agora Media Engine to handle the remaining video process; such as video filtering and publishing the video to the RTC connection.

Once you have implemented this interface, the Agora Media Engine will automatically release its ownership of the current video input device and pass it on to you, so that you can use the same video input device to capture the video stream.

1. Definition of the IVideoSource Interface

The IVideoSource interface is defined as follows:

public interface IVideoSource {
  boolean onInitialize(IVideoFrameConsumer consumer);
  boolean onStart();
  void onStop()
  void onDispose();
  int getBufferType();
}

The IVideoSource interface defines a set of protocols to create customized video source objects and pass them to the underlying media engine to replace the default camera source.

The media engine includes the interface to set the video source, see Set the Video Source (setVideoSource).

The media engine receives data from the video source, then encodes and sends it. The video source produces the video frame and the media engine manipulates the video frame.

When the video source is initialized, the media engine will pass an IVideoFrameConsumer object to the video source, then the video source will send the video frame to the media engine through the IVideoFrameConsumer interface method.

IVideoFrameConsumer

IVideoFrameConsumer supports the video data in three buffer types: ByteBuffer, ByteArray, and Texture. Call the Get the Buffer Type (getBufferType) method to specify a buffer type. The video data can only be transmitted in the corresponding type.

public interface IVideoFrameConsumer {
  void consumeByteBufferFrame(ByteBuffer buffer, int format, int width, int height, int rotation, long timestamp);
  void consumeByteArrayFrame(byte[] data, int format, int width, int height, int rotation, long timestamp);
  void consumeTextureFrame(int textureId, int format, int width, int height, int rotation, long timestamp, float[] matrix);
 }
Output the Video Frame’s Byte Buffer Information (consumeByteBufferFrame)
void consumeByteBufferFrame(
  ByteBuffer buffer,
  int format,
  int width,
  int height,
  int rotation,
  long timestamp)

This method outputs the video frame’s byte buffer information.

Name Description
buffer Buffer type (ByteBuffer)
format Pixel format of the video frame, see Set the Pixel Format (AgoraVideoPixelFormat).
width Width of the video frame.
height Height of the video frame.
rotation Clockwise rotating angle (0, 90, 180, and 270 degrees) of the video frame.
timestamp Timestamp (ms) of the video frame.
Output the Video Frame’s Byte Array Information (consumeByteArrayFrame)
void consumeByteArrayFrame(
  byte[] data,
  int format,
  int width,
  int height,
  int rotation,
  long timestamp)

This method outputs the video frame’s byte array information.

Name Description
data Byte type (data)
format Pixel format of the video frame, see Set the Pixel Format (AgoraVideoPixelFormat).
width Width of the video frame.
height Height of the video frame.
rotation Clockwise rotating angle (0, 90, 180, and 270 degrees) of the video frame.
timestamp Timestamp of the video frame (ms).
Output the Video Frame’s Texture Information (consumeTextureFrame)
void consumeTextureFrame(
  int textureId,
  int format,
  int width,
  int height,
  int rotation,
  long timestamp,
  float[] matrix)

This method outputs the video frame’s texture information.

Name Description
textureId ID of the texture.
format Pixel format of the video frame, see Set the Pixel Format (AgoraVideoPixelFormat).
width Width of the video frame.
height Height of the video frame.
rotation Clockwise rotating angle (0, 90, 180, and 270 degrees) of the video frame.
timestamp Timestamp of the video frame (ms).
matrix Matrix of the texture. The float is between 0 and 1, such as 0.1, 0.2…

Note

When passing the texture frame to the media engine, use the consumeTextureFrame() method in the GL environment in which the textureId is created.

2. Description of the IVideoSource Interface

The IVideoSource interface consists of the following methods:

  • Initialize the Video Source (onInitialize)
  • Enable the Video Source (onStart)
  • Disable the Video Source (onStop)
  • Remove the Video Source (onDispose)
  • Get the Buffer Type (getBufferType)
Initialize the Video Source (onInitialize)
boolean onInitialize(IVideoFrameConsumer consumer);

This callback function initializes the video source. Pass one of the following return values to the media engine:

  • true: If the external video source is initialized
  • false: If the external video source is not ready or fails to be initialized; the media engine will stop and report the error.

You can also initialize the video input device before this method is called and return true to the media engine in this method.

Name Description
IVideoFrameConsumer Video source consumer object which implements the IVideoFrameConsumer interface. Use this access point to consume the video frame after Enable the Video Source (onStart) is called. The frame will be kept until Release the Video Source (onDispose) is called.
Enable the Video Source (onStart)
boolean OnStart();

This callback function is called when the media engine is ready to start video streaming. You should start the video source to capture the video frame. Once the frame is ready, use the IVideoFrameConsumer to consume the video frame.

Pass one of the following return values to the media engine:

  • true: If the external video source is enabled and IVideoFrameConsumer is called to receive video frames.
  • false: If the external video source is not ready or fails to be enabled; the media engine will stop and report the error.

After true is returned, video frames can be passed to the media engine through the preset IVideoFrameConsumer interface method.

Disable the Video Source (onStop)
void OnStop();

This callback function is called when the media engine stops streaming. You should then stop capturing the video frame and consuming it. After this method, the video frames will be discarded by the media engine.

Release the Video Source (onDispose)
void OnDispose();

This callback function is called when the IVideoFrameConsumer is released by the media engine. You can now release the video source as well as the IVideoFrameconsumer .

Get the Buffer Type (getBufferType)
int getBufferType();

Pass the buffer type that you previously set in Set the Buffer Type (AgoraVideoBufferType) to the media engine. This buffer type is used to set up the correct media engine environment.

Name Description
Return value
  • 1: Byte buffer
  • 2: Byte array buffer
  • 3: Texture

Note

  • All methods in the IVideoSource interface are callback methods. The media engine maintains a finite state machine and uses these functions at the right time. Do not use these functions directly in the app.
  • These functions are synchronized.
  • If the media engine restarts during the process, these functions can be repeated for a couple of times.
  • These functions are not in the primary thread.

When using the IVideoSource interface, call the following two callback functions to set the buffer type and pixel format of the transmitted video.

Set the Buffer Type (BufferType)

public enum BufferType {

    BYTE_BUFFER(1),
    BYTE_ARRAY(2),
    TEXTURE(3);

    public int intValue() {
        return this.value;
    }
 }

This enumeration defines the buffer type for video transmission. Choose one of the following three types:

Name Description
BYTE_BUFFER Use a byte buffer to transmit the video frame.
BYTE_ARRAY Use a byte array to transmit the video frame.
TEXTURE Use OpenGL texture to transmit the video frame (for interactive broadcast only).

Set the Pixel Format (PixelFormat)

public enum PixelFormat {
    I420(1),
    NV21(3),
    RGBA(4),
    TEXTURE_2D(10),
    TEXTURE_OES(11);

    public int intValue() {
        return this.value;

    }
 }

This enumeration defines the pixel format of the video frame. Agora supports on the Android platform the following 5 pixel format: NV21, I420, RGBA, TEXTURE_2D and TEXTURE_OES. For detailed information on the YUV format, see:

http://www.fourcc.org/yuv.php

https://msdn.microsoft.com/en-us/library/windows/desktop/dd206750(v=vs.85).aspx

3. Example on Enabling the External Video Source

By using the callback functions, developers enable the media engine to manage the video source. An example is shown in the following steps:

Step 1. Implement The IVideoSource Interface to create the customized video source class:

Step 2. Implement the IVideoSource interface to create a customized video source object.

Step 3. Pass the external video source to the media engine by Set the Video Source (setVideoSource).

Step 4. The media engine implements the methods in The IVideoSource Interface.

Note

The video source and renderer can be customized by switching on/off the video frame input based on the the media engine callback. This simpler method can be used if an App has its own video module and only needs the Agora SDK for real-time communications. See Advanced: Customizing the Video Source and Renderer.

The IVideoSink Interface

The IVideoSink interface allows you to implement the custom video source.

By default, when you try to enable real-time communication, Agora SDK enables the default video sink start video rendering. Now, buy calling Set the Local Video Renderer (setLocalVideoRenderer) and Set the Remote Video Renderer (setRemoteVideoRenderer), however, you can change the default video sink.

Once you have implemented this interface, you will receive callbacks from the media engine to indicate the state of the custom video sink and the underlying media engine and enable their synchronization. Follow each callback to handle resource allocation and release and receive the video frame from the media engine.

1. Definition of the IVideoSink Interface

The IVideosink interface is defined as follows:

public interface IVideoSink extends IVideoFrameConsumer {
    boolean onInitialize();
    boolean onStart();
    void onStop();
    void onDispose();
    int getBufferType();
    int getPixelFormat();
}

The IVideoSink interface defines a set of protocols to create customized video sink. The IVideoFrameConsumer interface passes the video frames to the media engine, who then passes them to the renderer.

For details on IVideoFrameConsumer, see IVideoFrameConsumer.

After a customized video sink is created, the App passes it to the media engine, see Set the Local Video Renderer (setLocalVideoRenderer) and Set the Remote Video Renderer (setRemoteVideoRenderer).

2. Description the of IVideoSink Interface

The IVideoSink interface consists of the following callback functions:

  • Initialize the Video Sink (onInitialize)
  • Enable the Video Sink (onStart)
  • Disable the Video Sink (onStop)
  • Release the Video Sink (onDispose)
  • Get the EGLContext Handle (getEGLContextHandle)
  • Get the Buffer Type (getBufferType)
  • Get the Pixel Format (getPixelFormat)

Note

All methods defined in the IVideoSink interface are callback methods. The media engine uses these methods to inform the customized renderer of its internal changes.

Initialize the Video Sink (onInitialize)
boolean onInitialize();

This callback function initializes the video sink. Pass one of the following return values to the media engine:

  • true: If the video sink is initialized
  • false: If the video sink is not ready or fails to be initialized; the media engine stops and reports the error.

You can also initialize the video sink before this method is called and return true in this method.

Enable the Video Sink (onStart)
boolean onStart();

This callback function is called when the media engine starts streaming.

  • true: If the video sink is ready. The media engine will then provide video frame to the custom video sink by calling the IVideoFrameConsumer interface
  • false: If the video sink is not ready
Disable the Video Sink (onStop)
void onStop();

This callback function is called when the media engine stops video streaming. You should then stop the video sink.

Release the Video Sink (onDispose)
void onDispose();

This callback function is called when the media engine wants to release the video sink.

Get the EGLContextHandle (getEGLContextHandle)
public long getEGLContextHandle();

This callback returns the native handle of the EGLContext if you are using OpenGL and have your own EGLContxt. You can share your EGLContext using the Texture ID returned by the SDK in your own EGLContext.

If you do not have an EGLContext, this callback returns 0; if the buffer type is set to Texture, the media engine will create an EGLContext internally and return the Texture ID.

Get the Buffer Type (getBufferType)
void getBufferType();

Pass the buffer type specified in Set the Buffer Type (AgoraVideoBufferType) to the media engine.

Get the Pixel Format (getPixelFormat)
void getPixelFormat()

Pass the pixel format specified in Set the Pixel Format (AgoraVideoPixelFormat) to the media engine.

Note

All methods in the IVideoSink interface callback methods. The media engine maintains a finite state machine and uses these methods at the right time. Do not use these methods directly in the app.

3. Example on Customizing the Video Sink

An example is shown in the following steps to customize the video sink:

Step 1. Call Get the Buffer Type (getBufferType) and Get the Pixel Format (getPixelFormat) to set the buffer type and pixel format of the video frame.

Step 2. Implement Initialize the Video Sink (onInitialize), Enable the Video Sink (onStart), Disable the Video Sink (onStop), and Release the Video Sink (onDispose) to manage the customized video sink.

Step 3. Implement the buffer type and pixel format as specified in Step 1 of IVideoFrameConsumer, see IVideoFrameConsumer.

Step 4. Create the customized video sink object.

Step 5. Call the Set the Local Video Renderer (setLocalVideoRenderer) and Set the Remote Video Renderer (setRemoteVideoRenderer) methods to set the local and remote renderers.

Step 6. The media engine will call functions in The IVideoSink Interface according to its internal state.

Note

Agora provides a sample demo for developers to create and integrate a customized renderer. See Advanced: Customizing the Video Source and Renderer.

Configuring the External Data API

The External Audio Data

Note

If you use the external audio data API to capture audio, you may encounter echo problems. Developers need to take care of such problems themselves.

Set the External Audio Source (setExternalAudioSource)

public abstract int setExternalAudioSource(boolean enabled, int sampleRate, int channels);

This method sets the external audio source.

Name Description
enabled Enable the function of the external audio source: true/false.
sampleRate Sampling rate of the external audio source.
channelsPerFrame Number of the external audio source channels (two channels maximum).

Push the External Audio Frame (pushExternalAudioFrameRawData)

public abstract int pushExternalAudioFrame(byte[] data, long timestamp);

This method pushes the external audio frame to the Agora SDK for encoding.

Name Description
data External audio data.
timestamp Timestamp of the external audio frame to be synchronized with the external video source.

Configuring the External Video Data

This section describes how to configure an external video source to capture the video data. The external video source function in libvideoprp is merged with the texture encoding interfaces. The YUV/RGBA/textured images are sent to the Agora SDK for encoding.

Note

It is recommended to use this method for apps under development.

Check Whether Texture Encoding Is Supported (isTextureEncodeSupported)

public abstract boolean isTextureEncodeSupported();

This method checks whether texture encoding is supported.

Name Description
Return Value

True: Texture encoding is supported.

False: Texture encoding is not supported.

Configure the External Video Source (setExternalVideoSource)

public abstract void setExternalVideoSource(boolean enable, boolean useTexture, boolean pushMode);

This method sets whether to use an external video source:

  • If yes, whether to use the texture as an input;
  • If yes, whether to push the video frame to the Agora SDK for encoding.
Name Description
enable [1]

Whether to use an external video source:

  • True: Use external video source.
  • False: Do not use external video source.
useTexture

Whether to use the texture as an input:

  • True: Use the texture as an input.
  • False: Do not use the texture as an input.
pushMode

Whether the external video source needs to call PushExternalVideoFrame to send the video frame to the Agora SDK:

  • True: Use the push mode.
  • False: Use the pull mode (not supported yet).

Footnotes

[1]Currently, the Agora SDK does not support switching video sources dynamically in the channel. If an external video source is enabled and you are in a channel, if you want to switch to an internal video source, you must exit the channel. Then call this method to set enable as false, and join the channel again.

Push the External Video Frame (pushExternalVideoFrame)

public abstract boolean pushExternalVideoFrame(AgoraVideoFrame frame);

This method pushes the video frame using the AgoraVideoFrame class and passes it to the Agora SDK.

Call setExternalVideoSource and set the pushMode parameter as true before calling this method. Otherwise, it will return a failure after calling this method.

Name Description
frame Video frame that contains the video data to be encoded by the Agora SDK.
Return Value
  • True: Frame is pushed successfully
  • False: Failed to push the frame

The definition of the AgoraVideoFrame class:

public AgoraVideoFrame() {

  //mandatory
    public int format;
    public long timeStamp;
    public int stride;
    public int height;

    // Texture related
    public javax.microedition.khronos.egl.EGLContext eglContext11;
    public android.opengl.EGLContext eglContext14;
    public int textureID;
    public boolean syncMode;
    public float[] transform;

    // Raw data related
    public byte[] buf;
    public int cropLeft;
    public int cropTop;
    public int cropRight;
    public int cropBottom;
    public int rotation;
}

An explanation of the fields are listed in the following table:

format Mandatory Field

Format of the incoming video frame, which must be specified as one of the following:

  • AgoraVideoFrame.FORMAT_TEXTURE_2D
  • AgoraVideoFrame.FORMAT_TEXTURE_OES
  • AgoraVideoFrame.FORMAT_I420
  • AgoraVideoFrame.FORMAT_NV21
  • AgoraVideoFrame.FORMAT_RGBA
timestamp Mandatory Field Timestamp of the incoming video frame (ms). An incorrect timestamp will result in frame loss or unsynchronized audio and video.
stride Mandatory Field Line spacing of the incoming video frame, which must be in pixels instead of bytes. For textures, it is the width of the texture.
height Mandatory Field Height of the incoming video frame.
eglContext11 Texture Related Field When using the OpenGL interface (javax.microedition.khronos.egl.*) defined by Khronos, you need to set the EGLContext to this field.
eglContext14 Texture Related Field When using the OpenGL interface (android.opengl.*) defined by Android, you need to set the EGLContext to this field.
textureID Texture Related Field Texture ID used by the video frame.
transform Texture Related Field Incoming 4 x 4 transformational matrix. The typical value is a unit matrix.
syncMode Texture Related Field

(Optional) Sets whether to wait for the encoding completion of the previous one video frame:

  • True: Wait.
  • False: Do not wait.
buf Raw Data Field Content data of the incoming video frame.
cropLeft, cropTop, cropRight, cropBottom Raw Data Field (Optional) Specifies the number of pixels trimmed in each direction, which is set as 0 by default.
rotation Raw Data Field (Optional) Specifies whether to rotate the incoming video group. Optional values: 0, 90, 180, or 270 clockwise. It is set as 0 by default.

Sample Code:

When the format of the incoming video frame is TEXTURE_2D and is EGLContext defined by Khronos:

import io.agora.rtc.video.AgoraVideoFrame;

private final static float[] UNIQUE_MAT = {
    1.0f, 0.0f, 0.0f, 0.0f,
    0.0f, 1.0f, 0.0f, 0.0f,
    0.0f, 0.0f, 1.0f, 0.0f,
    0.0f, 0.0f, 0.0f, 1.0f
};

AgoraVideoFrame vf = new AgoraVideoFrame();
vf.format = AgoraVideoFrame.FORMAT_TEXTURE_2D;
vf.timeStamp = System.currentTimeMillis();
vf.stride = 352;
vf.height = 640;
vf.textureID = textureId;
vf.eglContext11 = eglContext;
vf.transform = UNIQUE_MAT;
mRtcEngine.pushExternalVideoFrame(vf);

When the format of the incoming video frame is NV21 YUV:

AgoraVideoFrame vf = new AgoraVideoFrame();
vf.format = AgoraVideoFrame.FORMAT_NV21;
vf.timeStamp = System.currentTimeMillis();
vf.stride = 640;
vf.height = 480;
vf.rotation = rotation;
vf.buf = data;
mRtcEngine.pushExternalVideoFrame(vf);

Raw Data API

Methods

Set the Recording Audio Format (setRecordingAudioFrameParameters)

public abstract int setRecordingAudioFrameParameters(int sampleRate,
                                                     int channel,
                                                     int mode,
                                                     int samplesPerCall);

This method sets the format of the callback data in onRecordAudioFrame.

Name Description
sampleRate Specifies the sampling rate in the callback data returned by onRecordAudioFrame, which can set be as 8000, 16000, 32000, 44100, or 48000.
channel

Specifies the number of channels in the callback data returned by onRecordAudioFrame, which can be set as 1 or 2:

  • 1: Mono
  • 2: Dual-track
mode

Specifies the use mode of the onRecordAudioFrame callback:

  • AgoraAudioRawFrameOperationModeReadOnly = 0: Read-only mode; users only read the AudioFrame data without modifying anything. For example,

    Users can use this mode if they acquire data with the Agora SDK, then push RTMP streams by themselves.

  • AgoraAudioRawFrameOperationModeWriteOnly = 1: Write-only mode; users replace the AudioFrame data with their own data and pass to the SDK for encoding. For example,

    Users can use this mode if they acquire data.

  • AgoraAudioRawFrameOperationModeReadWrite = 2: Read and write mode; users read the data from AudioFrame, modify it and then play it. For example,

    Users can use this mode if they have their own sound-effect processing module, and want to do some voice pre-processing, such as a voice change.

samplesPerCall Specifies the sampling points in the called data returned in onRecordAudioFrame. For example, it is usually set as 1024 for stream pushing.

Set the Playback Audio Format (setPlaybackAudioFrameParameters)

public abstract int setPlaybackAudioFrameParameters(int sampleRate,
                                                    int channel,
                                                    int mode,
                                                    int samplesPerCall);

This method sets the format of the callback data in onPlaybackAudioFrame.

Name Description
sampleRate Specifies the sampling rate in the callback data returned by onPlaybackAudioFrame, which can set be as 8000, 16000, 32000, 44100, or 48000.
channel

Specifies the number of channels in the callback data returned by onPlaybackAudioFrame, which can be set as 1 or 2:

  • 1: Mono
  • 2: Dual-track
mode

Specifies the use mode of the onPlaybackAudioFrame callback:

  • AgoraAudioRawFrameOperationModeReadOnly = 0: Read-only mode; users only read the AudioFrame data without modifying anything. For example,

    Users can set this mode when they acquire data with the Agora SDK, then push RTMP streams by themselves.

  • AgoraAudioRawFrameOperationModeWriteOnly = 1: Write-only mode; users replace the AudioFrame data with their own data. For example,

    Users can use this mode when they acquire data by themselves.

  • AgoraAudioRawFrameOperationModeReadWrite = 2: Read and write mode; users read the data from AudioFrame, modify it, and then play it. For example,

    Users can use this mode when they have their own sound-effect processing module, and want to do some voice post-processing, such as a voice change.

samplesPerCall Specifies the sampling points in the called data returned in onPlaybackAudioFrame. For example, it is usually set as 1024 for stream pushing.

Set the Mixed Recording and Playback Data Format (setMixedAudioFrameParameters)

public abstract int setMixedAudioFrameParameters(int sampleRate, int samplesPerCall);

This method sets the format of the callback data in onMixedAudioFrame.

Name Description
sampleRate Specifies the sampling rate in the callback data returned by onMixedAudioFrame, which can set be as 8000, 16000, 32000, 44100, or 48000.
samplesPerCall Specifies the sampling points in the called data returned in onMixedAudioFrame, for example, it is usually set as 1024 for stream pushing.

Register the Audio Observer Object (registerAudioFrameObserver)

public abstract int registerAudioFrameObserver(IAudioFrameObserver observer);

This method registers the audio observer object. When you need the engine to return the callback of onRecordAudioFrame, onPlaybackAudioFrame, or onPlaybackAudioFrameObserver, call this method to register the callback.

Name Description
observer

Interface object instance.

Set the value to NULL to cancel registering if necessary.

Register the Video Observer Object (registerVideoFrameObserver)

int registerVideoFrameObserver(agora::media::IVideoFrameObserver *observer);

This method registers the video observer object. When you need the engine to return the callback of onCaptureVideoFrame or onRenderVideoFrame, call this method to register the callback.

Name Description
observer

Interface object instance.

Set the value to NULL to cancel registering if necessary.

On Android platforms, registerVideoFrameObserver is defined in libHDACEngine.so, which need to be loaded manually.

Callback Events

Get the Recorded Audio Frame (onRecordAudioFrame)

virtual  bool  onRecordAudioFrame(AudioFrame&audioFrame)  override {
return  true;
}

This method gets the recorded audio frame.

Name Description
AudioFrame
  • samples: Number of samples in the frame.
  • bytesPerSample: Number of bytes per sample: 2 for PCM 16.
  • channels: Number of channels (stereo data is interleaved).
  • samplesPerSec: Sampling rate.
  • buffer: Data buffer.
  • renderTimeMs: Timestamp to render the audio stream. This timestamp is used to synchronize the audio stream renderer while rendering the audio streams. [2]

Footnotes

[2](1, 2) This timestamp is for rendering the audio stream, and is not the timestamp of capturing the audio stream.
struct  AudioFrame  {
AUDIO_FRAME_TYPE  type;
int  samples;
int  bytesPerSample;
int  channels;
int  samplesPerSec;
void*  buffer;
int64_t renderTimeMs;
}

Get the Playback Audio Frame (onPlaybackAudioFrame)

virtual  bool  onPlaybackAudioFrame(AudioFrame&  audioFrame)  override {
return  true;
}

This method gets the playback audio frame. The parameter description is the same as onRecordAudioFrame.

Get the Playback Audio Frame of a Specific User (onPlaybackAudioFrameBeforeMixing)

virtual  bool  onPlaybackAudioFrameBeforeMixing(unsigned  int  uid,  AudioFrame&  audioFrame)  override
{
return  true;
}

This method gets the playback audio frame of a specific user. The parameter description is the same as onRecordAudioFrame.

Name Description
uid UID of a specified user.

Get the Mixed Recording and Playback Audio Data (onMixedAudioFrame)

virtual bool onMixedAudioFrame(AudioFrame& audioFrame) override
{
return true;
}

This method gets the mixed recording and playback audio data. It only returns the single-channel data.

Name Description
AudioFrame
  • samples: Number of samples in the frame.
  • bytesPerSample: Number of bytes per sample: 2 for PCM 16.
  • channels: Number of channels (stereo data is interleaved).
  • samplesPerSec: Sampling rate
  • buffer: Data buffer
  • renderTimeMs: Timestamp to render the audio stream. Use this timestamp to synchronize the audio stream renderer while rendering the audio streams. [2]

Get the Captured Video Frame (onCaptureVideoFrame)

virtual bool onCaptureVideoFrame(VideoFrame&videoFrame)

This method gets the camera’s captured image.

Name Description
VideoFrame
  • yBuffer: Pointer to the Y buffer pointer in the YUV data.
  • uBuffer: Pointer to the U buffer pointer in the YUV data.
  • vBuffer: Pointer to the V buffer pointer in the YUV data.
  • width: Video pixel width.
  • height: Video pixel height.
  • yStride: Line span of the Y buffer in the YUV data.
  • uStride: Line span of the U buffer in the YUV data.
  • vStride: Line span of the V buffer in the YUV data.
  • rotation: Set the rotation of this frame before rendering the video; it supports 0, 90, 180, and 270 degrees clockwise.
  • renderTimeMs: Timestamp to render the video stream. Use this timestamp to synchronize the video stream renderer while rendering the video streams. [3]
Return Value None

Footnotes

[3]This timestamp is for rendering the video stream, not for capturing the video stream.

The video data format is YUV420. The buffer provides a pointer to a pointer. However, the pointer of the buffer cannot be modified. Only the contents of the buffer can be modified.

struct  VideoFrame  {
VIDEO_FRAME_TYPE  type;
int  width;
int  height;
int  yStride;
int  uStride;
int  vStride;
void*  yBuffer;
void*  uBuffer;
void*  vBuffer;
int rotation; // rotation of this frame (0, 90, 180, 270)
int64_t renderTimeMs;
};

Get the Video Frame of the Other User (onRenderVideoFrame)

virtual bool onRenderVideoFrame(unsigned int uid, VideoFrame& videoFrame)

This method processes the received image of the other user (post-processing).

Name Description
uid UID of a specified user.
VideoFrame
  • yBuffer: Pointer to the Y buffer pointer in the YUV data.
  • uBuffer: Pointer to the U buffer pointer in the YUV data.
  • vBuffer: Pointer to the V buffer pointer in the YUV data.
  • width: Video pixel width.
  • height: Video pixel height.
  • yStride: Line span of the Y buffer in the YUV data.
  • uStride: Line span of the U buffer in the YUV data.
  • vStride: Line span of the V buffer in the YUV data.
Return Value None
struct  VideoFrame  {
VIDEO_FRAME_TYPE  type;
int  width;
int  height;
int  yStride;
int  uStride;
int  vStride;
void*  yBuffer;
void*  uBuffer;
void*  vBuffer;
};

Error Code and Warning code

See Error Codes and Warning Codes .