Skip to main content

Raw video and audio processing

In some scenarios, raw audio and video captured through the camera and microphone must be processed to achieve the desired functionality or to enhance the user experience. Video SDK enables you to pre-process and post-process the captured audio and video data for implementation of custom playback effects.

Understand the tech

You can use the raw data processing functionality in Video SDK to process the feed according to your particular scenario. This feature enables you to pre-process the captured signal before sending it to the encoder, or to post-process the decoded signal before playback. To implement processing of raw video and audio data in your app, take the following steps.

  • Set up video and audio frame observers.
  • Register video and audio frame observers before joining a channel.
  • Set the format of audio frames captured by each callback.
  • Implement callbacks in the frame observers to process raw video and audio data.
  • Unregister the frame observers before you leave a channel.

The figure below shows the workflow you need to implement to process raw video and audio data in your app.

Process raw audio and video

Prerequisites

To follow this procedure you must have implemented Get Started with Video Calling.

Project setup

To create the environment necessary to integrate processing of raw audio and video data in your app, open the Get Started with Video Calling project you created previously.

Implement raw data processing

When a user captures or receives video and audio data, the data is available to the app for processing before it is played. This section shows how to retrieve this data and process it, step-by-step.

Implement the user interface

To enable or disable processing of captured raw video data, add a button to the user interface. In /app/res/layout/activity_main.xml add the following lines before </RelativeLayout>:


_9
<Button
_9
android:id="@+id/ZoomButton"
_9
android:layout_width="wrap_content"
_9
android:layout_height="wrap_content"
_9
android:layout_below="@+id/JoinButton"
_9
android:layout_alignEnd="@id/LeaveButton"
_9
android:layout_alignStart="@id/JoinButton"
_9
android:onClick="setZoom"
_9
android:text="Zoom In" />

Handle the system logic

This sections describes the steps required to use the relevant libraries, declare the necessary variables, and set up access to the UI elements.

  1. Import the required Android and Agora libraries

    To integrate Video SDK frame observer libraries into your app and access the button object, add the following statements after the last import statement in /app/java/com.example.<projectname>/MainActivity.


    _5
    import io.agora.rtc2.video.IVideoFrameObserver;
    _5
    import io.agora.rtc2.IAudioFrameObserver;
    _5
    import io.agora.base.VideoFrame;
    _5
    import java.nio.ByteBuffer;
    _5
    import android.widget.Button;

  2. Define a variable to manage video processing

    In /app/java/com.example.<projectname>/MainActivity, add the following declaration to the MainActivity class:


    _1
    private boolean isZoomed = false;

Implement processing of raw video and audio data

To register and use video and audio frame observers in your app, take the following steps:

  1. Set up the audio frame observer

    IAudioFrameObserver gives you access to each audio frame after it is captured or access to each audio frame before it is played back. To setup the IAudioFrameObserver, add the following lines to the MainActivity class after variable declarations:


    _61
    private final IAudioFrameObserver iAudioFrameObserver = new IAudioFrameObserver() {
    _61
    @Override
    _61
    public boolean onRecordAudioFrame(String channelId, int type, int samplesPerChannel,
    _61
    int bytesPerSample, int channels, int samplesPerSec, ByteBuffer buffer, long renderTimeMs, int avsync_type) {
    _61
    // Gets the captured audio frame.
    _61
    // Add code here to process the recorded audio.
    _61
    return false;
    _61
    }
    _61
    _61
    @Override
    _61
    public boolean onPlaybackAudioFrame(String channelId, int type, int samplesPerChannel,
    _61
    int bytesPerSample, int channels, int samplesPerSec, ByteBuffer buffer, long renderTimeMs, int avsync_type) {
    _61
    // Gets the audio frame for playback.
    _61
    // Add code here to process the playback audio.
    _61
    return false;
    _61
    }
    _61
    _61
    @Override
    _61
    public boolean onMixedAudioFrame(String channelId, int type, int samplesPerChannel,
    _61
    int bytesPerSample, int channels, int samplesPerSec, ByteBuffer buffer, long renderTimeMs, int avsync_type) {
    _61
    // Retrieves the mixed captured and playback audio frame.
    _61
    return false;
    _61
    }
    _61
    _61
    @Override
    _61
    public boolean onEarMonitoringAudioFrame(int type, int samplesPerChannel, int bytesPerSample, int channels, int samplesPerSec, ByteBuffer buffer, long renderTimeMs, int avsync_type) {
    _61
    return false;
    _61
    }
    _61
    _61
    @Override
    _61
    public boolean onPlaybackAudioFrameBeforeMixing(String channelId, int userId, int type, int samplesPerChannel,
    _61
    int bytesPerSample, int channels, int samplesPerSec, ByteBuffer buffer, long renderTimeMs, int avsync_type) {
    _61
    // Retrieves the audio frame of a specified user before mixing.
    _61
    return false;
    _61
    }
    _61
    _61
    @Override
    _61
    public int getObservedAudioFramePosition() {
    _61
    return 0;
    _61
    }
    _61
    _61
    @Override
    _61
    public AudioParams getRecordAudioParams() {
    _61
    return null;
    _61
    }
    _61
    _61
    @Override
    _61
    public AudioParams getPlaybackAudioParams() {
    _61
    return null;
    _61
    }
    _61
    _61
    @Override
    _61
    public AudioParams getMixedAudioParams() {
    _61
    return null;
    _61
    }
    _61
    _61
    @Override
    _61
    public AudioParams getEarMonitoringAudioParams() {
    _61
    return null;
    _61
    }
    _61
    };

  2. Set up the video frame observer

    IVideoFrameObserver gives you access to each local video frame after it is captured and access to each remote video frame before it is played back. In this example, your modify the captured video frame buffer to crop and scale the frame and play a zoomed-in version of the video. To set up IVideoFrameObserver, add the following lines to the MainActivity class after the variable declarations:


    _65
    private final IVideoFrameObserver iVideoFrameObserver = new IVideoFrameObserver() {
    _65
    @Override
    _65
    public boolean onCaptureVideoFrame(VideoFrame videoFrame) {
    _65
    if (isZoomed) {
    _65
    VideoFrame.Buffer buffer = videoFrame.getBuffer();
    _65
    int w = buffer.getWidth();
    _65
    int h = buffer.getHeight();
    _65
    int cropX = (w - 320)/2, cropY = (h - 240)/2, cropWidth = 320, cropHeight = 240, scaleWidth = 320, scaleHeight = 240;
    _65
    buffer = buffer.cropAndScale(cropX, cropY, cropWidth, cropHeight, scaleWidth, scaleHeight);
    _65
    videoFrame.replaceBuffer(buffer, 270, videoFrame.getTimestampNs());
    _65
    }
    _65
    return true;
    _65
    }
    _65
    _65
    @Override
    _65
    public boolean onPreEncodeVideoFrame(VideoFrame videoFrame) {
    _65
    return false;
    _65
    }
    _65
    _65
    @Override
    _65
    public boolean onScreenCaptureVideoFrame(VideoFrame videoFrame) {
    _65
    return false;
    _65
    }
    _65
    _65
    @Override
    _65
    public boolean onPreEncodeScreenVideoFrame(VideoFrame videoFrame) {
    _65
    return false;
    _65
    }
    _65
    _65
    @Override
    _65
    public boolean onMediaPlayerVideoFrame(VideoFrame videoFrame, int i) {
    _65
    return false;
    _65
    }
    _65
    _65
    @Override
    _65
    public boolean onRenderVideoFrame(String s, int i, VideoFrame videoFrame) {
    _65
    return false;
    _65
    }
    _65
    _65
    @Override
    _65
    public int getVideoFrameProcessMode() {
    _65
    // The process mode of the video frame. 0 means read-only, and 1 means read-and-write.
    _65
    return 1;
    _65
    }
    _65
    _65
    @Override
    _65
    public int getVideoFormatPreference() {
    _65
    return 1;
    _65
    }
    _65
    _65
    @Override
    _65
    public boolean getRotationApplied() {
    _65
    return false;
    _65
    }
    _65
    _65
    @Override
    _65
    public boolean getMirrorApplied() {
    _65
    return false;
    _65
    }
    _65
    _65
    @Override
    _65
    public int getObservedFramePosition() {
    _65
    return 0;
    _65
    }
    _65
    };

    Note that you must set the return value in getVideoFrameProcessMode to 1 in order for your raw data changes to take effect.

  3. Register the video and audio frame observers

    To receive callbacks declared in IVideoFrameObserver and IAudioFrameObserver, you must register the video and audio frame observers with the Agora Engine before joining a channel. To specify the format of audio frames captured by each IAudioFrameObserver callback, use the setRecordingAudioFrameParameters, setMixedAudioFrameParameters and setPlaybackAudioFrameParameters methods. To do this, add the following lines after if (checkSelfPermission()) { in the joinChannel method:


    _11
    agoraEngine.registerVideoFrameObserver(iVideoFrameObserver);
    _11
    agoraEngine.registerAudioFrameObserver(iAudioFrameObserver);
    _11
    _11
    // Set the format of the captured raw audio data.
    _11
    int SAMPLE_RATE = 16000, SAMPLE_NUM_OF_CHANNEL = 1, SAMPLES_PER_CALL = 1024;
    _11
    _11
    agoraEngine.setRecordingAudioFrameParameters(SAMPLE_RATE, SAMPLE_NUM_OF_CHANNEL,
    _11
    Constants.RAW_AUDIO_FRAME_OP_MODE_READ_WRITE,SAMPLES_PER_CALL);
    _11
    agoraEngine.setPlaybackAudioFrameParameters(SAMPLE_RATE, SAMPLE_NUM_OF_CHANNEL,
    _11
    Constants.RAW_AUDIO_FRAME_OP_MODE_READ_WRITE,SAMPLES_PER_CALL);
    _11
    agoraEngine.setMixedAudioFrameParameters(SAMPLE_RATE, SAMPLE_NUM_OF_CHANNEL, SAMPLES_PER_CALL);

  4. Unregister the video and audio observers when you leave a channel

    When you leave a channel, you unregister the frame observers by calling the register frame observer method again with a null argument. To do this, add the following lines to the leaveChannel(View view) method before agoraEngine.leaveChannel();:


    _2
    agoraEngine.registerVideoFrameObserver(null);
    _2
    agoraEngine.registerAudioFrameObserver(null);

  5. Start and stop video processing

    When a user presses the button, enable or disable video processing. To do this, add the following method to the MainActivity class:


    _9
    public void setZoom (View view){
    _9
    isZoomed = !isZoomed;
    _9
    _9
    Button button = (Button) view;
    _9
    if (isZoomed)
    _9
    button.setText("Zoom Out");
    _9
    else
    _9
    button.setText("Zoom In");
    _9
    }

Test your implementation

To ensure that you have implemented raw data processing into your app:

  1. Generate a temporary token in Agora Console.

  2. In your browser, navigate to the Agora web demo and update App ID, Channel, and Token with the values for your temporary token, then click Join.

  3. In Android Studio, open app/java/com.example.<projectname>/MainActivity, and update appId, channelName and token with the values for your temporary token.

  4. Connect a physical Android device to your development device.

  5. In Android Studio, click Run app. A moment later you see the project installed on your device.

    If this is the first time you run the project, grant microphone and camera access to your app.

  6. Press Join to see the video feed from the web app.

  7. Test processing of raw video data.

    Press Zoom In. You see that the local video captured by your device camera is cropped and scaled to give a zoom-in effect. The processed video is displayed both locally and remotely. Pressing the button again stops processing of raw video data and restores the original video.

  8. Test processing of raw audio data.

    Edit the iAudioFrameObserver definition by adding code that processes the raw audio data you receive in the following callbacks:

    • onRecordAudioFrame: Gets the captured audio frame data

    • onPlaybackAudioFrame: Gets the audio frame for playback

Reference

This section contains information that completes the information in this page, or points you to documentation that explains other aspects to this product.