Raw Video Data
During the video transmission process, you can pre- and post-process the captured video data to achieve the desired playback effect.
Agora provides the raw data function for you to process the video data per your application scenario. This function enables you to pre-process the captured video frames before sending it to the encoder, or to post-process the decoded video frames.
Agora provides an open-source sample project on GitHub. You can view the source code on Github or download the project to try it out.
Before using the raw data function, ensure that you have implemented the basic real-time communication functions in your project. See Start a Video Call or Start Interactive Live Video Streaming for details.
IVideoFrameObserverclass to capture and modify raw video data. Therefore, you must use Java to call Agora's C++ interface via the JNI (Java Native Interface). Since the Video SDK for Java encapsulates the Video SDK for C++, you can include the
.hfile in the SDK to directly call the C++ methods.
Follow these steps to implement the raw video data function in your project:
- Use the JNI and C++ interface files to generate a shared library in the project, and use Java to call the raw video data interface of the Agora C++ SDK.
- Before joining a channel, call the
registerVideoFrameObservermethod to register a video observer, and implement an
IVideoFrameObserverclass in this method.
- After you successfully register the observer, the SDK captures every video frame and sends the captured raw video data via the
- Process the captured raw video data according to your needs. Then, you can send it to the SDK via the callbacks mentioned in step 3.
When you use the SDK v3.0.1 or later, note that the SDK no longer guarantees that callback functions in
IVideoFrameObserver are reported in the same thread. The SDK only guarantees the sequence of these callbacks.
If you are using OpenGL to perform image enhancement on the raw video data, you need to actively switch the OpenGL context in the callback function in
IVideoFrameObserver to adapt to a multi-threaded scenario; otherwise, the image enhancement cannot work.
Call the Agora C++ API in a Java project
The following diagram shows the basic flow of calling the Agora C++ API in a Java project:
- The Java project loads the
.solibrary built from the C++ interface file (
.cppfile) via the Java interface file.
- The Java interface file generates a
.hfile with the
javac -h -jnicommand. The C++ interface file should include this file.
- The C++ interface file calls C++ methods of the
.solibrary in the Agora Android SDK by including the Agora Android SDK header file.
API call sequence
The following diagram shows how to implement the raw video data function in your project:
onRenderVideoFrameare all C++ methods and callbacks.
Create a JNI interface
Create a Java interface file and a C++ interface file separately via the JNI interface. Make sure to build the C++ interface file as a
- Create a Java interface file to call the C++ API. The interface file should declare the corresponding Java methods for calling C++. Refer to the
MediaPreProcessing.javafile in the sample project for the implementation.
- Run the following command to generate a
.hfile from the Java interface file:
- Create a C++ interface file to include the methods to be called from Java. The C++ interface file exports the corresponding methods from the C++ SDK based on the generated
.hfile. Refer to the
io_agora_advancedvideo_rawdata_MediaPreProcessing.cppfile in the sample project for the implementation.
- Build the C++ interface file via the NDK to generate a
.solibrary. Use the
System.loadLibrary()method to load the generated
.solibrary in the Java interface file. See the following CMake example:
Implement the raw video data function in a Java project
- Implement an interface that maps to the C++ methods in a Java interface file.
- Call the
setCallbackmethod calls the
registerVideoFrameObserverC++ method via JNI to register a video frame observer.
- Implement the
onPreEncodeVideoFramecallbacks. Get the video frame from the callbacks, and process the video frame.
Refer to Raw Audio Data if you want to implement the raw audio data function in your project.