Custom video source | Agora Docs

Custom video capture refers to the collection of a video stream from a custom source. Unlike the default video capture method, custom video capture enables you to control the capture source, and precisely adjust video attributes. You can dynamically adjust parameters such as video quality, resolution, and frame rate to adapt to various application use-cases. For example, you can capture video from high-definition cameras, and drone cameras.

Agora recommends default video capture for its stability, reliability, and ease of integration. Custom video capture offers flexibility and customization for specific video capture use-cases where default video capture does not fulfill your requirements.

Understand the tech

Video SDK provides a custom video track method for video self-collection. You can create and publish custom video tracks to one or more channels. You use the self-capture module to drive the capture device, and send the captured video frames to the SDK through the video track.

The following figure illustrates the video data transmission process when custom video capture is implemented:

Prerequisites

Ensure that you have implemented the SDK quickstart in your project.

Implement the logic

This section shows you how to implement custom video capture and custom video rendering in your app.

Custom video capture

The following figure shows the workflow you implement to capture and stream a custom video source in your app.

Custom video capture

Take the following steps to implement this workflow:

Create a custom video track

To create a custom video track and obtain the video track ID, call createCustomVideoTrack after initializing an instance of RtcEngine. To create multiple custom video tracks, call the method multiple times.

int videoTrackId = RtcEngine.createCustomVideoTrack();

val videoTrackId = RtcEngine.createCustomVideoTrack()

Join a channel and publish the custom video track

// Create a ChannelMediaOptions instance
  ChannelMediaOptions option = new ChannelMediaOptions();
  // Set the client role to BROADCASTER
  option.clientRoleType = Constants.CLIENT_ROLE_BROADCASTER;
  // Enable auto subscription of audio and video
  option.autoSubscribeAudio = true;
  option.autoSubscribeVideo = true;
  // Publish self-captured video stream
  option.publishCustomVideoTrack = true;
  // Set custom video track ID
  option.customVideoTrackId = videoTrackId;
  // Join a channel with the specified options
  int res = engine.joinChannel(accessToken, channelId, 0, option);

// Create a ChannelMediaOptions instance
 val option = ChannelMediaOptions().apply {
   // Set the client role to BROADCASTER
   clientRoleType = Constants.CLIENT_ROLE_BROADCASTER
   // Enable auto subscription of audio and video
   autoSubscribeAudio = true
   autoSubscribeVideo = true
   // Publish self-captured video stream
   publishCustomVideoTrack = true
   // Set custom video track ID
   customVideoTrackId = videoTrackId
 }

 // Join a channel with the specified options
 val res = engine.joinChannel(accessToken, channelId, 0, option)

Implement your self-capture module

Agora provides the VideoFileReader demo project that shows you how to read YUV format video data from a local file. In a production environment, create a custom video module for your device using Video SDK based on your business requirements.

Push video data to the SDK

Before sending captured video frames to Video SDK, integrate your video module with the VideoFrame. To ensure audio-video synchronization, best practice is to obtain the current monotonic time from Video SDK and pass it as the timestamp parameter in the VideoFrame.

To ensure audio-video synchronization, set the timestamp parameter of VideoFrame to the system's Monotonic Time. Use getCurrentMonotonicTimeInMs to obtain the current monotonic Time.

Call pushExternalVideoFrameById [2/2] to push the captured video frames through the video track to Video SDK. Ensure that the videoTrackId matches the track ID you specified when joining the channel. Customize parameters like pixel format, data type, and timestamp in the VideoFrame.

The following code samples demonstrate pushing I420, NV21, NV12, and Texture format video data:

I420

private void pushVideoFrameByI420(int trackId, byte[] yuv, int width, int height) {
   // Create an i420Buffer object and store the original YUV data in the buffer
   JavaI420Buffer i420Buffer = JavaI420Buffer.allocate(width, height);
   i420Buffer.getDataY().put(yuv, 0, i420Buffer.getDataY().limit());
   i420Buffer.getDataU().put(yuv, i420Buffer.getDataY().limit(), i420Buffer.getDataU().limit());
   i420Buffer.getDataV().put(yuv, i420Buffer.getDataY().limit() + i420Buffer.getDataU().limit(), i420Buffer.getDataV().limit());
   // Get the current monotonic time from the SDK
   long currentMonotonicTimeInMs = engine.getCurrentMonotonicTimeInMs();
   // Create a VideoFrame object, passing the I420 video frame to be pushed and the monotonic time of the video frame (in nanoseconds)
   VideoFrame videoFrame = new VideoFrame(i420Buffer, 0, currentMonotonicTimeInMs * 1000000);

   // Push the video frame to the SDK through the video track
   int ret = engine.pushExternalVideoFrameById(videoFrame, trackId);
   // Release the memory resources occupied by the i420Buffer object
   i420Buffer.release();

   if (ret != Constants.ERR_OK) {
    Log.w(TAG, "pushExternalVideoFrame error");
   }
 }

private fun pushVideoFrameByI420(trackId: Int, yuv: ByteArray, width: Int, height: Int) {
   // Create an i420Buffer object and store the original YUV data in the buffer
   val i420Buffer = JavaI420Buffer.allocate(width, height)
   i420Buffer.getDataY().put(yuv, 0, i420Buffer.getDataY().limit())
   i420Buffer.getDataU().put(yuv, i420Buffer.getDataY().limit(), i420Buffer.getDataU().limit())
   i420Buffer.getDataV().put(yuv, i420Buffer.getDataY().limit() + i420Buffer.getDataU().limit(), i420Buffer.getDataV().limit())

   // Get the current monotonic time from the SDK
   val currentMonotonicTimeInMs = engine.getCurrentMonotonicTimeInMs()

   // Create a VideoFrame object, passing the I420 video frame to be pushed and the monotonic time of the video frame (in nanoseconds)
   val videoFrame = VideoFrame(i420Buffer, 0, currentMonotonicTimeInMs * 1_000_000)

   // Push the video frame to the SDK through the video track
   val ret = engine.pushExternalVideoFrameById(videoFrame, trackId)

   // Release the memory resources occupied by the i420Buffer object
   i420Buffer.release()

   if (ret != Constants.ERR_OK) {
    Log.w(TAG, "pushExternalVideoFrame error")
   }
 }

NV21

private void pushVideoFrameByNV21(int trackId, byte[] nv21, int width, height) {
   // Create a frameBuffer object and store the original YUV data in the NV21 format buffer
   VideoFrame.Buffer frameBuffer = new NV21Buffer(nv21, width, height, null);

   // Get the current monotonic time from the SDK
   long currentMonotonicTimeInMs = engine.getCurrentMonotonicTimeInMs();
   // Create a VideoFrame object, pass the NV21 video frame to be pushed and the monotonic time of the video frame (in nanoseconds)
   VideoFrame videoFrame = new VideoFrame(frameBuffer, 0, currentMonotonicTimeInMs * 1000000);

   // Push the video frame to the SDK through the video track
   int ret = engine.pushExternalVideoFrameById(videoFrame, trackId);

   if (ret != Constants.ERR_OK) {
    Log.w(TAG, "pushExternalVideoFrame error");
   }
 }

private fun pushVideoFrameByNV21(trackId: Int, nv21: ByteArray, width: Int, height: Int) {
   // Create a frameBuffer object and store the original YUV data in the NV21 format buffer
   val frameBuffer: VideoFrame.Buffer = NV21Buffer(nv21, width, height, null)

   // Get the current monotonic time from the SDK
   val currentMonotonicTimeInMs = engine.getCurrentMonotonicTimeInMs()

   // Create a VideoFrame object, pass the NV21 video frame to be pushed and the monotonic time of the video frame (in nanoseconds)
   val videoFrame = VideoFrame(frameBuffer, 0, currentMonotonicTimeInMs * 1_000_000)

   // Push the video frame to the SDK through the video track
   val ret = engine.pushExternalVideoFrameById(videoFrame, trackId)

   if (ret != Constants.ERR_OK) {
    Log.w(TAG, "pushExternalVideoFrame error")
   }
 }

NV12

private void pushVideoFrameByNV12(int trackId, ByteBuffer nv12, int width, int height) {
   // Create a frameBuffer object and store the original YUV data in the NV12 format buffer
   VideoFrame.Buffer frameBuffer = new NV12Buffer(width, height, width, height, nv12, null);
   // Get the current monotonic time from the SDK
   long currentMonotonicTimeInMs = engine.getCurrentMonotonicTimeInMs();
   // Create a VideoFrame object, pass the NV12 video frame to be pushed and the monotonic time of the video frame (in nanoseconds)
   VideoFrame videoFrame = new VideoFrame(frameBuffer, 0, currentMonotonicTimeInMs * 1000000);

   // Push the video frame to the SDK through the video track
   int ret = engine.pushExternalVideoFrameById(videoFrame, trackId);
   if (ret != Constants.ERR_OK) {
      Log.w(TAG, "pushExternalVideoFrame error");
   }
 }

private fun pushVideoFrameByNV12(trackId: Int, nv12: ByteBuffer, width: Int, height: Int) {
   // Create a frameBuffer object and store the original YUV data in the NV12 format buffer
   val frameBuffer: VideoFrame.Buffer = NV12Buffer(width, height, width, height, nv12, null)
   // Get the current monotonic time from the SDK
   val currentMonotonicTimeInMs = engine.getCurrentMonotonicTimeInMs()
   // Create a VideoFrame object, pass the NV12 video frame to be pushed and the monotonic time of the video frame (in nanoseconds)
   val videoFrame = VideoFrame(frameBuffer, 0, currentMonotonicTimeInMs * 1_000_000)
   // Push the video frame to the SDK through the video track
   val ret = engine.pushExternalVideoFrameById(videoFrame, trackId)

   if (ret != Constants.ERR_OK) {
    Log.w(TAG, "pushExternalVideoFrame error")
   }
 }

Texture

private void pushVideoFrameByTexture(int trackId, int textureId, VideoFrame.TextureBuffer.Type textureType, int width, int height) {
   // Create a frameBuffer object to store the texture format video frame
   VideoFrame.Buffer frameBuffer = new TextureBuffer(
    EglBaseProvider.getCurrentEglContext(),
    width,
    height,
    textureType,
    textureId,
    new Matrix(),
    null,
    null,
    null
   );
   // Get the current monotonic time from the SDK
   long currentMonotonicTimeInMs = engine.getCurrentMonotonicTimeInMs();
   // Create a VideoFrame object, passing the texture video frame to be pushed and the monotonic time of the video frame (in nanoseconds)
   VideoFrame videoFrame = new VideoFrame(frameBuffer, 0, currentMonotonicTimeInMs * 1000000);
   // Push the video frame to the SDK through the video track
   int ret = engine.pushExternalVideoFrameById(videoFrame, trackId);
   if (ret != Constants.ERR_OK) {
    Log.w(TAG, "pushExternalVideoFrame error");
   }
 }

private fun pushVideoFrameByTexture(
    trackId: Int,
    textureId: Int,
    textureType: VideoFrame.TextureBuffer.Type,
    width: Int,
    height: Int
   ) {
   // Create a frameBuffer object to store the texture format video frame
   val frameBuffer: VideoFrame.Buffer = TextureBuffer(
    EglBaseProvider.getCurrentEglContext(),
    width,
    height,
    textureType,
    textureId,
    Matrix(),
    null,
    null,
    null
   )

   // Get the current monotonic time from the SDK
   val currentMonotonicTimeInMs = engine.getCurrentMonotonicTimeInMs()
   // Create a VideoFrame object, passing the texture video frame to be pushed and the monotonic time of the video frame (in nanoseconds)
   val videoFrame = VideoFrame(frameBuffer, 0, currentMonotonicTimeInMs * 1_000_000)
   // Push the video frame to the SDK through the video track
   val ret = engine.pushExternalVideoFrameById(videoFrame, trackId)

   if (ret != Constants.ERR_OK) {
    Log.w(TAG, "pushExternalVideoFrame error")
   }
 }

If the captured custom video format is Texture and remote users experience flickering or distortion in the captured video, it is recommended to first duplicate the video data and then send both the original and duplicated video data back to the Video SDK. This helps eliminate anomalies during internal data encoding processes.

Destroy custom video tracks

To stop custom video capture and destroy the video track, call destroyCustomVideoTrack.

// Destroy custom video track
 engine.destroyCustomVideoTrack(videoTrack);
 // Leave the channel
 engine.leaveChannelEx(connection);

// Destroy custom video track
 engine.destroyCustomVideoTrack(videoTrack)
 // Leave the channel
 engine.leaveChannelEx(connection)

Custom video rendering

To implement custom video rendering in your app, refer to the following steps:

Set up onCaptureVideoFrame or onRenderVideoFrame callback to obtain the video data to be played.
Implement video rendering and playback yourself.

Reference

This section contains content that completes the information on this page, or points you to documentation that explains other aspects to this product.

Applicable use-cases

Use custom video capture in the following industries and use-cases:

Specialized video processing and enhancement

In specific gaming or virtual reality use-cases, real-time effects processing, filter handling, or other enhancement effects necessitate direct access to the original video stream. Custom video capture facilitates this, enabling seamless real-time processing and enhances the overall gaming or virtual reality experience for a more realistic outcome.

High-precision video capture

In video surveillance applications, detailed observation and analysis of scene details is necessary. Custom video capture enables higher image quality and finer control over capture to meet the requirements of video monitoring.

Capture from specific video sources

Industries such as IoT and live streaming often require the use of specific cameras, monitoring devices, or non-camera video sources, such as video capture cards or screen recording data. In such situations, default Video SDK capture may not meet your requirements, necessitating use of custom video capture.

Seamless integration with specific devices or third-party applications

In smart home or IoT applications, transmitting video from devices to users' smartphones or computers for monitoring and control may require the use of specific devices or applications for video capture. Custom video capture facilitates seamless integration of specific devices or applications with the Video SDK.

Specific video encoding formats

In certain live streaming use-cases, specific video encoding formats may be needed to meet business requirements. In such cases, Video SDK default capture might not suffice, and custom video capture is required to capture and encode videos in specific formats.

Advantages

Using custom video capture offers the following advantages:

More types of video streams

Custom video capture allows the use of higher quality and a greater variety of capture devices and cameras, resulting in clearer and smoother video streams. This enhances the user viewing experience and makes the product more competitive.

More flexible video effects

Custom video capture enables you to implement richer and more personalized video effects and filters, enhancing the user experience. You can implement effects such as beautification filters and dynamic stickers.

Adaptation to diverse use-case requirements

Custom video capture helps applications better adapt to the requirements of various use-cases, such as live streaming, video conferencing, and online education. You can customize different video capture solutions based on the use-case requirements to provide a more robust application.

Sample projects

Agora provides the following open-source sample projects for your reference. Download the project or view the source code for a more detailed example.

MultiVideoSourceTracks: Video self-capture
CustomRemoteVideoRender: Custom remote video rendering

API reference

Understand the tech

The following figure illustrates the video data transmission process when custom video capture is implemented:

Prerequisites

Ensure that you have implemented the SDK quickstart in your project.

Implement the logic

This section shows you how to implement custom video capture and custom video rendering in your app.

Custom video capture

The figure below shows the workflow you need to implement to stream a custom video source in your app.

Custom video capture

Take the following steps to implement this workflow:

Create a custom video track

To create a custom video track and obtain the video track ID, call createCustomVideoTrack after initializing an instance of AgoraRtcEngineKit. To create multiple custom video tracks, call the method multiple times.

customCamera?.trackId = agoraKit.createCustomVideoTrack()

Join a channel and publish the custom video track

// Create an AgoraRtcChannelMediaOptions instance
let option = AgoraRtcChannelMediaOptions()
// Set the client role to broadcaster
option.clientRoleType = .broadcaster
// Enable auto subscription for audio and video
option.autoSubscribeAudio = true
option.autoSubscribeVideo = true
// Publish the self-collected video stream
option.publishCustomVideoTrack = true
// Set the custom video track ID
option.customVideoTrackId = Int(customCamera?.trackId ?? 0)
// Generate a token for the channel
NetworkManager.shared.generateToken(channelName: channel, success: { token in
  // Join a channel with the specified options
  let result = self.agoraKit.joinChannel(byToken: token, channelId: channel, uid: 0, mediaOptions: option)
  if result != 0 {
    self.isProcessing = false
    self.showAlert(title: "Error", message: "joinChannel call failed: \(result), please check your params")
  }
})

Implement your self-capture module

Agora provides the CustomVideoSourcePushMulti demo project that shows you how to read YUV format video data from a local file. In a production environment, create a custom video module for your device using Video SDK based on your business requirements.

Push video data to the SDK

Use pushExternalVideoFrame to send the captured video frame to the SDK through the video track. Ensure that the videoTrackId matches the video track ID specified when joining the channel. In AgoraVideoFrame, you can set the pixel format, data type, timestamp, and other parameters of the video frame.

The following code demonstrates how to push video data in the CVPixelBufferRef format. For other supported video frame formats, refer to the format field in AgoraVideoFrame.

let videoFrame = AgoraVideoFrame()
videoFrame.format = 12
videoFrame.textureBuf = buffer
videoFrame.rotation = Int32(rotation)
agoraKit?.pushExternalVideoFrame(videoFrame, videoTrackId: trackId)

To ensure audio and video synchronization, Agora recommends setting the time (timestamp) of AgoraVideoFrame to the system monotonic time. To retrieve the current monotonic time, call getCurrentMonotonicTimeInMs.

Destroy custom video track

To stop publishing the custom video track, call destroyCustomVideoTrack.

// Destroy the custom video track
agoraKit.destroyCustomVideoTrack(UInt(userModel?.trackId ?? 0))
// Leave the channel
agoraKit.leaveChannelEx(connection) { state in
  LogUtils.log(message: "warning: \(state.description)", level: .info)
}

Custom video rendering

To implement custom video rendering in your app, refer to the following steps:

Set up onCaptureVideoFrame or onRenderVideoFrame callback to obtain the video data to be played.
Implement video rendering and playback yourself.
In renderPixelBuffer or renderRawData, the rotation parameter of the video frame may not be 0. You need to set the rotation parameters yourself.

Reference

This section contains content that completes the information on this page, or points you to documentation that explains other aspects to this product.

Applicable use-cases

Use custom video capture in the following industries and use-cases:

Specialized video processing and enhancement

High-precision video capture

Capture from specific video sources

Seamless integration with specific devices or third-party applications

Specific video encoding formats

Advantages

Using custom video capture offers the following advantages:

More types of video streams

More flexible video effects

Adaptation to diverse use-case requirements

Sample projects

Agora provides the following open-source sample projects for your reference. Download the project or view the source code for a more detailed example.

CustomVideoSourcePush: Video self-capture
CustomVideoRender: Custom video rendering
CustomVideoSourcePushMulti.swift: Read YUV format video data from a local file.

API reference

Understand the tech

The following figure illustrates the video data transmission process when custom video capture is implemented:

Prerequisites

Ensure that you have implemented the SDK quickstart in your project.

Implement the logic

This section shows you how to implement custom video capture and custom video rendering in your app.

Custom video capture

The figure below shows the workflow you need to implement to stream a custom video source in your app.

Custom video capture

Take the following steps to implement this workflow:

Create a custom video track

customCamera?.trackId = agoraKit.createCustomVideoTrack()

Join a channel and publish the custom video track

// Create an AgoraRtcChannelMediaOptions instance
let option = AgoraRtcChannelMediaOptions()
// Set the client role to broadcaster
option.clientRoleType = .broadcaster
// Enable auto subscription for audio and video
option.autoSubscribeAudio = true
option.autoSubscribeVideo = true
// Publish the self-collected video stream
option.publishCustomVideoTrack = true
// Set the custom video track ID
option.customVideoTrackId = Int(customCamera?.trackId ?? 0)
// Generate a token for the channel
NetworkManager.shared.generateToken(channelName: channel, success: { token in
  // Join a channel with the specified options
  let result = self.agoraKit.joinChannel(byToken: token, channelId: channel, uid: 0, mediaOptions: option)
  if result != 0 {
    self.isProcessing = false
    self.showAlert(title: "Error", message: "joinChannel call failed: \(result), please check your params")
  }
})

Implement your self-capture module

Push video data to the SDK

The following code demonstrates how to push video data in the CVPixelBufferRef format. For other supported video frame formats, refer to the format field in AgoraVideoFrame.

let videoFrame = AgoraVideoFrame()
videoFrame.format = 12
videoFrame.textureBuf = buffer
videoFrame.rotation = Int32(rotation)
agoraKit?.pushExternalVideoFrame(videoFrame, videoTrackId: trackId)

Destroy custom video track

To stop publishing the custom video track, call destroyCustomVideoTrack.

// Destroy the custom video track
agoraKit.destroyCustomVideoTrack(UInt(userModel?.trackId ?? 0))
// Leave the channel
agoraKit.leaveChannelEx(connection) { state in
  LogUtils.log(message: "warning: \(state.description)", level: .info)
}

Custom video rendering

To implement custom video rendering in your app, refer to the following steps:

Set up onCaptureVideoFrame or onRenderVideoFrame callback to obtain the video data to be played.
Implement video rendering and playback yourself.
In renderPixelBuffer or renderRawData, the rotation parameter of the video frame may not be 0. You need to set the rotation parameters yourself.

Reference

This section contains content that completes the information on this page, or points you to documentation that explains other aspects to this product.

Applicable use-cases

Use custom video capture in the following industries and use-cases:

Specialized video processing and enhancement

High-precision video capture

Capture from specific video sources

Seamless integration with specific devices or third-party applications

Specific video encoding formats

Advantages

Using custom video capture offers the following advantages:

More types of video streams

More flexible video effects

Adaptation to diverse use-case requirements

Sample projects

Agora provides the following open-source sample projects for your reference. Download the project or view the source code for a more detailed example.

CustomVideoSourcePush: Video self-capture
CustomVideoRender: Custom video rendering

API reference

Try out the online demo for Custom video source.

Understand the tech

Video SDK provides support for creating local video tracks by passing in a MediaStreamTrack object to createCustomVideoTrack.

Prerequisites

Ensure that you have implemented the SDK quickstart in your project.

Implement the logic

This section shows you how to create and publish a custom video track from a media streams or a canvas stream.

Create a custom video track a media stream

You can implement custom video capture or preprocessing by obtaining a MediaStreamTrack object. For example, manually call the getUserMedia method to obtain a MediaStreamTrack, then use the createCustomVideoTrack method to create a local video track object for use in Video SDK.

async function createAndPublishCustomVideoTrack() {
  try {
    // Get the video media stream
    const mediaStream = await navigator.mediaDevices.getUserMedia({ video: true, audio: false });

    // Extract the video track from the media stream
    const videoMediaStreamTrack = mediaStream.getVideoTracks()[0];

    // Create a custom video track
    const customVideoTrack = await AgoraRTC.createCustomVideoTrack({
      mediaStreamTrack: videoMediaStreamTrack,
    });

    // Store the custom video track for later use in a shared object
    rtc.localVideoTrack = customVideoTrack;

    // Publish the custom video track to the RTC channel
    await rtc.client.publish([rtc.localVideoTrack]);

    console.log("Custom video track published successfully!");
  } catch (error) {
    console.error("Failed to create or publish custom video track:", error);
  }
}

You can also use HTMLMediaElement.captureStream or HTMLCanvasElement.captureStream to obtain the MediaStreamTrack object.

The MediaStreamTrack object refers to the browser-native MediaStreamTrack API. For details on usage and browser support, see th MediaStreamTrack API documentation.

Create a custom video track from a canvas stream

To create and publish a custom video track generated from a canvas stream, follow these steps:

Create a canvas element and set its dimensions.
Continuously draw content on the canvas using requestAnimationFrame to ensure the stream updates in real time.
Capture a media stream from the canvas.
Create a custom video track using the canvas stream.
Publish the custom video track to the channel.

Refer to the following example:

// Function to create and publish a custom video track
async function createAndPublishCustomVideoTrack() {
  try {
    // Create and draw on the canvas
    const canvas = createAndDrawCanvas();

    // Get the media stream from the canvas
    const canvasStream = canvas.captureStream();
    const canvasMediaStreamTrack = canvasStream.getVideoTracks()[0];

    // Create a custom video track from the canvas stream
    rtc.localVideoTrack = await AgoraRTC.createCustomVideoTrack({
      mediaStreamTrack: canvasMediaStreamTrack,
    });

    // Publish the custom video track to the RTC channel
    await rtc.client.publish([rtc.localVideoTrack]);

    console.log("Custom video track published successfully!");
  } catch (error) {
    console.error("Failed to create or publish custom video track:", error);
  }
}

// Function to create a canvas and start drawing
function createAndDrawCanvas() {
  const canvas = document.createElement("canvas");
  canvas.width = 640;
  canvas.height = 480;
  const context = canvas.getContext("2d");

  // Continuously draw on the canvas
  function drawCanvas() {
    // Clear previous drawings
    context.clearRect(0, 0, canvas.width, canvas.height);
    context.fillStyle = "blue";
    context.fillRect(0, 0, canvas.width, canvas.height);

    // Add your code to draw on the canvas

    // Keep refreshing the canvas
    requestAnimationFrame(drawCanvas);
  }
  drawCanvas();

  return canvas;
}

// Call the function to create and publish the track
createAndPublishCustomVideoTrack();

Reference

This section contains content that completes the information on this page, or points you to documentation that explains other aspects to this product.

API reference

Understand the tech

The following figure illustrates the video data transmission process when custom video capture is implemented:

Prerequisites

Ensure that you have implemented the SDK quickstart in your project.

Implement the logic

This section shows you how to implement custom video capture and custom video rendering in your app.

Custom video capture

The following figure shows the workflow you implement to capture and stream a custom video source in your app.

Custom video capture

Take the following steps to implement this workflow:

Create a custom video track

To create a custom video track and obtain the video track ID, call createCustomVideoTrack after initializing an instance of IRtcEngine. To create multiple custom video tracks, call the method multiple times.

// For creating multiple custom video tracks, call createCustomVideoTrack multiple times
int videoTrackId = m_rtcEngine->createCustomVideoTrack();
m_trackVideoTrackIds[trackIndex] = videoTrackId;

Join a channel and publish the custom video track

// Create a ChannelMediaOptions instance
ChannelMediaOptions mediaOptions;
// Set the client role to broadcaster
mediaOptions.clientRoleType = CLIENT_ROLE_BROADCASTER;
// Publish the self-captured video stream
mediaOptions.publishCustomVideoTrack = true;
// Disable auto subscription for video and audio
mediaOptions.autoSubscribeVideo = false;
mediaOptions.autoSubscribeAudio = false;
// Set the custom video track ID
mediaOptions.customVideoTrackId = videoTrackId;
// Join the channel with the specified options
int ret = m_rtcEngine->joinChannel(APP_TOKEN, szChannelId.data(), 0, mediaOptions);

Implement self-capture module

Agora provides the YUVReader.cpp and YUVReader.h demo projects that show you how to read YUV format video data from a local file. In a production environment, create a custom video module for your device using Video SDK based on your business requirements.

// Use the custom YUVReader class to continuously read YUV-format video data in the YUVReader thread and pass the data to the OnYUVRead callback for further processing
m_yuvReaderHandlers[trackIndex].Setup(m_rtcEngine, m_mediaEngine.get(), videoTrackId);
m_yuvReaders[trackIndex].start(std::bind(&MultiVideoSourceTracksYUVReaderHander::OnYUVRead, m_yuvReaderHandlers[trackIndex], std::placeholders::_1, std::placeholders::_2, std::placeholders::_3, std::placeholders::_4));

Push video data to the SDK

Call pushVideoFrame to push the captured video frames through the video track to Video SDK. Ensure that the videoTrackId matches the track ID you specified when joining the channel. Customize parameters like pixel format, data type, and timestamp in the VideoFrame.

To ensure audio-video synchronization, set the timestamp parameter of VideoFrame to the system's Monotonic Time. Use getCurrentMonotonicTimeInMs to obtain the current Monotonic Time.

void MultiVideoSourceTracksYUVReaderHander::OnYUVRead(int width, int height,
                           unsigned char* buffer,
                           int size) {
  if (m_mediaEngine == nullptr || m_rtcEngine == nullptr) {
    return;
  }

  // Set the video pixel format to I420
  m_videoFrame.format = agora::media::base::VIDEO_PIXEL_I420;
  // Set the video data type to raw data
  m_videoFrame.type = agora::media::base::ExternalVideoFrame::
    VIDEO_BUFFER_TYPE::VIDEO_BUFFER_RAW_DATA;
  // Pass the width, height, and buffer of the captured YUV video data to videoFrame
  m_videoFrame.height = height;
  m_videoFrame.stride = width;
  m_videoFrame.buffer = buffer;
  // Get the current Monotonic Time from the SDK and assign it to the timestamp parameter of videoFrame
  m_videoFrame.timestamp = m_rtcEngine->getCurrentMonotonicTimeInMs();
  // Push the video frame to the SDK
  m_mediaEngine->pushVideoFrame(&m_videoFrame, m_videoTrackId);
}

The sample code demonstrates converting YUV format to raw video data in I420 format. Agora video self-capture supports pushing external video frames in other formats; refer to VIDEO_PIXEL_FORMAT.

Destroy custom video tracks

To stop custom video capture and destroy the video track, call destroyCustomVideoTrack. To destroy multiple video tracks, call the method for each track.

// Stop self-captured video data
m_yuvReaders[trackIndex].stop();
m_yuvReaderHandlers[trackIndex].Release();
// Destroy the custom video track
m_rtcEngine->destroyCustomVideoTrack(m_trackVideoTrackIds[trackIndex]);
// Leave the channel
m_rtcEngine->leaveChannelEx(m_trackConnections[trackIndex]);

Custom video rendering

To implement custom video rendering in your app, refer to the following steps:

Set up onCaptureVideoFrame or onRenderVideoFrame callback to obtain the video data to be played.
Implement video rendering and playback yourself.

Reference

This section contains content that completes the information on this page, or points you to documentation that explains other aspects to this product.

Applicable use-cases

Use custom video capture in the following industries and use-cases:

Specialized video processing and enhancement

High-precision video capture

Capture from specific video sources

Seamless integration with specific devices or third-party applications

Specific video encoding formats

Advantages

Using custom video capture offers the following advantages:

More types of video streams

More flexible video effects

Adaptation to diverse use-case requirements

Sample project

Agora provides the following open-source sample projects for your reference. Download the project or view the source code for a more detailed example.

MultiVideoSourceTracks: Video self-capture
CustomVideoCapture: Custom video rendering

API reference

Understand the tech

The following figure illustrates the video data transmission process when custom video capture is implemented:

Prerequisites

Ensure that you have implemented the SDK quickstart in your project.

Implement the logic

This section shows you how to implement custom video capture and custom video rendering in your game.

Initialize MediaEngine

Before implementing custom video features, obtain the MediaEngine interface from the initialized RtcEngine:

void InitAgoraEngine(FString APP_ID, FString TOKEN, FString CHANNEL_NAME)
{
  // Initialize RtcEngine context and event handler
  agora::rtc::RtcEngineContext RtcEngineContext;
  UserRtcEventHandler = MakeShared<FUserRtcEventHandler>(this);
  std::string StdStrAppId = TCHAR_TO_UTF8(*APP_ID);
  RtcEngineContext.appId = StdStrAppId.c_str();
  RtcEngineContext.eventHandler = UserRtcEventHandler.Get();
  RtcEngineContext.channelProfile = agora::CHANNEL_PROFILE_TYPE::CHANNEL_PROFILE_LIVE_BROADCASTING;

  // Initialize the RtcEngine
  int ret = AgoraUERtcEngine::Get()->initialize(RtcEngineContext);

  // Query for the MediaEngine interface
  int ret = AgoraUERtcEngine::Get()->queryInterface(INTERFACE_ID_TYPE::AGORA_IID_MEDIA_ENGINE, (void**)&MediaEngineManager);
}

The MediaEngine is not created directly. Instead, it's obtained as an interface from the initialized RtcEngine using the queryInterface method with AGORA_IID_MEDIA_ENGINE.

Custom video capture

The following figure shows the workflow you implement to capture and stream a custom video source in your game.

Custom video capture

Take the following steps to implement this workflow:

Set up the external video source

To use custom video frames instead of the camera feed, configure the SDK to accept external video data.

The following code enables the external video source mode and specifies the type as VIDEO_FRAME, indicating you will push raw pixel data to the SDK.

void SetExternalVideoSource()
{
  agora::rtc::SenderOptions sendoptions;
  int ret = MediaEngineManager->setExternalVideoSource(true, false, agora::media::EXTERNAL_VIDEO_SOURCE_TYPE::VIDEO_FRAME, sendoptions);
  UBFL_Logger::Print(FString::Printf(TEXT("%s setExternalVideoSource ret %d"), *FString(FUNCTION_MACRO), ret), LogMsgViewPtr);
}

Join a channel

Enable audio and video, set the client role, and join the channel.

void JoinChannel()
{
  AgoraUERtcEngine::Get()->enableAudio();
  AgoraUERtcEngine::Get()->enableVideo();
  AgoraUERtcEngine::Get()->setClientRole(CLIENT_ROLE_BROADCASTER);
  int ret = AgoraUERtcEngine::Get()->joinChannel(TCHAR_TO_UTF8(*Token), TCHAR_TO_UTF8(*ChannelName), "", 0);
}

Implement custom video capture

Set up a callback to capture frames from Unreal Engine's rendering pipeline. Register for the back buffer ready event to capture rendered frames.

void InitAgoraWidget(FString APP_ID, FString TOKEN, FString CHANNEL_NAME)
{
  // ... other initialization code ...

  // Register for back buffer ready callback
  if (FSlateApplication::IsInitialized())
  {
    eventId = FSlateApplication::Get().GetRenderer()->OnBackBufferReadyToPresent().AddUObject(this, &UCustomCaptureVideoScene::OnBackBufferReady_RenderThread);
  }
}

Push video frames

Capture frames from Unreal Engine's rendering pipeline on the render thread, convert them to raw BGRA pixel format, and send them to the Agora SDK as external video frames.

Timestamp Sync

To synchronize audio and video streams, use system timestamp for the timestamp field in the ExternalVideoFrame.

void OnBackBufferReady_RenderThread(SWindow& window, const FTexture2DRHIRef& BackBuffer)
{
  FRHICommandListImmediate& RHICmdList = FRHICommandListExecutor::GetImmediateCommandList();
  auto width = BackBuffer->GetSizeX();
  auto height = BackBuffer->GetSizeY();
  FIntRect Rect(0, 0, BackBuffer->GetSizeX(), BackBuffer->GetSizeY());
  TArray<FColor> Data;

  RHICmdList.ReadSurfaceData(BackBuffer, Rect, Data, FReadSurfaceDataFlags());

  if (UserExternalVideoFrame == nullptr)
  {
    UserExternalVideoFrame = new agora::media::base::ExternalVideoFrame();
  }

  // Configure the video frame
  UserExternalVideoFrame->type = agora::media::base::ExternalVideoFrame::VIDEO_BUFFER_TYPE::VIDEO_BUFFER_RAW_DATA;
  UserExternalVideoFrame->format = agora::media::base::VIDEO_PIXEL_FORMAT::VIDEO_PIXEL_BGRA;
  UserExternalVideoFrame->stride = BackBuffer->GetSizeX();
  UserExternalVideoFrame->height = BackBuffer->GetSizeY();
  UserExternalVideoFrame->cropLeft = 10;
  UserExternalVideoFrame->cropTop = 10;
  UserExternalVideoFrame->cropRight = 10;
  UserExternalVideoFrame->cropBottom = 10;
  UserExternalVideoFrame->rotation = 0;
  UserExternalVideoFrame->timestamp = getTimeStamp();

  if (UserExternalVideoFrame->buffer == nullptr)
  {
    UserExternalVideoFrame->buffer = (uint8*)FMemory::Malloc(BackBuffer->GetSizeX() * BackBuffer->GetSizeY() * 4);
  }

  if (Data.Num() > 4)
  {
    FMemory::Memcpy(UserExternalVideoFrame->buffer, Data.GetData(), BackBuffer->GetSizeX() * BackBuffer->GetSizeY() * 4);
    if (MediaEngineManager != nullptr)
    {
      // Push the video frame to the SDK
      MediaEngineManager->pushVideoFrame(UserExternalVideoFrame);
    }
  }
}

std::time_t getTimeStamp()
{
  std::chrono::time_point<std::chrono::system_clock, std::chrono::milliseconds> tp =
    std::chrono::time_point_cast<std::chrono::milliseconds>(std::chrono::system_clock::now());
  std::time_t timestamp = tp.time_since_epoch().count();
  return timestamp;
}

The sample code demonstrates converting Unreal's BGRA format to raw video data. Agora video capture supports pushing external video frames in other formats; refer to VIDEO_PIXEL_FORMAT.

Cleanup

When finished unregister the callback and clean up resources.

void NativeDestruct()
{
  Super::NativeDestruct();

  // Remove the back buffer callback
  FSlateApplication::Get().GetRenderer()->OnBackBufferReadyToPresent().Remove(eventId);

  UnInitAgoraEngine();
}

void UnInitAgoraEngine()
{
  if (AgoraUERtcEngine::Get() != nullptr)
  {
    AgoraUERtcEngine::Get()->leaveChannel();
    AgoraUERtcEngine::Get()->unregisterEventHandler(UserRtcEventHandler.Get());
    AgoraUERtcEngine::Release();
    MediaEngineManager = nullptr;
  }
}

Custom video rendering

To implement custom video rendering in your game, refer to the following steps:

Set up onCaptureVideoFrame or onRenderVideoFrame callback to obtain the video data to be played.
Implement video rendering and playback yourself.

Reference

This section contains content that completes the information on this page, or points you to documentation that explains other aspects to this product.

Applicable use-cases

Use custom video capture in the following industries and use-cases:

Specialized video processing and enhancement

High-precision video capture

Capture from specific video sources

Seamless integration with specific devices or third-party applications

Specific video encoding formats

Advantages

Using custom video capture offers the following advantages:

More types of video streams

More flexible video effects

Adaptation to diverse use-case requirements

Sample projects

Agora provides a open-source sample project for your reference. Download CustomCaptureVideo or view the source code for a more detailed example.

Contact Us

Contact Us