# Custom video source (/en/realtime-media/rtc/build/capture-and-render-video/custom-video/unreal)

> For AI agents: see the complete documentation index at [llms.txt](/llms.txt).

Custom video capture refers to the collection of a video stream from a custom source. Unlike the default video capture method, custom video capture enables you to control the capture source, and precisely adjust video attributes. You can dynamically adjust parameters such as video quality, resolution, and frame rate to adapt to various application use-cases. For example, you can capture video from high-definition cameras, and drone cameras.

Agora recommends default video capture for its stability, reliability, and ease of integration. Custom video capture offers flexibility and customization for specific video capture use-cases where default video capture does not fulfill your requirements.

      
  
      
  
      
  
      
  
      
  
      
## Understand the tech

RTC SDK provides a custom video track method for video self-collection. You can create and publish custom video tracks to one or more channels. You use the self-capture module to drive the capture device, and send the captured video frames to the SDK through the video track.

The following figure illustrates the video data transmission process when custom video capture is implemented:

![Custom video source](https://assets-docs.agora.io/images/video-sdk/publish-custom-tracks-in-a-channel.svg)

## Prerequisites

Ensure that you have implemented the [SDK quickstart](/en/realtime-media/rtc/get-started-sdk) in your project.

## Implement the logic

This section shows you how to implement custom video capture and custom video rendering in your game.

### Initialize MediaEngine

Before implementing custom video features, obtain the `MediaEngine` interface from the initialized `RtcEngine`:

```cpp
void InitAgoraEngine(FString APP_ID, FString TOKEN, FString CHANNEL_NAME)
{
  // Initialize RtcEngine context and event handler
  agora::rtc::RtcEngineContext RtcEngineContext;
  UserRtcEventHandler = MakeShared<FUserRtcEventHandler>(this);
  std::string StdStrAppId = TCHAR_TO_UTF8(*APP_ID);
  RtcEngineContext.appId = StdStrAppId.c_str();
  RtcEngineContext.eventHandler = UserRtcEventHandler.Get();
  RtcEngineContext.channelProfile = agora::CHANNEL_PROFILE_TYPE::CHANNEL_PROFILE_LIVE_BROADCASTING;

  // Initialize the RtcEngine
  int ret = AgoraUERtcEngine::Get()->initialize(RtcEngineContext);

  // Query for the MediaEngine interface
  int ret = AgoraUERtcEngine::Get()->queryInterface(INTERFACE_ID_TYPE::AGORA_IID_MEDIA_ENGINE, (void**)&MediaEngineManager);
}
```

<CalloutContainer type="info">
  <CalloutDescription>
    The MediaEngine is not created directly. Instead, it's obtained as an interface from the initialized RtcEngine using the `queryInterface` method with `AGORA_IID_MEDIA_ENGINE`.
  </CalloutDescription>
</CalloutContainer>

### Custom video capture

The following figure shows the workflow you implement to capture and stream a custom video source in your game.

**Custom video capture**

![API call sequence](https://assets-docs.agora.io/images/video-sdk/custom-video-capture-unreal.svg)

Take the following steps to implement this workflow:

1. **Set up the external video source**

To use custom video frames instead of the camera feed, configure the SDK to accept external video data.

The following code enables the external video source mode and specifies the type as `VIDEO_FRAME`, indicating you will push raw pixel data to the SDK.

```cpp
void SetExternalVideoSource()
{
  agora::rtc::SenderOptions sendoptions;
  int ret = MediaEngineManager->setExternalVideoSource(true, false, agora::media::EXTERNAL_VIDEO_SOURCE_TYPE::VIDEO_FRAME, sendoptions);
  UBFL_Logger::Print(FString::Printf(TEXT("%s setExternalVideoSource ret %d"), *FString(FUNCTION_MACRO), ret), LogMsgViewPtr);
}
```

2. **Join a channel**

Enable audio and video, set the client role, and join the channel.

```cpp
void JoinChannel()
{
  AgoraUERtcEngine::Get()->enableAudio();
  AgoraUERtcEngine::Get()->enableVideo();
  AgoraUERtcEngine::Get()->setClientRole(CLIENT_ROLE_BROADCASTER);
  int ret = AgoraUERtcEngine::Get()->joinChannel(TCHAR_TO_UTF8(*Token), TCHAR_TO_UTF8(*ChannelName), "", 0);
}
```

3. **Implement custom video capture**

Set up a callback to capture frames from Unreal Engine's rendering pipeline. Register for the back buffer ready event to capture rendered frames.

```cpp
void InitAgoraWidget(FString APP_ID, FString TOKEN, FString CHANNEL_NAME)
{
  // ... other initialization code ...

  // Register for back buffer ready callback
  if (FSlateApplication::IsInitialized())
  {
    eventId = FSlateApplication::Get().GetRenderer()->OnBackBufferReadyToPresent().AddUObject(this, &UCustomCaptureVideoScene::OnBackBufferReady_RenderThread);
  }
}
```

4. **Push video frames**

Capture frames from Unreal Engine's rendering pipeline on the render thread, convert them to raw BGRA pixel format, and send them to the Agora SDK as external video frames.

<CalloutContainer type="info">
  <CalloutDescription>
    To synchronize audio and video streams, use system timestamp for the `timestamp` field in the `ExternalVideoFrame`.
  </CalloutDescription>
</CalloutContainer>

```cpp
void OnBackBufferReady_RenderThread(SWindow& window, const FTexture2DRHIRef& BackBuffer)
{
  FRHICommandListImmediate& RHICmdList = FRHICommandListExecutor::GetImmediateCommandList();
  auto width = BackBuffer->GetSizeX();
  auto height = BackBuffer->GetSizeY();
  FIntRect Rect(0, 0, BackBuffer->GetSizeX(), BackBuffer->GetSizeY());
  TArray<FColor> Data;

  RHICmdList.ReadSurfaceData(BackBuffer, Rect, Data, FReadSurfaceDataFlags());

  if (UserExternalVideoFrame == nullptr)
  {
    UserExternalVideoFrame = new agora::media::base::ExternalVideoFrame();
  }

  // Configure the video frame
  UserExternalVideoFrame->type = agora::media::base::ExternalVideoFrame::VIDEO_BUFFER_TYPE::VIDEO_BUFFER_RAW_DATA;
  UserExternalVideoFrame->format = agora::media::base::VIDEO_PIXEL_FORMAT::VIDEO_PIXEL_BGRA;
  UserExternalVideoFrame->stride = BackBuffer->GetSizeX();
  UserExternalVideoFrame->height = BackBuffer->GetSizeY();
  UserExternalVideoFrame->cropLeft = 10;
  UserExternalVideoFrame->cropTop = 10;
  UserExternalVideoFrame->cropRight = 10;
  UserExternalVideoFrame->cropBottom = 10;
  UserExternalVideoFrame->rotation = 0;
  UserExternalVideoFrame->timestamp = getTimeStamp();

  if (UserExternalVideoFrame->buffer == nullptr)
  {
    UserExternalVideoFrame->buffer = (uint8*)FMemory::Malloc(BackBuffer->GetSizeX() * BackBuffer->GetSizeY() * 4);
  }

  if (Data.Num() > 4)
  {
    FMemory::Memcpy(UserExternalVideoFrame->buffer, Data.GetData(), BackBuffer->GetSizeX() * BackBuffer->GetSizeY() * 4);
    if (MediaEngineManager != nullptr)
    {
      // Push the video frame to the SDK
      MediaEngineManager->pushVideoFrame(UserExternalVideoFrame);
    }
  }
}

std::time_t getTimeStamp()
{
  std::chrono::time_point<std::chrono::system_clock, std::chrono::milliseconds> tp =
    std::chrono::time_point_cast<std::chrono::milliseconds>(std::chrono::system_clock::now());
  std::time_t timestamp = tp.time_since_epoch().count();
  return timestamp;
}
```

<CalloutContainer type="info">
  <CalloutDescription>
    The sample code demonstrates converting Unreal's BGRA format to raw video data. Agora video capture supports pushing external video frames in other formats; refer to `VIDEO_PIXEL_FORMAT`.
  </CalloutDescription>
</CalloutContainer>

5. **Cleanup**

When finished unregister the callback and clean up resources.

```cpp
void NativeDestruct()
{
  Super::NativeDestruct();

  // Remove the back buffer callback
  FSlateApplication::Get().GetRenderer()->OnBackBufferReadyToPresent().Remove(eventId);

  UnInitAgoraEngine();
}

void UnInitAgoraEngine()
{
  if (AgoraUERtcEngine::Get() != nullptr)
  {
    AgoraUERtcEngine::Get()->leaveChannel();
    AgoraUERtcEngine::Get()->unregisterEventHandler(UserRtcEventHandler.Get());
    AgoraUERtcEngine::Release();
    MediaEngineManager = nullptr;
  }
}
```

### Custom video rendering

To implement custom video rendering in your game, refer to the following steps:

1. Set up `onCaptureVideoFrame` or `onRenderVideoFrame` callback to obtain the video data to be played.
2. Implement video rendering and playback yourself.

## Reference

This section contains content that completes the information on this page, or points you to documentation that explains other aspects to this product.

### Applicable use-cases

Use custom video capture in the following industries and use-cases:

**Specialized video processing and enhancement**

In specific gaming or virtual reality use-cases, real-time effects processing, filter handling, or other enhancement effects necessitate direct access to the original video stream. Custom video capture facilitates this, enabling seamless real-time processing and enhances the overall gaming or virtual reality experience for a more realistic outcome.

**High-precision video capture**

In video surveillance applications, detailed observation and analysis of scene details is necessary. Custom video capture enables higher image quality and finer control over capture to meet the requirements of video monitoring.

**Capture from specific video sources**

Industries such as IoT and live streaming often require the use of specific cameras, monitoring devices, or non-camera video sources, such as video capture cards or screen recording data. In such situations, default RTC SDK capture may not meet your requirements, necessitating use of custom video capture.

**Seamless integration with specific devices or third-party applications**

In smart home or IoT applications, transmitting video from devices to users' smartphones or computers for monitoring and control may require the use of specific devices or applications for video capture. Custom video capture facilitates seamless integration of specific devices or applications with the RTC SDK.

**Specific video encoding formats**

In certain live streaming use-cases, specific video encoding formats may be needed to meet business requirements. In such cases, RTC SDK default capture might not suffice, and custom video capture is required to capture and encode videos in specific formats.

### Advantages

Using custom video capture offers the following advantages:

**More types of video streams**

Custom video capture allows the use of higher quality and a greater variety of capture devices and cameras, resulting in clearer and smoother video streams. This enhances the user viewing experience and makes the product more competitive.

**More flexible video effects**

Custom video capture enables you to implement richer and more personalized video effects and filters, enhancing the user experience. You can implement effects such as beautification filters and dynamic stickers.

**Adaptation to diverse use-case requirements**

Custom video capture helps applications better adapt to the requirements of various use-cases, such as live streaming, video conferencing, and online education. You can customize different video capture solutions based on the use-case requirements to provide a more robust application.

### Sample projects

Agora provides a open-source sample project for your reference. Download [CustomCaptureVideo](https://github.com/AgoraIO-Extensions/Agora-Unreal-RTC-SDK/tree/main/Agora-Unreal-SDK-CPP-Example/Source/AgoraExample/Examples/Advanced/CustomCaptureVideo) or view the source code for a more detailed example.

### API reference

* [`setExternalVideoSource`](https://api-ref.agora.io/en/video-sdk/unreal-engine/4.x/API/class_imediaengine.html#api_imediaengine_setexternalvideosource)
* [`pushVideoFrame`](https://api-ref.agora.io/en/video-sdk/unreal-engine/4.x/API/class_imediaengine.html#api_imediaengine_pushvideoframe)