Video mixing

Server Gateway supports video mixing, enabling you to combine video tracks from multiple remote or local users and output the video in a customized format. This feature can be used to add watermarks, create picture-in-picture layouts or produce split-screen and multi-view displays.

Implementation

Refer to the following steps to implement video mixing.

1. Create an IVideoMixerSource object.

   agora::agora_refptr<agora::rtc::IVideoMixerSource> videoMixer = factory->createVideoMixer();

2. Call createMixedVideoTrack to create a mixed video track. Use the mixed video track to register a video frame observer.

   agora::agora_refptr<agora::rtc::ILocalVideoTrack> mixVideoTrack =
     service->createMixedVideoTrack(videoMixer);
   if (!mixVideoTrack) {
       AG_LOG(ERROR, "Failed to create video track!");
       return -1;
   }
   agora::rtc::VideoEncoderConfiguration encoderConfig;
   encoderConfig.codecType = agora::rtc::VIDEO_CODEC_H264;
   encoderConfig.dimensions.width = 1920;
   encoderConfig.dimensions.height = 1080;
   encoderConfig.frameRate = 15;
   
   mixVideoTrack->setVideoEncoderConfiguration(encoderConfig);
   
   // Register video frame observer to receive video stream
   agora::agora_refptr<agora::rtc::IVideoSinkBase> yuvFrameObserver =
     new agora::RefCountedObject<YuvFrameObserver>(options.videoFile);
   
   mixVideoTrack->addRenderer(yuvFrameObserver.get());

3. Add the video track to IVideoMixerSource for video mixing.

   if (remote_video_track_ && video_mixer_ && enable_video_mix_) {
       video_mixer_->addVideoTrack(userId, remote_video_track_);
       remote_video_track_map_[std::string(userId)] = remote_video_track_;
   }

4. When the SDK decodes the first YUV frame, call setBackground to set the background and refresh the IVideoMixerSource object.

   remote_source_map_[std::string(userId)] = calculate_layout(width,height);
   for (auto it = remote_source_map_.begin(); it != remote_source_map_.end(); it++) {
       video_mixer_->setStreamLayout(it->first.c_str(), it->second);
   }
   video_mixer_->setBackground(1920, 1080, 15);
   video_mixer_->refresh();