Optimize video experience in multi-host scenarios
Updated
Video experience optimization when multiple hosts are streaming.
In multi-host streaming scenarios where 4 or more hosts stream videos simultaneously, users with the audience role face the following challenges:
-
Device performance and bandwidth pressure: Subscribing to multiple video streams can put significant pressure on device performance and downstream bandwidth, especially for mid-to-low-end devices. Audience members might experience lag or frame drops, leading to a suboptimal viewing experience.
-
Flexible layout needs: A custom video layout is often required, allowing viewers to adjust the position of each host's video to suit their individual preferences.
-
Window switching and zooming: With multiple hosts, the video size of each host may be limited. Flexible switching and resizing allow for a more personalized viewing experience.
Agora offers a video experience optimization solution for multi-host video scenarios through a client-customized composite layout, designed to enhance user experience by focusing on smooth, personalized viewing.
This solution is ideal for scenarios such as:
- Team battle PK
- Multi-person conference
- Online education for large classes
Understand the tech
The following figure shows how the client-customized composite layout is used in conjunction with the Cloud Transcoding service:
After the audience receives the transcoded composite video stream, the custom composite layout is implemented locally as follows:
Prerequisites
Make sure you have:
-
Implemented basic real-time audio and video functionality with SDK version 4.5 or later. See the SDK quickstart.
-
Implemented Cloud Transcoding to forward composite video to other channels. Make sure you set the following parameters when you configure transcoding:
"outputs": [ { "seiOption": { "source": { "agora": { "uidvideoLayout": true } }, "sink": { "type": 100 } } } ]
Implementation
This section explains how to implement a client-customized composite layout, with the API call sequence illustrated in the diagram.
To implement a client-customized composite layout for the audience, follow these steps:
-
Call
joinChannelExto join the channel where the composite stream is located. -
Register an
RtcEngineEventHandlerand listen for theonTranscodedStreamLayoutInfocallback triggered by the SDK. Use it to obtain the following information about the composite video stream:- The width and height of the composite stream, and the user ID of the user publishing it.
- The user ID of each host publishing a sub-stream within the composite stream.
- The x and y coordinates of each sub-stream on the composite canvas.
- The width and height of each sub-stream.
- The rendering state of each sub-stream on the composite canvas:
- Normal rendering
- Displayed as a placeholder
- Displayed as a black image
-
For each sub-stream in the layout list, create an
AgoraVideoViewwidget usingVideoViewController.remoteand configure theVideoCanvaswith the following key parameters:sourceType: Set toVideoSourceType.videoSourceTranscoded, indicating the video source is a transcoded composite stream.uid: The user ID of the user forwarding the composite stream to the current channel, obtained from theonTranscodedStreamLayoutInfocallback.subviewUid: The user ID corresponding to a specific sub-stream within the composite stream.renderMode: Set toRenderModeType.renderModeFit.
- When
subviewUidis not0, the default value ofsetupModeisVideoViewSetupMode.videoViewSetupReplace(add a view). - If
subviewUidis set, the setting ofcropAreadoes not take effect.
The following sample code demonstrates this in a multi-channel scenario. The usage in a single-channel scenario is similar.
// Register the event handler _engine.registerEventHandler( RtcEngineEventHandler( onTranscodedStreamLayoutInfo: (RtcConnection connection, int uid, int width, int height, int layoutCount, List<VideoLayout> layoutlist) { setState(() { _compositeUid = uid; _layoutList = layoutlist; }); }, ), ); // Join the channel containing the composite stream await _engine.joinChannelEx( token: token, connection: RtcConnection(channelId: compositeChannelId, localUid: localUid), options: const ChannelMediaOptions( autoSubscribeVideo: true, autoSubscribeAudio: true, clientRoleType: ClientRoleType.clientRoleAudience, ), ); // Build AgoraVideoView widgets for each sub-stream Widget _buildSubStreamViews() { if (_layoutList == null || _compositeUid == null) { return const SizedBox.shrink(); } return Stack( children: _layoutList!.map((layout) { return AgoraVideoView( controller: VideoViewController.remote( rtcEngine: _engine, canvas: VideoCanvas( // User ID of the user forwarding the composite stream uid: _compositeUid, // User ID corresponding to a specific sub-stream subviewUid: layout.uid, sourceType: VideoSourceType.videoSourceTranscoded, renderMode: RenderModeType.renderModeFit, ), connection: RtcConnection(channelId: compositeChannelId), ), ); }).toList(), ); } // To remove a specific sub-stream view, rebuild the widget tree // without that sub-stream's AgoraVideoView, or use setupMode: VideoCanvas( uid: _compositeUid, subviewUid: layout.uid, setupMode: VideoViewSetupMode.videoViewSetupRemove, ) -
When the layout of the composite stream changes, the
onTranscodedStreamLayoutInfocallback fires again. Each time it fires, update your widget state with the new layout list and rebuild theAgoraVideoViewwidgets to reflect the updated layout.Layout changes can occur in the following situations:
- The coordinates of a sub-stream on the composite canvas change.
- The width or height of the composite stream or a sub-stream changes.
- The state of a sub-stream changes. When the host publishing a sub-stream leaves the channel, the
videoStateparameter for that sub-stream changes from0(normal) to1(no video available). Remove the correspondingAgoraVideoViewfrom the widget tree to stop rendering that host's video.
-
Listen for the
onUserOfflinecallback. When the user forwarding the composite stream leaves the channel, clear the layout list and stop rendering the composite stream.
Note
- Client-customized composite layout is currently supported on mobile platforms only.
- If a watermark is applied to the composite stream, the watermark renders within its designated region and may appear split across sub-stream boundaries.
Reference
This section contains content that completes the information on this page, or points you to documentation that explains other aspects of this product.
