Skip to main content
Conversational AI Engine: Create advanced voice apps by combining Agora's real-time audio with powerful conversational AI.

Optimize video experience in multi-host scenarios

In multi-host streaming scenarios where 4 or more hosts stream videos simultaneously, users with the audience role face the following challenges:

  • Device performance and bandwidth pressure: Subscribing to multiple video streams can put significant pressure on device performance and downstream bandwidth, especially for mid-to-low-end devices. Audience members might experience lag or frame drops, leading to a suboptimal viewing experience.

  • Flexible layout needs: A custom video layout is often required, allowing viewers to adjust the position of each host's video to suit their individual preferences.

  • Window switching and zooming: With multiple hosts, the video size of each host may be limited. Flexible switching, and resizing allow for a more personalized "thousand people, thousand faces" viewing experience.

Agora offers a video experience optimization solution for multi-host video scenarios through a client-customized composite layout, designed to enhance user experience by focusing on smooth, personalized viewing.

This solution is ideal for scenarios such as:

  • Team battle PK
  • Multi-person conference
  • Online education for large classes

Understand the tech​

The following figure shows how the client-customized composite layout is used in conjunction with the Cloud Transcoding service:

MultihostVideoStreamingOptimization

After the audience receives the transcoded composite video stream, the custom composite layout is implemented locally as follows:

MultihostVideoStreamingOptimization

Prerequisites​

Make sure you have:

  • Implemented basic real-time audio and video functionality with SDK version 4.5 or above. See the SDK quickstart guide.

  • Implemented Cloud Transcoding to forward composite video to other channels. Make sure you set the following parameters when you configure transcoding:


    _14
    "outputs": [
    _14
    {
    _14
    "seiOption":{
    _14
    "source":{
    _14
    "agora": {
    _14
    "uidvideoLayout": true
    _14
    }
    _14
    },
    _14
    "sink":{
    _14
    "type": 100
    _14
    }
    _14
    }
    _14
    }
    _14
    ]

Implementation​

This section explains how to implement a client-customized composite layout, with the API call sequence illustrated in the diagram.

Multihost workflow

To implement a client-customized composite layout for the audience, follow these steps:

  1. Call joinChannel [2/2] to join the channel where the composite stream is located.

  2. Obtain the following information about the composite video stream through the onTranscodedStreamLayoutInfo callback triggered by the SDK:

    • The width and height of the composite stream, and the user ID of the user publishing it.
    • The user ID of each host publishing a sub-stream within the composite stream.
    • The x and y coordinates of each sub-stream on the composite canvas.
    • The width and height of each sub-stream.
    • The rendering state of each sub-stream on the composite canvas:
      • Normal rendering
      • Displayed as a placeholder
      • Displayed as a black image

  3. Call setupRemoteVideo to configure local rendering of the remote video stream. Set the following key parameters:

    • uid: The user ID of the user forwarding the composite stream to the current channel, obtained from the onTranscodedStreamLayoutInfo callback.
    • view: The remote video view displayed locally. Use this parameter to set the display position of each sub-stream in the local view.
    • subviewUid: The user ID corresponding to a specific sub-stream within the composite stream.
    Information
    • When subviewUid is not 0, the default value of setupMode is VIEW_SETUP_MODE_REPLACE (add a view).
    • If subviewUid is set, the setting of rect does not take effect.

    Use the following sample code as a reference for calling setupRemoteVideoEx in a multi-channel scenario. Calling setupRemoteVideo in a single-channel scenario is similar.


    _46
    mRtcEngine.joinChannelEx(null, connection, mediaOptions, new IRtcEngineEventHandler() {
    _46
    @Override
    _46
    public void onTranscodedStreamLayoutInfo(final int uid, final VideoLayoutInfo info) {
    _46
    super.onTranscodedStreamLayoutInfo(uid, info);
    _46
    runOnUiThread(new Runnable() {
    _46
    @Override
    _46
    public void run() {
    _46
    int size = info.layoutNumber;
    _46
    viewGrids.setChildCount(info.layoutNumber);
    _46
    for (int i = 0; i < size; i++) {
    _46
    if (mUserViews.containsKey(info.layoutList[i].uid)) {
    _46
    continue;
    _46
    }
    _46
    _46
    // Dynamically add the view
    _46
    TextureView textureView = new TextureView(VideoChatViewActivity.this);
    _46
    textureView.setLayoutParams(new ViewGroup.LayoutParams(
    _46
    ViewGroup.LayoutParams.MATCH_PARENT,
    _46
    ViewGroup.LayoutParams.MATCH_PARENT));
    _46
    users.add(info.layoutList[i].uid);
    _46
    views.add(textureView);
    _46
    mUserViews.put(info.layoutList[i].uid, textureView);
    _46
    viewGrids.addView(textureView);
    _46
    _46
    // Configure VideoCanvas; set uid and subviewUid as shown,
    _46
    // and configure other parameters as needed
    _46
    VideoCanvas canvas = new VideoCanvas(textureView);
    _46
    // User ID of the user forwarding the composite stream to the current channel
    _46
    canvas.uid = uid;
    _46
    // User ID corresponding to a specific sub-stream
    _46
    canvas.subviewUid = info.layoutList[i].uid;
    _46
    canvas.renderMode = Constants.RENDER_MODE_FIT;
    _46
    mRtcEngine.setupRemoteVideoEx(canvas, connection);
    _46
    _46
    // To remove a specific sub-stream, use the following code
    _46
    // Do not include this in the same flow as the add code above.
    _46
    VideoCanvas canvasToRemove = new VideoCanvas(textureView); // Set textureView to the view you want to remove
    _46
    canvasToRemove.uid = uid;
    _46
    canvasToRemove.subviewUid = info.layoutList[i].uid;
    _46
    canvasToRemove.setupMode = VideoCanvas.VIEW_SETUP_MODE_REMOVE;
    _46
    mRtcEngine.setupRemoteVideoEx(canvasToRemove);
    _46
    }
    _46
    }
    _46
    });
    _46
    }
    _46
    });

  4. When the layout of the composite stream changes, the onTranscodedStreamLayoutInfo callback fires again. Each time it fires, call setupRemoteVideo with the updated information to adjust the local rendering layout.

    Information

    Layout changes can occur in the following situations:

    • The coordinates of a sub-stream on the composite canvas change.
    • The width or height of the composite stream or a sub-stream changes.
    • The state of a sub-stream changes. When the host publishing a sub-stream leaves the channel, the videoState parameter for that sub-stream changes from 0 (normal) to 1 (no video available). To stop rendering that host's video, call setupRemoteVideo with subviewUid set to that host's user ID and setupMode set to VIEW_SETUP_MODE_REMOVE.

  5. Listen for the onUserOffline callback. When the user forwarding the composite stream leaves the channel, stop rendering the composite stream on the receiving end.

Note
  • Client-customized composite layout is currently supported on mobile platforms only.
  • If a watermark is applied to the composite stream, the watermark renders within its designated region and may appear split across sub-stream boundaries.

Reference​

This section contains content that completes the information on this page, or points you to documentation that explains other aspects to this product.