How to solve SEI-related issues?
The Agora SEI
By default, Agora adds the encoding information of the current video to the transcoded H264/H265 SEI (Supplemental Enhancement Information) during Media Push. The encoding information is a JSON string. The following is the sample code:
The definition of the parameters:
Parameter | Description |
---|---|
canvas | The canvas information. It contains the following properties:w : The width of the canvas in pixels. The canvas width field is set in the client or server bypass streaming API.h : The height of the canvas in pixels. The canvas height field is set in the client or server bypass streaming API.bgnd : The background color (RGB) of the canvas, represented by a hexademical integer defined in RGB. The color field of the canvas set in the client or server bypass streaming API. |
regions | The layout information of the host. It corresponds to the transcodingUsers member in the LiveTranscoding class. It contains the following properties:suid : (Optional)The string user account of the host in this region. This parameter applies to scenarios where string user accounts are used to identify the host.uid : UID of the host in this region. It corresponds to the uid member in the TranscodingUser class.alpha : The transparency of the video frame of the host. The value range is [0.0, 1.0]. It corresponds to the alpha member in TranscodingUser .zorder : The layout position of the video frame of the host. The value range is [0, 100]. It corresponds to the zOrder member in TranscodingUser .volume : The volume (dB) of the host. The value range is [0, 100].x : The horizontal position of the video frame of the broadacaster from the top left corner of the Media Push. It corresponds to the x member in TranscodingUser .y : The vertical position of the video frame of the host from the top left corner of the Media Push. It corresponds to the y member in TranscodingUser .w : Width (pixel) of the video frame of the host. It corresponds to the width member in TranscodingUser .h : Height (pixel) of the video frame of the host. It corresponds to the height member in TranscodingUser . |
ver | The version of the SEI protocol. The current version is 20190611. |
ts | Timestamp (ms) of the current encoding information. |
app_data | Extra user-defined information. It corresponds to the transcodingExtraInfo member in the client or server bypass streaming API. |
The structure of SEI
-
H.264
A snippet of an H.264 SEI frame is as follows:
The transcoded H.264 SEI information is as below:
-
H.265
Here's a snippet of an H.265 SEI frame:
The transcoded H.265 SEI information is as follows:
In which:
06
: SEI frames in H.264 format.4E01
: SEI frames in H.265 format.64
: The SEI frame type defined by the user. Here we define it as 100.bd
: The length of the SEI frame. The following are some sample calculations rendered in decimal and hexadecimal:- If the length is 922, because 922 can be divided by 255 (
0xff
) three times and the remainder is 157 (0x9d
), thenbd
isffffff9d
. - If the length is 572, because 572 can be divided by 255 (
0xff
) two times and the remainder is 62 (0x3e
), thenbd
isffff3e
. - If the length is 234, because 234 divided by 255 (
0xff
) gives 0 and the remainder is 234 (0xea
), thenbd
isea
.
- If the length is 922, because 922 can be divided by 255 (
- Other digits: Content of the SEI frame.
FAQs
Q: Does this mean that if SEI is used here, the layout cannot be transmitted via signaling? Does the SDK transmit the same field, and can the transmission method (signaling or SEI) only be one or the other?
A: The information added to the SEI frame of H264/H265 is done when the server pushes the stream. This is different from the data sent by the app via uplink. The only field related to the data sent by the app uplink is the app_data
field.
In the new live broadcast system, the only valid method is through the LiveTranscoding
configuration. The related interfaces from the old live broadcast system are no longer effective.
Q: Isn't layout information transmitted through signaling? Why do we need to include layout information here and not just volume information?
A: Agora has always sent SEI-related information in the combined image streaming, but it was never standardized. This update standardizes the SEI format and adds volume information for forward compatibility.