Release notes

This page provides the release notes for the following:

• Video SDK

• Notifications

• AI Noise Suppression

• Virtual Background

The list of known issues page is continuously updated as the systems evolve. Agora suggests you regularly upgrade to the latest version of the SDK, which includes new features, bug fixes and improvements.

• Unity SDK v4.4.0

  MacOS 15 Sequoia beta system issue

  When the app path (including the App name) contains non-English characters (such as Chinese characters), the compiled Metal shader library may not contain any methods, causing the app to crash. See this article for the workaround.

• Unity SDK v4.2.6

  AirPods Pro Bluetooth connection issue (iOS)

  AirPods Pro does not support A2DP protocol in communication audio mode, which may lead to connection failure in that mode.

• Unity SDK v4.2.3

  Android 14 screen sharing issue

  Due to changes in the screen sharing behavior of Android 14 system, using devices with this version for screen sharing may entail the following issues:

  • Switching between the landscape and portrait mode during screen sharing can interrupt the current screen sharing process and a window will pop up asking if you want to start recording the screen. Once confirmed, screen sharing can be started.

  • When integrating the SDK, setting the Android targetSdkVersion to 34 may cause the screen sharing feature to be unavailable or even cause the application to crash.

v4.5.1 was released on March 13, 2025.

1. AI conversation scenario

   This version adds the AUDIO_SCENARIO_AI_CLIENT audio scenario specifically designed for interacting with the conversational AI agent created by Conversational AI Engine. This scenario optimizes the audio transmission algorithm based on AI-generated voice characteristics, ensuring stable voice data transmission even in weak network conditions with up to 80% packet loss. It enhances conversation continuity and reliability while adapting to complex network conditions.

• Reduces the SDK initialization time on specific device models. (iOS)

This release fixes the following issues:

• Apps that integrated the Agora SDK and set the targetSdkVersion to 34 encountered crashes when attempting to enable screen sharing for the first time on an Android 14 system. (Android)
• When joining two or more channels simultaneously, and calling the TakeSnapshotEx [1/2] method to take screenshots of the local video streams in each channel consecutively, the screenshot of the first channel failed.
• When using the Pause method to pause playback, then calling Seek to move to a specified position, and finally calling Play to continue playback, the Media Player resumed from the position where it was paused, not the new specified position.
• When using the Media Player, the file path of the media resource returned by the GetPlaySrc did not change after calling the SwitchSrc method to switch to a new media resource.
• When using Bluetooth headphones on specific device models for audio and video interactions, adjusting the phone volume would occasionally change the media volume instead of the Bluetooth volume. (Android)
• During audio and video interactions, the local user occasionally experienced a black screen when watching the video streams of remote users. (Android)
• On specific device models, after calling SetCameraExposureFactor to set the exposure coefficient of the current camera at a specific angle of the device, the video screen occasionally became dark when the device was moved to another angle. (Android)
• When playing a CDN live stream, the video occasionally froze for an extended period after recovering from an interruption. (Android)
• When pushing video frames in i420 format to the channel, using CVPixelBuffer to handle these frames caused a crash. (iOS)
• Calling SetupLocalVideo to set up two views, then calling EnableFaceDetection to start face detection, no face information can be detected in the subsequently passed views. (iOS)
• In a screen sharing scenario, the receiving-end user saw a green line on the shared image. (iOS)
• In the interactive live streaming scenario, after joining a channel to watch live streams using string user id, the audience members occasionally saw that the audio was not synchronized with the video.
• Plugins sometimes did not work when using AI noise suppression and AI echo cancellation plugins at the same time.

v4.5.0 was released on December 10, 2024.

This version includes optimizations to some features, including changes to SDK behavior, API renaming and deletion. To ensure normal operation of the project, update the code in the app after upgrading to this release.

1. Member Parameter Type Changes

   This version has made the following modifications to some API members or parameters:

   API	Members/Parameters	Change
   StartScreenCaptureByWindowId	windowId	Changed from view_t to long
   ScreenCaptureConfiguration	displayId windowId	Changed from unit to long
   ScreenCaptureSourceInfo	sourceDisplayId sourceId	Changed from view_t to long

2. Changes in strong video noise suppression implementation

   This version adjusts the implementation of strong video noise suppression. VIDEO_DENOISER_LEVEL_STRENGTH is removed from VIDEO_DENOISER_LEVEL.

   Instead, after enabling video noise suppression by calling SetVideoDenoiserOptions, you can call the SetBeautyEffectOptions method to enable the beauty skin smoothing feature. Using both together will achieve better video noise suppression effects. For strong noise suppression, it is recommended to set the skin smoothing parameters as detailed in SetVideoDenoiserOptions.

   Additionally, due to this adjustment, to achieve the best low-light enhancement effect with a focus on image quality, you need to enable video noise suppression first and use specific settings as detailed in SetLowlightEnhanceOptions.

3. Changes in camera plug and unplug status (macOS, Windows)

   In previous versions, when the camera was unplugged and replugged, the OnVideoDeviceStateChanged callback would report the device status as MEDIA_DEVICE_STATE_ACTIVE(1) (device in use). Starting from this version, after the camera is replugged, the device status will change to MEDIA_DEVICE_STATE_IDLE(0) (device ready).

4. Changes in video encoding preferences

   To enhance the user’s video interaction experience, this version optimizes the default preferences for video encoding:

   • In the COMPRESSION_PREFERENCE enumeration class, a new PREFER_COMPRESSION_AUTO (-1) enumeration is added, replacing the original PREFER_QUALITY (1) as the default value. In this mode, the SDK will automatically choose between PREFER_LOW_LATENCY or PREFER_QUALITY based on your video scene settings to achieve the best user experience.
   • In the DEGRADATION_PREFERENCE enumeration class, a new MAINTAIN_AUTO (-1) enumeration is added, replacing the original MAINTAIN_QUALITY (1) as the default value. In this mode, the SDK will automatically choose between MAINTAIN_FRAMERATE, MAINTAIN_BALANCED, or MAINTAIN_RESOLUTION based on your video scene settings to achieve the optimal overall quality experience (QoE).

5. 16 KB memory page size

   Starting from Android 15, the system adds support for 16 KB memory page size, as detailed in Support 16 KB page sizes. To ensure the stability and performance of the app, starting from this version, the SDK supports 16 KB memory page size, ensuring seamless operation on devices with both 4 KB and 16 KB memory page sizes, enhancing compatibility and preventing crashes.

1. Live show scenario

   This version adds the APPLICATION_SCENARIO_LIVESHOW(3) (Live Show) enumeration to VIDEO_APPLICATION_SCENARIO_TYPE. You can call SetVideoScenario to set the video business scenario to show room. To meet the high requirements for first frame rendering time and image quality in this scenario, the SDK has optimized strategies to significantly improve the first frame rendering experience and image quality, while enhancing the image quality in weak network environments and on low-end devices.

2. Maximum frame rate for video rendering

   This version adds the SetLocalRenderTargetFps and SetRemoteRenderTargetFps methods, which support setting the maximum frame rate for video rendering locally and remotely. The actual frame rate for video rendering by the SDK will be as close to this value as possible.

   In use-cases where the frame rate requirement for video rendering is not high (e.g., screen sharing, online education) or when the remote end uses mid-to-low-end devices, you can use this set of methods to limit the video rendering frame rate, thereby reducing CPU consumption and improving system performance.

3. Filter effects

   This version introduces the SetFilterEffectOptions method. You can pass a cube map file (.cube) in the config parameter to achieve custom filter effects such as whitening, vivid, cool, black and white, etc. Additionally, the SDK provides a built-in built_in_whiten_filter.cube file for quickly achieving a whitening filter effect.

4. Local audio mixing

   This version introduces the local audio mixing feature. You can call the StartLocalAudioMixer method to mix the audio streams from the local microphone, media player, sound card, and remote audio streams into a single audio stream, which can then be published to the channel. When you no longer need audio mixing, you can call the StopLocalAudioMixer method to stop local audio mixing. During the mixing process, you can call the UpdateLocalAudioMixerConfiguration method to update the configuration of the audio streams being mixed.

   Example use cases for this feature include:

   • By utilizing the local video mixing feature, the associated audio streams of the mixed video streams can be simultaneously captured and published.
   • In live streaming use-cases, users can receive audio streams within the channel, mix multiple audio streams locally, and then forward the mixed audio stream to other channels.
   • In educational use-cases, teachers can mix the audio from interactions with students locally and then forward the mixed audio stream to other channels.

5. External MediaProjection (Android)

   This version introduces the SetExternalMediaProjection method, which allows you to set an external MediaProjection and replace the MediaProjection applied by the SDK.

   If you have the capability to apply for MediaProjection on your own, you can use this feature to achieve more flexible screen capture.

6. EGL context (Android)

   This version introduces the SetExternalRemoteEglContext method, which is used to set the EGL context for rendering remote video streams. When using Texture format video data for remote video self-rendering, you can use this method to replace the SDK's default remote EGL context, achieving unified EGL context management.

7. Color space settings

   This version adds the colorSpace parameter to VideoFrame and ExternalVideoFrame. You can use this parameter to set the color space properties of the video frame. By default, the color space uses Full Range and BT.709 standard configuration. You can flexibly adjust according to your own capture or rendering needs, further enhancing the customization capabilities of video processing.

8. Other features

   • OnLocalVideoStateChanged callback adds the LOCAL_VIDEO_STREAM_REASON_DEVICE_DISCONNECTED enumeration, indicating that the currently used video capture device has been disconnected (e.g., unplugged). (Windows)
   • MEDIA_DEVICE_STATE_TYPE adds the MEDIA_DEVICE_STATE_PLUGGED_IN enumeration, indicating that the device has been plugged in. (Windows)

1. Virtual background algorithm optimization

   This version upgrades the virtual background algorithm, making the segmentation between the portrait and the background more accurate. There is no background exposure, the body contour of the portrait is complete, and the detail recognition of fingers is significantly improved. Additionally, the edges between the portrait and the background are more stable, reducing edge jumping and flickering in continuous video frames.

2. Snapshot at specified video observation points

   This version introduces the TakeSnapshot [2/2] and TakeSnapshotEx [2/2] methods. You can use the config parameter when calling these methods to take snapshots at specified video observation points, such as before encoding, after encoding, or before rendering, to achieve more flexible snapshot effects.

3. Custom audio capture improvements

   This version adds the enableAudioProcessing member parameter to AudioTrackConfig, which is used to control whether to enable 3A audio processing for custom audio capture tracks of the AUDIO_TRACK_DIRECT type. The default value of this parameter is false, meaning that audio processing is not enabled. Users can enable it as needed, enhancing the flexibility of custom audio processing.

4. Other improvements

   • In use-cases where Alpha transparency effects are achieved by stitching video frames and Alpha data, the rendering performance on the receiving end has been improved, effectively reducing stuttering and latency. (Android, iOS)
   • Optimizes the logic for calling QueryDeviceScore to obtain device score levels, improving the accuracy of the score results.
   • Supports using virtual cameras in YV12 format as video capture devices. (Windows)
   • When calling SwitchSrc to switch between live streams or on-demand streams of different resolutions, smooth and seamless switching can be achieved. An automatic retry mechanism has been added in case of switching failures. The SDK will automatically retry 3 times after a failure. If it still fails, the OnPlayerEvent callback will report the PLAYER_EVENT_SWITCH_ERROR event, indicating an error occurred during media resource switching.
   • When calling SetPlaybackSpeed to set the playback speed of an audio file, the minimum supported speed is 0.3x.

This version fixes the following issues:

• When calling StartScreenCaptureByWindowId to share the screen, the window capture area specified by regionRect was inaccurate, resulting in incorrect width and height of the screen sharing window seen by the receiving end. (Windows)
• When the video source type of the sender is in JPEG format, the frame rate on the receiving end occasionally falls below expectations. (Android, iOS)
• During audio and video interaction, after being interrupted by a system call, the user volume reported by the OnAudioVolumeIndication callback was incorrect. (Android)
• When the receiving end subscribes to the low-quality video stream by default and does not automatically subscribe to any video stream when joining the channel, calling MuteRemoteVideoStream (uid, false) after joining the channel to resume receiving the video stream results in receiving the high-quality video stream. (Android)
• Occasional errors of not finding system files during audio and video interaction on Windows 7 systems. (Windows)
• When calling FollowSystemRecordingDevice or FollowSystemPlaybackDevice to set the audio capture or playback device used by the SDK to not follow the system default audio playback device, the local audio state callback OnLocalAudioStateChanged is not triggered when the audio device is removed. (Windows)
• Occasional instances where the receiving end cannot hear the sender during audio and video interaction. (iOS)
• During audio and video interaction, if the sender's device system version is iOS 17, the receiving end occasionally cannot hear the sender. (iOS)
• In live streaming use-cases, the time taken to reconnect to the live room after the audience end disconnects due to network switching is longer than expected. (iOS)
• No sound when playing online media resources using the media player after the app starts. (iOS)
• Occasional instances of no sound in audio capture after resuming from being interrupted by other system apps during audio and video interaction. (iOS)
• Calling StartAudioMixing [1/2] and then immediately calling PauseAudioMixing to pause the music file playback does not take effect.
• Occasional crashes during audio and video interaction. (Android)

v4.4.0 was released on August 29, 2024.

This version includes optimizations to some features, including changes to SDK behavior and API renaming and deletion. To ensure normal operation of the project, update the code in the app after upgrading to this release.

1. To distinguish context information in different extension callbacks, this version removes the original extension callbacks and adds corresponding callbacks that contain context information (see the table below). You can identify the extension name, the user ID, and the service provider name through ExtensionContext in each callback.

   Original callback	New callback
   onExtensionEvent	OnExtensionEventWithContext
   onExtensionStarted	OnExtensionStartedWithContext
   onExtensionStopped	OnExtensionStoppedWithContext
   onExtensionError	OnExtensionErrorWithContext

2. This version renames the following members in ExternalVideoFrame:

• d3d11_texture_2d is renamed to d3d11Texture2d.
• texture_slice_index is renamed to textureSliceIndex.
• metadata_buffer is renamed to metadataBuffer.
• metadata_size is renamed to metadataSize.

1. Alpha transparency effects

   This version introduces the Alpha transparency effects feature. It supports the transmission and rendering of Alpha channel data in video frames for SDK capture and custom capture use-cases, enabling transparent gift effects, custom backgrounds on the receiver end, and so on:

   • VideoFrame and ExternalVideoFrame add the alphaBuffer member, which sets the Alpha channel data.
   • ExternalVideoFrame adds the fillAlphaBuffer member, which sets whether to automatically extract the Alpha channel data and fill it into alphaBuffer for BGRA or RGBA formatted video data.
   • VideoFrame and ExternalVideoFrame add the alphaStitchMode member, which sets the relative position of alphaBuffer and video frame stitching.

   Additionally, AdvanceOptions adds a new member encodeAlpha, which is used to set whether to encode and send Alpha information to the remote end. By default, the SDK does not encode and send Alpha information; if you need to encode and send Alpha information to the remote end (for example, when virtual background is enabled), explicitly call SetVideoEncoderConfiguration to set the video encoding properties and set encodeAlpha to true.

2. Voice AI tuner

   This version introduces the voice AI tuner feature, which can enhance the sound quality and tone, similar to a physical sound card. You can enable the voice AI tuner feature by calling the EnableVoiceAITuner method. Pass in the sound effect types supported in the VOICE_AI_TUNER_TYPE enum to achieve effects like deep voice, cute voice, husky singing voice, and so on.

3. 1v1 video call scenario

   This version adds APPLICATION_SCENARIO_1V1 (1v1 video call) in VIDEO_APPLICATION_SCENARIO_TYPE. You can call SetVideoScenario to set the video application scenario to 1v1 video call. The SDK optimizes performance to achieve low latency and high video quality, enhancing image quality, first frame rendering, latency on mid-to-low-end devices, and smoothness under poor network conditions.

1. Adaptive hardware decoding support (Android, Windows)

   This release introduces adaptive hardware decoding support, enhancing rendering smoothness on low-end devices and effectively reducing system load.

2. Rendering performance enhancement (Windows)

   DirectX 11 renderer is now enabled by default on Windows devices, providing high-performance and high-quality graphics rendering capabilities.

3. Facial region beautification

   To avoid losing details in non-facial areas during heavy skin smoothing, this version improves the skin smoothing algorithm. The SDK now recognizes various parts of the face, applying smoothing to facial skin areas excluding the mouth, eyes, and eyebrows. In addition, the SDK supports smoothing up to two faces simultaneously.

4. Other improvements

   This version also includes the following improvements:

   • Optimizes the transmission strategy: Calling EnableInstantMediaRendering no longer impacts the security of the transmission link.
   • The LOCAL_VIDEO_STREAM_REASON_SCREEN_CAPTURE_DISPLAY_DISCONNECTED enumerator is added in OnLocalVideoStateChanged callback, indicating that the display used for screen capture has been disconnected (Windows, macOS).
   • Optimizes the video link for window sharing, reducing CPU usage (macOS).
   • Improves echo cancellation for screen sharing use-cases.
   • Adds the channelId parameter to Metadata, which is used to get the channel name from which the metadata is sent.
   • Deprecates redundant enumeration values CLIENT_ROLE_CHANGE_FAILED_REQUEST_TIME_OUT and CLIENT_ROLE_CHANGE_FAILED_CONNECTION_FAILED in CLIENT_ROLE_CHANGE_FAILED_REASON.

This release fixed the following issues:

• Occasional app crashes occurred when multiple remote users joined the channel simultaneously during real-time interaction (iOS).
• Remote video occasionally froze or displayed corrupted images when the app returned to the foreground after being in the background for a while (iOS).
• After the sender called StartDirectCdnStreaming to start direct CDN streaming, frequent switching or toggling of the network occasionally resulted in a black screen on the receiver's end without a streaming failure callback on the sender's end (iOS).
• Audio playback failed when pushing external audio data using PushAudioFrame and the sample rate was not set to a recommended value, such as 22050 Hz and 11025 Hz.

v4.3.2 was released on June 6, 2024.

This release enhances the usability of the setRemoteSubscribeFallbackOption method by removing the timing requirements for invocation. It can now be called both before and after joining the channel to dynamically switch audio and video stream fallback options in weak network conditions.

This version fixed the following issues:

• The remote video froze or became pixelated when the app returned to the foreground after being in the background for a long time (iOS).
• The local preview image rotated by 90° on some iPad devices, which did not meet expectations (iOS).
• Occasional video smoothness issues during audio and video interactions.
• The app occasionally crashed when the decoded video resolution on the receiving end was an odd number.
• The app crashed when opening the app and starting screen sharing after the first installation or system reboot (Android).
• Local audio capture failed after joining a channel while answering a system phone call and hanging up, causing remote users to not hear any sound (Android).
• During the interaction process on certain devices (for example, Redmi Note8), after answering and hanging up a system call, local media files were played without sound and no sound was heard from the remote end (Android).
• The app occasionally crashed when remote users left the channel.
• The screen occasionally flickered on the receiver's side when sharing a PPT window using startScreenCaptureByWindowId and playing PPT animations (Windows).
• The window border did not retain its original size after exiting the presentation and then maximizing the PPT window when sharing a WPS PPT window on Windows 7 using startScreenCaptureByWindowId and setting enableHighLight in ScreenCaptureParameters to true (Windows).
• The specified window could not be brought to the foreground if it was covered by other windows when sharing a window using startScreenCaptureByWindowId and setting windowFocus and enableHighLight in ScreenCaptureParameters to true (Windows).
• Clicking on the desktop widget caused the outlined part to flicker when sharing and highlighting a window on a Windows 7 device (Windows).
• When playing an audio file using StartAudioMixing [1/2] and the playing finished, the SDK sometimes failed to trigger the OnAudioMixingStateChanged (AUDIO_MIXING_STATE_STOPPED, AUDIO_MIXING_REASON_ALL_LOOPS_COMPLETED) callback that reports that the playing is completed (iOS).
• When calling the PlayEffect method to play sound effect files shorter than 1 second with loopCount set to 0, there was no sound (iOS).
• When using the Agora media player to play a video and stopping it during playing, sometimes there was no sound for a short time after the playback was resumed (iOS).

v4.3.1 was released on May 9, 2024.

1. Privacy manifest file (iOS)

   To meet Apple's safety compliance requirements for app publication, the SDK now includes a privacy manifest file, PrivacyInfo.xcprivacy, detailing the SDK's API calls that access or use user data, along with a description of the types of data collected.

   Note

   If you need to publish an app with SDK versions prior to v4.3.1 to the Apple App Store, manually add the PrivacyInfo.xcprivacy file to your Xcode project.

2. Portrait center stage (iOS, macOS)

   To enhance the presentation effect in online meetings, shows, and online education use-cases, this version introduces the EnableCameraCenterStage method to activate the portrait center stage feature. This ensures that presenters, regardless of movement, always remain centered in the video frame, achieving better presentation effects.

   Before enabling portrait center stage it is recommended to verify whether your current device supports this feature by calling IsCameraCenterStageSupported. A list of supported devices can be found in the API documentation at EnableCameraCenterStage.

3. Camera stabilization (iOS)

   To improve video stability in mobile filming, low-light environments, and hand-held shooting use-cases, this version introduces a camera stabilization feature. You can activate this feature and select an appropriate stabilization mode by calling SetCameraStabilizationMode, achieving more stable and clearer video footage.

4. Wide and ultra-wide cameras (Android, iOS)

   To allow users to capture a broader field of view and more complete scene content, this release introduces support for wide and ultra-wide cameras. You can first call QueryCameraFocalLengthCapability to check the device's focal length capabilities, and then call SetCameraCapturerConfiguration and set cameraFocalLengthType to the supported focal length types, including wide and ultra-wide.

5. Multi-camera capture (Android)

   This release introduces additional functionalities for Android camera capture:

   1. Support for capturing and publishing video streams from the third and fourth cameras:

      • VIDEO_SOURCE_CAMERA_THIRD (11) and VIDEO_SOURCE_CAMERA_FOURTH (12) in VIDEO_SOURCE_TYPE add support for Android, specifically for the third and fourth camera sources. This change allows you to specify up to four camera streams when initiating camera capture by calling StartCameraCapture.
      • publishThirdCameraTrack and publishFourthCameraTrack in ChannelMediaOptions add support for Android. Set these parameters to true when joining a channel with JoinChannel [2/2] to publish video streams captured from the third and fourth cameras.

   2. Support for specifying cameras by camera ID:

      A new parameter cameraId is added to CameraCapturerConfiguration. For devices with multiple cameras, where cameraDirection cannot identify or access all available cameras, you can obtain the camera ID through Android's native system APIs and specify the desired camera by calling with the specific cameraId.

6. Data stream encryption

   This version adds datastreamEncryptionEnabled to EncryptionConfig for enabling data stream encryption. You can set this when you activate encryption with EnableEncryption. If there are issues causing failures in data stream encryption or decryption, these can be identified by the newly added ENCRYPTION_ERROR_DATASTREAM_DECRYPTION_FAILURE and ENCRYPTION_ERROR_DATASTREAM_ENCRYPTION_FAILURE enumerations.

7. Adaptive configuration for low-quality video streams

   This version introduces adaptive configuration for low-quality video streams. When you activate the dual-stream mode and set up low-quality video streams on the sending side using SetDualStreamMode [2/2], the SDK defaults to the following behaviors:

   • The default encoding resolution for low-quality video streams is set to 50% of the original video encoding resolution.
   • The bitrate for the low-quality streams is automatically matched based on the video resolution and frame rate, eliminating the need for manual specification.

8. Other features

   • A new method EnableEncryptionEx is added for enabling media stream or data stream encryption in multi-channel use-cases.
   • A new method SetAudioMixingPlaybackSpeed is introduced for setting the playback speed of audio files.
   • A new method GetCallIdEx is introduced for retrieving call IDs in multi-channel use-cases.

9. Beta features

   • Speech driven avatar is released in beta. See beta documentation for details.

1. Optimization for game use-case screen sharing (Windows)

   This version specifically optimizes screen sharing for game use-cases, enhancing performance, stability, and clarity in ultra-high definition (4K, 60 fps) game use-cases, resulting in a clearer, smoother, and more stable gaming experience for players.

2. Audio device type detection (macOS)

   This version adds the following methods to obtain the information and type of audio playback and recording devices:

   • GetPlaybackDefaultDevice [2/2]
   • GetRecordingDefaultDevice [2/2]
   • GetPlaybackDeviceInfo [2/2]
   • GetRecordingDeviceInfo [2/2]

3. Virtual background algorithm optimization

   To enhance the accuracy and stability of human segmentation when activating virtual backgrounds against solid colors, this version optimizes the green screen segmentation algorithm:

   • Supports recognition of any solid color background, no longer limited to green screens.
   • Improves accuracy in recognizing background colors and reduces the background exposure during human segmentation.
   • After segmentation, the edges of the human figure (especially around the fingers) are more stable, significantly reducing flickering at the edges.

4. CPU consumption reduction of in-ear monitoring

   This release adds an enumerator EAR_MONITORING_FILTER_REUSE_POST_PROCESSING_FILTER in EAR_MONITORING_FILTER_TYPE. For complex audio processing use-cases, you can specify this option to reuse the audio filter post sender-side processing in in-ear monitoring, thereby reducing CPU consumption. Note that this option may increase the latency of in-ear monitoring, which is suitable for latency-tolerant use-cases requiring low CPU consumption.

5. Other improvements

   This version also includes the following improvements:

   • Optimization of video encoding and decoding strategies in non-screen sharing use-cases to save system performance overhead (Windows).
   • Enhanced media player capabilities to handle the WebM format videos, including support for rendering Alpha channels.
   • In AUDIO_EFFECT_PRESET, a new enumeration ROOM_ACOUSTICS_CHORUS (chorus effect) is added, enhancing the spatial presence of vocals in chorus use-cases.
   • In RemoteAudioStats, a new e2eDelay field is added to report the delay from when the audio is captured on the sending end to when the audio is played on the receiving end.

This version fixed the following issues:

• Fixed an issue where SEI data output did not synchronize with video rendering when playing media streams containing SEI data using the media player.
• In screen sharing use-cases, when the app enabled sound card capture with EnableLoopbackRecording to capture audio from the shared screen, the transmission of sound card captured audio failed after a local user manually disabled the local audio capture device, causing remote users to not hear the shared screen's audio (Windows).
• Hosts using certain models of devices in the speaker mode experienced occasional local audio capture failures when switching the app process to the background and then back to the foreground, preventing remote users from hearing the host's audio (Android).
• An occasional echo was observed when playing the audio stream of a specified user before mixing (macOS, Windows).
• During interactions, when a local user set the system default playback device to speakers, there was no sound from the remote end (Windows).
• On devices with Android 8.0, enabling screen sharing occasionally caused the app to crash (Android).
• When sharing an Excel document window, remote users occasionally saw a green screen (Windows).
• In use-cases using camera capture for local video, when the app was moved to the background and DisableVideo or StopPreview was called to stop video capture, camera capture was unexpectedly activated when the app was brought back to the foreground (Android).
• When the network conditions of the sender deteriorated (for example, in poor network environments), the receiver occasionally experienced a decrease in video smoothness and an increase in lag.

Added

• EnableCameraCenterStage (iOS, macOS)
• IsCameraCenterStageSupported (iOS, macOS)
• SetCameraStabilizationMode (iOS)
• CAMERA_STABILIZATION_MODE (iOS)
• RegisterFaceInfoObserver
• UnregisterFaceInfoObserver
• IFaceInfoObserver
• OnFaceInfo
• The publishLipSyncTrack member in ChannelMediaOptions
• MEDIA_SOURCE_TYPE adds SPEECH_DRIVEN_VIDEO_SOURCE
• VIDEO_SOURCE_TYPE adds VIDEO_SOURCE_SPEECH_DRIVEN
• EncryptionConfig adds datastreamEncryptionEnabled
• ENCRYPTION_ERROR_TYPE adds the following enumerations:
  • ENCRYPTION_ERROR_DATASTREAM_DECRYPTION_FAILURE
  • ENCRYPTION_ERROR_DATASTREAM_ENCRYPTION_FAILURE
• GetPlaybackDefaultDevice [2/2] (macOS),
• GetRecordingDefaultDevice [2/2] (macOS),
• GetPlaybackDeviceInfo [2/2] (macOS)
• GetRecordingDeviceInfo [2/2] (macOS)
• RemoteAudioStats adds e2eDelay
• ERROR_CODE_TYPE adds ERR_DATASTREAM_DECRYPTION_FAILED
• AUDIO_EFFECT_PRESET adds ROOM_ACOUSTICS_CHORUS, enhancing the spatial presence of vocals in chorus use-cases.
• GetCallIdEx
• EnableEncryptionEx
• SetAudioMixingPlaybackSpeed
• QueryCameraFocalLengthCapability(Android, iOS)
• FocalLengthInfo (Android, iOS)
• CAMERA_FOCAL_LENGTH_TYPE (Android, iOS)
• CameraCapturerConfiguration adds a new member cameraFocalLengthType (Android, iOS)
• CameraCapturerConfiguration adds a new member cameraId (Android)
• EAR_MONITORING_FILTER_TYPE adds a new enumeration EAR_MONITORING_FILTER_BUILT_IN_AUDIO_FILTERS

v4.3.0 was released on February 28, 2024.

This release has optimized the implementation of some functions, involving renaming or deletion of some APIs. To ensure the normal operation of the project, you need to update the code in the app after upgrading to this release.

1. Renaming parameters in callbacks

   In order to make the parameters in some callbacks and the naming of enumerations in enumeration classes easier to understand, the following modifications have been made in this release. Please modify the parameter settings in the callbacks after upgrading to this release.

   Callback	Original parameter name	New parameter name
   OnLocalAudioStateChanged	error	reason
   onLocalVideoStateChanged	error	reason
   OnDirectCdnStreamingStateChanged	error	reason
   OnRtmpStreamingStateChanged	errCode	reason

   Original enumeration class	New enumeration class
   LOCAL_AUDIO_STREAM_ERROR	LOCAL_AUDIO_STREAM_REASON
   LOCAL_VIDEO_STREAM_ERROR	LOCAL_VIDEO_STREAM_REASON
   DIRECT_CDN_STREAMING_ERROR	DIRECT_CDN_STREAMING_REASON
   MEDIA_PLAYER_ERROR	MEDIA_PLAYER_REASON
   RTMP_STREAM_PUBLISH_ERROR	RTMP_STREAM_PUBLISH_REASON

   Note: For specific renaming of enumerations, please refer to API changes.

2. Channel media relay

   To improve interface usability, this release removes some methods and callbacks for channel media relay. Use the alternative options listed in the table below:

   Deleted methods and callbacks	Alternative methods and callbacks
   startChannelMediaRelay updateChannelMediaRelay	startOrUpdateChannelMediaRelay
   startChannelMediaRelayEx updateChannelMediaRelayEx	startOrUpdateChannelMediaRelayEx
   onChannelMediaRelayEvent	onChannelMediaRelayStateChanged

3. Reasons for local video state changes

   This release makes the following modifications to the enumerations in the LOCAL_VIDEO_STREAM_REASON class:

   • The value of LOCAL_VIDEO_STREAM_REASON_SCREEN_CAPTURE_PAUSED (formerly LOCAL_VIDEO_STREAM_ERROR_SCREEN_CAPTURE_PAUSED) has been changed from 23 to 28.
   • The value of LOCAL_VIDEO_STREAM_REASON_SCREEN_CAPTURE_RESUMED (formerly LOCAL_VIDEO_STREAM_ERROR_SCREEN_CAPTURE_RESUMED) has been changed from 24 to 29.
   • The LOCAL_VIDEO_STREAM_ERROR_CODEC_NOT_SUPPORT enumeration has been changed to LOCAL_VIDEO_STREAM_REASON_CODEC_NOT_SUPPORT.

4. Audio route

   Starting with this release, routeBluetooth in AudioRoute is renamed to ROUTE_BLUETOOTH_DEVICE_HFP, representing a Bluetooth device using the HFP protocol. ROUTE_BLUETOOTH_DEVICE_A2DP(10) is added to represent a Bluetooth device using the A2DP protocol

5. Audio loopback capturing (Windows, macOS)

   • Before v4.3.0, if you called the DisableAudio method to disable the audio module, audio loopback capturing would not be disabled.
   • As of v4.3.0, if you call the DisableAudio method to disable the audio module, audio loopback capturing will be disabled as well. If you need to enable audio loopback capturing, you need to enable the audio module by calling the EnableAudio method and then call EnableLoopbackRecording.

6. Log encryption behavior changes

   For security and performance reasons, as of this release, the SDK encrypts logs and no longer supports printing plaintext logs via the console.

   Refer to the following solutions for different needs:

   • If you need to know the API call status, please check the API logs and print the SDK callback logs yourself.
   • For any other special requirements, please contact technical support and provide the corresponding encrypted logs.

1. Local preview with multiple views

   This release supports local preview with simultaneous display of multiple frames, where the videos shown in the frames are positioned at different observation positions along the video link. Examples of usage are as follows:

   1. Call SetupLocalVideo to set the first view: Set the position parameter to POSITION_POST_CAPTURER_ORIGIN (introduced in this release) in VideoCanvas. This corresponds to the position after local video capture and before preprocessing. The video observed here does not have preprocessing effects.
   2. Call SetupLocalVideo to set the second view: Set the position parameter to POSITION_POST_CAPTURER in VideoCanvas, the video observed here has the effect of video preprocessing.
   3. Observe the local preview effect: The first view is the original video of a real person; the second view is the virtual portrait after video preprocessing (including image enhancement, virtual background, and local preview of watermarks) effects.

2. Custom mixed video layout on the receiving end (Android, iOS)

   To facilitate customized layout of mixed video stream at the receiver end, this release introduces the OnTranscodedStreamLayoutInfo callback. When the receiver receives the channel's mixed video stream sent by the video mixing server, this callback is triggered, reporting the layout information of the mixed video stream and the layout information of each sub-video stream in the mixed stream. The receiver can set a separate view for rendering the sub-video stream (distinguished by subviewUid) in the mixed video stream when calling the SetupRemoteVideo method, achieving a custom video layout effect.

   When the layout of the sub-video streams in the mixed video stream changes, this callback will also be triggered to report the latest layout information in real time.

   Through this feature, the receiver end can flexibly adjust the local view layout. When applied in a multi-person video use-case, the receiving end only needs to receive and decode a mixed video stream, which can effectively reduce the CPU usage and network bandwidth when decoding multiple video streams on the receiving end.

3. Query Device Score

   This release adds the QueryDeviceScore method to query the device's score level to ensure that the user-set parameters do not exceed the device's capabilities. For example, in HD or UHD video use-cases, you can first call this method to query the device's score. If the returned score is low (for example, below 60), you need to lower the video resolution to avoid affecting the video experience. The minimum device score required for different business use-cases is varied. For specific score recommendations, please contact technical support.

4. Select different audio tracks for local playback and streaming

   This release introduces the SelectMultiAudioTrack method that allows you to select different audio tracks for local playback and streaming to remote users. For example, in use-cases like online karaoke, the host can choose to play the original sound locally and publish the accompaniment in the channel. Before using this function, you need to open the media file through the OpenWithMediaSource method and enable this function by setting the enableMultiAudioTrack parameter in MediaSource.

5. Others

   This release has passed the test verification of the following APIs and can be applied to the entire series of RTC 4.x SDK.

   • SetRemoteSubscribeFallbackOption: Sets fallback option for the subscribed video stream in weak network conditions.
   • OnRemoteSubscribeFallbackToAudioOnly: Occurs when the subscribed video stream falls back to audio-only stream due to weak network conditions or switches back to the video stream after the network conditions improve.
   • GetRecordingDeviceVolume(Windows): Sets the volume of the audio capturing device.
   • SetPlayerOption and SetPlayerOption: Sets media player options for providing technical previews or special customization features.
   • EnableCustomAudioLocalPlayback: Sets whether to enable the local playback of external audio source.

1. SDK task processing scheduling optimization

   This release optimizes the scheduling mechanism for internal tasks within the SDK, with improvements in the following aspects:

   • The speed of video rendering and audio playback for both remote and local first frames improves by 10% to 20%.
   • The API call duration and response time are reduced by 5% to 50%.
   • The SDK's parallel processing capability significantly improves, delivering higher video quality (720P, 24 fps) even on lower-end devices. Additionally, image processing remains more stable in use-cases involving high resolutions and frame rates.
   • The stability of the SDK is further enhanced, leading to a noticeable decrease in the crash rate across various specific use-cases.

2. In-ear monitoring volume boost

   This release provides users with more flexible in-ear monitoring audio adjustment options, supporting the ability to set the in-ear monitoring volume to four times the original volume by calling SetInEarMonitoringVolume.

3. Spatial audio effects usability improvement

   • This release optimizes the design of the SetZones method, supporting the ability to set the zones parameter to NULL, indicating the clearing of all echo cancellation zones.
   • As of this release, it is no longer necessary to unsubscribe from the audio streams of all remote users within the channel before calling the methods in ILocalSpatialAudioEngine.

4. Other Improvements

   This release also includes the following improvements:

   • This release optimizes the SDK's domain name resolution strategy, improving the stability of calling to resolve domain names in complex network environments.
   • When passing in an image with transparent background as the virtual background image, the transparent background can be filled with customized color.
   • This release adds the earMonitorDelay and aecEstimatedDelay members in LocalAudioStats to report ear monitor delay and acoustic echo cancellation (AEC) delay, respectively.
   • The OnPlayerCacheStats callback is added to report the statistics of the media file being cached. This callback is triggered once per second after file caching is started.
   • The OnPlayerPlaybackStats callback is added to report the statistics of the media file being played. This callback is triggered once per second after the media file starts playing. You can obtain information like the audio and video bitrate of the media file through PlayerPlaybackStats.

This release fixed the following issues:

• When sharing two screen sharing video streams simultaneously, the reported captureFrameRate in the OnLocalVideoStats callback is 0, which is not as expected.

Added

• OnTranscodedStreamLayoutInfo
• VideoLayout
• The subviewUid member in VideoCanvas
• EnableCustomAudioLocalPlayback
• QueryDeviceScore
• The CUSTOM_VIDEO_SOURCE enumeration in MEDIA_SOURCE_TYPE
• The ROUTE_BLUETOOTH_DEVICE_A2DP enumeration in AudioRoute
• SelectMultiAudioTrack
• OnPlayerCacheStats
• OnPlayerPlaybackStats
• PlayerPlaybackStats

Modified

• All ERROR fields in the following enumerations are changed to REASON:

  • LOCAL_AUDIO_STREAM_ERROR_OK
  • LOCAL_AUDIO_STREAM_ERROR_FAILURE
  • LOCAL_AUDIO_STREAM_ERROR_DEVICE_NO_PERMISSION
  • LOCAL_AUDIO_STREAM_ERROR_DEVICE_BUSY
  • LOCAL_AUDIO_STREAM_ERROR_RECORD_FAILURE
  • LOCAL_AUDIO_STREAM_ERROR_ENCODE_FAILURE
  • LOCAL_AUDIO_STREAM_ERROR_RECORD_INVALID_ID
  • LOCAL_AUDIO_STREAM_ERROR_PLAYOUT_INVALID_ID
  • LOCAL_VIDEO_STREAM_ERROR_OK
  • LOCAL_VIDEO_STREAM_ERROR_FAILURE
  • LOCAL_VIDEO_STREAM_ERROR_DEVICE_NO_PERMISSION
  • LOCAL_VIDEO_STREAM_ERROR_DEVICE_BUSY
  • LOCAL_VIDEO_STREAM_ERROR_CAPTURE_FAILURE
  • LOCAL_VIDEO_STREAM_ERROR_CODEC_NOT_SUPPORT
  • LOCAL_VIDEO_STREAM_ERROR_DEVICE_NOT_FOUND
  • LOCAL_VIDEO_STREAM_ERROR_DEVICE_DISCONNECTED
  • LOCAL_VIDEO_STREAM_ERROR_DEVICE_INVALID_ID
  • LOCAL_VIDEO_STREAM_ERROR_SCREEN_CAPTURE_WINDOW_MINIMIZED
  • LOCAL_VIDEO_STREAM_ERROR_SCREEN_CAPTURE_WINDOW_CLOSED
  • LOCAL_VIDEO_STREAM_ERROR_SCREEN_CAPTURE_WINDOW_OCCLUDED
  • LOCAL_VIDEO_STREAM_ERROR_SCREEN_CAPTURE_NO_PERMISSION
  • LOCAL_VIDEO_STREAM_ERROR_SCREEN_CAPTURE_PAUSED
  • LOCAL_VIDEO_STREAM_ERROR_SCREEN_CAPTURE_RESUMED
  • LOCAL_VIDEO_STREAM_ERROR_SCREEN_CAPTURE_WINDOW_HIDDEN
  • LOCAL_VIDEO_STREAM_ERROR_SCREEN_CAPTURE_WINDOW_RECOVER_FROM_HIDDEN
  • LOCAL_VIDEO_STREAM_ERROR_SCREEN_CAPTURE_WINDOW_RECOVER_FROM_MINIMIZED
  • LOCAL_VIDEO_STREAM_ERROR_SCREEN_CAPTURE_FAILURE
  • LOCAL_VIDEO_STREAM_ERROR_DEVICE_SYSTEM_PRESSURE
  • DIRECT_CDN_STREAMING_ERROR_OK
  • DIRECT_CDN_STREAMING_ERROR_FAILED
  • DIRECT_CDN_STREAMING_ERROR_AUDIO_PUBLICATION
  • DIRECT_CDN_STREAMING_ERROR_VIDEO_PUBLICATION
  • DIRECT_CDN_STREAMING_ERROR_NET_CONNECT
  • DIRECT_CDN_STREAMING_ERROR_BAD_NAME
  • PLAYER_ERROR_NONE
  • PLAYER_ERROR_INVALID_ARGUMENTS
  • PLAYER_ERROR_INTERNAL
  • PLAYER_ERROR_NO_RESOURCE
  • PLAYER_ERROR_INVALID_MEDIA_SOURCE
  • PLAYER_ERROR_UNKNOWN_STREAM_TYPE
  • PLAYER_ERROR_OBJ_NOT_INITIALIZED
  • PLAYER_ERROR_CODEC_NOT_SUPPORTED
  • PLAYER_ERROR_VIDEO_RENDER_FAILED
  • PLAYER_ERROR_INVALID_STATE
  • PLAYER_ERROR_URL_NOT_FOUND
  • PLAYER_ERROR_INVALID_CONNECTION_STATE
  • PLAYER_ERROR_SRC_BUFFER_UNDERFLOW
  • PLAYER_ERROR_INTERRUPTED
  • PLAYER_ERROR_NOT_SUPPORTED
  • PLAYER_ERROR_TOKEN_EXPIRED
  • PLAYER_ERROR_UNKNOWN
  • RTMP_STREAM_PUBLISH_ERROR_OK
  • RTMP_STREAM_PUBLISH_ERROR_INVALID_ARGUMENT
  • RTMP_STREAM_PUBLISH_ERROR_ENCRYPTED_STREAM_NOT_ALLOWED
  • RTMP_STREAM_PUBLISH_ERROR_CONNECTION_TIMEOUT
  • RTMP_STREAM_PUBLISH_ERROR_INTERNAL_SERVER_ERROR
  • RTMP_STREAM_PUBLISH_ERROR_RTMP_SERVER_ERROR
  • RTMP_STREAM_PUBLISH_ERROR_TOO_OFTEN
  • RTMP_STREAM_PUBLISH_ERROR_REACH_LIMIT
  • RTMP_STREAM_PUBLISH_ERROR_NOT_AUTHORIZED
  • RTMP_STREAM_PUBLISH_ERROR_STREAM_NOT_FOUND
  • RTMP_STREAM_PUBLISH_ERROR_FORMAT_NOT_SUPPORTED
  • RTMP_STREAM_PUBLISH_ERROR_NOT_BROADCASTER
  • RTMP_STREAM_PUBLISH_ERROR_TRANSCODING_NO_MIX_STREAM
  • RTMP_STREAM_PUBLISH_ERROR_NET_DOWN
  • RTMP_STREAM_PUBLISH_ERROR_INVALID_PRIVILEGE
  • RTMP_STREAM_UNPUBLISH_ERROR_OK

Deleted

• StartChannelMediaRelay
• UpdateChannelMediaRelay
• StartChannelMediaRelayEx
• UpdateChannelMediaRelayEx
• OnChannelMediaRelayEvent
• CHANNEL_MEDIA_RELAY_EVENT

v4.2.6 was released on November 24, 2023.

This release fixed the following issues:

• Issues occurring when using Android 14:

  • When switching between portrait and landscape modes during screen sharing, the screen sharing process was interrupted. To restart screen sharing, users needed to confirm recording the screen in the pop-up window.
  • When integrating the SDK, setting the Android targetSdkVersion to 34 could cause screen sharing to be unavailable or even cause the app to crash.
  • Calling StartScreenCapture[1/2] without sharing video (setting captureVideo to false) and then calling UpdateScreenCaptureParameters to share video (setting captureVideoto true) resulted in a frozen shared screen on the receiving end.
  • When screen sharing in the landscape mode, the shared screen seen by the audience was divided into two parts: One side of the screen was compressed, the other side was black.

• When using an iOS 16 or later device with Bluetooth headphones connected before joining the channel, the audio routing after joining the channel was not as expected: Audio was played from the speaker, not the Bluetooth headphones (iOS).

• In live streaming use-cases, the video on the audience end was occasionally distorted (Android).

• In specific use-cases, such as when the network packet loss rate was high or when the broadcaster left the channel without destroying the engine and then re-joined the channel, the video on the receiving end stuttered or froze.

v4.2.4 was released on October 25, 2023.

This release fixes the following issue:

The AgoraRtcWrapper version number in CFBundleShortVersionString on iOS and macOS is wrong, which interferes with submitting the App to the App Store.

v4.2.3 was released on October 20, 2023.

1. Update video screenshot and upload

   To facilitate the integration of third-party video moderation services from Agora Extensions Marketplace, this version has the following changes:

   • The CONTENT_INSPECT_IMAGE_MODERATION enumeration is added in CONTENT_INSPECT_TYPE which means using video moderation extensions from Agora Extensions Marketplace to take video screenshots and upload them.
   • An optional parameter serverConfig is added in ContentInspectConfig, which is for server-side configuration related to video screenshot and upload via extensions from Agora Extensions Marketplace. By configuring this parameter, you can integrate multiple third-party moderation extensions and achieve flexible control over extension switches and other features. For more details, please contact technical support.

   In addition, this version also introduces the EnableContentInspectEx method, which supports taking screenshots for multiple video streams and uploading them.

2. ID3D11Texture2D Rendering on Windows

   As of this release, the SDK supports video formats of type ID3D11Texture2D, improving the rendering effect of video frames in game use-cases. You can set format to VIDEO_TEXTURE_ID3D11TEXTURE2D when pushing external raw video frames to the SDK by calling PushVideoFrame. By setting the d3d11_texture_2d and texture_slice_index properties, you can determine the ID3D11Texture2D texture object to use.

3. Local video status error code update on Windows and macOS

   In order to help users understand the exact reasons for local video errors in screen sharing use-cases, the following sets of enumerations have been added to the OnLocalVideoStateChanged callback:

   • LOCAL_VIDEO_STREAM_ERROR_SCREEN_CAPTURE_PAUSED(23): Screen capture has been paused. Common use-cases for reporting this error code: The current screen may have been switched to a secure desktop, such as a UAC dialog box or Winlogon desktop.
   • LOCAL_VIDEO_STREAM_ERROR_SCREEN_CAPTURE_RESUMED(24): Screen capture has resumed from the paused state.
   • LOCAL_VIDEO_STREAM_ERROR_SCREEN_CAPTURE_WINDOW_HIDDEN(25): The window being captured on the current screen is in a hidden state and is not visible on the current screen.
   • LOCAL_VIDEO_STREAM_ERROR_SCREEN_CAPTURE_WINDOW_RECOVER_FROM_HIDDEN(26): The window for screen capture has been restored from the hidden state.
   • LOCAL_VIDEO_STREAM_ERROR_SCREEN_CAPTURE_WINDOW_RECOVER_FROM_MINIMIZED(27): The window for screen capture has been restored from the minimized state.

4. Check device support for advanced features

   This version adds the IsFeatureAvailableOnDevice method to check whether the capability of the current device meets the requirements of the specified advanced feature, such as virtual background and image enhancement.

   Before using advanced features, you can check whether the current device supports these features based on the call result. This helps to avoid performance degradation or unavailable features when enabling advanced features on low-end devices. Based on the return value of this method, you can decide whether to display or enable the corresponding feature button, or notify the user when the device's capabilities are insufficient.

   In addition, since this version, calling EnableVirtualBackground and SetBeautyEffectOptions automatically triggers a test on the capability of the current device. When the device is considered underperformed, the error code -4:ERR_NOT_SUPPORTED is returned, indicating the device does not support the feature.

1. Optimize virtual background memory usage

   This version has upgraded the virtual background algorithm, reducing the memory usage of the virtual background feature. Compared to the previous version, the memory consumption of the app during the use of the virtual background feature on low-end devices has been reduced by approximately 4% to 10%. Specific values may vary depending on the device model and platform.

2. Screen sharing use-case optimization

   This release optimizes the performance and encoding efficiency in ultra-high-definition (4K, 60 fps) game sharing use-cases, effectively reducing the system resource usage during screen sharing.

Other Improvements

This release includes the following additional improvements:

• Optimizes the logic of handling invalid parameters. When you call the SetPlaybackSpeed method to set the playback speed of audio files, if you pass an invalid parameter, the SDK returns the error code -2, which means that you need to reset the parameter.
• Optimizes the logic of Token parsing, in order to prevent an app from crashing when an invalid token is passed in.

This release fixed the following issues:

• Occasional crashes and dropped frames in screen sharing use-cases on Windows.
• Occasional crashes when joining a channel on macOS.
• Occasional failure of joining a channel when the local system time was not set correctly.
• When calling the PlayEffect method to play two audio files using the same soundId, the first audio file was sometimes played repeatedly.
• When the host called the StartAudioMixing [2/2] method to play music, sometimes the host couldn't hear the music while the remote users could hear it on Android.
• Occasional crashes occurred on certain Android devices.
• Calling TakeSnapshotEx once received the OnSnapshotTaken callback for multiple times.
• In channels joined by calling JoinChannelEx exclusively, calling SetEnableSpeakerphone did not result in switching audio route from the speaker to the headphone on Android.

Added

• The following enumerations in OnLocalVideoStateChanged on (Windows and macOS:
  • LOCAL_VIDEO_STREAM_ERROR_SCREEN_CAPTURE_PAUSED
  • LOCAL_VIDEO_STREAM_ERROR_SCREEN_CAPTURE_RESUMED
  • LOCAL_VIDEO_STREAM_ERROR_SCREEN_CAPTURE_WINDOW_HIDDEN
  • LOCAL_VIDEO_STREAM_ERROR_SCREEN_CAPTURE_WINDOW_RECOVER_FROM_HIDDEN
  • LOCAL_VIDEO_STREAM_ERROR_SCREEN_CAPTURE_WINDOW_RECOVER_FROM_MINIMIZED
• d3d11_texture_2d and texture_slice_index members in ExternalVideoFrame on Windows
• VIDEO_TEXTURE_ID3D11TEXTURE2D in VIDEO_PIXEL_FORMAT on Windows
• EnableContentInspectEx
• CONTENT_INSPECT_IMAGE_MODERATION in CONTENT_INSPECT_TYPE
• serverConfig in ContentInspectConfig
• IsFeatureAvailableOnDevice
• FeatureType

This version was released on August 1, 2023.

If you use the features mentioned in this section, ensure that you modify the implementation of the relevant features after upgrading the SDK.

1. Audio frame observer

The following methods in the IAudioFrameObserver class are deleted:

• GetObservedAudioFramePosition: Use the newly-added position parameter in RegisterAudioFrameObserver instead.
• GetPlaybackAudioParams: Use SetPlaybackAudioFrameParameters instead.
• GetRecordAudioParams: Use SetRecordingAudioFrameParameters instead.
• GetMixedAudioParams: Use SetMixedAudioFrameParameters instead.
• GetEarMonitoringAudioParams: Use SetEarMonitoringAudioFrameParameters instead.

2. Video frame observer

The following methods in the IVideoFrameObserver class are deleted:

• GetVideoFormatPreference: Use the newly-added formatPreference parameter in RegisterVideoFrameObserver.
• GetObservedFramePosition: Use the newly-added position parameter in RegisterVideoFrameObserver.

3. Metadata

This release deletes GetMaxMetadataSize and OnReadyToSendMetadata in the IMetadataObserver class. You can use the newly-added SetMaxMetadataSize and SendMetadata methods instead.

1. Wildcard token

   This release introduces wildcard tokens. Agora supports setting the channel name used for generating a token as a wildcard character. The token generated can be used to join any channel if you use the same user id. In use-cases involving multiple channels, such as switching between different channels, using a wildcard token can avoid repeated application of tokens every time users joining a new channel, which reduces the pressure on your token server. See Secure authentication with tokens.

   All 4.x SDKs support using wildcard tokens.

2. Preloading channels

   This release adds PreloadChannel[1/2] and PreloadChannel[2/2] methods, which allows a user whose role is set as audience to preload channels before joining one. Calling the method can help shortening the time of joining a channel, thus reducing the time it takes for audience members to hear and see the host.

   When preloading more than one channels, Agora recommends that you use a wildcard token for preloading to avoid repeated application of tokens every time you joining a new channel, thus saving the time for switching between channels. See Secure authentication with tokens.

3. Customized background color of video canvas

   In this release, the backgroundColor member has been added to VideoCanvas, which allows you to customize the background color of the video canvas when setting the properties of local or remote video display.

4. Publishing video streams from different sources (Windows, macOS)

   This release adds the following members in ChannelMediaOptions to allow you publish video streams captured from the third and fourth camera or screen:

   • publishThirdCameraTrack: Publishing the video stream captured from the third camera.
   • publishFourthCameraTrack: Publishing the video stream captured from the fourth camera.
   • publishThirdScreenTrack: Publishing the video stream captured from the third screen.
   • publishFourthScreenTrack: Publishing the video stream captured from the fourth screen.
   For one RtcConnection, Agora supports publishing multiple audio streams and one video stream at the same time.

1. Improved camera capture effect (Android, iOS)

   This release has improved camera capture effect in the following aspects:

   1. Support for camera exposure adjustment

      This release adds IsCameraExposureSupported to query whether the device supports exposure adjustment and SetCameraExposureFactor to set the exposure ratio of the camera.

   2. Optimization of default camera selection (iOS)

      Since this release, the default camera selection behavior of the SDK is aligned with that of the iOS system camera. If the device has multiple rear cameras, better shooting perspectives, zooming capabilities, low-light performance, and depth sensing can be achieved during video capture, thereby improving the quality of video capture.

2. Virtual Background Algorithm Upgrade

   This version has upgraded the portrait segmentation algorithm of the virtual background, which comprehensively improves the accuracy of portrait segmentation, the smoothness of the portrait edge with the virtual background, and the fit of the edge when the person moves. In addition, it optimizes the precision of the person's edge in use-cases such as meetings, offices, homes, and under backlight or weak light conditions.

3. Channel media relay

   The number of target channels for media relay has been increased to 6. When calling StartOrUpdateChannelMediaRelay and StartOrUpdateChannelMediaRelayEx, you can specify up to 6 target channels.

4. Enhancement in video codec query capability

   To improve the video codec query capability, this release adds the codecLevels member in CodecCapInfo. After successfully calling QueryCodecCapability, you can obtain the hardware and software decoding capability levels of the device for H.264 and H.265 video formats through codecLevels.

This release includes the following additional improvements:

1. To improve the switching experience between multiple audio routes, this release adds the SetRouteInCommunicationMode method. This method can switch the audio route from a Bluetooth headphone to the earpiece, wired headphone or speaker in communication volume mode (MODE_IN_COMMUNICATION). (Android)
2. The SDK automacially adjusts the frame rate of the sending end based on the screen sharing use-case. Especially in document sharing use-cases, this feature avoids exceeding the expected video bitrate on the sending end to improve transmission efficiency and reduce network burden.
3. To help users understand the reasons for more types of remote video state changes, the REMOTE_VIDEO_STATE_REASON_CODEC_NOT_SUPPORT enumeration has been added to the OnRemoteVideoStateChanged callback, indicating that the local video decoder does not support decoding the received remote video stream.

This release fixed the following issues:

• Occasionally, noise occurred when the local user listened to their own and remote audio after joining the channel. (macOS)
• Slow channel reconnection after the connection was interrupted due to network reasons.
• In screen sharing use-cases, the delay of seeing the shared screen was occasionally higher than expected on some devices.
• In custom video capturing use-cases, SetBeautyEffectOptions, SetLowlightEnhanceOptions, SetVideoDenoiserOptions, and SetColorEnhanceOptions could not load extensions automatically.
• In multi-device audio recording use-cases, after repeatedly plugging and unplugging or enabling/disabling the audio recording device, no sound could be heard occasionally when calling the StartRecordingDeviceTest to start an audio capturing device test. (Windows)

Added

• SetCameraExposureFactor (Android, iOS)
• IsCameraExposureSupported (Android, iOS)
• PreloadChannel[1/2]
• PreloadChannel[2/2]
• UpdatePreloadChannelToken
• SetRouteInCommunicationMode (Android)
• The following members in ChannelMediaOptions (Windows, macOS):
  • publishThirdCameraTrack
  • publishFourthCameraTrack
  • publishThirdScreenTrack
  • publishFourthScreenTrack
• CodecCapLevels
• VIDEO_CODEC_CAPABILITY_LEVEL
• backgroundColor in VideoCanvas
• codecLevels in CodecCapInfo
• REMOTE_VIDEO_STATE_REASON_CODEC_NOT_SUPPORT in REMOTE_VIDEO_STATE_REASON
• SetMaxMetadataSize
• SendMetadata
• position parameter in RegisterAudioFrameObserver
• formatPreference and position parameters in RegisterVideoFrameObserver

Deleted

• GetObservedAudioFramePosition
• GetPlaybackAudioParams
• GetRecordAudioParams
• GetMixedAudioParams
• GetEarMonitoringAudioParams
• GetVideoFormatPreference
• GetObservedFramePosition
• GetMaxMetadataSize
• OnReadyToSendMetadata

This version was released on June 21, 2023.

This version improves the network transmission strategy, enhancing the smoothness of audio and video interactions.

This version fixed the following issues:

• Inability to join channels caused by SDK's incompatibility with some older versions of AccessToken.
• After the sending end called SetAINSMode to activate AI noise reduction, occasional echo was observed by the receiving end.
• Brief noise occurred while playing media files using the media player.
• After enabling the screen sharing function on the sending end, there was high delay until the receiving end could see the shared screen. (macOS)
• Occasional crash after calling the DestroyMediaPlayer method. (iOS)
• When the sending end mixed and published two streams of video captured by two cameras locally, the video from the second camera was occasionally missing on the receiving end. (Windows)
• In screen sharing use-cases, some Android devices experienced choppy video on the receiving end. (Android)

v4.2.0 was released on May 29, 2023.

If you use the features mentioned in this section, ensure that you modify the implementation of the relevant features after upgrading the SDK.

1. Video capture (Windows, iOS)

This release optimizes the APIs for camera and screen capture function. As of v4.2.0, ensure you use the alternative methods listed in the table below and specify the video source by setting the sourceType parameter:

Deleted Methods	Alternative Methods
StartPrimaryCameraCapture（Windows） StartSecondaryCameraCapture（Windows，iOS）	StartCameraCapture
StopPrimaryCameraCapture（Windows） StopSecondaryCameraCapture（Windows，iOS）	StopCameraCapture
StartPrimaryScreenCapture （Windows） StartSecondaryScreenCapture（Windows，iOS）	StartScreenCapture[2/2]
StopPrimaryScreenCapture（Windows） StopSecondaryScreenCapture（Windows，iOS）	StopScreenCapture[2/2]

2. Video rendering

As of v4.2.0, Agora Unity SDK does not support using VideoSurface and VideoSurfaceYUV at the same time for video rendering. After one IRtcEngine is created, if the first view is rendered through VideoSurfaceYUV, all views need to be rendered using VideoSurfaceYUV throughout the lifecycle of the IRtcEngine.

3. Video data acquisition

The OnCaptureVideoFrame and OnPreEncodeVideoFrame callbacks are added with a new parameter called sourceType, which is used to indicate the specific video source type.

4. Channel media options

• PublishCustomAudioTrackEnableAec in ChannelMediaOptions is deleted. Use PublishCustomAudioTrack instead.
• PublishTrancodedVideoTrack in ChannelMediaOptions is renamed to PublishTranscodedVideoTrack.
• PublishCustomAudioSourceId in ChannelMediaOptions is renamed to PublishCustomAudioTrackId.

5. Local video mixing (Windows)

• The VideoInputStreams in LocalTranscoderConfiguration is changed to VideoInputStreams.
• The MEDIA_SOURCE_TYPE in TranscodingVideoStream is changed to VIDEO_SOURCE_TYPE.

6. Virtual sound card (macOS)

As of v4.2.0, Agora supports third-party virtual sound cards. You can use a third-party virtual sound card as the audio input or output device for the SDK. You can use the stateChanged callback to see whether the current input or output device selected by the SDK is a virtual sound card.

If you set AgoraALD or Soundflower as the default input or output device when joining a channel, you will not hear audio.

7. Default video encoding resolution

As of this release, the SDK optimizes the video encoder algorithm and upgrades the default video encoding resolution from 640 × 360 to 960 × 540 to accommodate improvements in device performance and network bandwidth, providing users with a full-link HD experience in various audio and video interaction use-cases.

You can call the SetVideoEncoderConfiguration method to set the expected video encoding resolution in the video encoding parameters configuration.

info

In use-cases such as 1-on-1 video calls and video group chats, the increase in the default resolution affects the aggregate resolution and thus the billing rate. For example, before this release, in a 1-on-1 video call use-case, the default aggregate resolution of subscribed video was 460,800 (640 × 360 × 2) and was billed as HD type; as of this release, the default aggregate resolution in the same use-case is 1,036,800 (960 × 540 × 2) and is billed as Full HD type.

8. Miscellaneous

• OnApiCallExecuted is deleted. Agora recommends getting the results of the API implementation through relevant channels and media callbacks.

• The IAudioFrameObserver class is renamed to IAudioPcmFrameSink, thus the prototypes of the following methods are updated accordingly:

  • OnFrame
  • RegisterAudioFrameObserver [1/2] and RegisterAudioFrameObserver[2/2] in IMediaPlayer

• EnableDualStreamMode[1/2] and EnableDualStreamMode[2/2] are deprecated. Use SetDualStreamMode[1/2] and SetDualStreamMode[2/2] instead.

• StartChannelMediaRelay, UpdateChannelMediaRelay, StartChannelMediaRelayEx, and UpdateChannelMediaRelayEx are deprecated. Use StartOrUpdateChannelMediaRelayAnd StartOrUpdateChannelMediaRelayEx instead.

1. AI Noise Suppression

This release introduces public APIs for the AI Noise Suppression function. Once enabled, the SDK automatically detects and reduces background noises. Whether in bustling public venues or real-time competitive arenas that demand lightning-fast responsiveness, this function guarantees optimal audio clarity, providing users with an elevated audio experience. You can enable this function through the newly-introduced setAINSMode method and set the noise suppression mode to balance, aggressive, or low latency according to your use-case.

info

Agora charges separately for this function. See AI Noise Suppression unit pricing.

2. Enhanced Virtual Background

To increase the fun of real-time video calls and protect user privacy, this version has enhanced the Virtual Background feature. You can now set custom backgrounds of various types by calling the EnableVirtualBackground method, including:

• Process the background as Alpha information without replacement, only separating the portrait and the background. This can be combined with the local video mixing feature to achieve a portrait-in-picture effect.
• Replace the background with various formats of local videos.

See Virtual Background documentation.

3. Video scenario settings

This release introduces SetVideoScenario for setting the video application scene. The SDK will automatically enable the best practice strategy based on different scenes, adjusting key performance indicators to optimize video quality and improve user experience. Whether it is a formal business meeting or a casual online gathering, this feature ensures that the video quality meets the requirements.

Currently, this feature provides targeted optimizations for real-time video conferencing use-cases, including:

• Automatically activate multiple anti-weak-network technologies to enhance the capability and performance of low-quality video streams in meeting use-cases where high bitrates are required, ensuring smoothness when multiple streams are subscribed by the receiving end.
• Monitor the number of subscribers for the high-quality and low-quality video streams in real time, dynamically adjusting the configuration of the high-quality stream and dynamically enabling or disabling the low-quality stream, to save uplink bandwidth and consumption.

4. Local video mixing

On Windows, this release adds the OnLocalVideoTranscoderError callback. When there is an error in starting or updating the local video mixing, the SDK triggers this callback to report the reason for the failure.

On Android, iOS, and macOS, this release adds the local video mixing feature. You can use the StartLocalVideoTranscoder method to mix and render multiple video streams locally, such as camera-captured video, screen sharing streams, video files, images, etc. This allows you to achieve custom layouts and effects, making it easy to create personalized video display effects to meet various use-case requirements, such as remote meetings, live streaming, online education, while also supporting features like portrait-in-picture effect.

Additionally, the SDK provides the UpdateLocalTranscoderConfiguration method and the OnLocalVideoTranscoderError callback. After enabling local video mixing, you can use the UpdateLocalTranscoderConfiguration method to update the video mixing configuration. Where an error occurs in starting the local video mixing or updating the configuration, you can get the reason for the failure through the OnLocalVideoTranscoderError callback.

Local video mixing requires more CPU resources. Therefore, Agora recommends enabling this function on devices with higher performance.

5. Cross-device synchronization

In real-time collaborative singing use-cases, network issues can cause inconsistencies in the downlinks of different client devices. To address this, this release introduces GetNtpWallTimeInMs for obtaining the current Network Time Protocol (NTP) time. By using this method to synchronize lyrics and music across multiple client devices, users can achieve synchronized singing and lyrics progression, resulting in a better collaborative experience.

6. Instant frame rendering

This release adds the EnableInstantMediaRendering method to enable instant rendering mode for audio and video frames, which can speed up the first video or audio frame rendering after the user joins the channel.

7. Video rendering tracing

This release adds the StartMediaRenderingTracing and StartMediaRenderingTracingEx methods. The SDK starts tracing the rendering status of the video frames in the channel from the moment this method is called and reports information about the event through the OnVideoRenderingTracingResult callback.

Agora recommends that you use this method in conjunction with the UI settings, such as buttons and sliders, in your app. For example, call this method when the user clicks the Join Channel button, and then get the indicators in the video frame rendering process through the OnVideoRenderingTracingResult callback. This enables developers to optimize the indicators and improve the user experience.

1. Voice changer

This release introduces the SetLocalVoiceFormant method that allows you to adjust the formant ratio to change the timbre of the voice. This method can be used together with the SetLocalVoicePitch method to adjust the pitch and timbre of voice at the same time, enabling a wider range of voice transformation effects.

2. Enhanced screen share (Android, iOS)

This release adds the QueryScreenCaptureCapability method, which is used to query the screen capture capabilities of the current device. To ensure optimal screen sharing performance, particularly in enabling high frame rates like 60 fps, Agora recommends you to query the device's maximum supported frame rate using this method beforehand.

This release also adds the SetScreenCaptureScenario method, which is used to set the scenario type for screen sharing. The SDK automatically adjusts the smoothness and clarity of the shared screen based on the scenario type you set.

3. Improved compatibility with audio file types (Android)

As of v4.2.0, you can use the following methods to open files with a URI starting with Content:// : StartAudioMixing [2/2], PlayEffect [3/3], Open, and OpenWithMediaSource.

4. Enhanced rendering compatibility (Windows)

This release enhances the rendering compatibility of the SDK. Issues like black screens caused by rendering failures on certain devices are fixed.

5. Audio and video synchronization

For custom video and audio capture use-cases, this release introduces GetCurrentMonotonicTimeInMs for obtaining the current Monotonic Time. By passing this value into the timestamps of audio and video frames, developers can accurately control the timing of their audio and video streams, ensuring proper synchronization.

6. Multi-camera capture and multi-screen capture

This release introduces StartScreenCapture [2/2] (PC only) and StartCameraCapture. By calling these methods multiple times and specifying the sourceType parameter, developers can start capturing video streams from multiple cameras and screens for local video mixing or multi-channel publishing. This is particularly useful for use-cases such as remote medical care and online education, where multiple cameras and displays need to be connected.

7. Channel media relay

This release introduces StartOrUpdateChannelMediaRelay and StartOrUpdateChannelMediaRelayEx, allowing for a simpler and smoother way to start and update media relay across channels. With these methods, developers can easily start the media relay across channels and update the target channels for media relay with a single method. Additionally, the internal interaction frequency has been optimized, effectively reducing latency in function calls.

8. Custom audio tracks

To better meet the needs of custom audio capture use-cases, this release adds CreateCustomAudioTrack and DestroyCustomAudioTrack for creating and destroying custom audio tracks. Two types of audio tracks are also provided for users to choose from, further improving the flexibility of capturing external audio source:

• Mixable audio track: Supports mixing multiple external audio sources and publishing them to the same channel, suitable for multi-channel audio capture use-cases.
• Direct audio track: Only supports publishing one external audio source to a single channel, suitable for low-latency audio capture use-cases.

9. Super resolution (Android, iOS)

To optimize the usability of super resolution, this release removes EnableRemoteSuperResolution. Super resolution no longer needs to be enabled manually. The SDK now automatically optimizes the resolution of the remote video based on the performance of the user's device.

This release fixed the following issues:

Windows

• When using Agora Media Player to play RTSP video streams, the video images sometimes appeared pixelated.
• Adding an Alpha channel to an image in PNG or GIF format failed when the local client mixed video streams.
• After joining the channel, remote users saw a watermark even though the watermark was deleted.
• If a watermark was added after starting screen sharing, the watermark was not displayed on the screen.
• When joining a channel and accessing an external camera, calling SetDevice to specify the video capture device as the external camera did not take effect.
• When trying to outline the shared window and put it on top, the shared window did not stay on top of other windows.

Android

• Occasional crashes occurred on Android devices when users joined or left a channel.
• Occasional failure when enabling in-ear monitoring.
• Occasional echo.
• Crashes occurred after users set the video resolution to 3840 × 2160 and started CDN streaming on Xiaomi Redmi 9A devices.
• In real-time chorus use-cases, remote users heard noises and echoes when an OPPO R11 device joined the channel in the loudspeaker mode.
• When the playback of the local music finished, the OnAudioMixingFinished callback was not properly triggered.
• When using a video frame observer, the first video frame was occasionally missed on the receiver's end.
• When sharing screens in use-cases involving multiple channels, remote users occasionally saw black screens.
• Switching to the rear camera with the virtual background enabled occasionally caused the background to be inverted.
• Abnormal client status caused by an exception in the OnRemoteAudioStateChanged callback.

iOS

• Occasional loss of the OnFirstRemoteVideoFrame callback during channel media relay.
• The receiver actively subscribed to the high-quality stream but unexpectedly received a low-quality stream.
• Abnormal client status cased by an exception in the OnRemoteAudioStateChanged callback.

macOS

• The receiver was receiving the low-quality stream originally and automatically switched to the high-quality stream after a few seconds.
• Occasional screen jitter during screen sharing.
• If the rendering view of the player was set as a UIViewController's view, the video was zoomed from the bottom-left corner to the middle of the screen when entering the full-screen mode.
• When joining a channel and accessing an external camera, calling SetDevice to specify the video capture device as the external camera did not take effect.

All platforms

• When the host frequently switched the user role between broadcaster and audience in a short period of time, the audience members could not hear the audio of the host.
• Playing audio files with a sample rate of 48 kHz failed.
• When there were multiple video streams in a channel, calling some video enhancement APIs occasionally failed.

Added

• StartCameraCapture
• StopCameraCapture
• StartScreenCapture[2/2] (Windows, macOS)
• StopScreenCapture[2/2] (Windows, macOS)
• StartOrUpdateChannelMediaRelay
• StartOrUpdateChannelMediaRelayEx
• GetNtpWallTimeInMs
• SetVideoScenario
• GetCurrentMonotonicTimeInMs
• OnLocalVideoTranscoderError
• startLocalVideoTranscoder（macOS, iOS, Android）
• updateLocalTranscoderConfiguration（macOS, iOS, Android）
• QueryScreenCaptureCapability (iOS, Android)
• SetScreenCaptureScenario (iOS, Android)
• CreateAudioCustomTrack
• DestroyAudioCustomTrack
• AudioTrackConfig
• AUDIO_TRACK_TYPE
• VIDEO_APPLICATION_SCENARIO_TYPE
• SCREEN_CAPTURE_FRAMERATE_CAPABILITY
• The domainLimit and autoRegisterAgoraExtensions members in RtcEngineContext
• The sourceType parameter in OnCaptureVideoFrame and OnPreEncodeVideoFrame callbacks
• The BACKGROUND_NONE and BACKGROUND_VIDEO enumerators in BACKGROUND_SOURCE_TYPE
• EnableInstantMediaRendering
• StartMediaRenderingTracing
• StartMediaRenderingTracingEx
• OnVideoRenderingTracingResult
• MEDIA_TRACE_EVENT
• VideoRenderingTracingInfo

Deprecated

• EnableDualStreamMode[1/2]
• EnableDualStreamMode[2/2]
• StartChannelMediaRelay
• StartChannelMediaRelayEx
• UpdateChannelMediaRelay
• UpdateChannelMediaRelayEx
• OnChannelMediaRelayEvent
• CHANNEL_MEDIA_RELAY_EVENT

Deleted

• StartPrimaryScreenCapture (Windows)
• StartSecondaryScreenCapture (Windows)
• StopPrimaryScreenCapture (Windows)
• StopSecondaryScreenCapture (Windows)
• StartPrimaryCameraCapture (Windows)
• StartSecondaryCameraCapture (Windows, iOS)
• StopPrimaryCameraCapture (Windows)
• StopSecondaryCameraCapture (Windows, iOS)
• OnApiCallExecuted
• PublishCustomAudioTrackEnableAec in ChannelMediaOptions
• EnableRemoteSuperResolution
• superResolutionType in RemoteVideoStats

v4.1.0 was released on December 20, 2022.

This release deletes the EnableDualStreamMode [2/3] method and the sourceType parameter in EnableDualStreamMode [3/3], because the SDK supports enabling dual-stream mode for various video sources captured by custom capture or SDK; you no longer need to specify the video source type.

1. In-ear monitoring

This release adds support for in-ear monitoring. You can call EnableInEarMonitoring to enable the in-ear monitoring function.

After successfully enabling the in-ear monitoring function, you can call RegisterAudioFrameObserver to register the audio observer, and the SDK triggers the OnEarMonitoringAudioFrame callback to report the audio frame data. You can use your own audio effect processing module to preprocess the audio frame data of the in-ear monitoring to implement custom audio effects. Agora recommends that you choose one of the following two methods to set the audio data format of the in-ear monitoring:

• Call the SetEarMonitoringAudioFrameParameters method to set the audio data format of in-ear monitoring. The SDK calculates the sampling interval based on the parameters in this method and triggers the OnEarMonitoringAudioFrame callback based on the sampling interval.
• Set the audio data format in the return value of the GetEarMonitoringAudioParams callback. The SDK calculates the sampling interval based on the return value of the callback and triggers the OnEarMonitoringAudioFrame callback based on the sampling interval.

To adjust the in-ear monitoring volume, you can call SetInEarMonitoringVolume.

2. Audio capture device test (Android)

This release adds support for testing local audio capture devices before joining a channel. You can call StartRecordingDeviceTest to start the audio capture device test. After the test is complete, call the StopPlaybackDeviceTest method to stop the audio capture device test.

3. Local network connection types

To make it easier for users to know the connection type of the local network at any stage, this release adds the GetNetworkType method. You can use this method to get the type of network connection in use. The available values are UNKNOWN, DISCONNECTED, LAN, WIFI, 2G, 3G, 4G, and 5G. When the local network connection type changes, the SDK triggers the OnNetworkTypeChanged callback to report the current network connection type.

4. Audio stream filter

This release introduces filtering audio streams based on volume. Once this function is enabled, the Agora server ranks all audio streams by volume and transports the three audio streams with the highest volumes to the receivers by default. The number of audio streams to be transported can be adjusted; contact support@agora.io to adjust this number according to your use-case.

Agora also supports publishers in choosing whether the audio streams being published are to be filtered based on volume. Streams that are not filtered bypass this filter mechanism and are transported directly to the receivers. In use-cases with a number of publishers, enabling this function helps reduce the bandwidth and device system pressure for the receivers.

To enable this function, contact support@agora.io.

5. Dual-stream mode

This release optimizes the dual-stream mode. You can call EnableDualStreamMode and EnableDualStreamModeEx before and after joining a channel.

The implementation of subscribing to a low-quality video stream is expanded. The SDK enables the low-quality video stream auto mode on the sender by default (the SDK does not send low-quality video streams). Follow these steps to enable sending low-quality video streams:

1. The host at the receiving end calls SetRemoteVideoStreamType or SetRemoteDefaultVideoStreamType to initiate a low-quality video stream request.
2. After receiving the application, the sender automatically switches to sending low-quality video stream mode.

If you want to modify this default behavior, you can call SetDualStreamMode[1/2] or SetDualStreamMode[2/2] and set the mode parameter to DISABLE_SIMULCAST_STREAM (never send low-quality video streams) or ENABLE_SIMULCAST_STREAM (always send low-quality video streams).

6. Loopback device (Windows)

The SDK uses the playback device as the loopback device by default. As of 4.1.0, you can specify a loopback device separately and publish the captured audio to the remote end.

• SetLoopbackDevice: Specifies the loopback device. If you do not want the current playback device to be the loopback device, you can call this method to specify another device as the loopback device.
• GetLoopbackDevice: Gets the current loopback device.
• FollowSystemLoopbackDevice: Whether the loopback device follows the default playback device of the system.

7. Spatial audio effect

This release adds the following features applicable to spatial audio effect use-cases, which can effectively enhance the user's sense-of-presence experience in virtual interactive use-cases.

• Sound insulation area: You can set a sound insulation area and sound attenuation parameter by calling SetZones. When the sound source (which can be a user or the media player) and the listener belong to the inside and outside of the sound insulation area, the listener experiences an attenuation effect similar to that of the sound in the real environment when it encounters a building partition. You can also set the sound attenuation parameter for the media player and the user by calling SetPlayerAttenuation and SetRemoteAudioAttenuation respectively, and specify whether to use that setting to force an override of the sound attenuation parameter in SetZones.
• Doppler sound: You can enable Doppler sound by setting the enable_doppler parameter in SpatialAudioParams. The receiver experiences noticeable tonal changes in the event of a high-speed relative displacement between the source and receiver (such as in a racing game use-case).
• Headphone equalizer: You can use a preset headphone equalization effect by calling the SetHeadphoneEQPreset method to improve the audio experience for users with headphones.

8. Multiple cameras for video capture (iOS)

This release supports multi-camera video capture. You can call EnableMultiCamera to enable multi-camera capture mode, call StartSecondaryCameraCapture to start capturing video from the second camera, and then publish the captured video to the second channel.

To stop using multi-camera capture, you need to call StopSecondaryCameraCapture to stop the second camera capture, then call EnableMultiCamera and set enabled to false.

9. Headphone equalization effect

This release adds the SetHeadphoneEQParameters method, which is used to adjust the low- and high-frequency parameters of the headphone EQ. This is mainly useful in spatial audio use-cases. If you cannot achieve the expected headphone EQ effect after calling SetHeadphoneEQPreset, you can call setHeadphoneEQParameters to adjust the EQ.

10. Encoded video frame observer

This release adds the SetRemoteVideoSubscriptionOptions and SetRemoteVideoSubscriptionOptionsEx methods. When you call the RegisterVideoEncodedFrameObserver method to register a video frame observer for the encoded video frames, the SDK subscribes to the encoded video frames by default. If you want to change the subscription options, you can call these new methods to set them.

For more information about registering video observers and subscription options, see API reference.

11. MPUDP (MultiPath UDP) (Beta)

As of this release, the SDK supports MPUDP protocol, which enables you to connect and use multiple paths to maximize the use of channel resources based on the UDP protocol. You can use different physical NICs on both mobile and desktop and aggregate them to effectively combat network jitter and improve transmission quality.

To enable this feature, contact support@agora.io.

12. Register extensions (Windows)

This release adds the RegisterExtension method for registering extensions. When using a third-party extension, you need to call the extension-related APIs in the following order:

loadExtensionProvider -> registerExtension -> setExtensionProviderProperty -> enableExtension

13. Device management (Windows,macOS)

This release adds a series of callbacks to help you better understand the status of your audio and video devices:

• OnVideoDeviceStateChanged: Occurs when the status of the video device changes.
• OnAudioDeviceStateChanged: Occurs when the status of the audio device changes.
• OnAudioDeviceVolumeChanged: Occurs when the volume of an audio device or app changes.

14. Camera capture options

This release adds the followEncodeDimensionRatio member in CameraCapturerConfiguration, which enables you to set whether to follow the video aspect ratio already set in SetVideoEncoderConfiguration when capturing video with the camera.

15. Multi-channel management

This release adds a series of multi-channel related methods that you can call to manage audio and video streams in multi-channel use-cases.

• The MuteLocalAudioStreamEx and MuteLocalVideoStreamEx methods are used to cancel or resume publishing a local audio or video stream, respectively.
• The MuteAllRemoteAudioStreamsEx and MuteAllRemoteVideoStreamsEx are used to cancel or resume the subscription of all remote users to audio or video streams, respectively.
• The StartRtmpStreamWithoutTranscodingEx, StartRtmpStreamWithTranscodingEx, UpdateRtmpTranscodingEx, and StopRtmpStreamEx methods are used to implement Media Push in multi-channel use-cases.
• The StartChannelMediaRelayEx, UpdateChannelMediaRelayEx, PauseAllChannelMediaRelayEx, ResumeAllChannelMediaRelayEx, and StopChannelMediaRelayEx methods are used to relay media streams across channels in multi-channel use-cases.
• Adds the LeaveChannelEx [2/2] method. Compared with the LeaveChannelEx [1/2] method, a new options parameter is added, which is used to choose whether to stop recording with the microphone when leaving a channel in a multi-channel use-case.

16. Video encoding preferences

In general use-cases, the default video encoding configuration meets most requirements. For certain specific use-cases, this release adds the advanceOptions member in VideoEncoderConfiguration for advanced settings of video encoding properties:

• compressionPreference: The compression preferences for video encoding, which is used to select low-latency or high-quality video preferences.
• encodingPreference: The video encoder preference, which is used to select adaptive preference, software encoder preference, or hardware encoder video preferences.

17. Client role switching

In order to enable users to know whether the switched user role is low-latency or ultra-low-latency, this release adds the newRoleOptions parameter to the OnClientRoleChanged callback. The value of this parameter is as follows:

• AUDIENCE_LATENCY_LEVEL_LOW_LATENCY (1): Low latency.
• AUDIENCE_LATENCY_LEVEL_ULTRA_LOW_LATENCY (2): Ultra-low latency.

18. Brand-new AI Noise Suppression

The SDK supports a new version of AI noise reduction (in comparison to the basic AI noise reduction in v3.7.x). The new AI noise reduction has better vocal fidelity, cleaner noise suppression, and adds a dereverberation option. To experience this feature, contact support@agora.io.

1. Video information change callback

This release optimizes the trigger logic of OnVideoSizeChanged, which can also be triggered and report the local video size change when StartPreview is called separately.

2. First video frame rendering（Windows）

This release speeds up the first video frame rendering time to improve the video experience.

3. Screen sharing

In addition to the usability enhancements detailed in the fixed issued section, this release includes a number of functional improvements to screen sharing, as follows:

Windows:

• New minimizeWindow member in ScreenCaptureSourceInfo to indicate whether the target window is minimized.
• New enableHighLight, highLightColor, and highLightWidth members in ScreenCaptureParameters so that you can place a border around the target window or screen when screen sharing.
• Compatibility with a greater number of mainstream apps, including WPS Office, Microsoft Office PowerPoint, Visual Studio Code, Adobe Photoshop, Windows Media Player, and Scratch.
• Compatibility with additional devices and operating systems, including: Window 8 systems, devices without discrete graphics cards, and dual graphics devices.
• Support for Ultra HD video (4K, 60 fps) on devices that meet the requirements. Agora recommends a device with an Intel Core i7-9750H CPU @ 2.60 GHz or better.

macOS:

• Compatibility with additional devices and use-cases, including dual graphics devices and screen sharing using external screens.
• Support for Ultra HD video (4K, 60 fps) on devices that meet the requirements. Agora recommends a 2021 16" M1 Macbook Pro or better.

4. Bluetooth permissions (Android)

To simplify integration, as of this release, you can use the SDK to enable Android users to use Bluetooth normally without adding the BLUETOOTH_CONNECTpermission.

5. Relaying media streams across channels

This release optimizes the UpdateChannelMediaRelay method as follows:

• Before v4.1.0: If the target channel update fails due to internal reasons in the server, the SDK returns the error code RELAY_EVENT_PACKET_UPDATE_DEST_CHANNEL_REFUSED(8), and you need to call the UpdateChannelMediaRelay method again.
• v4.1.0 and later: If the target channel update fails due to internal server reasons, the SDK retries the update until the target channel update is successful.

6. Reconstructed AIAEC algorithm

This release reconstructs the AEC algorithm based on the AI method. Compared with the traditional AEC algorithm, the new algorithm can preserve the complete, clear, and smooth near-end vocals under poor echo-to-signal conditions, significantly improving the system's echo cancellation and dual-talk performance. This gives users a more comfortable call and live-broadcast experience. AIAEC is suitable for conference calls, chats, karaoke, and other use-cases.

7. Virtual background

This release optimizes the virtual background algorithm. Improvements include the following:

• The boundaries of virtual backgrounds are handled in a more nuanced way and image matting is is now extremely thin.
• The stability of the virtual background is improved whether the portrait is still or moving, effectively eliminating the problem of background flickering and exceeding the range of the picture.
• More application use-cases are now supported, and a user obtains a good virtual background effect day or night, indoors or out.
• A larger variety of postures are now recognized, when half the body is motionless, the body is shaking, the hands are swinging, or there is fine finger movement. This helps to achieve a good virtual background effect in conjunction with many different gestures.

Other improvements

This release includes the following additional improvements:

• Reduces the latency when pushing external audio sources.
• Improves the performance of echo cancellation when using the AUDIO_SCENARIO_MEETING scenario.
• Improves the smoothness of SDK video rendering.
• Enhances the ability to identify different network protocol stacks and improves the SDK's access capabilities in multiple-operator network use-cases.
• Reduces the CPU usage and power consumption of the local device when the host calls the MuteLocalVideoStream method. (Windows, macOS)

This release fixes the following issues:

Screen sharing issues

• macOS

  • In screen sharing use-cases on Mac devices, when the user minimized or closed a shared application window, another window of the same application was automatically shared.
  • In screen sharing use-cases, the system volume of the local user occasionally decreased.
  • In screen sharing use-cases, the shared window of a split screen was not highlighted correctly.
  • In screen sharing use-cases, the screen seen by the remote user occasionally crashed, lagged, or displayed a black screen.
  • The uplink network quality reported by the OnNetworkQuality callback was inaccurate for the user who was sharing a screen.

• Windows

  • In screen sharing use-cases, when the user minimized and then restored the shared window, the remote video occasionally switched to the low-quality stream.
  • When the host started screen sharing during live streaming, the audience members sometimes heard echoes.
  • In screen sharing use-cases, the system volume of the local user occasionally decreased.
  • In screen sharing use-cases, a black screen appeared when sharing a screen between a landscape monitor and a portrait monitor.
  • In screen sharing use-cases with a window excluded, the application crashed when the specified shared area exceeded the screen resolution.
  • The application failed to exclude a window using the StartScreenCaptureByDisplayId method for screen sharing.
  • In screen sharing use-cases, the screen seen by the remote user occasionally crashed, lagged, or displayed a black screen.
  • In screen sharing use-cases, when the user shared the screen by window, the mouse in the shared screen was not in its actual position.
  • When switching from a non-screen sharing use-case to a screen sharing one, the application occasionally crashed if the user did not switch the resolution accordingly.

Other issues

• All

  • The call GetExtensionProperty failed and returned an empty string.
  • When entering a live streaming room that has been played for a long time as an audience, the time for the first frame to be rendered was shortened.
  • When calling SetVideoEncoderConfigurationEx in the channel to increase the resolution of the video, it occasionally failed.
  • When using the Agora media player to play videos, after you played and paused the video, and then called the seek method to specify a new position for playback, the video image occasionally remained unchanged; if you called the Resume method to resume playback, the video was sometimes played at a speed faster than the original one.
  • Audience members heard buzzing noises when the host switched between speakers and earphones during live streaming.
  • Crashes occurred if you call the RegisterAudioEncodedFrameObserver method after failing to initialize RtcEngine.

• Windows

  • When StopPreview was called to disable the local video preview, the virtual background that had been set up was occasionally invalidated.
  • Crashes occasionally occurred when exiting a channel and joining it multiple times with virtual background enabled and set to blur effect.
  • If the local client used a camera with a resolution set to 1920 × 1080 as the video capture source, the resolution of the remote video was occasionally inconsistent with the local client.
  • When capturing video through the camera, if the video aspect ratio set in CameraCapturerConfiguration was inconsistent with that set in SetVideoEncoderConfiguration, the aspect ratio of the local video preview was not rendered according to the latter setting.

• Android

  • After calling SetCloudProxy to set the cloud proxy, calling JoinChannelEx to join multiple channels failed.
  • In online meeting use-cases, the local user and the remote user occasionally could not hear each other after the local user was interrupted by a call.

• iOS

  • Calling StartAudioMixing to play music files in the ipod-library://item path failed.
  • Different timestamps for audio and video data were obtained simultaneously and separately via OnRecordAudioFrame and OnCaptureVideoFrame callbacks.

• macOS

  • After starting and stopping the audio capture device test, there was no sound when the audio playback device was subsequently started.
  • The OnVideoPublishStateChanged callback reported an inaccurate video source type.
  • When capturing video through the camera, if the video aspect ratio set in CameraCapturerConfiguration was inconsistent with that set in SetVideoEncoderConfiguration, the aspect ratio of the local video preview was not rendered according to the latter setting.

Added

• EnableInEarMonitoring
• SetEarMonitoringAudioFrameParameters
• OnEarMonitoringAudioFrame
• SetInEarMonitoringVolume
• GetEarMonitoringAudioParams
• StartRecordingDeviceTest (Android)
• StopRecordingDeviceTest (Android)
• GetNetworkType
• SetRecordingDeviceVolume (Windows)
• isAudioFilterable in ChannelMediaOptions
• SetDualStreamMode [1/2]
• SetDualStreamMode [2/2]
• SetDualStreamModeEx
• SIMULCAST_STREAM_MODE
• SetLoopbackDevice
• GetLoopbackDevice
• FollowSystemLoopbackDevice
• SetZones
• SetPlayerAttenuation
• SetRemoteAudioAttenuation
• MuteRemoteAudioStream
• SpatialAudioParams
• SetHeadphoneEQPreset
• HEADPHONE_EQUALIZER_PRESET
• EnableMultiCamera (iOS)
• StartSecondaryCameraCapture (iOS)
• StopSecondaryCameraCapture (iOS)
• SetHeadphoneEQParameters
• SetRemoteVideoSubscriptionOptions
• SetRemoteVideoSubscriptionOptionsEx
• LeaveChannelEx [2/2]
• VideoSubscriptionOptions
• MuteLocalAudioStreamEx
• MuteLocalVideoStreamEx
• MuteAllRemoteAudioStreamsEx
• MuteAllRemoteVideoStreamsEx
• StartRtmpStreamWithoutTranscodingEx
• StartRtmpStreamWithTranscodingEx
• UpdateRtmpTranscodingEx
• StopRtmpStreamEx
• StartChannelMediaRelayEx
• UpdateChannelMediaRelayEx
• PauseAllChannelMediaRelayEx
• ResumeAllChannelMediaRelayEx
• StopChannelMediaRelayEx
• followEncodeDimensionRatio in CameraCapturerConfiguration
• hwEncoderAccelerating in LocalVideoStats
• advanceOptions in VideoEncoderConfiguration
• newRoleOptions in OnClientRoleChanged
• AdjustUserPlaybackSignalVolumeEx
• EnableAudioVolumeIndicationEx
• OnVideoDeviceStateChanged (Windows,macOS)
• OnAudioDeviceStateChanged (Windows,macOS)
• OnAudioDeviceVolumeChanged (Windows,macOS)
• SetParameters in IRtcEngine (Windows)

Modified

• EnableDualStreamMode [3/3]

Deprecated

• StartEchoTest [2/3]
• OnApiCallExecuted. Use the callbacks triggered by specific methods instead.

Deleted

• EnableDualStreamMode [2/3]
• Removes deprecated member parameters backgroundImage and watermark in LiveTranscoding class (Android/iOS/macOS)
• Removes RELAY_EVENT_PACKET_UPDATE_DEST_CHANNEL_REFUSED(8) in OnChannelMediaRelayEvent callback

v4.0.0 was released on September 28, 2022.

This release has optimized the implementation of some features, resulting in incompatibility with v3.7.x. The following are the main features with compatibility changes:

• Multiple channel
• Media stream publishing control
• Warning codes

After upgrading the SDK, you need to update the code in your app according to your business use-cases. For details, see Migration guide.

1. Multiple media tracks

This release supports one IRtcEngine instance to collect multiple audio and video sources at the same time and publish them to the remote users by setting IRtcEngineEx and ChannelMediaOptions.

• After calling JoinChannel to join the first channel, call JoinChannelEx multiple times to join multiple channels, and publish the specified stream to different channels through different user ID (localUid) and ChannelMediaOptions settings.
• You can simultaneously publish multiple sets of video streams captured by multiple cameras or screen sharing by setting publishSecondaryCameraTrack and publishSecondaryScreenTrack in ChannelMediaOptions. (Windows)

This release adds CreateCustomVideoTrack method to implement video custom capture. You can refer to the following steps to publish multiple custom captured video in the channel:

1. Create a custom video track: Call the CreateCustomVideoTrack method to create a video track, and get the video track ID.
2. Set the custom video track to be published in the channel: In each channel's ChannelMediaOptions, set the customVideoTrackId parameter to the ID of the video track you want to publish, and set publishCustomVideoTrack to true.
3. Pushing an external video source: Call PushVideoFrame, and specify videoTrackId as the ID of the custom video track in step 2 in order to publish the corresponding custom video source in multiple channels.

You can also experience the following features with the multi-channel capability:

• Publish multiple sets of audio and video streams to the remote users through different user IDs (uid).
• Mix multiple audio streams and publish to the remote users through a user ID (uid).
• Combine multiple video streams and publish them to the remote users through a user ID (uid).

2. Ultra HD resolution (Beta) (Windows, macOS)

In order to improve the interactive video experience, the SDK optimizes the whole process of video capture, encoding, decoding and rendering, and now supports 4K resolution. The improved FEC (Forward Error Correction) algorithm enables adaptive switches according to the frame rate and number of video frame packets, which further reduces the video stuttering rate in 4K scenes.

Additionally, you can set the encoding resolution to 4K (3840 × 2160) and the frame rate to 60 fps when calling SetVideoEncoderConfiguration. The SDK supports automatic fallback to the appropriate resolution and frame rate if your device does not support 4K.

This feature has certain requirements with regards to device performance and network bandwidth, and the supported upstream and downstream frame rates vary on different platforms. To experience this feature, contact support@agora.io.

The increase in the default resolution affects the aggregate resolution and thus the billing rate. See Pricing.

3. Full HD and Ultra HD resolution (Beta) (Android, iOS)

In order to improve the interactive video experience, the SDK optimizes the whole process of video capturing, encoding, decoding and rendering. Starting from this version, it supports Full HD (FHD) and Ultra HD (UHD) video resolutions. You can set the dimensions parameter to 1920 × 1080 or higher resolution when calling the SetVideoEncoderConfiguration method. If your device does not support high resolutions, the SDK will automatically fall back to an appropriate resolution.

The UHD resolution (4K, 60 fps) is currently in beta and requires certain device performance and network bandwidth. If you want to enable this feature, contact technical support.

High resolution typically means higher performance consumption. To avoid a decrease in experience due to insufficient device performance, Agora recommends that you enable FHD and UHD video resolutions on devices with better performance.

The increase in the default resolution affects the aggregate resolution and thus the billing rate. See Pricing.

4. Agora media player

To make it easier for users to integrate the Agora SDK and reduce the SDK's package size, this release introduces the Agora media player. You can then call the methods in the IMediaPlayer class to experience a series of functions:

• Plays local and online media files.
• Preloads media files.
• Changes the CDN route for playing media files according your network conditions.
• Shares the audio and video streams being played with remote users.
• Caches data when playing media files.

5. Ultra-high audio quality

To make the audio clearer and restore more details, this release adds the ULTRA_HIGH_QUALITY_VOICE enumeration. In use-cases that mainly feature the human voice, such as chat or singing, you can call SetVoiceBeautifierPreset and use this enumeration to experience ultra-high audio quality.

6. Spatial audio

NOTE: This feature is in experimental status. To enable this feature, contact Technical Support.

You can set the spatial audio for the remote user as following:

• Local Cartesian Coordinate System Calculation Solution: This solution uses the ILocalSpatialAudioEngine class to implement spatial audio by calculating the spatial coordinates of the remote user. You need to call UpdateSelfPosition and UpdateRemotePosition to update the spatial coordinates of the local and remote users respectively, so that the local user can hear the spatial audio of the remote user.

• Local Cartesian Coordinate System Calculation Solution: This solution uses the ILocalSpatialAudioEngine class to implement spatial audio. You need to call UpdateSelfPosition and UpdatePlayerPositionInfo to update the spatial coordinates of the local user and media player, respectively, so that the local user can hear the spatial audio effect of media player.

7. Real-time chorus

This release gives real-time chorus the following abilities:

• Two or more choruses are supported.
• Each singer is independent of each other. If one singer fails or quits the chorus, the other singers can continue to sing.
• Very low latency experience. Each singer can hear each other in real time, and the audience can also hear each singer in real time.

This release adds the AUDIO_SCENARIO_CHORUS enumeration in AUDIO_SCENARIO_TYPE. With this enumeration, users can experience ultra-low latency in real-time chorus when the network conditions are good.

8. Enhanced channel management

To meet the channel management requirements of various business use-cases, this release adds the following functions to the ChannelMediaOptions structure:

• Sets or switches the publishing of multiple audio and video sources.
• Sets or switches channel profile and user role.
• Sets or switches the stream type of the subscribed video.
• Controls audio publishing delay.

Set ChannelMediaOptions when calling JoinChannel or JoinChannelEx to specify the publishing and subscription behavior of a media stream, for example, whether to publish video streams captured by cameras or screen sharing, and whether to subscribe to the audio and video streams of remote users. After joining the channel, call UpdateChannelMediaOptions to update the settings in ChannelMediaOptions at any time, for example, to switch the published audio and video sources.

9. Screen sharing

This release optimizes the screen sharing function. You can enable this function through the following ways.

On Windows and macOS:

• Call the StartScreenCaptureByDisplayId method before joining a channel, and then call JoinChannel[2/2] to join a channel and set publishScreenTrackor publishSecondaryScreenTrack as true.
• Call the StartScreenCaptureByDisplayId method after joining a channel, and then call UpdateChannelMediaOptions to set publishScreenTrack or publishSecondaryScreenTrack as true.

On Android and iOS:

• Call the StartScreenCapture method before joining a channel, and then call JoinChannel[2/2] to join a channel and set publishScreenCaptureVideo as true.
• Call the StartScreenCapture method after joining a channel, and then call UpdateChannelMediaOptions to set publishScreenCaptureVideo as true.

10. Subscription allowlists and blocklists

This release introduces subscription allowlists and blocklists for remote audio and video streams. You can add the user ID that you want to subscribe to in your allowlist, or in your blocklist if you do not want to subscribe to. You can experience this feature through the following APIs, and in use-cases that involve multiple channels, you can call the following methods in the IRtcEngineEx interface.

• SetSubscribeAudioBlacklist：Set the audio subscription blocklist.
• SetSubscribeAudioWhitelist：Set the audio subscription allowlist.
• SetSubscribeVideoBlacklist：Set the video subscription blocklist.
• SetSubscribeVideoWhitelist：Set the video subscription allowlist.

If a user is added in a blocklist and allowlist at the same time, only the blocklist takes effect.

11. Set audio scenarios

To make it easier to change audio scenarios, this release adds the SetAudioScenario method. For example, if you want to change the audio scenario from AUDIO_SCENARIO_DEFAULT to AUDIO_SCENARIO_GAME_STREAMING when you are in a channel, you can call this method.

12. Replace video feeds with images

This release supports replacing video feeds with images when publishing video streams. You can call the EnableVideoImageSource method to enable this function and choose your own images through the options parameter. If you disable this function, the remote users see the video feeds that you publish.

13. Local video transcoder

This release introduces local video transcoder with which you can locally merge multiple video streams into one. Common use-cases are as follows:

When streaming or using media push, you can merge the frames of multiple anchors into one locally.

To do this, you should merge multi-channel video streams collected locally (such as video captured by camera, screen sharing stream, video files and pictures) into one video stream. Then, release the merged stream in the channel.

You can call the StartLocalVideoTranscoder method to enable the local video transcoder and call the StopLocalVideoTranscoder method to stop the local video transcoder; After the local video transcoder is enabled, you can call UpdateLocalTranscoderConfiguration to update the configuration of the local video transcoder.

14. Video device management

A video capture device may support a variety of video formats, each of which supports a different combination of video frame width, video frame height, and frame rate.

This release introduces the NumberOfCapabilities and GetCapability method to obtain the number of video formats supported by video capture devices and the details of video frames under specified video formats. When you call the StartPrimaryCameraCapture or StartSecondaryCameraCapture method to capture video using camera, you can obtain the video with the specified video format. The SDK will automatically select the video format for video capture equipment according to your settings in VideoEncoderConfiguration. In general, you don't need to use this set of new methods.

1. Fast channel switching

This release can achieve the same switching speed as SwitchChannel in v3.7.x through theLeaveChannel and JoinChannel methods so that you don't need to take the time to call the SwitchChannel method.

2. Push external video frames

This releases supports pushing video frames in I422 format. You can call the PushVideoFrame method to push such video frames to the SDK.

3. Monitor the width and height of videos

This adds the OnTextureSizeModify callback that can be used to monitor any change in the width and height of videos.

4. Voice pitch of the local user This release adds voicePitch in AudioVolumeInfo of OnAudioVolumeIndication. You can use voicePitch to get the local user's voice pitch and perform business functions such as rating for singing.

5. Video preview

This release improves the implementation logic of StartPreview. You can call the StartPreview method to enable video preview at any time.

6. Video types of subscription

You can call the SetRemoteDefaultVideoStreamType method to choose the video stream type when subscribing streams.

• After you enable Notifications, your server receives the events that you subscribe to in the form of HTTPS requests.
• To improve communication security between the Notifications and your server, Agora SD-RTN™ uses signatures for identity verification.
• As of this release, you can use Notifications in conjunction with this product.

Agora charges additionally for this extension. See Pricing.

This release improves the calculation performance of the AI-powered noise suppression algorithm.

This release adds the following APIs and parameters:

• APIs:
  • checkCompatibility: Checks whether the AI Noise Suppression extension is supported on the current browser.
  • setMode: Sets the noise suppression mode as AI noise suppression or stationary noise suppression.
  • setLevel: Sets the AI noise suppression level.
• Parameters:
  • elapsedTime in onoverload: Reports the time in ms that the extension needs to process one audio frame.

For API details, see AI Noise Suppression.

This release brings the following changes:

• AI Noise Suppression supports Agora Video SDK for Web v4.15.0 or later.
• The extension has Wasm dependencies only. Because JS dependencies are removed, you need to publish the Wasm files located in the node_modules/agora-extension-ai-denoiser/external directory again. If you have enabled the Content Security Policy (CSP), you need to modify the CSP configuration. See AI Noise Suppression for details.
• The audio data is dumped in PCM format instead of WAV format.
• To adjust the intensity of noise suppression, best practice is to call setLevel.

First release.

agora-conversational-ai-denoiser v1.0.0-beta2 was released on March 5, 2025, to enhance the user experience in applications powered by Agora Conversational AI Engine.

caution

The agora-conversational-ai-denoiser plugin has high CPU usage. Agora recommend using it only for AI-related use cases.

• Significantly improved voice protection, especially for singing.
• Reduced risk of audio capture and playback stuttering during operation.
• When AI noise suppression is enabled, the default maximum AGC gain is reduced from 24dB to 12dB.

• Seamless Integration.

  • Integrate directly through an HTML <script> tag or through npm using:

    import `VirtualBackgroundExtension` from "agora-extension-virtual-background"
    

  • Works within Agora WebRTC video processing pipelines.

• Support for multiple background types.

  • Blurred Background: Users can blur their actual background with low, medium, or high blur intensity.
  • Image Background: Allows replacing the background with an image (HTMLImageElement).
  • Video Background: Supports setting a video as the virtual background.
  • Solid Color Background: Allows setting a solid color as the background.

• Safari & Firefox – Limited Support

  While the virtual background feature may work on Safari and Firefox, it is not officially supported due to performance limitations. Safari users may experience reduced performance due to browser-specific constraints, while Firefox users may encounter video freezing when minimizing the browser or switching tabs. For optimal performance and reliability, we recommend using the latest version of Chrome on desktop.

• Mobile Browsers - Not optimal

  The virtual background feature requires significant CPU and GPU processing power, which exceeds the capabilities of most mobile devices. Due to these hardware-intensive requirements, performance on mobile browsers is inconsistent and varies across devices.

v2.0.0 was released on December 18, 2024.

This release upgrades the extension algorithm to version 2.0.0. Virtual Background 2.0.0 provides a more distinct face edge and clearer and more stable video images.

• Support for VP9 codec, which provides better video compression and quality.

• Dynamic background switching. Users can change backgrounds without restarting the video stream.
• Ability to disable/re-enable virtual backgrounds dynamically.
• Processor optimization for better real-time segmentation.

Version 2.0.0 is suitable for users who prioritize quality over frame rate especially in use-cases requiring:

• Superior portrait edge detection
• Complex background handling
• Premium video quality

It is recommended for high-end devices such as:

• Apple M1/M2
• Latest iPhones (A13 Bionic and newer)
• Premium Android devices (Snapdragon 865 or equivalent)

Version 1.3.0 is suitable for mid-range devices and is recommended for users who prioritize smooth performance over edge precision such as in standard video conferencing. It is recommended for use cases requiring:

• Higher FPS
• Lower resource consumption
• Consistent performance

Implementation Guidelines:

• For premium applications, implement device detection. Use 2.0.0 for capable devices and default to 1.3.0 for others.

• For General applications, default to 1.3.0. Optionally, allow users to upgrade to 2.0.0 with a performance warning.

v1.2.0 was released on December 10, 2023.

As of this version, the Virtual Background extension incorporates the necessary Wasm module. You no longer need to publish the Wasm file separately, and pass the wasmDir parameter when calling the init method to initialize the extension.

After upgrading to this version, please modify your code accordingly.

This release upgrades the background segmentation algorithm of the extension, optimizing the segmentation effects on the subject, edges, and fingers in complex static and dynamic backgrounds.

This release fixed the issue that checkCompatibility could return inaccurate results on specific devices.

The wasmDir parameter of the init method is now optional.

This release fixes the occasional issue of jagged background images on Chrome for Android.

You can now specify the fit property when calling setOptions. This sets how the background is resized to fit the container. For API details, see Virtual background.

Virtual Background supports Agora Video SDK for Web v4.15.0 or later.

You can now call checkCompatibility and test if AI Noise Suppression extension is supported on the current browser. For API details, see Virtual background.

A black bar is no longer displayed to the left of the virtual background.

You can create multiple VirtualBackgroundProcessor instances to process multiple video streams.

First release.