Screen sharing

Agora screen sharing offers the following advantages:

• Ultra HD quality experience: Supports Ultra HD video (4K resolution, 60 FPS frame rate), giving users a smoother, high-definition, ultimate picture experience.
• Multi-app support: Compatible with many mainstream apps such as WPS Office, Microsoft Office Power Point, Visual Studio Code, Adobe Photoshop, Windows Media Player, and Scratch. This makes it convenient for users to directly share specific apps.
• Multi-device support: Supports multiple devices sharing at the same time. Screen sharing is compatible with Windows 8 systems, devices without independent graphics cards, dual graphics card devices, and external screen devices.
• Multi-platform adaptation: Supports iOS, Android, macOS, Windows, Web, Unity, Flutter, React Native, Unreal Engine, and other platforms.
• High security: Supports sharing only a single app or part of the screen. Also supports blocking specified app windows, effectively ensuring user information security.

This page shows you how to implement screen sharing in your app.

Understand the tech

The screen sharing feature provides the following screen sharing modes for use in various use-cases:

Screen sharing use-cases

• Share the entire screen: Share your entire screen, including all the information on the screen. This feature supports collecting and sharing information from two screens at the same time.
• Share an app window: If you don't want to share the entire screen with other users, you can share only the area within an app window.
• Share a designated screen area: If you only want to share a portion of the screen or app window, you can set a sharing area when starting screen sharing.

Screen sharing modes are available on different platforms as follows:

• Desktop (Windows and macOS): Supports all screen sharing features listed above.

• Mobile (Android and iOS): Only supports sharing the entire screen.

Prerequisites

• Ensure that you have implemented the SDK quickstart in your project.

Set up your project

Since Apple does not support capturing the screen in the main process of the app, you create a separate extension for the screen sharing stream. You use the iOS native ReplayKit framework in the extension to record the screen, and then send the screen sharing stream to the main process.

Screen sharing iOS tech

Take the following steps to set up your screen sharing project:

1. Go to your project folder and open the .xcodeproj file with Xcode.

2. Navigate to File > New > Target..., select Broadcast Upload Extension in the popup window, and then click Next:

   

3. Fill in the Product Name and other information in the popup window, uncheck Include UI Extension, and click Finish. Xcode automatically creates a folder for the extension that contains the SampleHandler.swift files.

4. Under Target, select the newly created extension. Click General and set the iOS version to 12.0 or later under Deployment Info.

5. Modify project settings to implement the screen sharing code logic, and choose one of the following three methods according to your business needs:

   • If you only need to use the functionality in the AgoraReplayKitExtension.xcframework dynamic library provided by Agora, select Target as the extension you just created, and change the Value in Info corresponding to NSExtension > NSExtensionPrincipalClass from SampleHandler to AgoraReplayKitHandler:

     

   • If you need to implement some additional business logic, you can inherit the AgoraReplayKitHandler class. Modify the SampleHandler.swift file as follows:

     import ReplayKit
     import AgoraReplayKitExtension
     
     class SampleHandler: AgoraReplayKitHandler {
       override func broadcastStarted(withSetupInfo setupInfo: [String : NSObject]?) {
         // User request to start broadcasting
         super.broadcastStarted(withSetupInfo: setupInfo)
       }
     }

   • If you want to use the iOS native SampleHandler without inheriting the AgoraReplayKitHandler class, refer to the following code to modify the SampleHandler.swift file:

     import ReplayKit
     import AgoraReplayKitExtension
     
     class SampleHandler: RPBroadcastSampleHandler, AgoraReplayKitExtDelegate {
       override func broadcastStarted(withSetupInfo setupInfo: [String : NSObject]?) {
         // User request to start broadcasting
         AgoraReplayKitExt.shareInstance().start(self)
       }
       override func broadcastPaused() {
         // User request to pause the broadcast
         AgoraReplayKitExt.shareInstance().pause()
       }
     
       override func broadcastResumed() {
         // User request to resume broadcasting
         AgoraReplayKitExt.shareInstance().resume()
       }
     
       override func broadcastFinished() {
         // User request to stop broadcasting
         AgoraReplayKitExt.shareInstance().stop()
       }
     
       override func processSampleBuffer(_ sampleBuffer: CMSampleBuffer, with sampleBufferType: RPSampleBufferType) {
         AgoraReplayKitExt.shareInstance().push(sampleBuffer, with: sampleBufferType)
       }
     
       func broadcastFinished(_ broadcast: AgoraReplayKitExt, reason: AgoraReplayKitExtReason) {
         var tip = ""
         switch reason {
         case AgoraReplayKitExtReasonInitiativeStop:
           tip = "AgoraReplayKitExtReasonInitiativeStop"
     
         case AgoraReplayKitExtReasonConnectFail:
           tip = "AgoraReplayKitExReasonConnectFail"
     
         case AgoraReplayKitExtReasonDisconnect:
           tip = "AgoraReplayKitExReasonDisconnect"
     
         default: break
         }
     
         let error = NSError(domain: NSCocoaErrorDomain,
                   code: 0,
                   userInfo: [NSLocalizedFailureReasonErrorKey: tip])
         self .finishBroadcastWithError(error as Error)
       }
     }

Automatically integrate dynamic libraries

When integrating the Agora RTC SDK through CocoaPods, you need to add the following content in the Podfile file to specify the integration of the screen sharing dynamic library AgoraReplayKitExtension.xcframework. Refer to the following sample code:

platform :ios, '9.0'
target 'Your App' do
# Integration of only base libraries and screen sharing dynamic libraries
# Please replace x.y.z in the following code with the specific SDK version number.
# You can get the latest version number from the release notes.
pod 'AgoraRtcEngine_iOS', 'x.y.z', :subspecs => ['RtcBasic', 'ReplayKit']
end

The screen sharing dynamic library encapsulates screen recording with Apple ReplayKit. Use the SDK capture function to get system recording data and send it to other users in the channel.

Implement screen sharing

This section introduces how to implement screen sharing in your project. The basic API call sequence is shown in the figure below:

Screen share process

Depending on the actual business use-case, you can choose either of the following methods to call the API to implement screen sharing:

• Call startScreenCapture before joining the channel, then call joinChannelByToken [2/4] to join the channel and set publishScreenCaptureVideo to true to start screen sharing.

• Call startScreenCapture after joining the channel, then call updateChannelWithMediaOptions to update the channel media options and set publishScreenCaptureVideo to true to start screen sharing.

Set up the audio scenario

Call the setAudioScenario method and set the audio scenario to AgoraAudioScenarioGameStreaming (high-quality scenario) to improve the success rate of capturing system audio during screen sharing.

Enable screen capture

1. Call startScreenCapture to start screen capture and set the parameters according to your application scenario:

   • captureVideo: Whether to capture system video during screen sharing.
   • captureAudio: Whether to capture system audio during screen sharing.
   • captureSignalVolume: The volume of the captured system audio.
   • dimensions: Resolution of video encoding.
   • frameRate: Video encoding frame rate (FPS).
   • bitrate: Video encoding bitrate (Kbps).
   • contentHint: Content type of screen sharing video.

   // Set parameters for screen capture
   private lazy var screenParams: AgoraScreenCaptureParameters2 = {
     let params = AgoraScreenCaptureParameters2()
     params.captureVideo = true
     params.captureAudio = true
   
     let audioParams = AgoraScreenAudioParameters()
     // Set the captured system volume
     audioParams.captureSignalVolume = 50
     params.audioParams = audioParams
   
     let videoParams = AgoraScreenVideoParameters()
     // Set the resolution of the shared screen
     videoParams.dimensions = screenShareVideoDimension()
     // Set the video encoding frame rate
     videoParams.frameRate = .FPS15
     // Set the video encoding bitrate
     videoParams.bitrate = AgoraVideoBitrateStandard
     params.videoParams = videoParams
   
     return params
   }()
   
   // Enable screen capture
   agoraKit.startScreenCapture(screenParams)

2. Combine this with a manual action by the user to make the app turn on screen sharing. There are two ways to do this:

   1. Prompt the user to long click the screen recording button in the Control Center on iOS and choose to use the extension you created to start recording.

   2. Use Apple's new RPSystemBroadcastPickerView in iOS 12.0 to make the Enable Screen Sharing button pop up on the app interface, prompting the user to click the button to start recording. The sample code is as follows:

      let frame = CGRect(x: 0, y:0, width: 60, height: 60)
      systemBroadcastPicker = RPSystemBroadcastPickerView(frame: frame)
      systemBroadcastPicker?.showsMicrophoneButton = false
      systemBroadcastPicker?.autoresizingMask = [.flexibleTopMargin, .flexibleRightMargin]
      // Get the Bundle Identifier of the main application
      let bundleId = Bundle.main.bundleIdentifier ?? ""
      // Set the Extension Name
      systemBroadcastPicker?.preferredExtension = "\(bundleId).Agora-ScreenShare-Extension"

   RPSystemBroadcastPickerView has some usage limitations and may become invalid in subsequent versions of iOS.

Publish a screen sharing video stream in a channel

After joining the channel, call updateChannelWithMediaOptions to publish the captured screen sharing video stream as follows:

// Define the behavior of rtcEngine when performing screen video capture (state is capturing, sourceType is screen)
func rtcEngine(_ engine: AgoraRtcEngineKit, localVideoStateChangedOf state: AgoraVideoLocalState, error: AgoraLocalVideoStreamError, sourceType: AgoraVideoSourceType) {
  switch (state, sourceType) {
  case (.capturing, .screen):
    // Publish screen captured videos in the channel
    option.publishScreenCaptureVideo = true
    // Publish screen captured videos in the channel
    option.publishScreenCaptureAudio = true
    // Avoid release of camera-captured video
    option.publishCameraTrack = false
    // Update the channel settings with the options set above
    agoraKit.updateChannel(with: option)

  default: break
  }
}

Set up a screen sharing scene (Optional)

Call the setScreenCaptureScenario method to set the screen sharing scenario. Choose the scenarioType that best fits your application from the following:

• AgoraScreenScenarioDocument: Document scene
• AgoraScreenScenarioGaming: Game scene
• AgoraScreenScenarioVideo: Video scene
• AgoraScreenScenarioRDC: Remote control scene

agoraKit.setScreenCaptureScenario(.video)

Update screen sharing (Optional)

If you want to update the screen sharing parameters, such as the video encoding resolution, frame rate, or bitrate, call updateScreenCapture to modify the parameter values.

agoraKit.updateScreenCapture(screenParams)

Stop screen sharing

Call stopScreenCapture to stop screen sharing within the channel.

agoraKit.stopScreenCapture()
// Stop sharing captured video in the channel
option.publishScreenCaptureVideo = false
// Stop sharing captured audio in the channel
option.publishScreenCaptureAudio = false
// Publish camera-captured video in the channel
option.publishCameraTrack = true
agoraKit.updateChannel(with: option)

Limitations

Be aware of the following limitations:

• After turning on screen sharing, Agora uses the resolution of the screen sharing video stream as the billing standard. Please see Pricing for details. The default resolution is 1280 × 720, but you can adjust it according to your business needs.
• Due to system limitations, screen sharing is only supported on iOS 12.0 and above.
• This feature requires a high level of device performance. Agora recommends that you use an iPhone X or above.
• After the user turns on screen sharing on an iOS device, the audio routing automatically switches to the earpiece due to system limitations.
  • If you are using call volume, you can manually switch the audio routing. For example, you can switch the audio routing to speaker as desired.
  • If you are using media volume, you cannot manually switch audio routing due to system limitations.

Reference

This section contains content that completes the information on this page, or points you to documentation that explains other aspects to this product.

Sample project

Agora provides open source sample projects on GitHub for your reference:

• Agora-ScreenShare-Extension: Implements the extension process.
• ScreenShare: Screen sharing in AgoraRtcEngineKit.

API reference

• startScreenCapture
• updateScreenCapture
• stopScreenCapture