Screen sharing

Since Apple does not support capturing the screen in the main process of the app, you create a separate extension for the screen sharing stream. You use the iOS native ReplayKit framework in the extension to record the screen, and then send the screen sharing stream to the main process.

API calling sequence

Take the following steps to set up your screen sharing project:

1. Go to your project folder and open the ios/.xcodeproj file with Xcode.

2. Navigate to File > New > Target..., select Broadcast Upload Extension in the popup window, and then click Next:

   

3. Fill in the Product Name and other information in the popup window, uncheck Include UI Extension, and click Finish. Xcode automatically creates a folder for the extension that contains the SampleHandler.m file.

4. Under Target, select the newly created extension. Click General and set the iOS version to 12.0 or later under Deployment Info. The memory usage of Broadcast Upload Extension is limited to 50 MB. Make sure that the memory usage of Screen Sharing Extension does not exceed 50 MB.

5. Modify project settings to implement the screen sharing code logic, and choose one of the following two methods according to your business needs:

   • If you only need to use the functionality in the AgoraReplayKitExtension.xcframework dynamic library provided by Agora, select Target as the extension you just created, and change the Value in Info corresponding to NSExtension > NSExtensionPrincipalClass from SampleHandler to AgoraReplayKitHandler:

     

   • If you need to implement some additional business logic, refer to the following code to modify the SampleHandler.m file:

     #import "SampleHandler.h"
     
     @implementation SampleHandler
     
     - (void)broadcastStartedWithSetupInfo:(NSDictionary<NSString *,NSObject *> *)setupInfo {
       // User requests to start broadcasting
     }
     
     - (void)broadcastPaused {
       // User requests a pause in broadcasting, screen sharing pauses
     }
     
     - (void)broadcastResumed {
       // User requests to resume broadcasting, screen sharing resumes
     }
     
     - (void)broadcastFinished {
       // User requests to stop broadcasting
     }
     
     - (void)processSampleBuffer:(CMSampleBufferRef)sampleBuffer withType:(RPSampleBufferType)sampleBufferType {
     
       switch (sampleBufferType) {
         case RPSampleBufferTypeVideo:
           break;
         case RPSampleBufferTypeAudioApp:
           break;
         case RPSampleBufferTypeAudioMic:
           break;
     
         default:
           break;
       }
     }
     
     @end

Agora currently supports the following two screen sharing options on the macOS and Windows platforms:

• Share a specific screen or a portion of a specified screen using displayId.
• Share a specific window or a portion of a specified window using windowId.

The API calling sequence is shown in the figure below:

API calling sequence

• Start screen sharing

  1. Set the parameters according to your application use-case and call startScreenCapture to start screen sharing:

     • captureVideo: Whether to capture system video during screen sharing.
     • captureAudio: Whether to capture system audio during screen sharing.
     • captureSignalVolume: The volume of the captured system audio.
     • dimensions: Resolution of video encoding.
     • frameRate: Video encoding frame rate (FPS).
     • bitrate: Video encoding bitrate (Kbps).
     • contentHint: Content type of screen sharing video.

     await rtcEngine.startScreenCapture(
       const ScreenCaptureParameters2(captureAudio: true, captureVideo: true));

  2. Combine the call to startScreenCapture with the user's action to enable screen sharing in your app. To do this, use one of the following methods:

     • Prompt the user to long click the screen recording button in the Control Center on iOS and choose to use the extension you created to start recording.

     • Use Apple's new RPSystemBroadcastPickerView in iOS 12.0 to make the Enable Screen Sharing button pop up on the app interface, prompting the user to click the button to start recording.

• Stop screen sharing

  Call stopScreenCapture to stop screen sharing within the channel.

  await _engine.stopScreenCapture();

When enabling screen sharing on Android, you only need to call the startScreenCapture method. Refer to the screen_sharing.dart file in the Agora RTC SDK to implement screen sharing.

1. Get the screen ID or the window ID

   To get the Display ID or Window ID, call the getScreenCaptureSources method provided in agora_rtc_engine.

   await rtcEngine.getScreenCaptureSources(
       thumbSize: thumbSize, iconSize: iconSize, includeScreen: true);

2. Share a specified screen or window

   To share a specified screen on macOS or Windows using Display ID, refer to the following code:

   await rtcEngine.startScreenCaptureByDisplayId(
     displayId: sourceId!,
     regionRect: const Rectangle(x: 0, y: 0, width: 0, height: 0),
     captureParams: const ScreenCaptureParameters(
      captureMouseCursor: true,
      frameRate: 30,
     ));

   To share a specified window using Window ID on macOS or Windows, use the following code:

   await rtcEngine.startScreenCaptureByWindowId(
    windowId: sourceId!,
    regionRect: const Rectangle(x: 0, y: 0, width: 0, height: 0),
    captureParams: const ScreenCaptureParameters(
     captureMouseCursor: true,
     frameRate: 30,
    ),
   );

3. Join the channel and publish a screen sharing stream

   To publish only the screen sharing stream, refer to the following code:

   await _engine.joinChannelEx(
     token: '',
     connection: RtcConnection(
       channelId: _controller.text, localUid: shareShareUid),
     options: const ChannelMediaOptions(
      autoSubscribeVideo: true,
      autoSubscribeAudio: true,
      publishScreenTrack: true,
      publishSecondaryScreenTrack: true,
      publishCameraTrack: false,
      publishMicrophoneTrack: false,
      publishScreenCaptureAudio: true,
      publishScreenCaptureVideo: true,
      clientRoleType: ClientRoleType.clientRoleBroadcaster,
     ));

   To publish the screen sharing stream and the video stream captured by local camera simultaneously, add the following code to your project:

   await _engine.joinChannelEx(
     token: '',
     connection:
       RtcConnection(channelId: _controller.text, localUid: localUid),
     options: const ChannelMediaOptions(
      publishCameraTrack: true,
      publishMicrophoneTrack: true,
      clientRoleType: ClientRoleType.clientRoleBroadcaster,
     ));
   
   await _engine.joinChannelEx(
     token: '',
     connection: RtcConnection(
       channelId: _controller.text, localUid: shareShareUid),
     options: const ChannelMediaOptions(
      autoSubscribeVideo: true,
      autoSubscribeAudio: true,
      publishScreenTrack: true,
      publishSecondaryScreenTrack: true,
      publishCameraTrack: false,
      publishMicrophoneTrack: false,
      publishScreenCaptureAudio: true,
      publishScreenCaptureVideo: true,
      clientRoleType: ClientRoleType.clientRoleBroadcaster,
     ));

• Before starting screen sharing, call the setAudioScenario method and set the audio scenario to audioScenarioGameStreaming (high sound quality scenario) to improve the success rate of capturing system audio during screen sharing.
• Due to system limitations, screen sharing is only supported on iOS 12.0 and above.
• This feature requires a high level of device performance. Agora recommends that you use an iPhone X or above.

• Before starting screen sharing, call the setAudioScenario method and set the audio scenario to audioScenarioGameStreaming (high sound quality scenario) to improve the success rate of capturing system audio during screen sharing.