Share the Screen

Introduction

Screen sharing enables the host of an interactive live streaming broadcast or video call to display what is on their screen to other users in the channel. This technology has many obvious advantages for communicating information, particularly in the following scenarios:

• During video conferencing, the speaker can share a local image, web page, or full presentation with other participants.
• For online instruction, the teacher can share slides or notes with students.

Implementation

Screen sharing on iOS is implemented by using the native iOS ReplayKit framework in an Extension to record the screen and by treating the screen sharing stream as a user joining the channel. As Apple does not support screen capture in the main app process, you need to create a separate Extension for the screen sharing stream.

Implementations for v3.7.0 and later

This section describes how to implement screen sharing using the iOS SDK v3.7.0 and later.

Before you begin, ensure that you understand how to start a video call or start interactive video streaming. For details, see Start a Video Call or Start Interactive Video Streaming.

1. Ensure that AgoraReplayKitExtension.framework is integrated into the project. This dynamic library wraps the following functionality:

   • Record the screen with Apple ReplayKit.

   • Use the custom video source feature of the SDK to capture the screen recording data and send it to other users in the channel.

2. Create a Broadcast Upload Extension to enable the screen sharing process:

   1. In Xcode, click File > New > Target..., select Broadcast Upload Extension in the pop-up window, and click Next.

      

   2. In the pop-up window, fill in Product Name and other fields, uncheck Include UI Extension, and click Finish. Xcode automatically creates the Extension folder, which contains the SampleHandler.h file.

   3. Modify the SampleHandler.h file to modify the code logic that implements screen sharing:

      • If you only need to use the functionality in AgoraReplayKitExtension.framework provided by Agora, modify it by checking Target for the Extension you created, and setting the Value of the NSExtension > NSExtensionPrincipalClass key in the Info settings from SampleHandler to AgoraReplayKitHandler.

        

      • If you need to add some business logic in addition to using the functionality in AgoraReplayKitExtension.framework provided by Agora, modify it by replacing the code in the SampleHandler.h file with the following code:

          // Objective-C
          #import "SampleHandler.h"
          #import "AgoraReplayKitExt.h"
          #import <sys/time.h>
        
          @interface SampleHandler ()<AgoraReplayKitExtDelegate>
        
          @end
        
          @implementation SampleHandler
        
          - (void)broadcastStartedWithSetupInfo:(NSDictionary<NSString *,NSObject *> *)setupInfo {
              // User has requested to start the broadcast. Setup info from the UI extension can be supplied but is optional.
              [[AgoraReplayKitExt shareInstance] start:self];
        
          }
        
          - (void)broadcastPaused {
              // User has requested to pause the broadcast. Samples are no longer delivered.
              NSLog(@"broadcastPaused");
              [[AgoraReplayKitExt shareInstance] pause];
          }
        
          - (void)broadcastResumed {
              // User has requested to resume the broadcast. Samples delivery resumes.
              NSLog(@"broadcastResumed");
              [[AgoraReplayKitExt shareInstance] resume];
        
          }
        
          - (void)broadcastFinished {
              // User has requested to finish the broadcast.
              NSLog(@"broadcastFinished");
              [[AgoraReplayKitExt shareInstance] stop];
        
          }
        
          - (void)processSampleBuffer:(CMSampleBufferRef)sampleBuffer withType:(RPSampleBufferType)sampleBufferType {
              [[AgoraReplayKitExt shareInstance] pushSampleBuffer:sampleBuffer withType:sampleBufferType];
          }
        
          #pragma mark - AgoraReplayKitExtDelegate
        
          - (void)broadcastFinished:(AgoraReplayKitExt *_Nonnull)broadcast reason:(AgoraReplayKitExtReason)reason {
              switch (reason) {
                  case AgoraReplayKitExtReasonInitiativeStop:
                      {
          //                NSDictionary *userInfo = @{NSLocalizedDescriptionKey : @"Host app stop srceen capture"};
          //                NSError *error = [NSError errorWithDomain:NSCocoaErrorDomain code:0 userInfo:userInfo];
          //                [self finishBroadcastWithError:error];
                          NSLog(@"AgoraReplayKitExtReasonInitiativeStop");
                      }
                      break;
                  case AgoraReplayKitExtReasonConnectFail:
                      {
          //                NSDictionary *userInfo = @{NSLocalizedDescriptionKey : @"Connect host app fail need startScreenCapture in host app"};
          //                NSError *error = [NSError errorWithDomain:NSCocoaErrorDomain code:0 userInfo:userInfo];
          //                [self finishBroadcastWithError:error];
                          NSLog(@"AgoraReplayKitExReasonConnectFail");
                      }
                      break;
        
                  case AgoraReplayKitExtReasonDisconnect:
                      {
          //                NSDictionary *userInfo = @{NSLocalizedDescriptionKey : @"disconnect with host app"};
          //                NSError *error = [NSError errorWithDomain:NSCocoaErrorDomain code:0 userInfo:userInfo];
          //                [self finishBroadcastWithError:error];
                          NSLog(@"AgoraReplayKitExReasonDisconnect");
                      }
                      break;
                  default:
                      break;
              }
          }
        
          @end

3. Call startScreenCapture, and prompt the user's manual action to make the app start screen sharing. You can choose either of the following two methods:

   • Method 1: Prompt the user to tap and hold the Screen Sharing button in the control center of the iOS system, and select the Extension you created to start screen sharing.

   • Method 2: Use RPSystemBroadcastPickerView, which is added by Apple in iOS 12.0, to pop up the "Start screen sharing" button in the app interface. It prompts the user to start screen sharing by tapping this button.

     RPSystemBroadcastPickerView is limited in use and might fail in later versions of iOS systems. Therefore, use the method 2 at your discretion.

Sample project

Agora provides an open-source sample project on GitHub, where you can refer to the following files:

• /APIExample/Examples/Advanced/ScreenShare/ScreenShare.swift
• /Agora-ScreenShare-Extension/SampleHandler.swift

API reference

Some restrictions and cautions exist for using the screen sharing feature, and charges can apply. Agora recommends that you read the following API references before calling the API:

• startScreenCapture
• updateScreenCapture
• stopScreenCapture

Implementations earlier than v3.7.0

This section describes how to implement screen sharing using the iOS SDK earlier than v3.7.0.

Before you begin, ensure that you understand how to start a video call or start interactive video streaming. For details, see Start a Video Call or Start Interactive Video Streaming.

1. Create an Extension for screen sharing

Create a Broadcast Upload Extension with the following steps:

1. Open the project file with Xcode, and select Editor > Add Target... in the menu bar.

2. In the pop-up window, choose Broadcast Upload Extension on the iOS page, and click Next.
   

3. In the Product Name field, enter a name for the Extension, for example, Agora-ScreenShare, and click Finish.
   You should now see a folder for the Extension in your project; this folder holds the code for implementing screen sharing.

   

4. Open the Podfile in the project, and add dependencies to the Extension.

   target 'Agora-ScreenShare-Extension' do
   
     use_frameworks!
   
     pod 'AgoraRtcEngine_iOS', '~> 3.1.1'
   
     # Use the following code to achieve media stream encryption
     # pod 'AgoraRtcEngine_iOS_Crypto', '~> 3.1.1'
   
   end

5. Run the pod install command in the project root directory to install the dependencies.

2. Use the Extension for screen sharing in the project

Add RPSystemBroadcastPickerView to the view of the app as a button to start screen recording.

// Swift
func prepareSystemBroadcaster() {
    if #available(iOS 12.0, *) {
        let frame = CGRect(x: 0, y:0, width: 60, height: 60)
        let systemBroadcastPicker = RPSystemBroadcastPickerView(frame: frame)
        systemBroadcastPicker.autoresizingMask = [.flexibleTopMargin, .flexibleRightMargin]
        if let url = Bundle.main.url(forResource: "Agora-ScreenShare-Extension", withExtension: "appex", subdirectory: "PlugIns") {
            if let bundle = Bundle(url: url) {
                systemBroadcastPicker.preferredExtension = bundle.bundleIdentifier
            }
        }
        broadcasterPickerContainer.addSubview(systemBroadcastPicker)
    } else {
        self.showAlert(message: "Minimum support iOS version is 12.0")
    }

}

3. Set up a custom video source

Use the custom video source function of the Agora SDK to send the system-recorded screen data to remote users for screen sharing.

Create an instance of AgoraRtcEngineKit for the screen sharing stream, and set up the custom video source function.

// Swift
private static let sharedAgoraEngine: AgoraRtcEngineKit = {
    let kit = AgoraRtcEngineKit.sharedEngine(withAppId: KeyCenter.AppId, delegate: nil)

    kit.enableVideo()
    // Use custom video sources
    kit.setExternalVideoSource(true, useTexture: true, pushMode: true)

    // Do not receive video and audio streams from all remote users
    kit.muteAllRemoteVideoStreams(true)
    kit.muteAllRemoteAudioStreams(true)

    return kit
}()

4. Get screen data

Use Apple's native ReplayKit framework for screen recording.

1. In the Info.plist file in the Extension for screen sharing, set the RPBroadcastProcessMode as RPBroadcastProcessModeSampleBuffer.

2. Start screen recording in the broadcastStarted callback.

   The channel name in the code example is set as ScreenShare. If you want the user to enter the channel name, you must use App Group to pass the parameter value of the main process to the Extension.

   // Swift
   override func broadcastStarted(withSetupInfo setupInfo: [String : NSObject]?) {
       AgoraUploader.startBroadcast(to: "ScreenShare")
   
       DispatchQueue.main.async {
           Timer.scheduledTimer(withTimeInterval: 0.1, repeats: true) {[weak self] (timer:Timer) in
               guard let weakSelf = self else {return}
               let elapse = Int64(Date().timeIntervalSince1970 * 1000) - weakSelf.lastSendTs
               print("elapse: \(elapse)")
               // If the inter-frame interval of the video is too long, resend the previous frame.
               if(elapse > 300) {
                   if let buffer = weakSelf.bufferCopy {
                       weakSelf.processSampleBuffer(buffer, with: .video)
                   }
               }
           }
       }
   }

3. Send the system-collected data to the Agora SDK.

   // Swift
   override func processSampleBuffer(_ sampleBuffer: CMSampleBuffer, with sampleBufferType: RPSampleBufferType) {
       DispatchQueue.main.async {[weak self] in
           switch sampleBufferType {
           case .video:
               if let weakSelf = self {
                   weakSelf.bufferCopy = sampleBuffer
                   weakSelf.lastSendTs = Int64(Date().timeIntervalSince1970 * 1000)
               }
               // Send video buffer
               AgoraUploader.sendVideoBuffer(sampleBuffer)
           @unknown default:
               break
           }
       }
   }

   Use pushExternalVideoFrame to enable sendVideoBuffer.

   // Swift
   static func sendVideoBuffer(_ sampleBuffer: CMSampleBuffer) {
       guard let videoFrame = CMSampleBufferGetImageBuffer(sampleBuffer)
               else {
           return
       }
   
       var rotation : Int32 = 0
       if let orientationAttachment = CMGetAttachment(sampleBuffer, key: RPVideoSampleOrientationKey as CFString, attachmentModeOut: nil) as? NSNumber {
           if let orientation = CGImagePropertyOrientation(rawValue: orientationAttachment.uint32Value) {
               switch orientation {
               case .up,    .upMirrored:    rotation = 0
               case .down,  .downMirrored:  rotation = 180
               case .left,  .leftMirrored:  rotation = 90
               case .right, .rightMirrored: rotation = 270
               default:   break
               }
           }
       }
   
       let time = CMTime(seconds: CACurrentMediaTime(), preferredTimescale: 1000)
   
       let frame = AgoraVideoFrame()
       frame.format = 12
       frame.time = time
       frame.textureBuf = videoFrame
       frame.rotation = rotation
       sharedAgoraEngine.pushExternalVideoFrame(frame)
   }

4. Call joinChannelByToken in the AgoraRtcEngineKit instance corresponding to the screen sharing stream to join the channel, and then start screen sharing.

   // Swift
   static func startBroadcast(to channel: String) {
       print("joining \(channel)")
       sharedAgoraEngine.joinChannel(byToken: KeyCenter.Token, channelId: channel, info: nil, uid: SCREEN_SHARE_UID, joinSuccess: nil)
   }

Sample project

Implementing screen sharing using the iOS SDK earlier than v3.7.0 is complicated. To reduce the difficulty of using the sample project, this section describes the files or folders used in the sample project.

Considerations

• If you use Cocoapods, add the following lines to the Podfile file to add dependencies for your screen sharing Extension. Remember to replace ScreenSharing with the target name of your screen sharing Extension.

  target 'ScreenSharing' do
     pod 'AgoraRtcEngine_iOS', '3.7.0'
  end

• The memory limit of a Broadcast Upload Extension is 50 MB. Ensure that your memory usage does not exceed this limit.

• In the screen sharing process, you need to call muteAllRemoteVideoStreams and muteAllRemoteAudioStreams to cancel receiving streams from remote users and avoid double subscriptions.

• For the main process, the screen sharing stream is equivalent to a remote user in the channel. To avoid unnecessary costs, you can exclude the screen sharing stream when setting up the remote user view.

  // Swift
  // Determine if the uid is used for a screen sharing stream
  func isScreenShareUid(uid: UInt) -> Bool {
      return uid >= SCREEN_SHARE_UID_MIN && uid <= SCREEN_SHARE_UID_MAX
  }
  
  // The callback of adding remote users to the channel
  func rtcEngine(_ engine: AgoraRtcEngineKit, didJoinedOfUid uid: UInt, elapsed: Int) {
      LogUtils.log(message: "remote user join: \(uid) \(elapsed)ms", level: .info)
  
      // In the case of a uid for screen sharing, ignore
      if(isScreenShareUid(uid: uid)) {
          LogUtils.log(message: "Ignore screen share uid", level: .info)
          return
      }
  
      // Set up the remote user view
      ......
  }