Record captions

Real-Time STT enables you to record caption text, generate a .vtt file, and store it in the configured cloud storage. These recorded captions are designed for seamless playback alongside audio and video recordings. Additionally, you can use the .vtt file to generate meeting minutes and topic summaries, perform sentiment analysis and content moderation. Caption recording does not incur any additional fees.

This page explains how to record captions and synchronize them with the audio or video files generated by Cloud Recording.

info

Since Cloud Recording and transcription tasks operate on different servers, you must enable the NTP timestamp for Cloud Recording to synchronize the timestamps with the transcription.

To follow this procedure, you must:

• Have a valid Agora Account.

• Have a valid Agora project with an app ID and a temporary token or a token server. For details, see Agora account management.

• Have a computer with access to the internet. If your network has a firewall, follow the steps in Firewall requirements.

• Join a Video SDK channel as a host and start streaming. Refer to the Voice SDK quickstart guide.

• Make sure Real-Time STT is enabled for your app.

• Set the enableNTPtimestamp parameter to true in the body of the Cloud Recording start request to synchronize the timestamps with the transcription.

  Sample Cloud Recording start request body

  
  

  _26

  "recordingConfig": {

  _26

  "maxIdleTime": 30,

  _26

  "streamTypes": 2,

  _26

  "channelType": 1,

  _26

  "extensionParams": {

  _26

  "enableNTPtimestamp": true

  _26

  },

  _26

  "transcodingConfig": {

  _26

  "width": 1280,

  _26

  "height": 720,

  _26

  "fps": 15,

  _26

  "bitrate": 2400,

  _26

  "mixedVideoLayout": 3,

  _26

  "layoutConfig": [

  _26

  {

  _26

  "alpha": 1,

  _26

  "height": 1,

  _26

  "render_mode": 1,

  _26

  "uid": "22228",

  _26

  "width": 1,

  _26

  "x_axis": 0,

  _26

  "y_axis": 0

  _26

  }

  _26

  ]

  _26

  }

  _26

  },

  
  

Follow these steps to record captions and synchronize them with the corresponding recordings for seamless playback.

To record the captions, follow the API call sequence from the REST Quickstart and modify the start request to include caption recording parameters as follows:

_27

curl --location --request POST 'https://api.agora.io/api/speech-to-text/v1/projects/{appId}/join' \

_27

--header 'Content-Type: application/json' \

_27

--header 'Authorization: Basic <credentials>' \

_27

--data '{

_27

"name": "unique-agent-id",

_27

"languages": [

_27

"en-US"

_27

],

_27

"maxIdleTime": 50,

_27

"rtcConfig": {

_27

"channelName": "<YourChannelName>",

_27

"subBotUid": "<YourSubscribeUid>",

_27

"subBotToken": "<YourSubscribeToken>",

_27

"pubBotUid": "<YourPublishUid>",

_27

"pubBotToken": "<PublishToken-IfRequired>"

_27

},

_27

"captionConfig": {

_27

"storage": {

_27

"accessKey": "<YourOssAccessKey>",

_27

"secretKey": "<YourOssSecretKey>",

_27

"bucket": "<YourOssBucketName>",

_27

"vendor": <YourOssVendorCode>,

_27

"region": <YourOssRegionCode>,

_27

"fileNamePrefix": ["folder","sub-folder"]

_27

}

_27

}

_27

}'

The m3u8+vtt file generated by Real-Time STT and the m3u8+ts file generated by Cloud Recording are two independent files, with different time stamps. The Cloud Recording time stamp starts at 0, while Real-Time STT uses the system time stamp. If either process starts abnormally, the media files generated by the two services may be out of sync during playback.

Agora provides a post-processing script that lets you sync the m3u8+ts and m3u8+vtt files. To run this script, take the following steps:

1. Unzip the post-processing script to a local folder.

2. Run the script on your transcription files:

   
   

   _1

   python3 insert_subtitle.py --av audio_dir/audio_ts.m3u8 --subtitle subtitle_dir/subtitle.m3u8 --output output_dir/ --overwrite

   
   

   If ffmpeg/ffprob are not in your PATH, use -ffmpeg_path to specify the path.

3. Play the synchronized files:

   1. Run the following command to start the HTTP server:

      
      

      _1

      python3 -m http.server --bind 127.0.0.1 -d output_dir

      
      

   2. In your browser, enter the following URL:

      
      

      _1

      http://127.0.0.1:8000/player_demo.html

      
      

vendor: Number. The third-party cloud storage vendor. The following are supported:

• 1: Amazon S3
• 5: Microsoft Azure
• 6: Google Cloud

region: Number. The region information specified for the third-party cloud storage. The service only supports regions in the following lists:

• Amazon S3 (vendor = 1):
  • 0: US_EAST_1
  • 1: US_EAST_2
  • 2: US_WEST_1
  • 3: US_WEST_2
  • 4: EU_WEST_1
  • 5: EU_WEST_2
  • 6: EU_WEST_3
  • 7: EU_CENTRAL_1
  • 8: AP_SOUTHEAST_1
  • 9: AP_SOUTHEAST_2
  • 10: AP_NORTHEAST_1
  • 11: AP_NORTHEAST_2
  • 12: SA_EAST_1
  • 13: CA_CENTRAL_1
  • 14: AP_SOUTH_1
  • 15: CN_NORTH_1
  • 16: CN_NORTHWEST_1
  • 17: US_GOV_WEST_1
  • 20: AP_NORTHEAST_3
  • 21: EU_NORTH_1
  • 22: ME_SOUTH_1
  • 23: US_GOV_EAST_1
  • 24: AP_SOUTHEAST_3
  • 25: EU_SOUTH_1
  • 28: IL_CENTRAL_1

• Microsoft Azure (vendor = 5): The region parameter has no effect, whether set or not.
• Google Cloud (vendor = 6): The region parameter has no effect, whether set or not.