Parse transcription data

Agora uses Protocol Buffers (protobuf) to serialize transcription data. Protobuf, developed by Google, is a language-neutral, platform-independent way to serialize structured data. It enables efficient, consistent data handling across platforms by generating source code in multiple programming languages. Learn more at protobuf.dev.

Agora Real-Time STT provides the SttMessage.proto file that defines the message format for speech-to-text conversion. This format serializes transcribed text data into an efficient transmission format, such as binary or JSON, for transmission through the data stream. This guide explains how to generate target language code using the Protobuf Compiler protoc, deserialize the received data stream, and extract specific text fields from the deserialized data structure.

To follow this procedure, you must:

• Have a valid Agora Account.

• Have a valid Agora project with an app ID and a temporary token or a token server. For details, see Agora account management.

• Have a computer with access to the internet. If your network has a firewall, follow the steps in Firewall requirements.

• Join a Video SDK channel as a host and start streaming. Refer to the Voice SDK quickstart guide.

• Enable Real-Time STT for your app.

• Install the Protobuf compiler to generate code classes that process transcription text.

  info

  Since the format of Protobuf may vary across versions, best practice is to ensure that the Protobuf SDK versions used for generated code and client deserialization are consistent.

Follow these steps to write a script that calls the protoc compiler to generate code in different languages.

Protobuf allows you to generate source code in your preferred language based on the structure defined in the .proto file. Agora provides the following Protobuf definition for parsing data. To use the file for generating code:

1. Copy the following Protobuf definition to a local SttMessage.proto file:

   
   

   _37

   syntax = "proto3";

   _37

   _37

   package Agora.SpeechToText;

   _37

   option objc_class_prefix = "Stt";

   _37

   option csharp_namespace = "AgoraSTTSample.Protobuf";

   _37

   option java_package = "io.agora.rtc.speech2text";

   _37

   option java_outer_classname = "AgoraSpeech2TextProtobuffer";

   _37

   _37

   message Text {

   _37

   reserved 1 to 3, 5, 7 to 9, 11, 17;

   _37

   int64 uid = 4;

   _37

   int64 time = 6;

   _37

   repeated Word words = 10;

   _37

   int32 duration_ms = 12;

   _37

   string data_type = 13;

   _37

   repeated Translation trans = 14;

   _37

   string culture = 15;

   _37

   int64 text_ts = 16;

   _37

   OriginalTranscript original_transcript = 18;

   _37

   }

   _37

   _37

   message Word {

   _37

   reserved 2, 3, 5;

   _37

   string text = 1;

   _37

   bool is_final = 4;

   _37

   }

   _37

   _37

   message Translation {

   _37

   bool is_final = 1;

   _37

   string lang = 2;

   _37

   repeated string texts = 3;

   _37

   }

   _37

   _37

   message OriginalTranscript {

   _37

   string culture = 1;

   _37

   repeated Word words = 2;

   _37

   }

   
   

   For a description of each field in the SttMessage.proto file, browse the Reference section.

2. Edit the following properties in your .proto file to match your project:

   • package: The source code package namespace.
   • option: The desired language options.

Java

Create a shell script named generate_code.sh with the following content:

#!/bin/sh# Specify the path to the protoc compiler. In the example code# The Protobuf version used is 21.12. You can replace it according to your actual needs.PROTOC_PATH=./protoc-21.12-osx-aarch_64/bin/protoc# Specify the path to the .proto file. # The detailed description of the data structure can be found in the reference section.PROTO_FILE=./SttMessage.proto# Specify the output directory.JAVA_OUT_DIR=$(pwd)/code/java# Create the output directory (if it doesn't exist).mkdir -p $JAVA_OUT_DIR# Generate Java code.$PROTOC_PATH --java_out=$JAVA_OUT_DIR $PROTO_FILE# Output a message once code generation is finished.echo "Code generation completed."

Objective-C

Ensure Protobuf dependencies are installed. If the dependencies are already installed, skip this step.

Install Protobuf Dependencies

1. Edit your project’s Podfile to add the following line:

   # 3.21.12 indicates the Protobuf version. You can choose the appropriate version according to your actual needs.pod "Protobuf", "3.21.12"

2. Run the following command in the Podfile directory:

   pod install

Open the .xcworkspace file generated in the project folder to proceed in Xcode.

Create a shell script, named generate_code.sh and add the following code to it:

#!/bin/sh# Specify the path to the protoc compiler. In the example code, the Protobuf version used is 21.12. You can replace it according to your actual needs.PROTOC_PATH=./protoc-21.12-osx-aarch_64/bin/protoc# Specify the path to the .proto file. The detailed description of the data structure can be found in the reference information.PROTO_FILE=./SttMessage.proto# Specify the output directory.OBJC_OUT_DIR=$(pwd)/code/objective-c# Create the output directory (if it doesn't exist).mkdir -p $OBJC_OUT_DIR# Generate Objective-C code.$PROTOC_PATH --objc_out=$OBJC_OUT_DIR $PROTO_FILE# Output a message once code generation is finished.echo "Code generation completed."

C#

Create a shell script named generate_code.sh with the following content:

#!/bin/sh# Path to the protoc compilerPROTOC_PATH=./protoc-21.12-osx-aarch_64/bin/protoc# Path to the .proto filePROTO_FILE=./SttMessage.proto# Output directoryCSHARP_OUT_DIR=$(pwd)/code/csharp# Create output directory if it doesn't existmkdir -p $CSHARP_OUT_DIR# Generate C# code$PROTOC_PATH --csharp_out=$CSHARP_OUT_DIR $PROTO_FILEecho "C# code generation completed."

Replace ./SttMessage.proto with the correct path as explained in the Create a Protobuf definition section.

JavaScript

To generate JavaScript code, ensure that the necessary Protobuf dependencies are installed. Follow the steps below to install them.

Install Protobuf Dependencies

1. Open your project’s root directory and edit the package.json file to include the following dependencies:

   {   "dependencies": {     "protobufjs": "^7.2.5"   },   "devDependencies": {     "pbjs": "^0.0.14",     "protobufjs-cli": "^1.1.2"   } }

2. Run the following command to install the dependencies:

   npm install

Next, create a shell script:

1. Create a file named generate_code.sh.
2. Add the following content:

# Add the executable file path of protobufjs-cli to the PATH environment variable# Replace {absolute path of protobufjs-cli in your node_modules}/bin with the absolute path of protobufjs-cli in node_modulesexport "PATH=$PATH:{absolute path of protobufjs-cli in your node_modules}/bin"# Generate JavaScript example codepbjs -t json-module -w es6 ./SttMessage.proto > ./SttMessage_es6.jsecho "JavaScript code generation completed."

Replace ./SttMessage.proto with the path to the file you created in the Create a Protobuf definition file section.

To generate a Protobuf class, run these commands in your terminal:

_5

# Make the script executable

_5

chmod +x generate_code.sh

_5

_5

# Run the script

_5

./generate_code.sh

When transcription text is available, your Video SDK event handler receives the stream message callback. Use the generated Protobuf class to deserialize the received data and convert it back into a data structure or object.

Java

// Join a channel and add callback events rtcManager.joinChannel(roomName, localUid, agora_token, roleType.equals(ROLE_TYPE_BROADCAST), new RtcManager.OnChannelListener() {     ...     // Callback for receiving a stream message     @Override     public void onStreamMessage(int uid, int streamId, byte[] data) {         // Check if the remote user ID matches the specified streaming bot ID.          // If so, decode the stream data into a text object.         if (String.valueOf(uid).equalsIgnoreCase(RTC_UID_STT_STREAM)) {             AgoraSpeech2TextProtobuffer.Text text = STTManager.getInstance().parseTextByte(roomName, data);             // Convert the parsed text object to JSON format and print it to the log             LogUtil.d(originLogName, mGson.toJson(text));         }     }     ... }); public AgoraSpeech2TextProtobuffer.Text parseTextByte(String channel, byte[] data) {     // Declare a variable of type AgoraSpeech2TextProtobuffer.Text to store the deserialized object     AgoraSpeech2TextProtobuffer.Text textStream;     try {         // Deserialize the byte array data into an AgoraSpeech2TextProtobuffer.Text object         textStream = AgoraSpeech2TextProtobuffer.Text.parseFrom(data);     } catch (Exception ex) {         notifyErrorHandler(new ErrorInfo("parseTextByte", "-1", "parseTextByte parseFrom error >> " + ex.toString()));         return null;     }     ... }

C#

private void InitRtcEngine(){       // Create an RTC engine instance    RtcEngine = Agora.Rtc.RtcEngine.CreateAgoraRtcEngine();        // Create an instance of the event handler class    AgoraEventHandler handler = new AgoraEventHandler(this);        // Create the RtcEngineContext object and set the channel profile to live broadcasting    RtcEngineContext context = new RtcEngineContext(_appID, 0,        CHANNEL_PROFILE_TYPE.CHANNEL_PROFILE_LIVE_BROADCASTING,        AUDIO_SCENARIO_TYPE.AUDIO_SCENARIO_DEFAULT);        // Initialize the engine    RtcEngine.Initialize(context);        // Add callback events    RtcEngine.InitEventHandler(handler);}// Define a class to handle RTC-related callbacks, inheriting from IRtcEngineEventHandlerinternal class AgoraEventHandler : IRtcEngineEventHandler{       // Callback for receiving data stream messages    public override void OnStreamMessage(RtcConnection connection, uint remoteUid, int streamId, byte[] data, uint length, ulong sentTs)    {        // Debug.Log(String.Format("remoteUid: {0}", remoteUid));                // If the remote user ID equals the specified streaming bot ID        if (remoteUid == {pusher bot uid}) {            // Parse Protobuf data            AgoraSTTSample.Protobuf.Text t = ProtobufUtility.ParseProtobufData(data);            // ...        }    }}

JavaScript

import AgoraRTC from "agora-rtc-sdk-ng"import protoRoot from "@/protobuf/SttMessage_es6.js"// Create an RTC client instancethis.rtc.client = AgoraRTC.createClient({ mode: "live", codec: "vp8", role: this.role })// Listen for stream message events and bind the event handler functionthis.rtc.client.on("stream-message", this.onStreamMessage.bind(this))// Callback for receiving data stream messagesfunction onStreamMessage(uid, stream) {    // Check if the remote user ID is the specified streaming bot ID; if not, return directly and do not proceed with further processing    if (uid != {pusher bot uid}) {        return    }    // Use Protobuf to decode the received data stream    let textstream = protoRoot.Agora.SpeechToText.lookup("Text").decode(data)    // ...}

Objective-C

// Temp.h#import "AgoraRtcKit/AgoraRtcKit.h"#import "./Protobuff/SttMessage.pbobjc.h"NS_ASSUME_NONNULL_BEGIN@interface Temp : NSObject<AgoraRtcEngineDelegate>@endNS_ASSUME_NONNULL_END// Temp.m@implementation Temp// Callback for receiving data stream messages- (void)rtcEngine:(AgoraRtcEngineKit *)engine receiveStreamMessageFromUid:(NSUInteger)uid streamId:(NSInteger)streamId data:(NSData *)data {    // Check if the remote user ID is the specified streaming bot ID;     // if not, return directly and do not proceed with further processing    if (uid != pusherUid) {        return;    }    NSError* error;    // Decode the received data stream    SttText* st =  [SttText parseFromData: data error: &error];    // ...}@end

Swift

// Callback for receiving data stream messagesfunc rtcEngine(_engine: AgoraRtcEngineKit, receiveStreamMessageFromUid uid: UInt, streamId: Int, data: Data) {    // Check if the remote user ID is the specified streaming bot ID;     // if not, return directly and do not proceed with further processing    guard uid == {pusher bot uid} else {        return    }    // Decode the received data stream    let text = try? SttText.parse(from: data)    // ...}

This section contains content that completes the information on this page, or points you to documentation that explains other aspects to this product.

Agora provides an open-source Agora-RTT-Demo sample project. You can download it or view its source code.

The following tables describe the fields in the SttMessage.proto file.

Field name	Type	Description
uid	int64	The user ID associated with the text.
time	int64	The start time of the transcription segment. Has a value only when isFinal is true; otherwise, the value is 0.
words	repeated	An array of transcription results. See WordMessage type for details.
duration_ms	int32	The duration of the transcribed text in milliseconds.
data_type	string	The type of data: transcribe: Transcription translate: Text translation
trans	repeated	An array of translation results. See TranslationMessage type for details.
culture	string	The source language of the transcription.
text_ts	int64	The continuously incremented timestamp of the transcription result, used to align source and target text during real-time translation.
original_transcript	OriginalTranscript	The transcribed text used for translation.

Field Name	Type	Description
text	string	The transcription result.
is_final	bool	Indicates whether this sentence is the final transcription result. true: The transcription engine has determined the result for this sentence, and no further modifications are expected. This does not mean the sentence is semantically complete. false: The result is not yet final and may change.

Field Name	Type	Meaning
is_final	bool	Indicates whether this sentence is the final translation result. true: The translation engine has determined that the translation result is final and no further modification is required. This does not mean the sentence is semantically complete. false: The translation result is not yet final and may be updated.
lang	string	The target language of the translation.
texts	repeated	The translated text results.

Field Name	Type	Meaning
culture	string	The source language of the transcription.
words	repeated	An array of transcription results.

This section shows sample output from the STT service for transcribed and translated sentences.

Protobuf

time: 1753359518654 words {   text: "Hello, how are you?"   is_final: true } duration_ms: 770 data_type: "transcribe" culture: "en-US" text_ts: 1753359520754

JSON

{    "transcript": {        "uid": 222,        "language": "zh-CN",        "text": "北选手",        "isFinal": false,        "offset": 1751438272384,        "duration": 760,        "textTs": 1751438273939    }}

Protobuf

time: 1753359518654duration_ms: 770data_type: "translate"trans {  is_final: true  lang: "es-ES"  texts: "Hola, ¿cómo estás? "}text_ts: 1753359520754original_transcript {  culture: "en-US"  words {    text: "Hello, how are you?"    is_final: true  }}

JSON

{  "translation": {      "uid": 222,      "isFinal": true,      "offset": 1751438270274,      "duration": 1320,      "textTs": 1751438272825,      "results0": {          "language": "ja-JP",          "texts": [              "4月13日。 "          ]      },      "original_transcript": {          "language": "zh-CN",          "text": "4月13日。"      }    }}