Parse transcription data

Agora uses Protocol Buffers (protobuf) to serialize transcription data. Protobuf, developed by Google, is a language-neutral, platform-independent way to serialize structured data. It enables efficient, consistent data handling across platforms by generating source code in multiple programming languages. Learn more at protobuf.dev.

Understand the tech

Agora Real-Time STT supports transmitting transcription and translation results through the data stream in either Protobuf or JSON format. Protobuf is used by default. If you set enableJsonProtocol to true when calling join, the service uses JSON format instead. This guide explains how to parse both Protobuf and JSON data on the receiving end, and how to extract specific text fields from the parsed data structure.

Prerequisites

To follow this procedure, you must:

• Have a valid Agora Account.

• Have a valid Agora project with an app ID and a temporary token or a token server. For details, see Agora account management.

• Have a computer with access to the internet. If your network has a firewall, follow the steps in Firewall requirements.

• Join an RTC channel as a host and start streaming. Refer to the Voice SDK quickstart guide.

• Enable Real-Time STT for your app.

• Install the Protobuf compiler to generate code classes that process transcription text.

Parse transcription data using Protobuf

This section explains how to use protoc to generate sample code in different languages to deserialize received Protobuf data. If you use JSON format instead, skip directly to JSON protocol data structure to parse the messages.

Follow these steps to write a script that calls the protoc compiler to generate code in different languages.

Create a Protobuf definition file

Protobuf allows you to generate source code in your preferred language based on the structure defined in the .proto file. Agora provides the following Protobuf definition for parsing RTT data. To use the file for generating code:

1. Copy the following Protobuf definition to a local SttMessage.proto file:

   syntax = "proto3";
   
   package Agora.SpeechToText;
   option objc_class_prefix = "Stt";
   option csharp_namespace = "AgoraSTTSample.Protobuf";
   option java_package = "io.agora.rtc.speech2text";
   option java_outer_classname = "AgoraSpeech2TextProtobuffer";
   
   message Text {
     reserved 1 to 3, 5, 7 to 9, 11, 17;
     int64 uid = 4;
     int64 time = 6;
     repeated Word words = 10;
     int32 duration_ms = 12;
     string data_type = 13;
     repeated Translation trans = 14;
     string culture = 15;
     int64 text_ts = 16;
     OriginalTranscript original_transcript = 18;
     int64 sentence_id = 19;
   }
   
   message Word {
     reserved 2, 3, 5;
     string text = 1;
     bool is_final = 4;
   }
   
   message Translation {
     bool is_final = 1;
     string lang = 2;
     repeated string texts = 3;
   }
   
   message OriginalTranscript {
     string culture = 1;
     repeated Word words = 2;
   }

   For a description of each field in the SttMessage.proto file, browse the Reference section.

2. Edit the following properties in your .proto file to match your project:

   • package: The source code package namespace.
   • option: The desired language options.

Generate source code script

#!/bin/sh

# Specify the path to the protoc compiler. In the example code
# The Protobuf version used is 21.12. You can replace it according to your actual needs.
PROTOC_PATH=./protoc-21.12-osx-aarch_64/bin/protoc

# Specify the path to the .proto file.
# The detailed description of the data structure can be found in the reference section.
PROTO_FILE=./SttMessage.proto

# Specify the output directory.
JAVA_OUT_DIR=$(pwd)/code/java

# Create the output directory (if it doesn't exist).
mkdir -p $JAVA_OUT_DIR

# Generate Java code.
$PROTOC_PATH --java_out=$JAVA_OUT_DIR $PROTO_FILE

# Output a message once code generation is finished.
echo "Code generation completed."

#!/bin/sh

# Specify the path to the protoc compiler. In the example code, the Protobuf version used is 21.12. You can replace it according to your actual needs.
PROTOC_PATH=./protoc-21.12-osx-aarch_64/bin/protoc

# Specify the path to the .proto file. The detailed description of the data structure can be found in the reference information.
PROTO_FILE=./SttMessage.proto

# Specify the output directory.
OBJC_OUT_DIR=$(pwd)/code/objective-c

# Create the output directory (if it doesn't exist).
mkdir -p $OBJC_OUT_DIR

# Generate Objective-C code.
$PROTOC_PATH --objc_out=$OBJC_OUT_DIR $PROTO_FILE

# Output a message once code generation is finished.
echo "Code generation completed."

#!/bin/sh

# Path to the protoc compiler
PROTOC_PATH=./protoc-21.12-osx-aarch_64/bin/protoc

# Path to the .proto file
PROTO_FILE=./SttMessage.proto

# Output directory
CSHARP_OUT_DIR=$(pwd)/code/csharp

# Create output directory if it doesn't exist
mkdir -p $CSHARP_OUT_DIR

# Generate C# code
$PROTOC_PATH --csharp_out=$CSHARP_OUT_DIR $PROTO_FILE

echo "C# code generation completed."

Run the script

To generate a Protobuf class, run these commands in your terminal:

# Make the script executable
chmod +x generate_code.sh

# Run the script
./generate_code.sh

Deserialize transcription data

When transcription text is available, your RTC SDK event handler receives the stream message callback. Use the generated Protobuf class to deserialize the received data and convert it back into a data structure or object.

// Join a channel and add callback events
 rtcManager.joinChannel(roomName, localUid, agora_token, roleType.equals(ROLE_TYPE_BROADCAST), new RtcManager.OnChannelListener() {
     ...
     // Callback for receiving a stream message
     @Override
     public void onStreamMessage(int uid, int streamId, byte[] data) {
         // Check if the remote user ID matches the specified streaming bot ID.
         // If so, decode the stream data into a text object.
         if (String.valueOf(uid).equalsIgnoreCase(RTC_UID_STT_STREAM)) {
             AgoraSpeech2TextProtobuffer.Text text = STTManager.getInstance().parseTextByte(roomName, data);
             // Convert the parsed text object to JSON format and print it to the log
             LogUtil.d(originLogName, mGson.toJson(text));
         }
     }
     ...
 });

 public AgoraSpeech2TextProtobuffer.Text parseTextByte(String channel, byte[] data) {
     // Declare a variable of type AgoraSpeech2TextProtobuffer.Text to store the deserialized object
     AgoraSpeech2TextProtobuffer.Text textStream;
     try {
         // Deserialize the byte array data into an AgoraSpeech2TextProtobuffer.Text object
         textStream = AgoraSpeech2TextProtobuffer.Text.parseFrom(data);
     } catch (Exception ex) {
         notifyErrorHandler(new ErrorInfo("parseTextByte", "-1", "parseTextByte parseFrom error >> " + ex.toString()));
         return null;
     }
     ...
 }

Reference

This section contains content that completes the information on this page, or points you to documentation that explains other aspects to this product.

Sample project

Agora provides an open-source Agora-RTT-Demo sample project. You can download it or view its source code.

SttMessage.proto fields

The following tables describe the fields in the SttMessage.proto file.

Text message fields

Word message fields

Translation message fields

Original transcript message fields

JSON protocol data structure

If you set enableJsonProtocol to true when calling join, the service pushes caption data in JSON format instead. The top-level JSON object contains a transcript or translation field:

• transcript: The transcription result.
• translation: The translation result.

The JSON protocol uses results[] to explicitly carry one or more segments. For a single segment, results[] has a length of 1. For a mixed result, a single message can contain both a stabilized prefix segment and a segment that may still change, in which case results[] has a length greater than 1.

transcript fields

transcript.results fields

translation fields

translation.results fields

original_transcript fields

Translate and transcribe examples

This section shows sample output from the STT service for transcribed and translated sentences, in both Protobuf and JSON format.

Transcribe

time: 1753359518654
 words {
   text: "Hello, how are you?"
   is_final: true
 }
 duration_ms: 770
 data_type: "transcribe"
 culture: "en-US"
 text_ts: 1753359520754
 sentence_id: 1753359518654

Translate

time: 1753359518654
duration_ms: 770
data_type: "translate"
trans {
  is_final: true
  lang: "es-ES"
  texts: "Hola, ¿cómo estás? "
}
text_ts: 1753359520754
sentence_id: 1753359518654
original_transcript {
  culture: "en-US"
  words {
    text: "Hello, how are you?"
    is_final: true
  }
}