# Parse transcription data (/en/realtime-media/speech-to-text/build/process-transcription-data/parse-data)
> For AI agents: see the complete documentation index at [llms.txt](/llms.txt).
Agora uses Protocol Buffers (protobuf) to serialize transcription data. Protobuf, developed by Google, is a language-neutral, platform-independent way to serialize structured data. It enables efficient, consistent data handling across platforms by generating source code in multiple programming languages. Learn more at [protobuf.dev](https://protobuf.dev/).
## Understand the tech [#understand-the-tech]
Agora Real-Time STT provides the `SttMessage.proto` file that defines the message format for speech-to-text conversion. This format serializes transcribed text data into an efficient transmission format, such as binary or JSON, for transmission through the data stream. This guide explains how to generate target language code using the Protobuf Compiler `protoc`, deserialize the received data stream, and extract specific text fields from the deserialized data structure.
## Prerequisites [#prerequisites]
To follow this procedure, you must:
* Have a valid [Agora Account](https://console.agora.io/v2).
* Have a valid Agora project with an app ID and a temporary token or a token
server. For details, see [Agora account management](/en/realtime-media/voice/manage-agora-account).
* Have a computer with access to the internet. If your network has a firewall, follow the steps in [Firewall requirements](../../reference/firewall).
* Join a Video SDK channel as a host and start streaming. Refer to the [Voice SDK quickstart](/en/realtime-media/voice/quickstart) guide.
* Enable Real-Time STT for your app.
* Install the [Protobuf compiler](https://github.com/protocolbuffers/protobuf#protobuf-compiler-installation) to generate code classes that process transcription text.
Since the format of Protobuf may vary across versions, best practice is to ensure that the Protobuf SDK versions used for generated code and client deserialization are consistent.
## Parse transcription data using Protobuf [#parse-transcription-data-using-protobuf]
Follow these steps to write a script that calls the `protoc` compiler to generate code in different languages.
### Create a Protobuf definition file [#create-a-protobuf-definition-file]
Protobuf allows you to generate source code in your preferred language based on the structure defined in the `.proto` file. Agora provides the following Protobuf definition for parsing RTT data. To use the file for generating code:
1. Copy the following Protobuf definition to a local `SttMessage.proto` file:
```js
syntax = "proto3";
package Agora.SpeechToText;
option objc_class_prefix = "Stt";
option csharp_namespace = "AgoraSTTSample.Protobuf";
option java_package = "io.agora.rtc.speech2text";
option java_outer_classname = "AgoraSpeech2TextProtobuffer";
message Text {
reserved 1 to 3, 5, 7 to 9, 11, 17;
int64 uid = 4;
int64 time = 6;
repeated Word words = 10;
int32 duration_ms = 12;
string data_type = 13;
repeated Translation trans = 14;
string culture = 15;
int64 text_ts = 16;
OriginalTranscript original_transcript = 18;
int64 sentence_id = 19;
}
message Word {
reserved 2, 3, 5;
string text = 1;
bool is_final = 4;
}
message Translation {
bool is_final = 1;
string lang = 2;
repeated string texts = 3;
}
message OriginalTranscript {
string culture = 1;
repeated Word words = 2;
}
```
For a description of each field in the `SttMessage.proto` file, browse the [Reference](#reference) section.
2. Edit the following properties in your `.proto` file to match your project:
* `package`: The source code package namespace.
* `option`: The desired language [options](https://protobuf.dev/programming-guides/proto3/#options).
### Generate source code script [#generate-source-code-script]
Java
Objective-C
C#
JavaScript
Create a shell script named `generate_code.sh` with the following content:
```bash
#!/bin/sh
# Specify the path to the protoc compiler. In the example code
# The Protobuf version used is 21.12. You can replace it according to your actual needs.
PROTOC_PATH=./protoc-21.12-osx-aarch_64/bin/protoc
# Specify the path to the .proto file.
# The detailed description of the data structure can be found in the reference section.
PROTO_FILE=./SttMessage.proto
# Specify the output directory.
JAVA_OUT_DIR=$(pwd)/code/java
# Create the output directory (if it doesn't exist).
mkdir -p $JAVA_OUT_DIR
# Generate Java code.
$PROTOC_PATH --java_out=$JAVA_OUT_DIR $PROTO_FILE
# Output a message once code generation is finished.
echo "Code generation completed."
```
Ensure Protobuf dependencies are installed. If the dependencies are already installed, skip this step.
Install Protobuf Dependencies
1. Edit your project’s `Podfile` to add the following line:
```bash
# 3.21.12 indicates the Protobuf version. You can choose the appropriate version according to your actual needs.
pod "Protobuf", "3.21.12"
```
2. Run the following command in the `Podfile` directory:
```bash
pod install
```
Open the `.xcworkspace` file generated in the project folder to proceed in Xcode.
Create a shell script, named `generate_code.sh` and add the following code to it:
```bash
#!/bin/sh
# Specify the path to the protoc compiler. In the example code, the Protobuf version used is 21.12. You can replace it according to your actual needs.
PROTOC_PATH=./protoc-21.12-osx-aarch_64/bin/protoc
# Specify the path to the .proto file. The detailed description of the data structure can be found in the reference information.
PROTO_FILE=./SttMessage.proto
# Specify the output directory.
OBJC_OUT_DIR=$(pwd)/code/objective-c
# Create the output directory (if it doesn't exist).
mkdir -p $OBJC_OUT_DIR
# Generate Objective-C code.
$PROTOC_PATH --objc_out=$OBJC_OUT_DIR $PROTO_FILE
# Output a message once code generation is finished.
echo "Code generation completed."
```
Create a shell script named `generate_code.sh` with the following content:
```bash
#!/bin/sh
# Path to the protoc compiler
PROTOC_PATH=./protoc-21.12-osx-aarch_64/bin/protoc
# Path to the .proto file
PROTO_FILE=./SttMessage.proto
# Output directory
CSHARP_OUT_DIR=$(pwd)/code/csharp
# Create output directory if it doesn't exist
mkdir -p $CSHARP_OUT_DIR
# Generate C# code
$PROTOC_PATH --csharp_out=$CSHARP_OUT_DIR $PROTO_FILE
echo "C# code generation completed."
```
Replace `./SttMessage.proto` with the correct path as explained in the [Create a Protobuf definition](#create-a-protobuf-definition-file) section.
To generate JavaScript code, ensure that the necessary Protobuf dependencies are installed. Follow the steps below to install them.
Install Protobuf Dependencies
1. Open your project’s root directory and edit the `package.json` file to include the following dependencies:
```json
{
"dependencies": {
"protobufjs": "^7.2.5"
},
"devDependencies": {
"pbjs": "^0.0.14",
"protobufjs-cli": "^1.1.2"
}
}
```
2. Run the following command to install the dependencies:
```bash
npm install
```
Next, create a shell script:
1. Create a file named `generate_code.sh`.
2. Add the following content:
```bash
# Add the executable file path of protobufjs-cli to the PATH environment variable
# Replace {absolute path of protobufjs-cli in your node_modules}/bin with the absolute path of protobufjs-cli in node_modules
export "PATH=$PATH:{absolute path of protobufjs-cli in your node_modules}/bin"
# Generate JavaScript example code
pbjs -t json-module -w es6 ./SttMessage.proto > ./SttMessage_es6.js
echo "JavaScript code generation completed."
```
Replace `./SttMessage.proto` with the path to the file you created in the [Create a Protobuf definition file](/en/realtime-media/speech-to-text/build/process-transcription-data/parse-data#create-a-protobuf-definition-file) section.
### Run the script [#run-the-script]
To generate a Protobuf class, run these commands in your terminal:
```bash
# Make the script executable
chmod +x generate_code.sh
# Run the script
./generate_code.sh
```
### Deserialize transcription data [#deserialize-transcription-data]
When transcription text is available, your Video SDK event handler receives the stream message callback. Use the generated Protobuf class to deserialize the received data and convert it back into a data structure or object.
```java
// Join a channel and add callback events
rtcManager.joinChannel(roomName, localUid, agora_token, roleType.equals(ROLE_TYPE_BROADCAST), new RtcManager.OnChannelListener() {
...
// Callback for receiving a stream message
@Override
public void onStreamMessage(int uid, int streamId, byte[] data) {
// Check if the remote user ID matches the specified streaming bot ID.
// If so, decode the stream data into a text object.
if (String.valueOf(uid).equalsIgnoreCase(RTC_UID_STT_STREAM)) {
AgoraSpeech2TextProtobuffer.Text text = STTManager.getInstance().parseTextByte(roomName, data);
// Convert the parsed text object to JSON format and print it to the log
LogUtil.d(originLogName, mGson.toJson(text));
}
}
...
});
public AgoraSpeech2TextProtobuffer.Text parseTextByte(String channel, byte[] data) {
// Declare a variable of type AgoraSpeech2TextProtobuffer.Text to store the deserialized object
AgoraSpeech2TextProtobuffer.Text textStream;
try {
// Deserialize the byte array data into an AgoraSpeech2TextProtobuffer.Text object
textStream = AgoraSpeech2TextProtobuffer.Text.parseFrom(data);
} catch (Exception ex) {
notifyErrorHandler(new ErrorInfo("parseTextByte", "-1", "parseTextByte parseFrom error >> " + ex.toString()));
return null;
}
...
}
```
```csharp
private void InitRtcEngine()
{
// Create an RTC engine instance
RtcEngine = Agora.Rtc.RtcEngine.CreateAgoraRtcEngine();
// Create an instance of the event handler class
AgoraEventHandler handler = new AgoraEventHandler(this);
// Create the RtcEngineContext object and set the channel profile to live broadcasting
RtcEngineContext context = new RtcEngineContext(_appID, 0,
CHANNEL_PROFILE_TYPE.CHANNEL_PROFILE_LIVE_BROADCASTING,
AUDIO_SCENARIO_TYPE.AUDIO_SCENARIO_DEFAULT);
// Initialize the engine
RtcEngine.Initialize(context);
// Add callback events
RtcEngine.InitEventHandler(handler);
}
// Define a class to handle RTC-related callbacks, inheriting from IRtcEngineEventHandler
internal class AgoraEventHandler : IRtcEngineEventHandler
{
// Callback for receiving data stream messages
public override void OnStreamMessage(RtcConnection connection, uint remoteUid, int streamId, byte[] data, uint length, ulong sentTs)
{
// Debug.Log(String.Format("remoteUid: {0}", remoteUid));
// If the remote user ID equals the specified streaming bot ID
if (remoteUid == {pusher bot uid}) {
// Parse Protobuf data
AgoraSTTSample.Protobuf.Text t = ProtobufUtility.ParseProtobufData(data);
// ...
}
}
}
```
```javascript
import AgoraRTC from "agora-rtc-sdk-ng"
// Create an RTC client instance
this.rtc.client = AgoraRTC.createClient({ mode: "live", codec: "vp8", role: this.role })
// Listen for stream message events and bind the event handler function
this.rtc.client.on("stream-message", this.onStreamMessage.bind(this))
// Callback for receiving data stream messages
function onStreamMessage(uid, stream) {
// Check if the remote user ID is the specified streaming bot ID; if not, return directly and do not proceed with further processing
if (uid != {pusher bot uid}) {
return
}
// Use Protobuf to decode the received data stream
let textstream = protoRoot.Agora.SpeechToText.lookup("Text").decode(data)
// ...
}
```
```objc
// Temp.h
#import "AgoraRtcKit/AgoraRtcKit.h"
#import "./Protobuff/SttMessage.pbobjc.h"
NS_ASSUME_NONNULL_BEGIN
@interface Temp : NSObject
@end
NS_ASSUME_NONNULL_END
// Temp.m
@implementation Temp
// Callback for receiving data stream messages
- (void)rtcEngine:(AgoraRtcEngineKit *)engine receiveStreamMessageFromUid:(NSUInteger)uid streamId:(NSInteger)streamId data:(NSData *)data {
// Check if the remote user ID is the specified streaming bot ID;
// if not, return directly and do not proceed with further processing
if (uid != pusherUid) {
return;
}
NSError* error;
// Decode the received data stream
SttText* st = [SttText parseFromData: data error: &error];
// ...
}
@end
```
```swift
// Callback for receiving data stream messages
func rtcEngine(_engine: AgoraRtcEngineKit, receiveStreamMessageFromUid uid: UInt, streamId: Int, data: Data) {
// Check if the remote user ID is the specified streaming bot ID;
// if not, return directly and do not proceed with further processing
guard uid == {pusher bot uid} else {
return
}
// Decode the received data stream
let text = try? SttText.parse(from: data)
// ...
}
```
## Reference [#reference]
This section contains content that completes the information on this page, or points you to documentation that explains other aspects to this product.
### Sample project [#sample-project]
Agora provides an open-source [Agora-RTT-Demo](https://github.com/AgoraIO-Community/Agora-RTT-Demo/tree/main) sample project. You can download it or view its source code.
### `SttMessage.proto` fields [#sttmessageproto-fields]
The following tables describe the fields in the `SttMessage.proto` file.
#### Text message fields [#text-message-fields]
| Field name | Type | Description |
| --------------------- | -------------------- | -------------------------------------------------------------------------------------------------------------------------------------- |
| `uid` | `int64` | The user ID associated with the text. |
| `time` | `int64` | The start time of this sentence segment transcription . Has a value only when `isFinal` is `true`; otherwise, the value is `0`. |
| `words` | `repeated` | An array of transcription results. See **WordMessage** type for details. |
| `duration_ms` | `int32` | The duration of the transcribed text in milliseconds. |
| `data_type` | `string` | The type of data:- `transcribe`: Transcription
- `translate`: Text translation |
| `trans` | `repeated` | An array of translation results. See **TranslationMessage** type for details. |
| `culture` | `string` | The source language of the transcription. |
| `text_ts` | `int64` | The continuously incremented timestamp of the transcription result, used to align source and target text during real-time translation. |
| `original_transcript` | `OriginalTranscript` | The transcribed text used for translation. |
| `sentence_id` | `int64` | A unique identifier for the subtitle in the data stream used for precise alignment between the original and translated subtitles. |
#### Word message fields [#word-message-fields]
| **Field Name** | **Type** | **Description** |
| -------------- | -------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `text` | `string` | The transcription result. |
| `is_final` | `bool` | Indicates whether this sentence is the final transcription result.- `true`: The transcription engine has determined the result for this sentence, and no further modifications are expected. This does not mean the sentence is semantically complete.
- `false`: The result is not yet final and may change. |
#### Translation message fields [#translation-message-fields]
| **Field Name** | **Type** | **Meaning** |
| -------------- | ---------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `is_final` | `bool` | Indicates whether this sentence is the final translation result.- `true`: The translation engine has determined that the translation result is final and no further modification is required. This does **not** mean the sentence is semantically complete.
- `false`: The translation result is not yet final and may be updated. |
| `lang` | `string` | The target language of the translation. |
| `texts` | `repeated` | The translated text results. |
#### Original transcript message fields [#original-transcript-message-fields]
| Field Name | Type | Meaning |
| ---------- | -------- | ----------------------------------------- |
| `culture` | string | The source language of the transcription. |
| `words` | repeated | An array of transcription results. |
### Translate and transcribe examples [#translate-and-transcribe-examples]
This section shows sample output from the STT service for transcribed and translated sentences.
#### Transcribe [#transcribe]
```js
time: 1753359518654
words {
text: "Hello, how are you?"
is_final: true
}
duration_ms: 770
data_type: "transcribe"
culture: "en-US"
text_ts: 1753359520754
sentence_id: 1753359518654
```
```json
{
"transcript": {
"uid": 222,
"language": "zh-CN",
"text": "北选手",
"isFinal": false,
"offset": 1751438272384,
"duration": 760,
"textTs": 1751438273939
}
}
```
#### Translate [#translate]
```js
time: 1753359518654
duration_ms: 770
data_type: "translate"
trans {
is_final: true
lang: "es-ES"
texts: "Hola, ¿cómo estás? "
}
text_ts: 1753359520754
sentence_id: 1753359518654
original_transcript {
culture: "en-US"
words {
text: "Hello, how are you?"
is_final: true
}
}
```
```json
{
"translation": {
"uid": 222,
"isFinal": true,
"offset": 1751438270274,
"duration": 1320,
"textTs": 1751438272825,
"results0": {
"language": "ja-JP",
"texts": [
"4月13日。 "
]
},
"original_transcript": {
"language": "zh-CN",
"text": "4月13日。"
}
}
}
```