Profile performance

Performance profiling helps identify bottlenecks and optimize resource usage in TEN applications. This guide covers how to profile CPU usage and memory consumption in TEN projects using profiling tools and analyze the resulting performance data.

Go projects

TEN supports profiling for Go projects using the following tools:

gperftools: Google Performance Tools for detailed CPU and heap memory profiling
pprof: Go's built-in profiler for CPU and memory analysis

These tools provide detailed performance insights through data collection, analysis, and visualization options to help you optimize your TEN Go applications effectively.

Install gperftools

Set up gperftools and create the required library links on Ubuntu:

apt update && apt install -y google-perftools

ln -s /usr/lib/x86_64-linux-gnu/libtcmalloc.so.4 /usr/lib/libtcmalloc.so

ln -s /usr/lib/x86_64-linux-gnu/libprofiler.so.0 /usr/lib/libprofiler.so

The symbolic links ensure that the profiling libraries are accessible in standard locations for your applications.

Profile CPU usage with gperftools

To profile your program's CPU usage, follow these steps:

Set up environment variables and run your program:
```
export LD_PRELOAD=/usr/lib/libtcmalloc.so
export CPUPROFILE=/path/to/cpu

exec /path/to/program
```
When the program exits, CPU performance data is generated and saved to the /path/to/cpu.0001.prof file.
Generate CPU performance data periodically (optional):

To generate CPU profiling data at regular intervals, set the CPUPROFILE_FREQUENCY environment variable. For example, to collect data every 30 seconds:
```
LD_PRELOAD=/usr/lib/libtcmalloc.so CPUPROFILE=/path/to/cpu CPUPROFILE_FREQUENCY=30 exec /path/to/program
```

Analyze CPU performance data

To analyze CPU performance data, run the following commands:

google-pprof --text /.../program /.../cpu.0001.prof > /.../cpu.0001.prof.txt

google-pprof --text --base=/.../cpu.0001.prof /.../program /.../cpu.0002.prof > /.../diff.txt

Profile heap memory usage with gperftools

To profile your program's heap memory usage, run the following commands:

export LD_PRELOAD=/usr/lib/libtcmalloc.so
export HEAPPROFILE=/.../heap
export HEAP_PROFILE_TIME_INTERVAL=30

exec /.../program

Set the HEAP_PROFILE_TIME_INTERVAL environment variable to periodically generate heap memory profiling data. All data is stored in the directory specified by HEAPPROFILE.

Analyze heap memory performance data

Each generated heap memory performance data file has a .heap extension. To analyze the data, convert the .heap files to a human-readable text format. For example, to convert a .heap file in /.../heap to text format, run the following command:

google-pprof --text /.../program /.../heap.0001.heap > /.../heap.0001.heap.txt

The performance data looks like this:

Total: 19.7 MB
16.0  81.3%  81.3%     16.0  81.3% lws_zalloc
2.9  14.6%  95.9%      3.3   16.7% setAgoraStreamChannelParameters
...

You can also convert the .heap file to other formats such as PDF or SVG. For example, to convert a .heap file in /.../heap to PDF format, run the following command:

google-pprof --pdf /.../program /.../heap.0001.heap > /.../heap.0001.heap.pdf

The PDF output contains the same performance data as the text format but presented graphically.

You can also compare two sets of heap memory performance data. For example, to compare the data in /.../heap.0001.heap and /.../heap.0002.heap, run the following command:

google-pprof --pdf --base=/.../heap.0001.heap /.../program /.../heap.0002.heap > /.../diff.pdf

The PDF shows the differences in heap memory usage between the two data sets.

If you generate heap memory performance data periodically, use the dump_heap_info_to_excel.py script to analyze all heap memory performance data in a directory and generate an Excel file showing the memory usage trends. For example, to analyze all heap memory performance data in /.../heap, run the following command:

python3 tools/profiler/gperftools/dump_heap_info_to_excel.py -heap_dir=/.../heap -bin=/.../program -output=/.../heap.xlsx -sample_interval=30

The script generates raw performance data with symbols in the /.../raw directory, human-readable text performance data in the /.../text directory, and an Excel file in /.../heap.xlsx.

To plot a line chart to display the heap memory usage trends, use:

python3 tools/profiler/gperftools/draw_line_chart.py -excel_file=/.../heap.xlsx -output_file=/.../heap_line_chart.png -title=HEAP_MEM -x=time/s -y=total_heap_size/MB

Profile with pprof

Use pprof to profile programs written in Go. The simplest way to profile a Go program is to use pprof_app_go as your TEN application. When activated, the app reads environment variables to decide whether to activate the pprof server and heap profiler.

export TEN_HEAP_DUMP_DIR=/data/prof
export HEAP_PROFILE_TIME_INTERVAL=30
export TEN_PROFILER_SERVER_PORT=6060

Analyze pprof performance data

To convert heap performance data to a text format, use the following command:

python3 tools/profiler/pprof/dump_heap_files_to_text.py -heap_dir=/data/prof -text_dir=/data/text

To analyze all heap performance data and generate an Excel file, run:

python3 tools/profiler/pprof/dump_heap_info_to_excel.py -heap_dir=/data/prof -output=/data/heap.xlsx

To see a visual representation of Go heap performance data trends, plot a line chart:

python3 tools/profiler/gperftools/draw_line_chart.py -excel_file=/data/heap.xlsx -output_file=/data/heap_line_chart.png -title=HEAP_MEM -x=time/s -y=total_heap_size/MB