# Profile performance (/en/ai/ten-agent/develop/profile-performance)

> For AI agents: see the complete documentation index at [llms.txt](/llms.txt).

Performance profiling helps identify bottlenecks and optimize resource usage in TEN applications. This guide covers how to profile CPU usage and memory consumption in TEN projects using profiling tools and analyze the resulting performance data.

## Go projects [#go-projects]

TEN supports profiling for Go projects using the following tools:

* `gperftools`: Google Performance Tools for detailed CPU and heap memory profiling
* `pprof`: Go's built-in profiler for CPU and memory analysis

These tools provide detailed performance insights through data collection, analysis, and visualization options to help you optimize your TEN Go applications effectively.

### Install gperftools [#install-gperftools]

Set up `gperftools` and create the required library links on Ubuntu:

```bash
apt update && apt install -y google-perftools

ln -s /usr/lib/x86_64-linux-gnu/libtcmalloc.so.4 /usr/lib/libtcmalloc.so

ln -s /usr/lib/x86_64-linux-gnu/libprofiler.so.0 /usr/lib/libprofiler.so
```

The symbolic links ensure that the profiling libraries are accessible in standard locations for your applications.

### Profile CPU usage with gperftools [#profile-cpu-usage-with-gperftools]

To profile your program's CPU usage, follow these steps:

1. **Set up environment variables and run your program:**

   ```bash
   export LD_PRELOAD=/usr/lib/libtcmalloc.so
   export CPUPROFILE=/path/to/cpu

   exec /path/to/program
   ```

   When the program exits, CPU performance data is generated and saved to the `/path/to/cpu.0001.prof` file.

2. **Generate CPU performance data periodically (optional):**

   To generate CPU profiling data at regular intervals, set the `CPUPROFILE_FREQUENCY` environment variable. For example, to collect data every 30 seconds:

   ```bash
   LD_PRELOAD=/usr/lib/libtcmalloc.so CPUPROFILE=/path/to/cpu CPUPROFILE_FREQUENCY=30 exec /path/to/program
   ```

#### Analyze CPU performance data [#analyze-cpu-performance-data]

To analyze CPU performance data, run the following commands:

```bash
google-pprof --text /.../program /.../cpu.0001.prof > /.../cpu.0001.prof.txt

google-pprof --text --base=/.../cpu.0001.prof /.../program /.../cpu.0002.prof > /.../diff.txt
```

### Profile heap memory usage with gperftools [#profile-heap-memory-usage-with-gperftools]

To profile your program's heap memory usage, run the following commands:

```bash
export LD_PRELOAD=/usr/lib/libtcmalloc.so
export HEAPPROFILE=/.../heap
export HEAP_PROFILE_TIME_INTERVAL=30

exec /.../program
```

Set the `HEAP_PROFILE_TIME_INTERVAL` environment variable to periodically generate heap memory profiling data. All data is stored in the directory specified by `HEAPPROFILE`.

#### Analyze heap memory performance data [#analyze-heap-memory-performance-data]

Each generated heap memory performance data file has a `.heap` extension. To analyze the data, convert the `.heap` files to a human-readable text format. For example, to convert a `.heap` file in `/.../heap` to text format, run the following command:

```bash
google-pprof --text /.../program /.../heap.0001.heap > /.../heap.0001.heap.txt
```

The performance data looks like this:

```text
Total: 19.7 MB
16.0  81.3%  81.3%     16.0  81.3% lws_zalloc
2.9  14.6%  95.9%      3.3   16.7% setAgoraStreamChannelParameters
...
```

You can also convert the `.heap` file to other formats such as PDF or SVG. For example, to convert a `.heap` file in `/.../heap` to PDF format, run the following command:

```bash
google-pprof --pdf /.../program /.../heap.0001.heap > /.../heap.0001.heap.pdf
```

The PDF output contains the same performance data as the text format but presented graphically.

You can also compare two sets of heap memory performance data. For example, to compare the data in `/.../heap.0001.heap` and `/.../heap.0002.heap`, run the following command:

```bash
google-pprof --pdf --base=/.../heap.0001.heap /.../program /.../heap.0002.heap > /.../diff.pdf
```

The PDF shows the differences in heap memory usage between the two data sets.

If you generate heap memory performance data periodically, use the `dump_heap_info_to_excel.py` script to analyze all heap memory performance data in a directory and generate an Excel file showing the memory usage trends. For example, to analyze all heap memory performance data in `/.../heap`, run the following command:

```bash
python3 tools/profiler/gperftools/dump_heap_info_to_excel.py -heap_dir=/.../heap -bin=/.../program -output=/.../heap.xlsx -sample_interval=30
```

The script generates raw performance data with symbols in the `/.../raw` directory, human-readable text performance data in the `/.../text` directory, and an Excel file in `/.../heap.xlsx`.

To plot a line chart to display the heap memory usage trends, use:

```bash
python3 tools/profiler/gperftools/draw_line_chart.py -excel_file=/.../heap.xlsx -output_file=/.../heap_line_chart.png -title=HEAP_MEM -x=time/s -y=total_heap_size/MB
```

### Profile with pprof [#profile-with-pprof]

Use `pprof` to profile programs written in Go. The simplest way to profile a Go program is to use `pprof_app_go` as your TEN application. When activated, the app reads environment variables to decide whether to activate the `pprof` server and heap profiler.

```bash
export TEN_HEAP_DUMP_DIR=/data/prof
export HEAP_PROFILE_TIME_INTERVAL=30
export TEN_PROFILER_SERVER_PORT=6060
```

### Analyze pprof performance data [#analyze-pprof-performance-data]

To convert heap performance data to a text format, use the following command:

```bash
python3 tools/profiler/pprof/dump_heap_files_to_text.py -heap_dir=/data/prof -text_dir=/data/text
```

To analyze all heap performance data and generate an Excel file, run:

```bash
python3 tools/profiler/pprof/dump_heap_info_to_excel.py -heap_dir=/data/prof -output=/data/heap.xlsx
```

To see a visual representation of Go heap performance data trends, plot a line chart:

```bash
python3 tools/profiler/gperftools/draw_line_chart.py -excel_file=/data/heap.xlsx -output_file=/data/heap_line_chart.png -title=HEAP_MEM -x=time/s -y=total_heap_size/MB
```