Test extensions and applications

Testing ensures the reliability and correctness of TEN framework applications and extensions. This guide covers comprehensive testing approaches for both TEN framework developers and users building applications with the framework.

The TEN framework supports multiple testing strategies depending on your role and testing needs:

For framework developers:

• Unit tests: Test individual components and functions using gtest
• Smoke tests: Verify basic functionality and system stability using gtest
• Integration tests: Validate end-to-end scenarios using pytest

For application developers:

• Standalone extension testing: Test individual extensions in isolation without requiring full TEN app setup

Each testing approach provides specific tools and workflows to help you validate functionality, catch regressions, and ensure your TEN applications work correctly across different scenarios and environments.

Unit tests are located in tests/ten_runtime/unit. To add new unit test cases, place them in this directory.

After the build completes, navigate to the output directory for your target platform such as out/linux/x64/tests/standalone and run:

_1

./ten_runtime_unit_test

Smoke tests are located in tests/ten_runtime/smoke. To add new test cases, place them in this directory.

After the build completes, navigate to the output directory for your target platform such as out/linux/x64/tests/standalone and run:

_1

./ten_runtime_smoke_test

Run multiple rounds of testing to catch intermittent issues and verify stability. These commands execute tests 100 times and stop on the first failure.

• Unit tests only:

  
  

  _1

  failed=0; for i in {1..100}; do if [ ${failed} != 0 ]; then echo "error occurred:" ${failed}; break; fi; ./ten_runtime_unit_test; failed=$?; done;

  
  

• Smoke tests only:

  
  

  _1

  failed=0; for i in {1..100}; do if [ ${failed} != 0 ]; then echo "error occurred:" ${failed}; break; fi; ./ten_runtime_smoke_test; failed=$?; done;

  
  

• Both unit and smoke tests:

  
  

  _1

  failed=0; for i in {1..100}; do if [ ${failed} != 0 ]; then echo "error occurred:" ${failed}; break; fi; ./ten_runtime_unit_test; failed=$?; if [ ${failed} != 0 ]; then echo "error occurred:" ${failed}; break; fi; ./ten_runtime_smoke_test; failed=$?; done;

  
  

Integration tests validate real-world execution scenarios using black-box testing approaches. Tests are located in tests/ten_runtime/integration with each test case in its own directory:

_8

tests/ten_runtime/integration/

_8

├── test_1/

_8

│ ├── test_case.py

_8

│ └── ...

_8

├── test_2/

_8

│ ├── test_case.py

_8

│ └── ...

_8

└── ...

To execute the integration tests, navigate to your output directory and run:

_2

cd path/to/out/

_2

pytest tests/ten_runtime/integration/

Slow down TEN execution to expose timing-related bugs and race conditions:

_2

sudo apt install util-linux

_2

cpulimit -f -l 50 -- taskset 0x3 ...

CPU core options:

• taskset 0x1: Limits execution to 1 CPU core
• taskset 0x3: Limits execution to 2 CPU cores

This technique helps identify concurrency issues that may not appear under normal execution speeds.

The TEN framework provides a standalone extension testing mechanism that allows you to test individual extensions without relying on other TEN components, such as other extensions, graphs, or apps. This is especially useful when you need to verify an extension's behavior without running the entire TEN app.

The standalone testing framework follows three key principles:

• Works with any native testing framework used by the extension's language
  For example, if an extension is written in C++, you can use the Google gtest/gmock framework together with the standalone testing framework to test it independently.

• Requires no changes to the extension code under test
  You can test the exact same code used at runtime without modification.

• Provides a consistent design and usage model across languages
  Once you learn the testing workflow for one language, you can apply the same concepts to test extensions written in other languages.

The TEN extension standalone testing framework introduces two main concepts:

• extension_tester

  Serves as a testing driver, responsible for setting up and executing the entire testing process.

• ten_env_tester

  Acts like a typical TEN extension's ten_env instance, enabling you to invoke functionalities within the standalone testing framework such as sending messages to and receiving messages from the extension under test.

The API design of extension_tester and ten_env_tester mirrors that of TEN extension and ten_env, which is intentional. This design helps users familiar with extension development quickly adapt to the standalone testing framework and efficiently develop test cases for their extensions.

However, certain APIs and features are dedicated exclusively to testing. To prevent these test-specific functionalities from affecting the runtime API set, the standalone testing framework introduces types and APIs exclusive to testing. This separation ensures that runtime and testing APIs remain independent, avoiding potential conflicts.

The standalone testing framework automatically starts a test app and creates a graph with two extensions:

• Extension under test: The extension you want to validate
• Testing proxy extension: Handles message routing and test interactions

All input and output messages from the extension under test are redirected to the testing proxy, allowing you to customize inputs and verify outputs during the testing process.

The testing proxy acts as an intermediary between your extension and the test framework, facilitating message exchanges using the TEN environment's standard communication patterns.

The standalone testing workflow follows these steps:

1. Create an extension tester to manage the testing process
2. Set the testing mode, such as single extension testing
3. Start the test execution

• C++

  Following is an example of TEN extension standalone testing using Google gtest:

  
  

  _23

  class `extension_tester`_basic : public ten::extension_tester_t {

  _23

  public:

  _23

  void on_start(ten::ten_env_tester_t &ten_env) override {

  _23

  auto new_cmd = ten::cmd_t::create("hello_world");

  _23

  ten_env.send_cmd(std::move(new_cmd),

  _23

  [](ten::ten_env_tester_t &ten_env,

  _23

  std::unique_ptr<ten::cmd_result_t> result) {

  _23

  if (result->get_status_code() == TEN_STATUS_CODE_OK) {

  _23

  ten_env.stop_test();

  _23

  }

  _23

  });

  _23

  }

  _23

  };

  _23

  _23

  TEST(Test, Basic) {

  _23

  // 1. Create an extension tester to manage the entire standalone testing process.

  _23

  auto *tester = new extension_tester_basic();

  _23

  // 2. Set a testing mode, such as a mode for testing a single extension.

  _23

  tester->set_test_mode_single("default_extension_cpp");

  _23

  // 3. Start the testing.

  _23

  tester->run();

  _23

  delete tester;

  _23

  }

  
  

• Python

  
  

  _25

  class ExtensionTesterBasic(ExtensionTester):

  _25

  def check_hello(self, ten_env: TenEnvTester, result: CmdResult):

  _25

  statusCode = result.get_status_code()

  _25

  print("receive hello_world, status:" + str(statusCode))

  _25

  if statusCode == StatusCode.OK:

  _25

  ten_env.stop_test()

  _25

  _25

  def on_start(self, ten_env: TenEnvTester) -> None:

  _25

  new_cmd = Cmd.create("hello_world")

  _25

  print("send hello_world")

  _25

  ten_env.send_cmd(

  _25

  new_cmd,

  _25

  lambda ten_env, result: self.check_hello(ten_env, result),

  _25

  )

  _25

  print("tester on_start_done")

  _25

  ten_env.on_start_done()

  _25

  _25

  _25

  def test_basic():

  _25

  # 1. Create an extension tester to manage the entire standalone testing process.

  _25

  tester = ExtensionTesterBasic()

  _25

  # 2. Set a testing mode, such as a mode for testing a single extension.

  _25

  tester.set_test_mode_single("default_extension_python")

  _25

  # 3. Start the testing.

  _25

  tester.run()