The Benchmark application allows for measuring throughput, packet error rate, latency, and CPU utilization in different scenarios. The application exposes a CLI interface for efficient test management. Additionally, it supports several Bluetooth features:
Thanks to these features, it is possible to measure the performance impact of the multiprotocol solution.
See Testing multiprotocol scenarios for testing scenarios that use multiprotocol solutions and are based on this Benchmark Application example.
The Benchmark application provides two CLI interfaces:
A Thread application automatically creates or joins a network on channel 11.
In a Zigbee appllication, the creation and joining process must be performed manually. By default, the new Zigbee network is created on channel 16. For both types of applications, CLI commands can be run using a terminal application like PuTTY, configured according to the following CLI terminal settings.
The following table lists the parameters measured by the application.
Parameter | Output unit | Description |
---|---|---|
Throughput | kbps | Average throughput. Measured for a single direction. |
Latency | ms | Average and maximum round trip time. Calculated in two-directional modes (echo and ACK). |
PER | % | Percentage of lost or incorrect app layer frames. Measured for a single direction (TX). |
MAC PER | % | Percentage of lost or incorrect MAC layer frames. Measured for a single direction in the Unidirectional mode (TX) and for two directions (TX sender + TX receiver) in the remaining modes. |
CPU utilization | % | Percentage of CPU utilization. |
The application supports the following modes:
You can use the commands from the following table to configure your tests.
Command | Description |
---|---|
test configure ack-timeout [time-out] | Set the response timeout for ACK and echo mode in milliseconds. This command does not apply for ACK mode in Zigbee, because it uses a regular APS ACK mechanism, for which the timeout value is specified by Zigbee specification. |
test configure count [number] | Set the number of packets sent during the test. |
test configure length [length] | Set the application payload length in bytes sent during the test. Due to the lack of APS fragmentation in the Zigbee stack, the maximum payload length is limited to 79 bytes. |
test configure mode [unidirectional/echo/ack] | Set the test mode. |
test info | Display the current test configuration. |
test peer discover | Discover other boards running the benchmark application within the same Thread or Zigbee network. |
test peer list | List the discovered peers. |
test peer results | Display the results of the last test from the selected peer. |
test peer select [number] | Select the peer to participate in the test. The first discovered peer is selected by default. |
test start | Start the test. The result is printed after it is finished. |
test stop | Stop the test. |
Use the commands from the following table to configure and run the BLE features supported by this example.
Command | Description |
---|---|
ble info | Display the current Bluetooth configuration. |
Use the commands from the following table to configure the BLE advertising during the tests:
Command | Description |
---|---|
ble adv info | Display the current Bluetooth advertising configuration. |
ble adv interval [interval] | Set the advertising interval in ms. |
ble adv start | Start Bluetooth advertising. |
ble adv stop | Stop Bluetooth advertising. |
ble adv window [length] | Set the advertising data length in bytes. |
Use the commands from the following table to configure the BLE scanning during the tests:
Command | Description |
---|---|
ble scan info | Display the current Bluetooth scanning configuration. |
ble scan interval [interval] | Set the scanning interval in ms. |
ble scan start | Start Bluetooth scanning. |
ble scan stop | Stop Bluetooth scanning. |
ble scan window [length] | Set the scan window in ms. |
Use the commands from the following table to configure the BLE connection during the tests:
Command | Description |
---|---|
ble conn [bdaddr] | Get Bluetooth Device Address. |
ble conn dle [value] | Set data length extension value. See the note under the table. |
ble conn info | Display the current Bluetooth connection configuration. |
ble conn interval [interval] | Set the connection interval in ms. |
ble conn slave_latency [latency] | Set the slave latency in number of connection events. |
ble conn start [bdaddr] | Connect to the specified Bluetooth Device Address. |
ble conn stop [bdaddr] | Disconnect from the selected connected device. |
ble conn dle [value]
sets the maximum data length that can be sent in a single packet (L2CAP and ATT headers plus payload). The actual ATT payload that can be sent in a single frame is smaller by L2CAP and ATT header sizes (7 octets). This value must be set before a connection is established for both central and peripheral devices. Maximum value is 251.Use the commands from the following table to send diagnostic packets to the connected device.
Command | Description |
---|---|
ble ping [length] | Send a single burst of random data with the specified size. If no parameters are specified, a default size of 16 bytes is used. The connected device must respond with the ping response. |
Use the commands from the following table to configure and send diagnostic packets continuously to the connected device. The flood commands can only be used by the peripheral device.
Command | Description |
---|---|
ble flood | Start sending the specified number of random bytes continuously. If no parameters are specified, a default size of 16 bytes is used. |
ble flood autostart [enable/disable] | Enable or disable BLE flood autostart on benchmark test start. |
ble flood set [length] | Set flood packet length. |
ble flood stop | Stop sending packets over BLE. Shows real BLE data througput that has been achieved. |
The CPU utilization can be started and stopped multiple times. This allows measuring the average utilization from many non-continuous time periods. The clear
command removes stored data.
Use the commands from the following table to configure the CPU utilization measurement.
Command | Description |
---|---|
cpu_utilization clear | Clear the CPU utilization measurement. |
cpu_utilization deinit | Deinitialize the CPU utilization measurement. |
cpu_utilization get | Get the current CPU utilization measurement. |
cpu_utilization init | Initialize the CPU utilization measurement. |
cpu_utilization start | Start the CPU utilization measurement. |
cpu_utilization stop | Stop the CPU utilization measurement. |
The Thread variant of the Bechmark Application supports OpenThread CLI commands. For the complete list of supported commands, see OpenThread CLI reference.
Execute these commands in the Benchmark Application with the following formula:
You can also issue commands on the discovered peers. The command response is then forwarded to the origin of a remote command. This allows for efficient test management, especially when the test consists of multiple nodes.
Use the following command template to send commands on the discovered peers:
remote [peer_num] [cmd]
– Send command cmd to peer peer_num.Example:
You can find the source code and the makefile in the following folders:
<InstallFolder>/examples/multiprotocol/benchmark/{thread,zigbee}
<InstallFolder>/components/softdevice/s140
You can now start Running benchmark tests using the Thread protocol.
You can enable autostart of Thread network by changing .autostart_disable
from true
to false
in thread_protocol_api.c.
You can now start Running benchmark tests using the Zigbee protocol.
test info
command.test peer discover
command to connect to the second board.test start
command and observe the logs.You will be able to observe results similar to the following output:
=== Test Finished === Test duration: 72912ms Average CPU utilization: Local: 20.32% Remote: 23.50% Without retransmissions: PER: 0.00% Throughput: 43 kbps With retransmissions: PER: 0.00% Throughput: 43 kbps MAC PER: 0.12% Raw data: Config: Length: 79 ACK timeout: 200ms Count: 5000 Mode: ACK Status: Test in progress: False Reset counters: False ACKs lost: 0 Waiting for ACKs: 0 Packets left count: 0 Frame number: 5001 Local: CPU utilization: 20.32% Duration: 72912ms App counters: Bytes received: 0B Packets received: 0 RX error: 0 RX total: 5006 Mac counters: TX error: 5 TX total: 5011 Remote: CPU utilization: 23.50% Duration: 72917ms App counters: Bytes received: 405000B Packets received: 5000 RX error: 0 RX total: 5002 Mac counters: TX error: 8 TX total: 5014 Done