Description

The Benchmark application provides two CLI interfaces:

UART, which uses UART communication and SEGGER J-Link as a USB-UART converter.
RTT, which uses the SEGGER J-Link RTT transport.

A Thread application automatically creates or joins a network on channel 11.

In a Zigbee appllication, the creation and joining process must be performed manually. By default, the new Zigbee network is created on channel 16. For both types of applications, CLI commands can be run using a terminal application like PuTTY, configured according to the following CLI terminal settings.

Measured parameters

The following table lists the parameters measured by the application.

Parameter	Output unit	Description
Throughput	kbps	Average throughput. Measured for a single direction.
Latency	ms	Average and maximum round trip time. Calculated in two-directional modes (echo and ACK).
PER	%	Percentage of lost or incorrect app layer frames. Measured for a single direction (TX).
MAC PER	%	Percentage of lost or incorrect MAC layer frames. Measured for a single direction in the Unidirectional mode (TX) and for two directions (TX sender + TX receiver) in the remaining modes.
CPU utilization	%	Percentage of CPU utilization.

Test modes

The application supports the following modes:

Unidirectional – the application sends packets in one direction and tries to saturate the wireless traffic.

Benchmark Unidirectional test mode
ACK – the application sends a packet and waits for a short acknowledgment from the peer before sending the next packet. If no acknowledgment is received:
- In a Thread application, the ACK timeout value is used for triggering the timeout event of the waiting for acknowledgment state.
- In a Zigbee application, the APS ACK timeout mechanism (9 seconds) is used.
  
  Benchmark ACK test mode
Echo – the application sends a packet and waits for an echo from the peer before sending the next packet. If no echo is received, the ACK time-out value is used for triggering the time-out event of the waiting for the echo state.

Benchmark Echo test mode

Test configuration commands

You can use the commands from the following table to configure your tests.

Command	Description
`test configure ack-timeout [time-out]`	Set the response timeout for ACK and echo mode in milliseconds. This command does not apply for ACK mode in Zigbee, because it uses a regular APS ACK mechanism, for which the timeout value is specified by Zigbee specification.
`test configure count [number]`	Set the number of packets sent during the test.
`test configure length [length]`	Set the application payload length in bytes sent during the test. Due to the lack of APS fragmentation in the Zigbee stack, the maximum payload length is limited to 79 bytes.
`test configure mode [unidirectional/echo/ack]`	Set the test mode.
`test info`	Display the current test configuration.
`test peer discover`	Discover other boards running the benchmark application within the same Thread or Zigbee network.
`test peer list`	List the discovered peers.
`test peer results`	Display the results of the last test from the selected peer.
`test peer select [number]`	Select the peer to participate in the test. The first discovered peer is selected by default.
`test start`	Start the test. The result is printed after it is finished.
`test stop`	Stop the test.

Bluetooth configuration commands

Use the commands from the following table to configure and run the BLE features supported by this example.

Command	Description
`ble info`	Display the current Bluetooth configuration.

Bluetooth advertising configuration

Use the commands from the following table to configure the BLE advertising during the tests:

Command	Description
`ble adv info`	Display the current Bluetooth advertising configuration.
`ble adv interval [interval]`	Set the advertising interval in ms.
`ble adv start`	Start Bluetooth advertising.
`ble adv stop`	Stop Bluetooth advertising.
`ble adv window [length]`	Set the advertising data length in bytes.

Bluetooth scanning configuration

Use the commands from the following table to configure the BLE scanning during the tests:

Command	Description
`ble scan info`	Display the current Bluetooth scanning configuration.
`ble scan interval [interval]`	Set the scanning interval in ms.
`ble scan start`	Start Bluetooth scanning.
`ble scan stop`	Stop Bluetooth scanning.
`ble scan window [length]`	Set the scan window in ms.

Bluetooth connection configuration

Use the commands from the following table to configure the BLE connection during the tests:

Command	Description
`ble conn [bdaddr]`	Get Bluetooth Device Address.
`ble conn dle [value]`	Set data length extension value. See the note under the table.
`ble conn info`	Display the current Bluetooth connection configuration.
`ble conn interval [interval]`	Set the connection interval in ms.
`ble conn slave_latency [latency]`	Set the slave latency in number of connection events.
`ble conn start [bdaddr]`	Connect to the specified Bluetooth Device Address.
`ble conn stop [bdaddr]`	Disconnect from the selected connected device.

Note: The command ble conn dle [value] sets the maximum data length that can be sent in a single packet (L2CAP and ATT headers plus payload). The actual ATT payload that can be sent in a single frame is smaller by L2CAP and ATT header sizes (7 octets). This value must be set before a connection is established for both central and peripheral devices. Maximum value is 251.

Bluetooth ping configuration

Use the commands from the following table to send diagnostic packets to the connected device.

Command	Description
`ble ping [length]`	Send a single burst of random data with the specified size. If no parameters are specified, a default size of 16 bytes is used. The connected device must respond with the ping response.

Bluetooth flood configuration

Use the commands from the following table to configure and send diagnostic packets continuously to the connected device. The flood commands can only be used by the peripheral device.

Command	Description
`ble flood`	Start sending the specified number of random bytes continuously. If no parameters are specified, a default size of 16 bytes is used.
`ble flood autostart [enable/disable]`	Enable or disable BLE flood autostart on benchmark test start.
`ble flood set [length]`	Set flood packet length.
`ble flood stop`	Stop sending packets over BLE. Shows real BLE data througput that has been achieved.

CPU utilization commands

The CPU utilization can be started and stopped multiple times. This allows measuring the average utilization from many non-continuous time periods. The clear command removes stored data.

Use the commands from the following table to configure the CPU utilization measurement.

Command	Description
`cpu_utilization clear`	Clear the CPU utilization measurement.
`cpu_utilization deinit`	Deinitialize the CPU utilization measurement.
`cpu_utilization get`	Get the current CPU utilization measurement.
`cpu_utilization init`	Initialize the CPU utilization measurement.
`cpu_utilization start`	Start the CPU utilization measurement.
`cpu_utilization stop`	Stop the CPU utilization measurement.

OpenThread commands (Thread-only)

The Thread variant of the Bechmark Application supports OpenThread CLI commands. For the complete list of supported commands, see OpenThread CLI reference.

Execute these commands in the Benchmark Application with the following formula:

ot <OpenThread command>

Running commands remotely on discovered peers (Thread-only)

You can also issue commands on the discovered peers. The command response is then forwarded to the origin of a remote command. This allows for efficient test management, especially when the test consists of multiple nodes.

Use the following command template to send commands on the discovered peers:

remote [peer_num] [cmd] – Send command cmd to peer peer_num.

Example:

remote 0 test info

Setup

You can find the source code and the makefile in the following folders:

<InstallFolder>/examples/multiprotocol/benchmark/{thread,zigbee}
<InstallFolder>/components/softdevice/s140

Configuring the boards (Thread-only)

Prepare two nRF52840 Development Kits.
Flash the SoftDevice to both boards according to the instructions in Running examples.
Program the example by using one of the following options:
- Default instructions:
  1. Build and flash the example according to the instructions in Building examples.
- Precompiled hex file:
  1. Run the following commands on both boards to flash the application:
    $ cd <InstallFolder>/examples/multiprotocol/benchmark/thread/hex
    
    $ nrfjprog -f NRF52 --sectorerase --program nrf52840_xxaa_s140_pca10056.hex
Reset the boards.
Start a terminal emulator like PuTTY and connect to the used COM port with the following UART settings:
- Baud rate: 115200
- 8 data bits
- 1 stop bit
- No parity
- HW flow control: RTS/CTS
Run the following commands from the CLI to start Thread stack:
ot ifconfig up

ot thread start

You can now start Running benchmark tests using the Thread protocol.

You can enable autostart of Thread network by changing .autostart_disable from true to false in thread_protocol_api.c.

Configuring the boards (Zigbee-only)

Prepare two nRF52840 Development Kits.
Flash the SoftDevice to both boards according to the instructions in Running examples.
Program the example by using one of the following options:
- Default instructions:
  1. Build and flash the example according to the instructions in Building examples.
- Precompiled hex file:
  1. Run the following commands on both boards to flash the application:
    $ cd <InstallFolder>/examples/multiprotocol/benchmark/zigbee/hex
    
    $ nrfjprog -f NRF52 --sectorerase --program nrf52840_xxaa_s140_pca10056.hex
Reset the boards.
Start a terminal emulator like PuTTY and connect to the used COM port with the following UART settings:
- Baud rate: 115200
- 8 data bits
- 1 stop bit
- No parity
- HW flow control: RTS/CTS
Choose one of the boards and configure it as a Zigbee coordinator:
bdb role zc
Run the following command from the CLI coordinator node to start a new Zigbee network:
bdb start
Connect the second board to the newly created network:
bdb start

You can now start Running benchmark tests using the Zigbee protocol.

Running benchmark tests

Choose one of the boards and run the test info command.
Run the test peer discover command to connect to the second board.
Run the test start command and observe the logs.

You will be able to observe results similar to the following output:

    === Test Finished ===
Test duration: 72912ms

Average CPU utilization:
    Local: 20.32%
    Remote: 23.50%

Without retransmissions:
    PER: 0.00%
    Throughput: 43 kbps

With retransmissions:
    PER: 0.00%
    Throughput: 43 kbps

MAC PER: 0.12%

Raw data:
    Config:
        Length: 79
        ACK timeout: 200ms
        Count: 5000
        Mode: ACK
    Status:
        Test in progress: False
        Reset counters: False
        ACKs lost: 0
        Waiting for ACKs: 0
        Packets left count: 0
        Frame number: 5001
    Local:
        CPU utilization: 20.32%
        Duration: 72912ms
        App counters:
            Bytes received: 0B
            Packets received: 0
            RX error: 0
            RX total: 5006
        Mac counters:
            TX error: 5
            TX total: 5011
    Remote:
        CPU utilization: 23.50%
        Duration: 72917ms
        App counters:
            Bytes received: 405000B
            Packets received: 5000
            RX error: 0
            RX total: 5002
        Mac counters:
            TX error: 8
            TX total: 5014

Done

Table of Contents