Router Charts

Click for Router Charts

Router Ranker

Click for Router Ranker

NAS Charts

Click for NAS Charts

NAS Ranker

Click for NAS Ranker

More Tools

Click for More Tools

Wireless How To

Introduction

Since mesh / distributed Wi-Fi systems are here to stay, it's time to get a better test process in place for them. So we're replacing the open air method used in the Mesh Mashup and other previous reviews of Wi-Fi systems. The new process can handle products containing up to three nodes and can also test router / extender designs like NETGEAR's Orbi.

The process consists of three sets of benchmarks:

  • Wired routing
  • Wi-Fi Throughput vs. attenuation
  • Wi-Fi System

Routing Performance

We run three benchmarks from the Version 10 router process. Please refer to that document for details.

  • 30 second Routing Throughput test (using iperf3)
  • HTTP Score
  • Bufferbloat Score

Wi-Fi Throughput vs. Attenuation

Wi-Fi Systems consist of multiple copies of essentially the same hardware that act as routers or access points. So we first run our Version 10 Throughput vs. Attenuation test process on one of the Wi-Fi System modules. In extender-type systems, we run this test on the router module. Please refer to the Version 10 Wireless Test Process document for details.

Wi-Fi System Test

These tests assess the performance of Wi-Fi Systems as systems. They require the new test configuration shown below. Each mesh node (AP) is placed in an RF-tight octoScope Box-18. Three Box-18's are connected by five octoScope quadAttenuators and 4x4 MIMO 1:2 or 1:4 RF splitters.

Three of the quadAttens connect the Box-18's to an octoScope Pal-24, Pal-5 and Pal-245 via an octoScope 1:4 splitter. Note all RF connections contain four separate signal paths. Although only two of those paths are used for the test STA, Wi-Fi Systems, NETGEAR's Orbi specifically, may use up to four streams, which the testbed accommodates.

The Pals are octoScope's partner devices that can function in AP, STA or Monitor mode and support up to 32 virtual STAs. Each is based on a Qualcomm Atheros chipset, has an onboard processor and runs both an iperf3 server and IxChariot endpoint.

The Pal-245 is a selectable dual-band 3x3 device that supports up to three transmit / receive chains, 20, 40 and 80 MHz bandwidth and 256 QAM on both bands. It does not support MU-MIMO or 80+80 or 160 MHz bandwidth.

The Pal-24 and Pal-5 are 2.4 and 5 GHz 4x4 devices, respectively, that support up to four transmit / receive chains, 20, 40, 80, 80+80 and 160 MHz channel bandwidth and 256 QAM. Pal-5 also supports 1x1 and 2x2 MU-MIMO.

Wi-Fi system test configuration

Wi-Fi system test configuration

The Pal-245 is our test client (STA) for most tests. The quadAttens set path loss to simulate distance between the STA and each AP. The other two quadAttens set the box-to-box path loss to simulate distance between APs.

Box-18 Root has two gigabit Ethernet connections. One is used to connect the root mesh node to the internet, which most mesh products require for connection to their companion cloud service for proper operation. The second Ethernet connection goes to the root mesh node's LAN connection, which is where the test traffic generator is connected.

The dashed lines going to the other mesh nodes indicate optional connections for their LAN ports, for backhaul / wireless bridge throughput tests.

The smaller Box-18's make it possible to fit a lot of test flexibility into the same space occupied by our previous testbed. The new stack, shown below, handles throughput vs. attenuation (rate vs. range) testing in the top half and Wi-Fi System testing in the bottom.

SmallNetBuilder Wireless Testbed

SmallNetBuilder Wireless Testbed

The octoScope TS-MESH module sits with the Box-18's and holds the quadAttens, Pal and PoE gigabit switches used to provide control and power to them. It also contains the splitters that provide RF connections.

The inside of the Box-18 is pretty tight, so unity-gain dual-band dipole antennas are used, mounted as shown below. For low-profile products, the antennas are angled at 45 degrees as shown; for taller products, they are set straight up. This setup provides typical beacon RSSIs reported by the Pal devices of -30 dBm for 2.4 GHz and -45 dBm for 5 GHz with attenuators set to 0 dB.

Box 18 inside

Box 18 inside

Wi-Fi System Performance

Note: "Downlink" means data flows from mesh node (AP) to client device. "Uplink" means data flows from client device to mesh node.

Multi-node Wi-Fi systems are generally designed to be in control and provide few user-settable controls. These products use a single SSID and usually don't allow channels, bandwidth or any other Wi-Fi parameters to be set, other than the network name (SSID) and encryption passphrase. So our process lets each system run as it normally would, including having a live internet connection during testing. All test traffic is run on the LAN side, however, so router throughput does not affect results.

Products are installed using their companion app or web interface. Attenuators are first set as shown in the diagram below. The root node is then installed in the Root Box-18, with the door open so that it can communicate with the smartphone running its app via Bluetooth. Once the root node is connected, the chamber door is closed. The other nodes are added, one at a time, following the same process. If the app prompts for a firmware update, it is run until the app says all nodes are up to date. No other product settings are changed for testing.

Wi-Fi System test diagram

Wi-Fi System test diagram

iperf3 is used for each throughput test with default parameters, except window size is set to 2 MB and two test streams are used. Each test is run for 90 seconds, with the first 5 seconds treated as "training" period. The "training" period data is not included in the calculated average for the rest of the test run.

The goal of the process is to measure best-case average up and downlink throughput for both bands on each node. The general test process is:

  1. Associate Pal client device to the root node 2.4 GHz radio, run down and uplink throughput tests
  2. Associate Pal client device to the root node 5 GHz radio, run down and uplink throughput tests
  3. Repeat for the one-hop and two-hop test nodes.

Attenuators 1 and 2 provide the RF paths so the Root, Hop 1 and Hop 2 nodes can form their backhaul links. The attenuation values shown are relatively low, because the path loss provided by air gap between the node and chamber antennas provides most of the loss. Path loss from the Hop 1 node to Root and Hop 2 nodes has been measured to be typically 40 dB in 2.4 GHz and 65 dB in 5 GHz using the 3 dB values shown for Attenuators 1 and 2. The goal here is to allow the branch nodes to have a signal level high enough to not cause the product to indicate that a node needs to be moved.

Testing is controlled by a script that executes the following test plan (reference above diagram):

  1. Set the attenuators as shown in diagram above
  2. Associate Pal to Root node device on 2.4 GHz. Run downlink throughput test. Run uplink throughput test
  3. Associate Pal to Root node device on 5 GHz. Run downlink throughput test. Run uplink throughput test
  4. Disassociate Pal from Root node (set Atten 5 to 63 dB). Move Pal to Hop 1 device (set Atten 4 to 0 dB).
  5. Repeat Steps 2 and 3
  6. Disassociate Pal from Hop 1 node (set Atten 4 to 63 dB). Move Pal to Hop 2 device (set Atten 3 to 0 dB).
  7. Repeat Steps 2 and 3

Results are entered into four benchmarks: 2.4 GHz Downlink throughput; 2.4 GHz Uplink throughput; 5 GHz Downlink throughput; 5 GHz Uplink throughput. Each benchmark contains the values for Root, Hop 1 and Hop 2 nodes.

Wi-Fi System Performance - Extender

Some Wi-Fi systems, NETGEAR's Orbi most notably, use router / extender architectures. These systems require each extender to connect only to a base router, i.e. extenders cannot be daisy-chained. These two-module systems require a slightly modified test plan.

Instead of testing a second Hop, a second set of tests is run on the Hop 1 module with the Pal-connect attenuator changed from 0 dB to 21 dB. This simulates the test client moved farther away from the extender / satellite.

The router module is installed in the Root Box-18; the Extender / Satellite is installed in the Hop 1 Box-18.

  1. Set the attenuators as shown in diagram above
  2. Associate Pal to Root node device on 2.4 GHz. Run downlink throughput test. Run uplink throughput test
  3. Associate Pal to Root node device on 5 GHz. Run downlink throughput test. Run uplink throughput test
  4. Disassociate Pal from Root node (set Atten 5 to 63 dB). Move Pal to Hop 1 device (set Atten 4 to 0 dB).
  5. Repeat Steps 2 and 3
  6. Change Hop 1 Pal-connect attenuator from 0 to 21 dB.
  7. Repeat Steps 2 and 3

Wi-Fi System Performance - Backhaul

This test measures the bandwidth from the Root node Ethernet-connected iperf3 endpoint to an Ethernet-connected iperf3 endpoint connected to Hop 1, then Hop 2. This measures the bandwidth available at each node for client and backhaul connections.

This test uses a second computer running an iperf3 server, connected via Ethernet to each node, one at a time.

  1. Confirm that all nodes are operating normally.
  2. Connect iperf3 server to Ethernet port of Hop 1 node device. Run downlink throughput test. Run uplink throughput test.
  3. Move iperf3 server from Hop 1 node device to Ethernet port of Hop 2 node device. Run downlink throughput test. Run uplink throughput test.

More Wireless

Win This!

Win WD Red drives

You could win two pairs of 10 TB WD Red drives!

Learn How!
Giveaway sponsored by WD.

Top Ranked Routers

Support Us!

If you like what we do and want to thank us, just buy something on Amazon. We'll get a small commission on anything you buy. Thanks!

Over In The Forums

Hi,My friend got a Asus RT-AC88U router from an Auction and apparently it is faulty (lol).The problem with the router is that it does not connect to t...
Hi,Is there a way in Merlin's firm to measure monthly data usage? Not necessary per device and/or type of traffic, just total data.Thanks,Sent from my...
Hi Everyone,I'm trying to create a crontab in XVortex-Merlin (380.67) to wake up a server at a certain time. I have a script in /jffs/scripts called s...
I am currently running Asuswrt-Merlin 378.56_2 and want to upgrade to the most recent release . However after uploading the RT-N66U_380.67_0.trx file,...
Hello,I have asus ac88u running merlin and openvpn server. My clients connect successfully but are unable to ping or RDP to certain (seemingly random)...

Don't Miss These

  • 1
  • 2
  • 3
Get Backblaze Now!