slow your network down

Have you you ever wanted to deliberately slow down your network to test your web service over a flaky connection? You can do it with netem, the Network Emulator.

It’s built right in to the Linux kernel.

It’s nice to develop something locally, throw it behind Apache or NGINX and hit http://localhost:24242 to test it out. But what if you need to simulate a network that isn’t near-zero latency and 30-odd Gbps?

You have a couple of options:

Forward some ports and phone a friend
Host it on a cloud provider in a remote region
Get a second computer, tether it through your phone’s data plan, and test it that way (I know you have done this)
Use netem

With netem, you don’t need a phone, a second computer, a cloud host, or even friends.

Queuing disciplines

Since this is intended to be more of a field guide, I’m not going to go in to the details of the Linux networking subsystem. There are entire books on that. But it’s helpful to have a general overview.

netem works at the kernel TCP/IP stack. It’s part of a tool called tc, for traffic control, which has shipped with Linux for at least the last decade as a component of the iproute2 suite of networking power tools.

Luckily, the dude who wrote netem also wrote a paper explaining what it is and how to use it all the way back in 2005. In brief, netem is one of a set of queuing disciplines which sit between the kernel TCP/IP stack and the hardware (or virtualized) network interface. The role of a queuing discipline is to decide how to prioritize packets leaving the stack destined for the interface.

From the aforementioned paper:

netem block diagram — `netem` block diagram

Most queuing disciplines attempt to solve the problem of deciding how to prioritize outgoing packets based on a set of rules in order to ensure quality of service. netem “solves” this problem by limiting egress rate, delaying packets for random intervals and sending them in bursts, or other doing other aggravating things that happen in real-world networks.`

Examples

I wrote a little framework using Docker Compose to demonstrate how netem affects both bandwidth and latency by bringing up two containers, adding the netem queuing discipline to one of the containerized network interfaces, then running ping and iperf3. We’ll be using it to demonstrate the havoc netem can simulate.

First, try running it without any options:

$ NETEM='' docker-compose up

You should see typical localhost performance: sub-millisecond latency and >10 Gbps bandwidth.

Latency

Let’s sprinkle some latency on our perfect network:

$ NETEM='delay 60ms' docker-compose up

You’ll notice that not only was latency affected, but bandwidth too. By a lot.

The reason for this has to do with the fact that TCP is a guaranteed delivery protocol. Specifically, the sender must wait for the receiver to acknowledge receipt of data every X bytes, known as the receive window.`

In the real world (or our simulation of it) where latency exists, those acknowledgements also take time to reach the other end, slowing the whole transmission down.

Take a look at these two plots, generated by Wireshark:

Within the first two seconds, the client scales up the TCP receive window to roughly 3 MiB, which is the full capacity of the client’s receive buffer (limited by the kernel parameter net.ipv4.tcp_rmem). You can see quite clearly the corresponding linear increase in throughput.

We can also add latency with some random jitter to make our network act even more like the real world by adding a second parameter to delay, like this:

$ NETEM='delay 60ms 10ms' docker-compose up

Now the ping results dance between 55-65 ms.

There’s even a way to change the distribution curve of the latency filter for some seriously detailed testing. But I’ll leave that as an exercise for you, dear reader.

Throughput

We can also directly throttle the throughput of our network by specifying a target rate.

$ NETEM='rate 5mbit' docker-compose up

iperf3-client_1  | [  5]   4.00-5.00   sec   585 KBytes  4.80 Mbits/sec                  
iperf3-client_1  | [  5]   5.00-6.00   sec   581 KBytes  4.76 Mbits/sec                  
iperf3-client_1  | [  5]   6.00-7.00   sec   584 KBytes  4.78 Mbits/sec  
...

Why aren’t we hitting exactly 5 megabits per second?

As discussed earlier, TCP is a pretty complicated protocol. Every data packet ships with a header which carries information like the source IP address and port, the destination, a timestamp, the aforementioned receive window plus a 16-bit scaling factor, etc. All that extra stuff takes up space that the application, in this case iperf, doesn’t see.

As far as netem is concerned, it is limiting the throughput to exactly 5 Mbps, but your application will see a bit less because of the TCP overhead.

There’s even a technical term for throughput minus TCP and link layer overhead: goodput.

Honorable mention UDP

If TCP is registered mail, UDP is carpet bombing. UDP doesn’t care if packets make it to their destination at all. It’s up to the application to handle corruption, duplication, reordering, and all the stuff TCP does on its own.

The advantage is that UDP is fast, both in terms of latency and throughput. That’s why UDP is used more often for transmitting things like lossy video streams, where a couple bytes missing here and there won’t do too much harm, and for applications that encapsulate other traffic that already have error correction, like an encrypted VPN tunnel carrying TCP.

Also, if Google gets its way, a new UDP-based application protocol called QUIC will become the future of the internet. QUIC will move all the complicated congestion control and error correction stuff now done by the operating system to the application level, giving developers more control over the networking stack.

We can run iperf3 in UDP mode, but we have to specify the bandwidth at the client, because with UDP the server doesn’t have the slightest idea whether the client is receiving any data at all. Carpet bomb.

By running iperf3 in UDP mode sending at 10 Mbps together with netem rate set to 5 Mbps, we can demonstrate packet loss.

iperf3-client_1  | [ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
iperf3-client_1  | [  5]   5.00-6.00   sec   592 KBytes  4.85 Mbits/sec  0.609 ms  443/862 (51%)  
iperf3-client_1  | [  5]   6.00-7.00   sec   594 KBytes  4.87 Mbits/sec  0.568 ms  444/864 (51%)  
iperf3-client_1  | [  5]   7.00-8.00   sec   592 KBytes  4.85 Mbits/sec  0.619 ms  444/863 (51%)

Right on the money.

Conclusion

netem can do a lot more than what was covered here, like introducing random data corruption, time slotting to simulate ‘bursty’ links where bandwidth is non-continuous, packet duplication, etc. I recommend exploring the netem man page (links below) to learn more.

We don’t live in a perfectly networked world. Tuning your app’s performance for sub-optimal network connections (sitting on the subway in a tunnel during rush hour, airport WiFi) can have a huge impact on user experience.

bad gateway