Side note: Using a secondary network interface is recommended since the following commands could make a remote machine unreachable.

This a blog post about the basics of netem and tc on how to modify the outgoing traffic. You could modify the incoming traffic with an Intermediate Functional Block pseudo-device in Linux, but I am not too familiar with it and is out of scope for now.

Reasons to simulate an unreliable network connection

There are various reasons why you want to modify the traffic between devices. The last time we had to ensure that a streaming server in Frankfurt could handle incoming video streams with a high latency over an unreliable connection from the US. The other time we had to provide proof that some SAP modules can't handle the additional latency of a VPN and that the problem is on their side and not ours.

Some additional reasons besides troubleshooting could be:

testing your network monitoring solution

whether your application or server handles unreliable connections well

simply do some research.

This post tries to provide enough information to help you troubleshoot various problems quickly and simulate certain scenarios.

Network shaping options #

tc and netem provide are variety of options to shape the outgoing traffic.

We are going to cover the basics of the following options in this post:

adding latency

adding jitter

sending duplicated packets

adding a percentage of packet loss

sending corrupt packets

Those options can be combined and will cover most of the cases.

Basics of tc

tc stands for 'traffic control' and, as the name implies, is used to configure the traffic control of the Linux kernel and is part of the iproute2 package. Netem stands for 'network emulator' and is controlled by the tc command.

You can check quickly whether tc is available by typing tc -V.

kuser@pleasejustwork:~$ tc -V
tc utility, iproute2-5.15.0, libbpf 0.5.0

Show or delete current options #

Show the currently applied options for an interface:

tc -p qdisc ls dev eth0

kuser@pleasejustwork:~$ sudo tc -p qdisc ls dev eth0
qdisc netem 8001: root refcnt 2 limit 1000 delay 100ms  50ms

You can delete all options for a specific interface with the following command:

tc qdisc del dev eth0 root

Side note: the following options are temporary and don't survive a reboot!

A breakdown of a common tc command can be found in the first example below.

Limiting to a specific IP or port #

Unfortunately, it is not that easy to limit the applied options to a specific IP or port. It is possible, but outside the scope of this basic guide. To avoid problems, it is therefore recommended to use a secondary network interface.

I might rework this section at some point. For further reading, feel free to check the official documentation for the filters.

Units used for Parameters for the netem options

Almost every 'nenum' option can have one or more parameters. I thought it would make sense to show you the available units before we start with the practical part.

Rates #

The bandwidths can be specified with the following units:

bit - Bits per second

kbit - Kilobits per second

mbit - Megabits per second

gbit - Gigabits per second

tbit - Terabits per second

bps - Bytes per second

kbps - Kilobytes per second

mbps - Megabytes per second

gbps - Gigabytes per second

tbps - Terabytes per second

Time #

The time for latency and other options can be specified as follows:

us - Microseconds

ms - Milliseconds

s - Seconds

Netem Options

I am going to explain the syntax in the first scenario.

As a reference, this is the normal ICMP request/ ping over a separate interface.

kuser@pleasejustwork:~$ ping -c 10 -I eth0 10.10.22.1 
PING 10.10.22.1 (10.10.22.1) from 10.10.22.51 eth0: 56(84) bytes of data.
64 bytes from 10.10.22.1: icmp_seq=1 ttl=255 time=0.458 ms
64 bytes from 10.10.22.1: icmp_seq=2 ttl=255 time=0.520 ms
64 bytes from 10.10.22.1: icmp_seq=3 ttl=255 time=0.453 ms
64 bytes from 10.10.22.1: icmp_seq=4 ttl=255 time=0.420 ms
64 bytes from 10.10.22.1: icmp_seq=5 ttl=255 time=0.513 ms
64 bytes from 10.10.22.1: icmp_seq=6 ttl=255 time=0.412 ms
64 bytes from 10.10.22.1: icmp_seq=7 ttl=255 time=0.550 ms
64 bytes from 10.10.22.1: icmp_seq=8 ttl=255 time=0.548 ms
64 bytes from 10.10.22.1: icmp_seq=9 ttl=255 time=0.402 ms
64 bytes from 10.10.22.1: icmp_seq=10 ttl=255 time=0.376 ms

--- 10.10.22.1 ping statistics ---
10 packets transmitted, 10 received, 0% packet loss, time 9202ms
rtt min/avg/max/mdev = 0.376/0.465/0.550/0.060 ms

Add Latency / Delay #

The netem latency will be added to the normal latency of the connection.

DELAY := delay TIME [ JITTER [ CORRELATION ]]

sudo tc qdisc add dev eth0 root netem delay 100ms

sudo # run command as sudo

tc # command stands for 'traffic control'

qdisc # stands for 'Queue discipline'

add|change|del # is the action that tc should perform to a option

dev eth0 # choosing the network interface

root # qdiscs ID

netem delay 100ms # 'netem' option + parameter

Results

kuser@pleasejustwork:~$ sudo tc qdisc add dev eth0 root netem delay 100ms
[sudo] password for kuser: 
kuser@pleasejustwork:~$ ping -c 10 -I eth0 10.10.22.1 
PING 10.10.22.1 (10.10.22.1) from 10.10.22.51 eth0: 56(84) bytes of data.
64 bytes from 10.10.22.1: icmp_seq=1 ttl=255 time=101 ms
64 bytes from 10.10.22.1: icmp_seq=2 ttl=255 time=100 ms
64 bytes from 10.10.22.1: icmp_seq=3 ttl=255 time=100 ms
64 bytes from 10.10.22.1: icmp_seq=4 ttl=255 time=100 ms
64 bytes from 10.10.22.1: icmp_seq=5 ttl=255 time=100 ms
64 bytes from 10.10.22.1: icmp_seq=6 ttl=255 time=100 ms
64 bytes from 10.10.22.1: icmp_seq=7 ttl=255 time=101 ms
64 bytes from 10.10.22.1: icmp_seq=8 ttl=255 time=100 ms
64 bytes from 10.10.22.1: icmp_seq=9 ttl=255 time=100 ms
64 bytes from 10.10.22.1: icmp_seq=10 ttl=255 time=100 ms

--- 10.10.22.1 ping statistics ---
10 packets transmitted, 10 received, 0% packet loss, time 9013ms
rtt min/avg/max/mdev = 100.416/100.466/100.586/0.050 ms

To remove this tc rule, send the same command again, but replace add with del.

sudo tc qdisc del dev eth0 root netem delay 100ms

Add Jitter #

If you want to add more Jitter - or in other words - variance in latency, add another parameter at the end. This is a plus/minus value.

sudo tc qdisc change dev eth0 root netem delay 100ms 50ms

Results

kuser@pleasejustwork:~$ ping -c 10 -I eth0 10.10.22.1 
PING 10.10.22.1 (10.10.22.1) from 10.10.22.51 eth0: 56(84) bytes of data.
64 bytes from 10.10.22.1: icmp_seq=1 ttl=255 time=105 ms
64 bytes from 10.10.22.1: icmp_seq=2 ttl=255 time=88.6 ms
64 bytes from 10.10.22.1: icmp_seq=3 ttl=255 time=108 ms
64 bytes from 10.10.22.1: icmp_seq=4 ttl=255 time=109 ms
64 bytes from 10.10.22.1: icmp_seq=5 ttl=255 time=130 ms
64 bytes from 10.10.22.1: icmp_seq=6 ttl=255 time=54.5 ms
64 bytes from 10.10.22.1: icmp_seq=7 ttl=255 time=141 ms
64 bytes from 10.10.22.1: icmp_seq=8 ttl=255 time=102 ms
64 bytes from 10.10.22.1: icmp_seq=9 ttl=255 time=124 ms
64 bytes from 10.10.22.1: icmp_seq=10 ttl=255 time=146 ms

--- 10.10.22.1 ping statistics ---
10 packets transmitted, 10 received, 0% packet loss, time 9011ms
rtt min/avg/max/mdev = 54.495/110.797/145.590/25.366 ms

The added latency will be in a range from 50-150ms from now on.

Send duplicate packets #

Sending random duplicate packets over a specific interface:

sudo tc qdisc change dev eth0 root netem duplicate 1%

Simulate Packet loss #

There are various reasons for packet loss: an unreliable network connection, network congestion, bugs, and so on.

To drop random packets of a specific interface, simply use the following command:

sudo tc qdisc add dev eth0 root netem loss 20%

Results

kuser@pleasejustwork:~$ ping -c 10 -I eth0 10.10.22.1 
PING 10.10.22.1 (10.10.22.1) from 10.10.22.51 eth0: 56(84) bytes of data.
64 bytes from 10.10.22.1: icmp_seq=1 ttl=255 time=0.833 ms
64 bytes from 10.10.22.1: icmp_seq=2 ttl=255 time=0.414 ms
64 bytes from 10.10.22.1: icmp_seq=3 ttl=255 time=0.576 ms
64 bytes from 10.10.22.1: icmp_seq=4 ttl=255 time=0.443 ms
64 bytes from 10.10.22.1: icmp_seq=5 ttl=255 time=0.449 ms
64 bytes from 10.10.22.1: icmp_seq=6 ttl=255 time=0.510 ms
64 bytes from 10.10.22.1: icmp_seq=8 ttl=255 time=0.515 ms
64 bytes from 10.10.22.1: icmp_seq=10 ttl=255 time=0.302 ms

--- 10.10.22.1 ping statistics ---
10 packets transmitted, 8 received, 20% packet loss, time 9221ms
rtt min/avg/max/mdev = 0.302/0.505/0.833/0.145 ms

Corrupt packets #

Introduced an error at a random position of the packet.

sudo tc qdisc change dev eth0 root netem corrupt 10%

Conclusions

As mentioned before, there are more advanced options, but this blog post should cover the basics.

---