Shaping

An old boss of mine said ’network demand is unbounded’. I suppose that’s the ‘if you build it, they will come’ traffic theory of induced demand. I’ve not observed that in practice. When we upgrade capacity traffic doesn’t grow to meet it. But there are peaks involving Apple updates and PlayStation launches you should manage. And if you don’t have enough to meet normal demand as is, you should at least manage what you can’t do.

Shaped vs Unshaped

When you don’t do anything, things still work. TCP for example recognizes when packets are being lost and backs-off the speed. But the side effect is that meanwhile, packets are queued up in the router’s buffers before ultimately being dropped. When lots of users are involved they are all ‘detecting’ congestion on their own, leading to a lot of latency. On top of this, UDP packets have no built-in congestion control and may do who-knows-what. Basically, everything gets slow. Slower than it needs to be.

If you shape the traffic, usually by controlling what packets drop, you can minimize latency, make sure important traffic gets through, and apply some fairness between users.

Modern Shaping

The old days you would look at protocol and port to classify traffic, and choose how much to let though of any given class. Sometimes dynamically allocating bandwidth depending on the technology on-hand.

Modern methods use flow identification and heuristics. This prevents a bad actor from trying to mimic DNS requests to get around the rules.

Egress Only

You don’t have any control over what people are giving you, other than sending Explicit Congestion Notification (ECN) back to the sender. You can only applying shaping to packets leaving your queue. But on a router this is fine because packets are leaving your queue all time time as you forward them to their destination.

Linux

Linux uses the Traffic Control (TC) subsystem to manage traffic by assigning queues to the network interfaces. You choose the algorithm the queue uses. So the network packets get forwarded (or dropped) based on what you want.

CAKE

The Common Applications Kept Enhanced, or CAKE algorithm is a good choice. It prioritizes “sparse” flows over bulk downloads. The thought being that sparse flows represent gaming and other latency sensitive traffic. It also enforces fairness between hosts and sends congestion signals upstream. And it does all this with minimal configuration.

To use it, you simply set a threshold. No one is limited until you get close, where it starts dropping packets to keep you within that limit.

By default it puts traffic in three “tins”; Bulk, Best Effort, and Voice. It looks for packets that are explicitly marked ‘EF’ (expedited forwarding) by the PC for the Voice tin, but not all games do this, so many fall into Best Effort along with general web traffic. Continuous flows move down from that into the Bulk tin.

Optimal latency is automatically detected and when not being met, packets are dropped algorithmically. This reduces traffic until optimal latency is restored.

Implementing a Linux Shaper with CAKE

Debian makes a good platform and includes tc-cake by default.

# Given that you have two interfaces, enable forwarding.
sysctl -w net.ipv4.ip_forward=1

# and persist
echo "net.ipv4.ip_forward=1" >> /etc/sysctl.conf

# Apply cake to one of the interfaces - preferably the one
# closest to the bottleneck 
tc qdisc delete root dev eth0
tc qdisc add root dev eth0 cake bandwidth 1000Mbit ethernet

# See how the traffic looks
tc -s qdisc show dev eth0

This is all you need. Though if you might want a HA pair.

HA Shaping

The thought is you have a virtual IP address and if one of the boxes goes down the other will take it’s place.

You can use ucarp, keepalived or pacemaker. Since we only need a floating IP, let’s use ucarp as it’s the simplest.

Do the following on both servers

# Install ucarp
sudo apt -y install ucarp

# Add an interface on both servers to the bottom
vi /etc/network/interfaces

...
iface eth0:ucarp inet static
     address 10.1.1.30
     netmask 255.255.255.0

# Create a config file. Correct SOURCE_ADDRESS as needed.
# On the second server, increase ADVSKEW by 1.
vi /etc/ucarp/vip-common.conf 

BIND_INTERFACE="eth0"
PASSWORD="SomePassLessThan20Chars"
VIP_ADDRESS="10.1.1.30"
SOURCE_ADDRESS="10.1.1.4"
VHID="1"
ADVBASE="1"
ADVSKEW="10"
OPTIONS="-P -z"
UPSCRIPT="/usr/share/ucarp/vip-up"
DOWNSCRIPT="/usr/share/ucarp/vip-down"


# Create a service
vi /etc/systemd/system/ucarp.service

[Unit]
Description=UCARP virtual interface %I
After=network.target

[Service]
EnvironmentFile=-/etc/ucarp/vip-common.conf
ExecStart=/usr/sbin/ucarp $OPTIONS -i $BIND_INTERFACE -p $PASSWORD -v $VHID -a $VIP_ADDRESS -s $SOURCE_ADDRESS -b $ADVBASE -k $ADVSKEW -u $UPSCRIPT -d $DOWNSCRIPT
KillMode=control-group

[Install]
WantedBy=multi-user.target


# And enable
systemctl enable --now ucarp

Sources

https://ndk.sytes.net/wordpress/?p=1296 https://blog.no42.org/article/ucarp-high-availability/