This is the multi-page printable view of this section. Click here to print.

Return to the regular view of this page.

Routing

1: Linux
2: OpenWrt

2.1: Dynamic DNS
2.2: GeoIP
2.3: HA With Keepalived
2.4: OpenWrt in Incus
2.5: OpenWRT in PVE LXC
2.6: Syncing
2.7: WireGuard

3: OPNsense
4: VyOS

1 - Linux

If less is more, maybe nothing is the most¹. Since most appliances are just linux with a wrapper, let’s get rid of that wrapper. We’ll be left with more.

All humor aside, having the simplest thing possible that works is almost always the best solution. I’ve found after several years in production that a simple Debian router/traffic shaper is equal or better than any appliance.

Preparation

Network

The first problem is how to mock-up the WAN/LAN networks for this new router.

For the outside interface, let’s pretend that your existing LAN is the internet. Just plug eth0 in to whatever you’ve already got.

For the inside network, we will overlay a new LAN on top of your existing one. This is analogous to having different computers use the same physical wires but configuring different IP networks. They may see each others traffic at the physical layer, but will ignore each other logically.

OS

Create a new Debian instance with two interfaces, both on the existing LAN. On bare-metal, install a minimal Debian instance by using the netinst image and selecting only ‘common system tools’ near the bottom.

In a LAN/WAN situation eth0 is traditionally WAN². So leave the first interface with it’s default DHCP settings and configure the second one with a static address.

Assuming that your existing LAN is 192.168.1.* we’ll overlay our new LAN as 192.168.2.*

sudo vi /etc/systemd/network/eth1.network

[Match]
Name=eth1

[Network]
Address=192.168.2.1/24

sudo systemctl reload systemd-networkd

Installation

We’ll need the nft tools to do the basics and make sure curl is handy as well.

sudo apt install nftables curl

Configuration

Forwarding

The first step is to enable forwarding, the basic job of all routers.

# as root
echo "net.ipv4.ip_forward=1" >> /etc/sysctl.conf
sysctl -p

It’s worth knowing that the system (as is) doesn’t spend a lot of time thinking about where a packet came from. If someone gives it a packet, it consults the route table and sends it on. It doesn’t really care where it came from. This sounds dangerous, but in practice it’s rarely a problem. Though you can look into policy based routing if things are complicated.

Masquerade

If one side is a private network (such as 192.168.*) you probably need to masquerade. This is different than the type attended by the Opera Ghost.

sudo vi /etc/nftables.conf

This is the default debian file. Add WAN and LAN at the top and a table at the bottom with the masquerade details.

It’s traditional to name this table ’nat’ and the chain ‘postrouting’. Those names are arbitrary but make sense as that’s its type and where it’s hooked.

#!/usr/sbin/nft -f

flush ruleset

# Define WAN and LAN 
define WAN = eth0
define LAN = eth1

table inet filter {
	chain input {
		type filter hook input priority filter; policy accept;
	}
	chain forward {
		type filter hook forward priority filter; policy accept;
	}
	chain output {
		type filter hook output priority filter; policy accept;
	}
}
table ip nat {
    chain postrouting {
        type nat hook postrouting priority srcnat; policy accept;
               
        # if the output interface is WAN, masquerade the traffic.
        oif $WAN masquerade 
    }
}

systemctl enable --now nftables.service

Test

To see if things are working so far, take another system and replace it’s .1 LAN address with something on the .2 and add the new router as it’s gateway.

# Replace your existing IP 
sudo ip addr replace 192.168.2.50/24 dev eth0

# Replace your existing route
ip route replace default via 192.168.2.1 dev eth0

ping 8.8.8.8

Firewall

Right now, you’re letting everything in and out, shouting ‘packets want to be free!’. But this is the real world so let’s add firewall to this router’s list of jobs. This is still a router, though. So allow controlled ping and traceroute. This helps avoid fragmentation and provides diagnostic benefits that outweigh attempts at obscurity.

Default Block

We are going to add a chain and edit two others. Create the chain early_drop at the top to get rid of any obvious garbage in the early prerouting stage. Then edit the input (packets destined to the router itself), and the forward chains (packets being sent on).

table inet filter {    

    chain early_drop {
        type filter hook prerouting priority filter; policy accept;
        ct state invalid drop
    }

    chain input {

        # Change the default policy to drop
        type filter hook input priority 0; policy drop;

        # Allow local and already established connections 
        iif lo accept
        ct state established,related accept

        # Allow standard ping with rate limiting and traceroute
        icmp type echo-request limit rate 5/second accept
        icmp type { destination-unreachable, time-exceeded } accept

        # Respond to Linux traceroute
        iif $WAN udp dport 33434-33534 reject with icmp type port-unreachable 

        # IPv6 equivalents 
        icmpv6 type { echo-request, nd-neighbor-solicit, nd-neighbor-advert, packet-too-big, time-exceeded } accept
    }

    chain forward {

        # Change the default policy to drop 
        type filter hook forward priority filter; policy drop;

        # Accept local and already established connections 
        ct state established,related accept

        # Accept connections from LAN to WAN
        iif $LAN oif $WAN accept
    }

nft -f /etc/nftables.conf

Congrats! you’ve just secured things and possibly locked yourself out! (don’t close your ssh session yet)

Accept SSH

To allow remote management, append a SSH rule to the bottom if your input chain. It’s best to start with LAN for this

    chain input {
        ...
        ...

        # Accept from any source to SSH
        iif $LAN tcp dport "ssh" accept
    }

Limit by IP

You may want to limit access based on IPs. The simplest and more performant way is with a set. That’s basically a list of IPs and networks to compare incoming packets against. We’ll put the details in it’s own file to keep the main config clean.

sudo mkdir -p /etc/nftables/sets
sudo vi /etc/nftables/sets/work_ips.nft

table inet filter {
    set work_ips {
        type ipv4_addr
        flags interval
        elements = {
            1.2.3.4,        # Hot in Cleveland
            5.6.7.8,        # WKRP in Cincinnati
            10.11.12.0/24,  # Buzz Beer headquarters
        }
    }
}

Put an include right after the definitions and use it in a rule like this.

vi /etc/nftables.conf

#!/usr/sbin/nft -f

flush ruleset

# Define WAN and LAN 
define WAN = eth0
define LAN = eth1

# Include all the config files in the folder
include "/etc/nftables/sets/*.nft"

table inet filter {

    chain input {
        ...
        ...

        # Accept from any source to SSH
        iif $LAN tcp dport "ssh" accept
        
        # Accept from work to SSH
        iif $WAN ip saddr @work_ips tcp dport "ssh" accept
    }

Reload the rules and take a look. You’ll see your new set displayed.

nft -f /etc/nftables.conf

sudo nft list ruleset

Note: You may have noticed that both the main and included files contain a table inet filter { block. These don’t replace each other, rather they are additive. That’s a nft thing.

Limit by GeoIP

Sometimes you don’t know what your IP will be, but you’re pretty sure you’re not leaving the country. In addition, the vast majority of attacks come from just a few countries. The wirefalls/geo-nft script will help by downloading a map of IP’s to countries.

sudo mkdir -p /etc/nftables/geo-nft
cd /etc/nftables/geo-nft
curl -LO  https://raw.githubusercontent.com/wirefalls/geo-nft/master/geo-nft.sh
chmod +x geo-nft.sh
sudo ./geo-nft.sh

Create a us.nft that incorporates the ipv4 data like this.

sudo vi /etc/nftables/sets/us.nft

include "/etc/nftables/geo-nft/countrysets/US.ipv4" 

table inet filter {
    set US_ipv4 {
        type ipv4_addr
        flags interval
        auto-merge
        elements = $US.ipv4
    }
}

Make use of it just like any set.

...
...
# Accept connections from US to SSH
iif $WAN ip saddr @US_ipv4 tcp dport "ssh" accept
...
...

Reload and observe the now much larger set. But don’t let the size scare you. I’m told speed is the same for ten or ten thousand as they are hashed.

nft -f /etc/nftables.conf
nft list ruleset | more

If you see an error about the Message size like this:

netlink: Error: Could not process rule: Message too long Please, rise /proc/sys/net/core/wmem_max on the host namespace. Hint: 4194304 bytes

You’ll need to increase your netlink buffer size. Do this on the host if running in an instance or container. Jumping up to a 4M buffer should insulate you from needing to change it again even as your set sizes change.

sudo vi /etc/sysctl.d/99-nftables-netlink.conf

net.core.wmem_max=4194304
net.core.wmem_default=524288

sudo sysctl -p /etc/sysctl.d/99-nftables-netlink.conf

Combining Sets

Sometimes you want to combine sets, like the worst 5 countries for cyber-attacks. The unfab 5. Just include them all with a semicolon after each include, and you can flatten them into a super-set. (That’s a hidden gotcha - the files don’t end with a newline and using ‘;’ isn’t well known)

sudo vi /etc/nftables/sets/unfab.nft

include "/etc/nftables/geo-nft/countrysets/CN.ipv4";
include "/etc/nftables/geo-nft/countrysets/IN.ipv4";
include "/etc/nftables/geo-nft/countrysets/NE.ipv4";
include "/etc/nftables/geo-nft/countrysets/NG.ipv4"; 
include "/etc/nftables/geo-nft/countrysets/RU.ipv4"; 

define UNFAB = { $CN.ipv4, $IN.ipv4, $NE.ipv4, $NG.ipv4, $RU.ipv4 }

table inet filter {
    set UNFAB_ipv4 {
        type ipv4_addr
        flags interval
        auto-merge
        elements = $UNFAB
    }
}

Inverting Sets

When you want to accept connections from everyplace and just exclude a few countries, it’s a lot easier to just list the exclusions. You do that with an inverse meaning accept any connection not on this list. If you’ve created the unfab set above, you’d use it inversely like this:

# Accept everyone but the unfab to connect to SSH
ip saddr != @UNFAB_ipv4 tcp dport "ssh" accept

Port Forwarding

Let’s say we are going to forward web traffic to a back-end server. This actually requires two things; an new prerouting chain and entry to catch incoming traffic to those ports and rewrite it with the back-end server’s address, and a forward chain entry where you accept the traffic.

sudo mkdir /etc/nftables/forwards

sudo vi /etc/nftables/forwards/web_server.nft

define WEB_SRV = 192.168.2.10

table ip nat {
    chain prerouting {
        type nat hook prerouting priority dstnat;

        # Rewrite any incoming traffic so they have the web servers address instead
        iif $WAN tcp dport { "http", "https" } dnat to $WEB_SRV
    }
}

table inet filter {
    chain forward {
        type filter hook forward priority filter;

        # Accept the web server traffic so it can be forwarded on.
        iif $WAN ip daddr $WEB_SRV tcp dport { "http", "https" } accept

        # Or maybe only accept traffic from the US
        iif $WAN ip saddr @US_ipv4 ip daddr $WEB_SRV tcp dport { "http", "https" } accept        
    }
}

Add an include in the main file after the sets and reload.

...
...

# Include any sets
include "/etc/nftables/sets/*.nft"

# Include any port forwards
include "/etc/nftables/forwards/*.nft"

...
...

nft -f /etc/nftables.conf

Change the IP and gateway on your web server temporarily and it should test correctly.

Note: It’s tempting to add more rules to the prerouting section. Resist this urge, however. According to the docs, some parts of a flow skip prerouting so you should “…never use this chain for filtering”³. The only exception being to drop obvious junk as we did at the beginning.

Port Rewriting

Say you’re forwarding the external port 222 to an internal server on port 22. It’s just a matter of adding that to the rewrite. The forward chain stays the same.

redefine EXTERNAL_PORT = 222
redefine INTERNAL_IP   = 192.168.2.4
redefine INTERNAL_PORT = 22

table ip nat {
    chain prerouting {
        type nat hook prerouting priority dstnat;
        iif $WAN tcp dport $EXTERNAL_PORT dnat to $INTERNAL_IP:$INTERNAL_PORT
    }
}
table inet filter {
    chain forward {
        type filter hook forward priority filter;
        iif $WAN ip daddr $INTERNAL_IP tcp dport $INTERNAL_PORT accept
    }
}

Note: Last time we defined the (probably) unique name for the variable of WEB_SRV, so that it wouldn’t conflict with other variables in other includes. But you can also just use redefine in a template and save a lot of editing.

Flow and Hardware Acceleration

Linux can bypass most of the nftables/conntrack path after a flow is established. If you have a decent ethernet card you can even get hardware acceleration.

Add A Flow Table

A flowtable is kernel-managed acceleration table you can add to your normal inet fitler table. It’s a “fastpath” that you can toss flows to that you’ve already looked at. This saves you from checking every packet when it’s already part of an established flow.

This will mostly offload web browsing/streaming and other normal TCP/UDP NAT traffic.

Add this at the top of the table inet filter.

table inet filter {

    flowtable ft {
            hook ingress priority filter;
            devices = { $LAN, $WAN };
    }
    ...
    ...

Then modify the forward chain to use it.

chain forward {

        type filter hook forward priority filter; policy drop;

        # Offload established/related flows
        ct state established,related flow add @ft accept

        # This line is now redundant but usually left in so you can 
        # comment the above hardware accel in and out easily
        ct state established,related accept

        # Accept connections from LAN to WAN
        iif $LAN oif $WAN accept
}

If you want to dig in more, you can add a counter to the rules above. Comment out the Offload rule and the flow_normal counter should increase steadily. Swap and the flow_offload should increase much more slowly as only the first packets get counted. The rest of the low bypasses.

        ct state established,related flow add @ft counter name flow_offload_counter accept
        ct state established,related accept counter name flow_normal_counter accept

Enable Hardware Offloading

Pro cards from Intel, Mellinox, Chelsio and others make drivers that provide hardware acceleration to the kernel. Check with this comand.

ethtool -k eth0 | grep offload

You’re mostly looking for hw-tc-offload: on as nftables flow offload can sometimes leverage TC hardware offload underneath. Enable the hardware offload with the flag offload.

flowtable ft {
        hook ingress priority filter;
        devices = { $LAN, $WAN };
        flags offload;
}

Note: if you saw l2-fwd-offload: on then you have a NIC that does autonomous Layer-2 forwarding/switching and you either don’t need to read this page, or need to follow up with another more advanced page!

eBGP and XDP

If this isn’t fast enough, you can use eBGP to take action on packets before the kernel starts working on them. If your card supports XDP, you can do that while they are still in the network card!

This is mostly useful for dropping trash and mitigating denial of service attacks without bothering the CPU. There’s also a case where you move packets around in a kubernetes cluster without looking at them closely for the sake of speed. Though Cloudflare utilizes the xt_bpf extension to embed eBPF programs with nftables, it’s not easy.

It’s not generally useful as in most cases you’ll have to look at the details of the packet anyway. In this case the best you can do is a flowtable you just created

Though if you’re under DDOS attacks consider xdp-filter and for large-scale load balancing you might look the other eBFP solutions.

If you’re running in a container of some kind, you can pass a card directly to the instance (SR-IOV or full PCI) to take advantage of it.

High Availability

This best done with a pair of routers that share a virtual IP managed by keepalived. It’s fairly simple to deploy as you just:

Set up the first router with all the configuration you need. (let’s assume you’ve done this already)
Install keepalived with a simple config.
Clone and tweak so the IPs, Hostname, Config works.

Change Admin IPs

Your router is still has a DHCP WAN address and that’s fine. But you’ll need to select something static for the virtual WAN ip address you’re about to create. I’ve used 192.168.1.2, but adjust as needed.

Install Keepalived

# Just install the core server, without perl and ipvsadm
sudo apt install --no-install-recommends  keepalived

Create a Virtual LAN and WAN Address

sudo vi /etc/keepalived/keepalived.conf

You’ll note that we’ve set the state to “BACKUP” and we’ll use that on both. This keeps them from flapping during any intermittent issues.

vrrp_instance WAN {
    state BACKUP
    interface eth0
    virtual_router_id 51
    virtual_ipaddress { 192.168.1.2/24 }
}

vrrp_instance LAN {
    state BACKUP
    interface eth1
    virtual_router_id 52
    virtual_ipaddress { 192.168.2.1/24 }
}

Add Firewall Rules

Add this at the bottom of your chain input section so you accept vrrp traffic. 224.0.0.18 is a reserved IPv4 multicast address specifically designated for the Virtual Router Redundancy Protocol

...
...

table inet filter {

    chain input {

        ...
        ...

        # Allow VRRP on both relevant interfaces. The 224 is
        iif { $WAN, $LAN } ip protocol vrrp accept
        iif { $WAN, $LAN } ip daddr 224.0.0.18 accept
    }
}

Test and Clone

Start the service and observe the log to see the status messages.

sudo systemctl start keepalived.service

sudo journalctl -u keepalived*

If things go well, you’ll see it start as backup, then quickly enter master when it doesn’t find any peers to say otherwise. An ip a will show the virtual IPs in place.

Clone the box (or the config), change the LAN IP so it doesn’t conflict, and bring both up. You can experiment by taking the service up and down to watch the IP shift.

Active vs Passive

This is an Active/Passive arrangement. For the most part, there’s no advantage to Active/Active. Either system can move traffic faster than the uplink permits. If you do have more traffic than either one can move on it’s own, you can’t take one down.

The main benefit of Active/Active is that you know it works. You don’t want to be surprised during an outage. It allows you to gracefully drain traffic for controlled maintenance. Also, you can’t always get a 200G system.

To setup Active/Active for capacity you create a team of three or more. Each is MASTER of a separate VIP and they all back each other up. DHCP passes out the gateway addresses round-robin to balance load. Keep in mind that traffic shaping with tc happens after nftables, so you can’t shape your way out of a capacity issue.

Keeping Them Synced

How do you keep them in sync? You can probably just remember to update rules only on router-1 and then run rsync. Though you’ll need to grant yourself (or a service account) permissions.

# On both routers

## Change the owner of the nftable data
sudo chown -R ${USER} /etc/nftables*

## Allow yourself to run nft without a password prompt so you can automate the reload
echo "$USER ALL=(root) NOPASSWD: /usr/sbin/nft" | sudo tee /etc/sudoers.d/nft
sudo chmod 440 /etc/sudoers.d/nft

# On your primary router

## Generate a key and copy it to the other router (make sure dns resolves it correctly)
ssh-keygen -t ed25519 -f ~/.ssh/id_ed25519 -N ''
ssh-copy-id router-2

## Run a sync 
rsync --archive --delete --inplace --verbose  /etc/nftables.conf /etc/nftables router-2:/etc/

## Trigger a reload on the other router
ssh router-2 sudo nft -f /etc/nftables.conf

You can probably put those last two commands into the bash file go-go-geo-nft to save typing them out. Even better, create a systemd watcher that detects changes and syncs automatically.

Election and Priority

The IP will only move when one system goes offline. If both come back up perfectly at once, they’ll have an election and the highest management IP wins. You can override this with an explicit priority or script it if you have reason to.

You can also change one to state MASTER and it will take over the role whenever it’s up. This is useful when you have a hardware preference or are running additional software that can’t do multi-master.

Elections are held via broadcast (the 224 address) but unicast is also supported if you’re in a cloud environment that blocks broadcast

If you have other software running and want to bind it to the virtual ip before it becomes instantiated, look at adding net.ipv4.ip_nonlocal_bind=1 to the /etc/sysctl.conf. Though for things like dnsmasq it’s not needed.

Connection Tracking

The only other issue with failing over is connection state. When you fail over, any session dependant applications (like SSH) or games get dropped. This isn’t a large issue as the web is (mostly) stateless. But you can deploy conntrackd if desired.

Scaling Up

Single Device

These days, you can Expect 100–400 concurrent connections per device at peak so you outpace a single device faster than you think. On a single device, you can optimize some kernetl settings.

# Create a drop-in configuration file in /etc/sysctl.d for these settings
net.core.netdev_max_backlog=250000
net.core.somaxconn=65535
net.ipv4.ip_local_port_range=1024 65535
net.ipv4.tcp_tw_reuse=1

net.netfilter.nf_conntrack_max=2000000
net.netfilter.nf_conntrack_tcp_timeout_established=86400
net.netfilter.nf_conntrack_generic_timeout=120

Smart Switches

On a old-style campus WAN, you have (expensive) layer 3 switches and routers that use the ECMP protocol to use multiple paths. This has wider compatibility than agregating ports with LAG. You’d think you could just distrubute multiple gateways as part of DHCP, but clients handle that randomly or not at all.

Become More Active

A more modern and cost effective strategy is to move from an Active/Passive HA pair, to an Active/Active/Active set of peers.

Create 3 or more routers
Create 3 or more VRRP instances, where a different router is MASTER in each and the others are BACKUP
Distrubute gateway via DHCP in round-robin to distribute load

IPv6 To The Rescue

If you can handle the new however, use and IPv6‑first design.

Design a full IPv6‑first + IPv4 fallback addressing plan
Browsers attempt IPv6 first
Mobile OSes heavily favor IPv6
Major content providers publish IPv6 natively
Less NAT to handle.

DNS and DHCP

DHCP and DNS are optional, but sometimes expected in small environments and frequently under other duties as assigned for such systems. Dnsmasq is the go-to for most. I’ve found that particular software doesn’t scale well beyond a few thousand clients, but if you’ve gotten to that point, you’d want a dedicated system anyway. So it’s fine for small scale.

This is covered in many other locations but I’ll toss a sample config here just in case.

# Remove systemd-resolved if installed by the default debian instance
sudo systemctl stop systemd-resolved.service 
sudo apt install dnsmasq

DNS

You’ll need to accept DNS requests by adding another accept at the bottom. UDP is traditional, but TCP is also used these days. Add this at the bottom of your input chain.

table inet filter {
    chain input {
        ...
        ...

        # Allow DNS queries 
        iif $LAN udp dport "domain" accept
        iif $LAN tcp dport "domain" accept
    }
}

By default, dnsmasq just fires up and starts work by caching DNS queries. It looks at your /etc/resolv/conf to see where to forward requests and your /etc/hosts for known hosts. You may want to tighten things up a bit with these.

sudo vi /etc/dnsmasq.conf

Just add these to the bottom.

# Don't forward plain names or non-routable IPs
domain-needed
bogus-priv

# Listen only on the LAN 
interface=lo
interface=eth1          

# Upstream DNS servers to forward requests to
server=8.8.8.8
server=8.8.4.4

# Cache size (1000 is a good start)
cache-size=1000

# Don't use the hosts file for entries
no-hosts

Create a seperate file for host entries as well. This will let you sync more easily later. Since debian’s packaging of dnsmasq makes use of a drop folder, let’s go ahead and embrace that and add entries in the native format.

sudo vi /etc/dnsmasq.d/hosts.conf

host-record=gateway,192.168.2.1
host-record=router-1,192.168.2.2
host-record=router-2,192.168.2.3
host-record=some-host,192.168.2.10
host-record=some-other-host,192.168.2.11
...
...

# Restart so dnsmasq reads the new config file.
sudo systemctl dnsmasq restart

# Tell the system to use itself for resolution as well.
echo "nameserver 127.0.0.1" | sudo tee /etc/resolv.conf

If you went to the trouble to set up sync above for nft, you can piggyback on that for dns.

# On both boxes
sudo test -f /etc/dnsmasq.d/hosts.conf || sudo touch /etc/dnsmasq.d/hosts.conf
sudo chgrp -R ${USER} /etc/dnsmasq.d/hosts.conf
sudo chmod -R 775 /etc/dnsmasq.d/hosts.conf
echo "$USER ALL=(root) NOPASSWD: /bin/systemctl restart dnsmasq, /bin/systemctl reload dnsmasq" | sudo tee /etc/sudoers.d/dnsmasq
sudo chmod 440 /etc/sudoers.d/dnsmasq

# On router-1, you can send the file and restart (reload doesn't load date from host.conf)
rsync --archive --delete --inplace --no-times --verbose  /etc/dnsmasq.d/hosts.conf router-2:/etc/dnsmasq.d/
ssh router-2 sudo systemctl restart dnsmasq


### DHCP

You can pass out addresses as well. Just follow up the last bit with this. Though you may want to leave these commented out until your ready, to avoid confusing the overlay networks.

```bash
dhcp-range=192.168.2.100,192.168.2.200,12h
dhcp-option=option:router,192.168.2.1
dhcp-authoritative

Bonus Points

You have a couple of things worth automating. Changes to nft rules and updates to the GeoIP database.

Automate GeoIP

Those IP sets are refreshed the 1st of every month on db-ip.com. We should refresh accordingly. On Debian 13, systemd timers are preferred (cron isn’t even installed).

sudo vi /etc/systemd/system/geo-nft-update.service

[Unit]
Description=Update geo-nft IP datasets
Wants=network-online.target
After=network-online.target

[Service]
Type=oneshot
ExecStart=/etc/nftables/geo-nft/geo-nft.sh

sudo nano /etc/systemd/system/geo-nft-update.timer

[Unit]
Description=Monthly geo-nft update (3rd day)

[Timer]
OnCalendar=*-*-03 03:30:00
Persistent=true
RandomizedDelaySec=30m

[Install]
WantedBy=timers.target

# Just enable the timer. We don't want the service to run unless the timer starts it.
sudo systemctl daemon-reload
sudo systemctl enable --now geo-nft-update.timer

Config Sync

You can also use systemd to watch for file changes and trigget a sync and reload. Note the User= setting. This must be your account (or whatever account you used in setting up the sync and ssh above) to work as the manual process did. Alternatly, create a service account.

sudo vi /etc/systemd/system/router-sync.service

[Unit]
Description=Sync router configuration to router-2
After=network-online.target
Wants=network-online.target

[Service]
Type=oneshot
User=YOU

ExecStart=/usr/bin/sudo /usr/sbin/nft -f /etc/nftables.conf
ExecStart=/usr/bin/rsync --archive --delete --inplace --no-times /etc/nftables.conf /etc/nftables router-2:/etc/
ExecStart=/usr/bin/ssh router-2 sudo nft -f /etc/nftables.conf

ExecStart=/usr/bin/sudo systemctl reload dnsmasq
ExecStart=/usr/bin/rsync --archive --delete --inplace --no-times /etc/dnsmasq.d/hosts.conf router-2:/etc/dnsmasq.d/
ExecStart=/usr/bin/ssh router-2 sudo systemctl reload dnsmasq

sudo vi /etc/systemd/system/router-sync.path

[Unit]
Description=Watch router config files

[Path]
PathChanged=/etc/nftables.conf
PathModified=/etc/nftables
PathChanged=/etc/dnsmasq.d/hosts.conf

Unit=router-sync.service

[Install]
WantedBy=multi-user.target

sudo systemctl daemon-reload
sudo systemctl enable --now router-sync.path

I put both services in this watcher as an example. A a better design is a sperate watch for each to reduce curn if you have frequent changes.

Replace GeoIP

The tool we’re using downloads the full set of countries as a .csv and then converts them into nft definitions. It’s a bit slow and heavy. You can replace it with something like this and get a more compact (and prehapse better) CIDR format like this.

COUNTRY=ad
OUT=/etc/nftables/sets/${COUNTRY}.nft

tmp=$(mktemp)

{
    echo "set ${COUNTRY}_ipv4 {"
    echo "    type ipv4_addr"
    echo "    flags interval"
    echo "    elements = {"

    curl -s https://www.ipdeny.com/ipblocks/data/countries/${COUNTRY}.zone | \
        awk '{print "        "$1","}' | sed '$ s/,$//'

    echo "    }"
    echo "}"
} > "$tmp"

mv "$tmp" "$OUT"

Troubleshooting

In a container, forwarding just stops. Restarting nftables has no affect but restarting systemd-networkd fixes it.

Incus especially likes to replace the bridge interfaces periodically.

sudo journalctl -u systemd-networkd -u NetworkManager --since "1 day ago"

May 28 12:55:50 shire1 systemd-networkd[486]: physSf5Ar4: Interface name change detected, renamed to vethe8067f8a.
May 28 12:55:50 shire1 systemd-networkd[486]: physnLFcl3: Interface name change detected, renamed to vethd880fdd5.
May 28 12:55:50 shire1 systemd-networkd[486]: vethc3f8a16b: Link UP
May 28 12:55:50 shire1 systemd-networkd[486]: vethc3f8a16b: Link DOWN
May 28 12:55:50 shire1 systemd-networkd[486]: veth243c40db: Link UP
May 28 12:55:50 shire1 systemd-networkd[486]: veth243c40db: Link DOWN
May 28 12:55:51 shire1 systemd-networkd[486]: veth24f2e389: Link UP
May 28 12:55:51 shire1 systemd-networkd[486]: vethff99fb0d: Link UP
May 28 12:55:51 shire1 systemd-networkd[486]: veth24f2e389: Gained carrier
May 28 12:55:51 shire1 systemd-networkd[486]: vethff99fb0d: Gained carrier

This causes a loss of static mappings to the interfaces and this breaks how iff and off in your rules work. You must instead us iffname which is dynamically mapped. It’s slightly slower, but needed in some containers.

sudo sed -i.bak -E \
    -e 's/\biif (\$WAN|\$LAN|\{ \$WAN)/iifname \1/g' \
    -e 's/\boif (\$WAN|\$LAN|\{ \$WAN)/oifname \1/g' \
    /etc/nftables.conf

I’ve read Rem Koolhaas meant this as mockey, so let’s think of it as “Simplify, then add lightness” as Chapman would say ↩︎
I don’t have a good attribution for this other than the many articles that assert this too ↩︎
https://wiki.nftables.org/wiki-nftables/index.php/Configuring_chains#Base_chain_types ↩︎

2 - OpenWrt

I once read a comparison of router distros (that I can no longer find) and was surprised to see OpenWrt score near the top in terms of speed and reliability. I’d thought it just a solution for small appliances. Turns out the developers spent significant time optimizing it to be a worthy alternative for large scale routing and firewall. And you can run in a container as opposed to needing a full VM like OPNsense.

2.1 - Dynamic DNS

The official docs seem complicated, but it’s fairly easy.

Installation

Go to System -> Software, Click the Update Lists button then search for “ddns” and simply click on the DDNS script for the service you have, then do the same for “luci-app-ddns” so it adds the menu item to the web GUI.

Configuration

In the Web GUI, select Services -> Dynamic DNS and delete the demo entries at the bottom. Add a new service, choose IPv4 and and select the service provider you added the script for above. In the subsequent screen, you’ll need to know

Lookup Hostname: This the FQDN, like “somehost.gattis.org”

     Domain: Similar, just in the form of "[email protected]"

```
   Username: "Bearer"
```

   Password: This is an API key you generated

Use HTTP Secure: Check this box, probably a good idea

If you’re behind NAT, you’ll also need to adjust the Advanced Settings

IP address source: URL

URL to detect: https://cloudflare.com/cdn-cgi/trace

Save and Apply, and then Reload. The Last Update/Next Update should change to a date.

2.2 - GeoIP

OpenWrt doesn’t ship with GeoIP capabilities, but you can add it with the IP set extras script. This is a somewhat legacy approach, but the GUI requires it, and it’s translated to modern nft named sets under the hood.

Warning

IP Sets don’t work on OpenWrt release 25.12. You’ll have to use the prior stable to use it.

https://forum.openwrt.org/t/ipset-extras-issues-in-25-12-0/247519

# At the command line
opkg update
opkg install ipset resolveip


# Install ipset-extras via their install script
wget -U "" -O ipset-extras.sh "https://openwrt.org/_export/code/docs/guide-user/advanced/ipset_extras?codeblock=0"
chmod +x ipset-extras.sh
./ipset-extras.sh

# Logout and back in to enable the extension from the /etc/profile.d folder
exit

# Configure an IP set for the US
uci set dhcp.us="ipset"
uci add_list dhcp.us.name="US"
uci add_list dhcp.us.name="US6"
uci add_list dhcp.us.geoip="us"
uci commit dhcp

# Populate IP sets
ipset setup

# Check set creation worked
nft list sets

# Install hotplug-extras for persistence
wget -U "" -O hotplug-extras.sh "https://openwrt.org/_export/code/docs/guide-user/advanced/hotplug_extras?codeblock=0"
chmod +x ./hotplug-extras.sh
./hotplug-extras.sh

When adding a port forward, the Advanced tab will now have the “Use ipset” populated and you can select “US”

You can also invert the rule by typing in “! US” - an important feature that doesn’t jump out at you.

This inversion is best with with a list of who you don’t want. Here’s an example of a set for the worst 5 countries for probes.

# Configure an IP set for the worst countries for probes and hacks - the axis of hacks
uci set dhcp.axis="ipset"
uci add_list dhcp.axis.name="axis"
uci add_list dhcp.axis.name="axis6"
uci add_list dhcp.axis.geoip="cn"
uci add_list dhcp.axis.geoip="in"
uci add_list dhcp.axis.geoip="ne"
uci add_list dhcp.axis.geoip="ng"
uci add_list dhcp.axis.geoip="ru"
uci commit dhcp

ipset setup

Troubleshooting

No NFT Sets Generated:

The script that generates the sets is sensitive to the value you put in `dhcp.axis.name=“value”. Try avoiding spaces and numbers and that ‘6’ is at the end of the second value.

Why are we adding these to the DHCP config section?:

I don’t know. The documentation adds them there. In the code, the script loads them from there. I tried adding them to the Firewall section just for fun and it didn’t work. I suspect if I knew more it would make sense.

Notes

The installation uses <www.ipdeny.com> and adds a cronjob that updates the list daily at 3 AM.

There are other tools, such as geopip-shell or ban ip, but these are more of an all-or-nothing solution and can’t be used with individual firewall rules. There is also the python utility from the netfilter team and misc bash scripts, but these lack easy OpenWrt integration.

The authors of nftables have a page[^5] on GeoIP, but it’s about tagging packets. OpenWrt expects named sets and you can’t easily construct that from a map.

[^5] https://wiki.nftables.org/wiki-nftables/index.php/GeoIP_matching

2.3 - HA With Keepalived

The simplest way to make routing highly-available is to have two routers and automate fail-over. The conventional way is with keepalived. It allows the gateway address to automatically move between your routers so clients can always connect.

Preparation

You need at least a pair of routers. You probably already have one running, so it’s easier (in a virtual world) to spin up two new ones and pull in the config later. If you can’t, it’s easy enough to translate the instructions here to add one.

Installation

Spin up two routers virtually or physically as needed. Name them gateway-1 and gateway-2. If your ISP only permits a single WAN address, you’ll need to start with them behind your existing router.

opkg update && opkg install keepalived
# The optional package keepalived-sync doesn't do anything useful in our case.
# Nor does the LuCI web interface for it.

Configuration

We are going to use a MASTER and BACKUP config as it’s easier to work with when getting started.

Set LAN IP

They need to stand on their own so assuming your existing router’s LAN gateway is .1 you’ll address the new routers as .2 and .3 in the /etc/config/network file. Once you’ve done that and restarted networking, you should test to ensure they work by setting a gateway manually on a test client.

Set Virtual LAN IP

The first step is to create the floating LAN interface. The configuration in OpenWrt is done with via the UCI files, not keepalived’s .conf files.

We’ll test with .4 before going live and potentially clobbering the existing gateway.

Router-1

This is our primary router.

# You don't really need to keep the old file, but it's interesting to look at.
mv /etc/config/keepalived ~/keepalived.orig

vi /etc/config/keepalived

config globals 'globals'
        option enabled '1'
        option router_id 'gateway-1'

config ipaddress
        option name 'vip_lan'
        option address '192.168.10.4/24'
        option device 'eth1'
        option scope 'global'

config vrrp_instance
        option name 'VI_LAN'
        option state 'MASTER'
        option interface 'eth1'
        option virtual_router_id '51'        
        option advert_int '1'
        list virtual_ipaddress 'vip_lan'

Router-2

This is our backup router

rm /etc/config/keepalived

vi /etc/config/keepalived

# Copy and paste the config from above but change two lines
...
...
        option router_id 'gateway-2'
...
...
        option state 'BACKUP'
...
...

Now restart keepalived on both systems and you’ll see your new address appear on the master. Stop keepalived on the master and you’ll see it appear on your backup system.

When you’re ready to move off your old router you can power it down, connect to the new and change the keepalived config to the virtual gateway. Doing this remotely is a fun magic trick to impress your friends and probably involves tmux and chained incus stop old router sleep incus reboot newrouter commands.

Set WAN IPs

There can be some issues when ISPs only allow one WAN address. Sometimes, it’s the first system that requests an address through the modem. Other times there’s double NAT and you need to ensure the WAN address stays the same for incoming port forwards.

If you have a static WAN address with double NAT or the ability to have multiple WAN addresses - i.e. your OpenWrt routers are behind another router - here’s how to handle it.

Change WAN To Static

# Check DHCP info if needed and set a static address for the main OpenWrt interface.

vi /etc/config/network

# Set this to something OTHER than your main wan address. 
config interface 'wan'
        option device 'eth0'
        option proto 'static'
        option ipaddr 'some.external.address'
        option netmask 'some.255.netmask'
        option gateway 'some.external.gateway'

/etc/init.d/network restart

Create a keepalived config

vi /etc/config/keepalived

# Add to the bottom of your existing file
config ipaddress
        option name 'vip_wan'
        option address 'the.main.wan.address/24'
        option device 'eth0'
        option scope 'global'

config vrrp_instance
        option name 'VI_WAN'
        option state 'MASTER'
        option interface 'eth0'
        option virtual_router_id '52'
        option advert_int '1'
        option garp_master_delay '1'
        option garp_master_repeat '5'
        list virtual_ipaddress 'vip_wan'

config vrrp_sync_group
        option name 'VG_1'
        list group 'VI_LAN'
        list group 'VI_WAN'

You’ll notice the sync_group we added at the bottom. That ensures that if either interface fails (like a cable failure) both are moved to the backup router.

Similar to before, copy the config to the backup router and change the option state to BACKUP.

Restart keepalived and you should see the WAN address move in tandem with the LAN address

Notes

This is a simple disaster recovery solution. We are not keeping connection state synced so at fail-over some connections will be interrupted. Though in testing most seem unaffected. A more enterprise solution is to set all nodes as backup and nopreempt, add conntrackd, and allow the addresses to move as part of maintenance or failure and minimize disruption moving them back.

This doesn’t keep your other config data in sync. There is a keepalived-sync package which installs rsync and some other utilities, but it seems to be aimed at keeping the keepalived config the same, not the rest of the router. To keep your settings consistent, you may want to setup ssh keys and rsync specific files from the master to the backup.

2.4 - OpenWrt in Incus

You don’t need a router with a Web GUI, but it’s convenient and Incus makes it easy by providing a base image you can spin up in a few seconds. This will already contain the incus-agent for integration with incus management commands.

Preparation

Incus provides a private network incusbr0 that handles the basics. But to to manage port forwarding, DNS and the like, you’ll need another. We created such a network when configuring Incus. Refer to that if you haven’t created one yet.

Installation

Just create a new instance and search for OpenWRT. The newest stable (non RC or snapshot) version is preferred. When creating, add a second network interface and change the name to eth1 before starting. Make sure both are connected to the bridge or vlan network you added when configuring Incus.

Configuration

Add LAN Interface

The first step is to get access. Use the Incus Web interface to inter the console and edit the config files.

# Configure the LAN interface
vi /etc/config/network

# Add this after the WAN entries
config interface 'lan'
    option device 'eth1'
    option proto 'static'
    option ipaddr '192.168.10.1'
    option netmask '255.255.255.0'

Then restart the network service to enable

/etc/init.d/network restart

Access The Web Interface

You need to be on the ‘inside’ to access the web interface. Since this is an overlay network where both interfaces are connected to the same LAN, you can simply change your workstation’s IP address manually to an address in that space.

If that’s not possible, you can stop the firewall service /etc/init.d/network restart to connect from the WAN side.

Access the web interface at http://192.168.10.1 and login as root (no password). Answer yes if prompted about checking for updates, and click Network -> Interfaces and answer yes again when prompted to allow it to update network names. Save and apply after accepting.

Notes

You may find that you’re running a release candidate despite not picking one. The image builds are not always tagged with ‘RC’ so you may check the latest actual release before grabbing one.

https://github.com/openwrt/openwrt/releases/latest

2.5 - OpenWRT in PVE LXC

When running lots of guests it helps to but them behind a virtual router. If you’re keeping things lean by using LXC containers you can put your router in a container too with OpenWRT.

The process in PVE is to:

Prepare Networking
Download OpenWRT
Create The Container
Edit The FW Init

Prepare Networking

You’re going to create a LAN inside of Proxmox and you can do it a couple of different ways;

Overlay
Additional Interface
VLAN

Overlay

The simplest thing to do is nothing. You just manually assign IPs and a gateway in a different range than your existing router and have two networks operating on the same physical LAN. The main downside is you can’t take advantage of DHCP because it would conflict with the original LAN.

Additional Interface

You can also install a second network card. This of course has a cost, though if you only have one PVE host you can cheat by just creating a new bridge interface that goes nowhere. But this isn’t helpful in a cluster.

VLAN

The best way is to add a Virtual LAN. Simply edit the config for vmbr0 and enable the VLAN aware checkbox. Then add an interface to the container and specify a VLAN Tag, such as “2”. Most network equipment is happy to pass it along to other cluster members so it just works.

Download OpenWRT

You want just the root file system, not the full image that includes the kernel. Happily, OpenWRT makes this available. Navigate to their releases, find the most recent, and drill down to targets / x86 / 64 / rootfs.tar.gz. It will save along the lines of “openwrt-24.10.1-x86-64-rootfs.tar.gz”.

Next, upload it to PVE with a secure copy to the root home folder like scp openwrt* root@pve01:

Create The Container

What we uploaded earlier isn’t actually a template, but it’s close enough as along as we create the container at PVE’s command line¹. The key here is that we provide an archive and set the OS type to unmanaged.

pct create \
 201 \
 ./openwrt* \
 --rootfs local-lvm:0.4 \
 --ostype unmanaged \
 --hostname openwrt \
 --arch amd64 \
 --cores 2 \
 --memory 256 \
 --swap 0 \
 --features nesting=1 \
 --net0 name=eth0,bridge=vmbr0,tag=2 \
 --net1 name=eth1,bridge=vmbr0

Also of note, we enable nesting so that dnsmasq will start² and set the VLAN tag on eth0 which comes up as the LAN interface by default on this image of OpenWRT. The container’s disk uses the rootfs syntax of STORAGE_ID:SIZE_IN_GiB, here being .4 Gigs.

Add Clients and Rules

When creating guests, make sure to change their network settings in PVE to have a VLAN tag of ‘2’ (or whatever you’re using).

In OpenWRT, add rules Network -> Firewall -> Port Forwards. There are no WAN rules discrete from port forwarding.

Updates

You should update by downloading new firmware, not by using the package manger. In fact: “Generally speaking, the use of opkg upgrade is very highly discouraged. It should be avoided in almost all circumstances³.”

But if you must;

opkg update
opkg list-upgradable | cut -f 1 -d ' ' | xargs opkg upgrade

A default install of PVE creates a single Linux Bridge, usually named vmbr0. Think of this as a virtual switch. The management interface is on that bridge, as well as any containers or guests. Most things just need one interface, but OpenWRT expects two. It is a router, after all.

In most cases, adding a VLAN is best, but there are other options. You can see and make changes in the Proxmox web GUI by changing to Server View, selecting a ProxMox Host, then going to System -> Network.

create a new bridge. Select new and allow it to select the name (which should be vmbr1). Leave the rest at the defaults (all blank with autostart checked). Important If you have a cluster you must actually connect this new bridge to a network adapter in the “Bridge ports” setting. Otherwise, it won’t be able to talk beyond the host it’s currently on. If you don’t have a second NIC, then this probably won’t do what you want.

2.6 - Syncing

OpenWrt doesn’t cluster. But you can get close by just syncing the configs. Assuming you’ve already deployed keepalived and contrackd of course.

Most of OpenWrt’s configuration is saved in a few text files in the /etc/config directory. It’s generally safe to sync the whole folder with the exception of the network file or others that have some instance-specific IP or names.

The simplest way is to generate SSH keys and use rsync on a schedule. If you want more control you can setup a realtime sync service via procd. But keep things simple if you can.

Scheduled Rsync

This works well if you have MASTER and BACKUP routers, ala keepalived.

Configure SSH Keys

Let’s use SSH keys to send changes on the other host and trigger a re-load.

# On the master router:
# Generate a key with modern cipher and no passphrase
mkdir .ssh
ssh-keygen -t ed25519 -f ~/.ssh/id_dropbear -N ""

# Copy that key to the other router. Manually, because ssh-copy-id isn't available. 
cat ~/.ssh/id_dropbear.pub | ssh [email protected] "cat >> /etc/dropbear/authorized_keys"

# Verify it worked
ssh [email protected]

Create a Cron Job

/etc/init.d/cron enable
/etc/init.d/cron start

crontab -e                

*/5 * * * * /usr/bin/rsync -avz --delete --exclude='keepalived' --exclude='network' /etc/config/ 192.168.1.2:/etc/config/ && ssh 192.168.1.2 "/sbin/reload_config"

Most changes will be replicated but it does leave out WireGuard clients. These are in the network file we are excluding. To tackle that you’ll want to perhaps use a template file. I’ll leave that to to you. Or you can tackle a realtime solution like below.

Realtime Sync

If need realtime sync, you can tap into OpenWrt’s init and daemon management system, procd. You can couple this with diff and patch to send just changes to things like the network config that you can’t copy wholesale.

Packages Needed

You’ll need the diff and patch programs.

opkg install diffutils patch

Configuration Storage

You also need to configure what you want to keep in sync. This is probably the:

firewall: port forwards and firewall rules
```
dhcp: hostnames and IP sets you create
```
network: wireguard peers you add

The OpenWrt way is to do this with a uci config file. This will let you change values from the command line or even a custom LuCI page later, should you aspire to that level of perfection.

# This file will be automatically detected. 
vi /etc/config/syncer

config settings 'main'
    option remote_ip '192.168.1.2'
    list configs 'firewall'
    list configs 'dhcp'
    list configs 'network'

Monitoring Service

Create service for procd to interact with. This service will be a run-and-exit approach as we have procd to keep an eye on things for us.

We’ll ask procd to call our reload function after specific config changes. We’ll to the actual work outside the service so we don’t become a blocker if something goes wrong (preventing a spinning Save and Apply action in the web interface).

vi /etc/init.d/syncer

#!/bin/sh /etc/rc.common

USE_PROCD=1
START=99  # Start last


service_triggers() {
 
    local CONFIGS=$(uci -q get syncer.main.configs)
    
    for CFG in $CONFIGS; do
        procd_add_config_trigger "config.change" "$CFG" /etc/init.d/config_syncer reload
    done
}

reload_service() {
    /usr/bin/syncer.sh &
}

start_service() {
    local CONFIGS=$(uci -q get syncer.main.configs)
    for CFG in $CONFIGS; do
        [ -f "/etc/config/$CFG" ] && cp "/etc/config/$CFG" "/tmp/$CFG.bak"
    done
}

Enable the service

chmod +x /etc/init.d/syncer
/etc/init.d/syncer enable
/etc/init.d/syncer start

You should now see a few .bak files in your /tmp directory.

Create The Sync Script

Now we need a simple script to create and send the patch files, then trigger a reload on the other router.

touch /usr/bin/syncer.sh
chmod +x /usr/bin/syncer.sh
vi /usr/bin/syncer.sh

#!/bin/sh

# Load settings using the 'uci' command
REMOTE_IP=$(uci -q get syncer.main.remote_ip)
CONFIGS=$(uci -q get syncer.main.configs)

# Exit if no IP is set
[ -z "$REMOTE_IP" ] && exit 1

for CFG in $CONFIGS; do

	# If the backup file exists create a patch against it
    if [ -f /tmp/$CFG.bak ]; then

        diff -u /tmp/$CFG.bak /etc/config/$CFG > /tmp/$CFG.patch

		# If the patch has data send it and patch
		if [ -s /tmp/$CFG.patch ]; then
				
			scp /tmp/$CFG.patch $REMOTE_IP:/tmp/ && ssh $REMOTE_IP "patch /etc/config/$CFG < /tmp/$CFG.patch"
		fi
    fi

    # Update backup file for next time. 
    cp /etc/config/$CFG /tmp/$CFG.bak
done

# Ask the other router to reload it's changed configs
ssh $REMOTE_IP "/sbin/reload_config"

Make sure to install diff and patch on the other side.

Notes

This on-demand approach works in my testing, but I haven’t run it for a long time. I can imagine the configs getting out of sync should one system be off-line when a change is made as there is no catch-up. It might benefit from error handling when the scp fails.

As an alternative to diff and patch, I tried out the message bus ubus. You can hook into that, but it turns out it doesn’t capture events from the web GUI.

For the GUI, you can find the staged files in /tmp and transmit them. This would work well, but there’s not a great way to know exactly when staged changes are being applied, other than to use ionotify on the config files and suck in the staged changes before they are erased. That’s just too hacky.

DHCP - On larger deployments, clients can step on each other after a fail-over as the second server can lease IPs to new clients that are already in use. Dnsmasq keeps the leases in memory and writes them to /tmp/dhcp.leases whenever a change occurs. So you’d periodically rsync the /tmp/dhcp.leases file to the BACKUP and then have keepalived issue a killall -HUP dnsmasq" when a fail-over event happens. Then handle it moving back.

Contrackd - clients will lose their stateful-ness during a fail over. While this is fine for some web traffic, games and some streaming will die.

Active/Active - Some sources suggest splitting the roles with one router having the internal gateway and the other having the main WAN address. But this causes asymmetric routing. A packet might go out through Router A but the reply comes back through Router B. If you have a firewall (like OpenWrt’s fw4/nftables) enabled, Router B will see a “reply” packet for a connection it never saw start. It will flag this as “Invalid” and drop the packet. Though If you’re using NAT you can simply pass out internal gateways round-robin for both, and move VIPs should one fail.

2.7 - WireGuard

The official docs work well and are summarized below.

Installation

In the GUI, go to System -> Software, click the Update Lists button and search for “luci-proto-wireguard”. Installing that will pull in the needed dependency. Restart the network services via System → Startup → Initscripts -> network → Restart.

Configuration

Add a WireGuard interface

Select “Network → Interfaces → Add new interface” Input the name wg0 and select WireGuard VPN.

In the subsequent screen, find and click the “Generate new key pair” button and enter 51820 for the listen port.

For IP addresses enter an address and network that will encompass your VPN, such as 10.0.0.1/24

You can add peers in this interface as well.

Add Traffic Rules

You’ll need to create a new zone that allows forwarding to the LAN, and a rule to allow the WireGuard traffic in. Refer to section 6 in the docs for that.

If you want to do it at the command line, edit your /etc/config/firewall file and add

config zone             
        option name 'WireGuardVPN'
        option input 'ACCEPT'
        option output 'ACCEPT'
        option forward 'ACCEPT'       
        option mtu_fix '1'
        list network 'wg0'

config rule
        option src 'wan'
        option name 'Wireguard-incoming'
        list proto 'udp'
        option dest_port '51820'
        option target 'ACCEPT'

And for the Interface, to the /etc/config/interfaces add:

config interface 'wg0'
        option proto 'wireguard'
        option private_key 'xxxxx'
        option listen_port '51820'
        list addresses '10.0.0.1/24'

3 - OPNsense

10G Speeds

When you set an OPNsense system up with supported 10G cards, say the Intel X540-AT2, you can move 6 to 8 Gb a second. Though this is better than in the past, but not line speed.

# iperf between two systems routed through a dial NIC on OPNsense

[ ID] Interval       Transfer     Bandwidth
[  1] 0.0000-10.0040 sec  8.04 GBytes  6.90 Gbits/sec

This is because the packet filter is getting involved. If you disable that you’ll get closer to line speeds

Firewall –> Settings –> Advanced –> Disable Firewall

[ ID] Interval       Transfer     Bandwidth
[  1] 0.0000-10.0067 sec  11.0 GBytes  9.40 Gbits/sec

4 - VyOS

VyOS gets a lot of respect as a network appliance. It’s a Debian-based router/firewall descended from the Vyatta project and has a command line config similar to JUNOS. It scores well in speed and reliability tests, a free version is available and commercial support is easy to get.

They do steer you toward the rather expensive commercial option by limiting access to the LTS versions, but you can always download the rolling release, the beta, or build it from source with a fairly straight-forward docker build process.

Downloading the beta, or stream as they call it, can be done with:

wget https://community-downloads.vyos.dev/stream/1.5-stream-2025-Q1/vyos-1.5-stream-2025-Q1-generic-amd64.iso

And a creation something like

qm create 200 \
--name vyos \
--memory 2048 \
--net0 virtio,bridge=vmbr0 \
--net1 virtio,bridge=vmbr0,tag=2 \
--ide2 media=cdrom,file=local:iso/live-image-amd64.hybrid.iso \
--virtio0 local-lvm:15

Then it’s just a matter of booting from the iso and running the very simple [install] and [quick-start] process. Assuming you’re going with the normal setup, hit the console and enter something like this.

Step 1 - Configure the interfaces and enable remote access.

# Enter 'configure' mode
configure

# For the address space 192.168.1.0/24

# Configure the LAN and WAN ports, with eth0 being the WAN
set interfaces ethernet eth0 address dhcp
set interfaces ethernet eth0 description 'OUTSIDE'
set interfaces ethernet eth1 address '192.168.1.1/24'
set interfaces ethernet eth1 description 'LAN'

# Enable remote login
# set service ssh listen-address '192.168.1.1' # Possibly don't listen on WAN if you don't need it
set service ssh port '22'

# Commit the changes and save if they work
commit
save

Step 2 - configure DNS/DHCP. There’s a lot of text, so we usually to SSH in to continue.

# Configure LAN DHCP services
ssh [email protected]
configure
set service dhcp-server shared-network-name LAN subnet 192.168.1.0/24 option default-router '192.168.1.1'
set service dhcp-server shared-network-name LAN subnet 192.168.1.0/24 option name-server '192.168.1.1'
set service dhcp-server shared-network-name LAN subnet 192.168.1.0/24 option domain-name 'vyos.net'
set service dhcp-server shared-network-name LAN subnet 192.168.1.0/24 lease '86400'
set service dhcp-server shared-network-name LAN subnet 192.168.1.0/24 range 0 start '192.168.1.9'
set service dhcp-server shared-network-name LAN subnet 192.168.1.0/24 range 0 stop '192.168.1.254'
set service dhcp-server shared-network-name LAN subnet 192.168.1.0/24 subnet-id '1'

# Configure DNS services
set service dns forwarding cache-size '0'
set service dns forwarding listen-address '192.168.1.1'
set service dns forwarding allow-from '192.168.1.0/24'

commit
save

Step 3 - enable NAT

# Enable masquerade - assuming you need it.
set nat source rule 100 outbound-interface name 'eth0'
set nat source rule 100 source address '192.168.1.0/24'
set nat source rule 100 translation address masquerade

commit
save

Step 4 - Set Names and Enable basic firewall rules

# Use the 'group' feature to give the interfaces more readable names than ethX
set firewall group interface-group WAN interface eth0
set firewall group interface-group LAN interface eth1
set firewall group network-group NET-INSIDE-v4 network '192.168.1.0/24'

# Typically you want a default drop as a global rule.
set firewall global-options state-policy established action accept
set firewall global-options state-policy related action accept
set firewall global-options state-policy invalid action drop






























[install]:https://docs.vyos.io/en/latest/installation/install.html#permanent-installation
[quick-start]:https://docs.vyos.io/en/latest/quick-start.html