OpenWrt

• 1: Dynamic DNS
• 2: GeoIP
• 3: HA With Keepalived
• 4: OpenWrt in Incus
• 5: OpenWRT in PVE LXC
• 6: Syncing
• 7: WireGuard

1 - Dynamic DNS

The official docs seem complicated, but it’s fairly easy.

Installation

Go to System -> Software, Click the Update Lists button then search for “ddns” and simply click on the DDNS script for the service you have, then do the same for “luci-app-ddns” so it adds the menu item to the web GUI.

Configuration

In the Web GUI, select Services -> Dynamic DNS and delete the demo entries at the bottom. Add a new service, choose IPv4 and and select the service provider you added the script for above. In the subsequent screen, you’ll need to know

• Lookup Hostname: This the FQDN, like “somehost.gattis.org”

•      Domain: Similar, just in the form of "[email protected]"
  

•    Username: "Bearer"
  

•    Password: This is an API key you generated
  

• Use HTTP Secure: Check this box, probably a good idea

If you’re behind NAT, you’ll also need to adjust the Advanced Settings

• IP address source: URL

• URL to detect: https://cloudflare.com/cdn-cgi/trace
  

Save and Apply, and then Reload. The Last Update/Next Update should change to a date.

2 - GeoIP

OpenWrt doesn’t ship with GeoIP capabilities, but you can add it with the IP set extras script. This is a somewhat legacy approach, but the GUI requires it, and it’s translated to modern nft named sets under the hood.

# At the command line
opkg update
opkg install ipset resolveip


# Install ipset-extras via their install script
wget -U "" -O ipset-extras.sh "https://openwrt.org/_export/code/docs/guide-user/advanced/ipset_extras?codeblock=0"
chmod +x ipset-extras.sh
./ipset-extras.sh

# Logout and back in to enable the extension from the /etc/profile.d folder
exit

# Configure an IP set for the US
uci set dhcp.us="ipset"
uci add_list dhcp.us.name="US"
uci add_list dhcp.us.name="US6"
uci add_list dhcp.us.geoip="us"
uci commit dhcp

# Populate IP sets
ipset setup

# Check set creation worked
nft list sets

# Install hotplug-extras for persistence
wget -U "" -O hotplug-extras.sh "https://openwrt.org/_export/code/docs/guide-user/advanced/hotplug_extras?codeblock=0"
chmod +x ./hotplug-extras.sh
./hotplug-extras.sh

When adding a port forward, the Advanced tab will now have the “Use ipset” populated and you can select “US”

You can also invert the rule by typing in “! US” - an important feature that doesn’t jump out at you.

This inversion is best with with a list of who you don’t want. Here’s an example of a set for the worst 5 countries for probes.

# Configure an IP set for the worst countries for probes and hacks - the axis of hacks
uci set dhcp.axis="ipset"
uci add_list dhcp.axis.name="axis"
uci add_list dhcp.axis.name="axis6"
uci add_list dhcp.axis.geoip="cn"
uci add_list dhcp.axis.geoip="in"
uci add_list dhcp.axis.geoip="ne"
uci add_list dhcp.axis.geoip="ng"
uci add_list dhcp.axis.geoip="ru"
uci commit dhcp

ipset setup

Troubleshooting

No NFT Sets Generated:

The script that generates the sets is sensitive to the value you put in `dhcp.axis.name=“value”. Try avoiding spaces and numbers and that ‘6’ is at the end of the second value.

Why are we adding these to the DHCP config section?:

I don’t know. The documentation adds them there. In the code, the script loads them from there. I tried adding them to the Firewall section just for fun and it didn’t work. I suspect if I knew more it would make sense.

Notes

The installation uses <www.ipdeny.com> and adds a cronjob that updates the list daily at 3 AM.

There are other tools, such as geopip-shell or ban ip, but these are more of an all-or-nothing solution and can’t be used with individual firewall rules. There is also the python utility from the netfilter team and misc bash scripts, but these lack easy OpenWrt integration.

The authors of nftables have a page[^5] on GeoIP, but it’s about tagging packets. OpenWrt expects named sets and you can’t easily construct that from a map.

[^5] https://wiki.nftables.org/wiki-nftables/index.php/GeoIP_matching

3 - HA With Keepalived

The simplest way to make routing highly-available is to have two routers and automate fail-over. The conventional way is with keepalived. It allows the gateway address to automatically move between your routers so clients can always connect.

Preparation

You need at least a pair of routers. You probably already have one running, so it’s easier (in a virtual world) to spin up two new ones and pull in the config later. If you can’t, it’s easy enough to translate the instructions here to add one.

Installation

Spin up two routers virtually or physically as needed. Name them gateway-1 and gateway-2. If your ISP only permits a single WAN address, you’ll need to start with them behind your existing router.

opkg update && opkg install keepalived
# The optional package keepalived-sync doesn't do anything useful in our case.
# Nor does the LuCI web interface for it.

Configuration

We are going to use a MASTER and BACKUP config as it’s easier to work with when getting started.

Set LAN IP

They need to stand on their own so assuming your existing router’s LAN gateway is .1 you’ll address the new routers as .2 and .3 in the /etc/config/network file. Once you’ve done that and restarted networking, you should test to ensure they work by setting a gateway manually on a test client.

Set Virtual LAN IP

The first step is to create the floating LAN interface. The configuration in OpenWrt is done with via the UCI files, not keepalived’s .conf files.

We’ll test with .4 before going live and potentially clobbering the existing gateway.

Router-1

This is our primary router.

# You don't really need to keep the old file, but it's interesting to look at.
mv /etc/config/keepalived ~/keepalived.orig

vi /etc/config/keepalived

config globals 'globals'
        option enabled '1'
        option router_id 'gateway-1'

config ipaddress
        option name 'vip_lan'
        option address '192.168.10.4/24'
        option device 'eth1'
        option scope 'global'

config vrrp_instance
        option name 'VI_LAN'
        option state 'MASTER'
        option interface 'eth1'
        option virtual_router_id '51'        
        option advert_int '1'
        list virtual_ipaddress 'vip_lan'

Router-2

This is our backup router

rm /etc/config/keepalived

vi /etc/config/keepalived

# Copy and paste the config from above but change two lines
...
...
        option router_id 'gateway-2'
...
...
        option state 'BACKUP'
...
...

Now restart keepalived on both systems and you’ll see your new address appear on the master. Stop keepalived on the master and you’ll see it appear on your backup system.

When you’re ready to move off your old router you can power it down, connect to the new and change the keepalived config to the virtual gateway. Doing this remotely is a fun magic trick to impress your friends and probably involves tmux and chained incus stop old router sleep incus reboot newrouter commands.

Set WAN IPs

There can be some issues when ISPs only allow one WAN address. Sometimes, it’s the first system that requests an address through the modem. Other times there’s double NAT and you need to ensure the WAN address stays the same for incoming port forwards.

If you have a static WAN address with double NAT or the ability to have multiple WAN addresses - i.e. your OpenWrt routers are behind another router - here’s how to handle it.

Change WAN To Static

# Check DHCP info if needed and set a static address for the main OpenWrt interface.

vi /etc/config/network

# Set this to something OTHER than your main wan address. 
config interface 'wan'
        option device 'eth0'
        option proto 'static'
        option ipaddr 'some.external.address'
        option netmask 'some.255.netmask'
        option gateway 'some.external.gateway'

/etc/init.d/network restart

Create a keepalived config

vi /etc/config/keepalived

# Add to the bottom of your existing file
config ipaddress
        option name 'vip_wan'
        option address 'the.main.wan.address/24'
        option device 'eth0'
        option scope 'global'

config vrrp_instance
        option name 'VI_WAN'
        option state 'MASTER'
        option interface 'eth0'
        option virtual_router_id '52'
        option advert_int '1'
        option garp_master_delay '1'
        option garp_master_repeat '5'
        list virtual_ipaddress 'vip_wan'

config vrrp_sync_group
        option name 'VG_1'
        list group 'VI_LAN'
        list group 'VI_WAN'

You’ll notice the sync_group we added at the bottom. That ensures that if either interface fails (like a cable failure) both are moved to the backup router.

Similar to before, copy the config to the backup router and change the option state to BACKUP.

Restart keepalived and you should see the WAN address move in tandem with the LAN address

Notes

This is a simple disaster recovery solution. We are not keeping connection state synced so at fail-over some connections will be interrupted. Though in testing most seem unaffected. A more enterprise solution is to set all nodes as backup and nopreempt, add conntrackd, and allow the addresses to move as part of maintenance or failure and minimize disruption moving them back.

This doesn’t keep your other config data in sync. There is a keepalived-sync package which installs rsync and some other utilities, but it seems to be aimed at keeping the keepalived config the same, not the rest of the router. To keep your settings consistent, you may want to setup ssh keys and rsync specific files from the master to the backup.

4 - OpenWrt in Incus

You don’t need a router with a Web GUI, but it’s convenient and Incus makes it easy by providing a base image you can spin up in a few seconds. This will already contain the incus-agent for integration with incus management commands.

Preparation

Incus provides a private network incusbr0 that handles the basics. But to to manage port forwarding, DNS and the like, you’ll need another. We created such a network when configuring Incus. Refer to that if you haven’t created one yet.

Installation

Just create a new instance and search for OpenWRT. The newest stable (non RC or snapshot) version is preferred. When creating, add a second network interface and change the name to eth1 before starting. Make sure both are connected to the bridge or vlan network you added when configuring Incus.

Configuration

Add LAN Interface

The first step is to get access. Use the Incus Web interface to inter the console and edit the config files.

# Configure the LAN interface
vi /etc/config/network

# Add this after the WAN entries
config interface 'lan'
    option device 'eth1'
    option proto 'static'
    option ipaddr '192.168.10.1'
    option netmask '255.255.255.0'

Then restart the network service to enable

/etc/init.d/network restart

Access The Web Interface

You need to be on the ‘inside’ to access the web interface. Since this is an overlay network where both interfaces are connected to the same LAN, you can simply change your workstation’s IP address manually to an address in that space.

If that’s not possible, you can stop the firewall service /etc/init.d/network restart to connect from the WAN side.

Access the web interface at http://192.168.10.1 and login as root (no password). Answer yes if prompted about checking for updates, and click Network -> Interfaces and answer yes again when prompted to allow it to update network names. Save and apply after accepting.

Notes

You may find that you’re running a release candidate despite not picking one. The image builds are not always tagged with ‘RC’ so you may check the latest actual release before grabbing one.

https://github.com/openwrt/openwrt/releases/latest

5 - OpenWRT in PVE LXC

When running lots of guests it helps to but them behind a virtual router. If you’re keeping things lean by using LXC containers you can put your router in a container too with OpenWRT.

The process in PVE is to:

• Prepare Networking
• Download OpenWRT
• Create The Container
• Edit The FW Init

Prepare Networking

You’re going to create a LAN inside of Proxmox and you can do it a couple of different ways;

• Overlay
• Additional Interface
• VLAN

Overlay

The simplest thing to do is nothing. You just manually assign IPs and a gateway in a different range than your existing router and have two networks operating on the same physical LAN. The main downside is you can’t take advantage of DHCP because it would conflict with the original LAN.

Additional Interface

You can also install a second network card. This of course has a cost, though if you only have one PVE host you can cheat by just creating a new bridge interface that goes nowhere. But this isn’t helpful in a cluster.

VLAN

The best way is to add a Virtual LAN. Simply edit the config for vmbr0 and enable the VLAN aware checkbox. Then add an interface to the container and specify a VLAN Tag, such as “2”. Most network equipment is happy to pass it along to other cluster members so it just works.

Download OpenWRT

You want just the root file system, not the full image that includes the kernel. Happily, OpenWRT makes this available. Navigate to their releases, find the most recent, and drill down to targets / x86 / 64 / rootfs.tar.gz. It will save along the lines of “openwrt-24.10.1-x86-64-rootfs.tar.gz”.

Next, upload it to PVE with a secure copy to the root home folder like scp openwrt* root@pve01:

Create The Container

What we uploaded earlier isn’t actually a template, but it’s close enough as along as we create the container at PVE’s command line¹. The key here is that we provide an archive and set the OS type to unmanaged.

pct create \
 201 \
 ./openwrt* \
 --rootfs local-lvm:0.4 \
 --ostype unmanaged \
 --hostname openwrt \
 --arch amd64 \
 --cores 2 \
 --memory 256 \
 --swap 0 \
 --features nesting=1 \
 --net0 name=eth0,bridge=vmbr0,tag=2 \
 --net1 name=eth1,bridge=vmbr0

Also of note, we enable nesting so that dnsmasq will start² and set the VLAN tag on eth0 which comes up as the LAN interface by default on this image of OpenWRT. The container’s disk uses the rootfs syntax of STORAGE_ID:SIZE_IN_GiB, here being .4 Gigs.

Add Clients and Rules

When creating guests, make sure to change their network settings in PVE to have a VLAN tag of ‘2’ (or whatever you’re using).

In OpenWRT, add rules Network -> Firewall -> Port Forwards. There are no WAN rules discrete from port forwarding.

Updates

You should update by downloading new firmware, not by using the package manger. In fact: “Generally speaking, the use of opkg upgrade is very highly discouraged. It should be avoided in almost all circumstances³.”

But if you must;

opkg update
opkg list-upgradable | cut -f 1 -d ' ' | xargs opkg upgrade

A default install of PVE creates a single Linux Bridge, usually named vmbr0. Think of this as a virtual switch. The management interface is on that bridge, as well as any containers or guests. Most things just need one interface, but OpenWRT expects two. It is a router, after all.

In most cases, adding a VLAN is best, but there are other options. You can see and make changes in the Proxmox web GUI by changing to Server View, selecting a ProxMox Host, then going to System -> Network.

create a new bridge. Select new and allow it to select the name (which should be vmbr1). Leave the rest at the defaults (all blank with autostart checked). Important If you have a cluster you must actually connect this new bridge to a network adapter in the “Bridge ports” setting. Otherwise, it won’t be able to talk beyond the host it’s currently on. If you don’t have a second NIC, then this probably won’t do what you want.

6 - Syncing

OpenWrt doesn’t cluster. But you can get close by just syncing the configs. Assuming you’ve already deployed keepalived and contrackd of course.

Most of OpenWrt’s configuration is saved in a few text files in the /etc/config directory. It’s generally safe to sync the whole folder with the exception of the network file or others that have some instance-specific IP or names.

The simplest way is to generate SSH keys and use rsync on a schedule. If you want more control you can setup a realtime sync service via procd. But keep things simple if you can.

Scheduled Rsync

This works well if you have MASTER and BACKUP routers, ala keepalived.

Configure SSH Keys

Let’s use SSH keys to send changes on the other host and trigger a re-load.

# On the master router:
# Generate a key with modern cipher and no passphrase
mkdir .ssh
ssh-keygen -t ed25519 -f ~/.ssh/id_dropbear -N ""

# Copy that key to the other router. Manually, because ssh-copy-id isn't available. 
cat ~/.ssh/id_dropbear.pub | ssh [email protected] "cat >> /etc/dropbear/authorized_keys"

# Verify it worked
ssh [email protected]

Create a Cron Job

/etc/init.d/cron enable
/etc/init.d/cron start

crontab -e                

*/5 * * * * /usr/bin/rsync -avz --delete --exclude='keepalived' --exclude='network' /etc/config/ 192.168.1.2:/etc/config/ && ssh 192.168.1.2 "/sbin/reload_config"

Most changes will be replicated but it does leave out WireGuard clients. These are in the network file we are excluding. To tackle that you’ll want to perhaps use a template file. I’ll leave that to to you. Or you can tackle a realtime solution like below.

Realtime Sync

If need realtime sync, you can tap into OpenWrt’s init and daemon management system, procd. You can couple this with diff and patch to send just changes to things like the network config that you can’t copy wholesale.

Packages Needed

You’ll need the diff and patch programs.

opkg install diffutils patch

Configuration Storage

You also need to configure what you want to keep in sync. This is probably the:

• firewall: port forwards and firewall rules

• dhcp: hostnames and IP sets you create
  

• network: wireguard peers you add

The OpenWrt way is to do this with a uci config file. This will let you change values from the command line or even a custom LuCI page later, should you aspire to that level of perfection.

# This file will be automatically detected. 
vi /etc/config/syncer

config settings 'main'
    option remote_ip '192.168.1.2'
    list configs 'firewall'
    list configs 'dhcp'
    list configs 'network'

Monitoring Service

Create service for procd to interact with. This service will be a run-and-exit approach as we have procd to keep an eye on things for us.

We’ll ask procd to call our reload function after specific config changes. We’ll to the actual work outside the service so we don’t become a blocker if something goes wrong (preventing a spinning Save and Apply action in the web interface).

vi /etc/init.d/syncer

#!/bin/sh /etc/rc.common

USE_PROCD=1
START=99  # Start last


service_triggers() {
 
    local CONFIGS=$(uci -q get syncer.main.configs)
    
    for CFG in $CONFIGS; do
        procd_add_config_trigger "config.change" "$CFG" /etc/init.d/config_syncer reload
    done
}

reload_service() {
    /usr/bin/syncer.sh &
}

start_service() {
    local CONFIGS=$(uci -q get syncer.main.configs)
    for CFG in $CONFIGS; do
        [ -f "/etc/config/$CFG" ] && cp "/etc/config/$CFG" "/tmp/$CFG.bak"
    done
}

Enable the service

chmod +x /etc/init.d/syncer
/etc/init.d/syncer enable
/etc/init.d/syncer start

You should now see a few .bak files in your /tmp directory.

Create The Sync Script

Now we need a simple script to create and send the patch files, then trigger a reload on the other router.

touch /usr/bin/syncer.sh
chmod +x /usr/bin/syncer.sh
vi /usr/bin/syncer.sh

#!/bin/sh

# Load settings using the 'uci' command
REMOTE_IP=$(uci -q get syncer.main.remote_ip)
CONFIGS=$(uci -q get syncer.main.configs)

# Exit if no IP is set
[ -z "$REMOTE_IP" ] && exit 1

for CFG in $CONFIGS; do

	# If the backup file exists create a patch against it
    if [ -f /tmp/$CFG.bak ]; then

        diff -u /tmp/$CFG.bak /etc/config/$CFG > /tmp/$CFG.patch

		# If the patch has data send it and patch
		if [ -s /tmp/$CFG.patch ]; then
				
			scp /tmp/$CFG.patch $REMOTE_IP:/tmp/ && ssh $REMOTE_IP "patch /etc/config/$CFG < /tmp/$CFG.patch"
		fi
    fi

    # Update backup file for next time. 
    cp /etc/config/$CFG /tmp/$CFG.bak
done

# Ask the other router to reload it's changed configs
ssh $REMOTE_IP "/sbin/reload_config"

Make sure to install diff and patch on the other side.

Notes

This on-demand approach works in my testing, but I haven’t run it for a long time. I can imagine the configs getting out of sync should one system be off-line when a change is made as there is no catch-up. It might benefit from error handling when the scp fails.

As an alternative to diff and patch, I tried out the message bus ubus. You can hook into that, but it turns out it doesn’t capture events from the web GUI.

For the GUI, you can find the staged files in /tmp and transmit them. This would work well, but there’s not a great way to know exactly when staged changes are being applied, other than to use ionotify on the config files and suck in the staged changes before they are erased. That’s just too hacky.

DHCP - On larger deployments, clients can step on each other after a fail-over as the second server can lease IPs to new clients that are already in use. Dnsmasq keeps the leases in memory and writes them to /tmp/dhcp.leases whenever a change occurs. So you’d periodically rsync the /tmp/dhcp.leases file to the BACKUP and then have keepalived issue a killall -HUP dnsmasq" when a fail-over event happens. Then handle it moving back.

Contrackd - clients will lose their stateful-ness during a fail over. While this is fine for some web traffic, games and some streaming will die.

Active/Active - Some sources suggest splitting the roles with one router having the internal gateway and the other having the main WAN address. But this causes asymmetric routing. A packet might go out through Router A but the reply comes back through Router B. If you have a firewall (like OpenWrt’s fw4/nftables) enabled, Router B will see a “reply” packet for a connection it never saw start. It will flag this as “Invalid” and drop the packet. Though If you’re using NAT you can simply pass out internal gateways round-robin for both, and move VIPs should one fail.

7 - WireGuard

The official docs work well and are summarized below.

Installation

In the GUI, go to System -> Software, click the Update Lists button and search for “luci-proto-wireguard”. Installing that will pull in the needed dependency. Restart the network services via System → Startup → Initscripts -> network → Restart.

Configuration

Add a WireGuard interface

Select “Network → Interfaces → Add new interface” Input the name wg0 and select WireGuard VPN.

In the subsequent screen, find and click the “Generate new key pair” button and enter 51820 for the listen port.

For IP addresses enter an address and network that will encompass your VPN, such as 10.0.0.1/24

You can add peers in this interface as well.

Add Traffic Rules

You’ll need to create a new zone that allows forwarding to the LAN, and a rule to allow the WireGuard traffic in. Refer to section 6 in the docs for that.

If you want to do it at the command line, edit your /etc/config/firewall file and add

config zone             
        option name 'WireGuardVPN'
        option input 'ACCEPT'
        option output 'ACCEPT'
        option forward 'ACCEPT'       
        option mtu_fix '1'
        list network 'wg0'

config rule
        option src 'wan'
        option name 'Wireguard-incoming'
        list proto 'udp'
        option dest_port '51820'
        option target 'ACCEPT'

And for the Interface, to the /etc/config/interfaces add:

config interface 'wg0'
        option proto 'wireguard'
        option private_key 'xxxxx'
        option listen_port '51820'
        list addresses '10.0.0.1/24'