Syncing

OpenWrt doesn’t cluster. But you can get close by just syncing the configs. Assuming you’ve already deployed keepalived and contrackd of course.

Most of OpenWrt’s configuration is saved in a few text files in the /etc/config directory. It’s generally safe to sync the whole folder with the exception of the network file or others that have some instance-specific IP or names.

The simplest way is to generate SSH keys and use rsync on a schedule. If you want more control you can setup a realtime sync service via procd. But keep things simple if you can.

Scheduled Rsync

This works well if you have MASTER and BACKUP routers, ala keepalived.

Configure SSH Keys

Let’s use SSH keys to send changes on the other host and trigger a re-load.

# On the master router:
# Generate a key with modern cipher and no passphrase
mkdir .ssh
ssh-keygen -t ed25519 -f ~/.ssh/id_dropbear -N ""

# Copy that key to the other router. Manually, because ssh-copy-id isn't available. 
cat ~/.ssh/id_dropbear.pub | ssh [email protected] "cat >> /etc/dropbear/authorized_keys"

# Verify it worked
ssh [email protected]

Create a Cron Job

/etc/init.d/cron enable
/etc/init.d/cron start

crontab -e                

*/5 * * * * /usr/bin/rsync -avz --delete --exclude='keepalived' --exclude='network' /etc/config/ 192.168.1.2:/etc/config/ && ssh 192.168.1.2 "/sbin/reload_config"

Most changes will be replicated but it does leave out WireGuard clients. These are in the network file we are excluding. To tackle that you’ll want to perhaps use a template file. I’ll leave that to to you. Or you can tackle a realtime solution like below.

Realtime Sync

If need realtime sync, you can tap into OpenWrt’s init and daemon management system, procd. You can couple this with diff and patch to send just changes to things like the network config that you can’t copy wholesale.

Packages Needed

You’ll need the diff and patch programs.

opkg install diffutils patch

Configuration Storage

You also need to configure what you want to keep in sync. This is probably the:

• firewall: port forwards and firewall rules

• dhcp: hostnames and IP sets you create
  

• network: wireguard peers you add

The OpenWrt way is to do this with a uci config file. This will let you change values from the command line or even a custom LuCI page later, should you aspire to that level of perfection.

# This file will be automatically detected. 
vi /etc/config/syncer

config settings 'main'
    option remote_ip '192.168.1.2'
    list configs 'firewall'
    list configs 'dhcp'
    list configs 'network'

Monitoring Service

Create service for procd to interact with. This service will be a run-and-exit approach as we have procd to keep an eye on things for us.

We’ll ask procd to call our reload function after specific config changes. We’ll to the actual work outside the service so we don’t become a blocker if something goes wrong (preventing a spinning Save and Apply action in the web interface).

vi /etc/init.d/syncer

#!/bin/sh /etc/rc.common

USE_PROCD=1
START=99  # Start last


service_triggers() {
 
    local CONFIGS=$(uci -q get syncer.main.configs)
    
    for CFG in $CONFIGS; do
        procd_add_config_trigger "config.change" "$CFG" /etc/init.d/config_syncer reload
    done
}

reload_service() {
    /usr/bin/syncer.sh &
}

start_service() {
    local CONFIGS=$(uci -q get syncer.main.configs)
    for CFG in $CONFIGS; do
        [ -f "/etc/config/$CFG" ] && cp "/etc/config/$CFG" "/tmp/$CFG.bak"
    done
}

Enable the service

chmod +x /etc/init.d/syncer
/etc/init.d/syncer enable
/etc/init.d/syncer start

You should now see a few .bak files in your /tmp directory.

Create The Sync Script

Now we need a simple script to create and send the patch files, then trigger a reload on the other router.

touch /usr/bin/syncer.sh
chmod +x /usr/bin/syncer.sh
vi /usr/bin/syncer.sh

#!/bin/sh

# Load settings using the 'uci' command
REMOTE_IP=$(uci -q get syncer.main.remote_ip)
CONFIGS=$(uci -q get syncer.main.configs)

# Exit if no IP is set
[ -z "$REMOTE_IP" ] && exit 1

for CFG in $CONFIGS; do

	# If the backup file exists create a patch against it
    if [ -f /tmp/$CFG.bak ]; then

        diff -u /tmp/$CFG.bak /etc/config/$CFG > /tmp/$CFG.patch

		# If the patch has data send it and patch
		if [ -s /tmp/$CFG.patch ]; then
				
			scp /tmp/$CFG.patch $REMOTE_IP:/tmp/ && ssh $REMOTE_IP "patch /etc/config/$CFG < /tmp/$CFG.patch"
		fi
    fi

    # Update backup file for next time. 
    cp /etc/config/$CFG /tmp/$CFG.bak
done

# Ask the other router to reload it's changed configs
ssh $REMOTE_IP "/sbin/reload_config"

Make sure to install diff and patch on the other side.

Notes

This on-demand approach works in my testing, but I haven’t run it for a long time. I can imagine the configs getting out of sync should one system be off-line when a change is made as there is no catch-up. It might benefit from error handling when the scp fails.

As an alternative to diff and patch, I tried out the message bus ubus. You can hook into that, but it turns out it doesn’t capture events from the web GUI.

For the GUI, you can find the staged files in /tmp and transmit them. This would work well, but there’s not a great way to know exactly when staged changes are being applied, other than to use ionotify on the config files and suck in the staged changes before they are erased. That’s just too hacky.

DHCP - On larger deployments, clients can step on each other after a fail-over as the second server can lease IPs to new clients that are already in use. Dnsmasq keeps the leases in memory and writes them to /tmp/dhcp.leases whenever a change occurs. So you’d periodically rsync the /tmp/dhcp.leases file to the BACKUP and then have keepalived issue a killall -HUP dnsmasq" when a fail-over event happens. Then handle it moving back.

Contrackd - clients will lose their stateful-ness during a fail over. While this is fine for some web traffic, games and some streaming will die.

Active/Active - Some sources suggest splitting the roles with one router having the internal gateway and the other having the main WAN address. But this causes asymmetric routing. A packet might go out through Router A but the reply comes back through Router B. If you have a firewall (like OpenWrt’s fw4/nftables) enabled, Router B will see a “reply” packet for a connection it never saw start. It will flag this as “Invalid” and drop the packet. Though If you’re using NAT you can simply pass out internal gateways round-robin for both, and move VIPs should one fail.