HA With Keepalived

The simplest way to make routing highly-available is to have two routers and automate fail-over. The conventional way is with keepalived. It allows the gateway address to automatically move between your routers so clients can always connect.

Preparation

You need at least a pair of routers. You probably already have one running, so it’s easier (in a virtual world) to spin up two new ones and pull in the config later. If you can’t, it’s easy enough to translate the instructions here to add one.

Installation

Spin up two routers virtually or physically as needed. Name them gateway-1 and gateway-2. If your ISP only permits a single WAN address, you’ll need to start with them behind your existing router.

opkg update && opkg install keepalived
# The optional package keepalived-sync doesn't do anything useful in our case.
# Nor does the LuCI web interface for it.

Configuration

We are going to use a MASTER and BACKUP config as it’s easier to work with when getting started.

Set LAN IP

They need to stand on their own so assuming your existing router’s LAN gateway is .1 you’ll address the new routers as .2 and .3 in the /etc/config/network file. Once you’ve done that and restarted networking, you should test to ensure they work by setting a gateway manually on a test client.

Set Virtual LAN IP

The first step is to create the floating LAN interface. The configuration in OpenWrt is done with via the UCI files, not keepalived’s .conf files.

We’ll test with .4 before going live and potentially clobbering the existing gateway.

Router-1

This is our primary router.

# You don't really need to keep the old file, but it's interesting to look at.
mv /etc/config/keepalived ~/keepalived.orig

vi /etc/config/keepalived

config globals 'globals'
        option enabled '1'
        option router_id 'gateway-1'

config ipaddress
        option name 'vip_lan'
        option address '192.168.10.4/24'
        option device 'eth1'
        option scope 'global'

config vrrp_instance
        option name 'VI_LAN'
        option state 'MASTER'
        option interface 'eth1'
        option virtual_router_id '51'        
        option advert_int '1'
        list virtual_ipaddress 'vip_lan'

Router-2

This is our backup router

rm /etc/config/keepalived

vi /etc/config/keepalived

# Copy and paste the config from above but change two lines
...
...
        option router_id 'gateway-2'
...
...
        option state 'BACKUP'
...
...

Now restart keepalived on both systems and you’ll see your new address appear on the master. Stop keepalived on the master and you’ll see it appear on your backup system.

When you’re ready to move off your old router you can power it down, connect to the new and change the keepalived config to the virtual gateway. Doing this remotely is a fun magic trick to impress your friends and probably involves tmux and chained incus stop old router sleep incus reboot newrouter commands.

Set WAN IPs

There can be some issues when ISPs only allow one WAN address. Sometimes, it’s the first system that requests an address through the modem. Other times there’s double NAT and you need to ensure the WAN address stays the same for incoming port forwards.

If you have a static WAN address with double NAT or the ability to have multiple WAN addresses - i.e. your OpenWrt routers are behind another router - here’s how to handle it.

Change WAN To Static

# Check DHCP info if needed and set a static address for the main OpenWrt interface.

vi /etc/config/network

# Set this to something OTHER than your main wan address. 
config interface 'wan'
        option device 'eth0'
        option proto 'static'
        option ipaddr 'some.external.address'
        option netmask 'some.255.netmask'
        option gateway 'some.external.gateway'

/etc/init.d/network restart

Create a keepalived config

vi /etc/config/keepalived

# Add to the bottom of your existing file
config ipaddress
        option name 'vip_wan'
        option address 'the.main.wan.address/24'
        option device 'eth0'
        option scope 'global'

config vrrp_instance
        option name 'VI_WAN'
        option state 'MASTER'
        option interface 'eth0'
        option virtual_router_id '52'
        option advert_int '1'
        option garp_master_delay '1'
        option garp_master_repeat '5'
        list virtual_ipaddress 'vip_wan'

config vrrp_sync_group
        option name 'VG_1'
        list group 'VI_LAN'
        list group 'VI_WAN'

You’ll notice the sync_group we added at the bottom. That ensures that if either interface fails (like a cable failure) both are moved to the backup router.

Similar to before, copy the config to the backup router and change the option state to BACKUP.

Restart keepalived and you should see the WAN address move in tandem with the LAN address

Notes

This is a simple disaster recovery solution. We are not keeping connection state synced so at fail-over some connections will be interrupted. Though in testing most seem unaffected. A more enterprise solution is to set all nodes as backup and nopreempt, add conntrackd, and allow the addresses to move as part of maintenance or failure and minimize disruption moving them back.

This doesn’t keep your other config data in sync. There is a keepalived-sync package which installs rsync and some other utilities, but it seems to be aimed at keeping the keepalived config the same, not the rest of the router. To keep your settings consistent, you may want to setup ssh keys and rsync specific files from the master to the backup.