This is the multi-page printable view of this section. Click here to print.

Return to the regular view of this page.

Networking

1 - Internet

1.1 - DNS

Web pages today are complex. Your browser will make on average 401 DNS queries to find the various parts of an average web page, so implementing a local DNS system is key to keeping things fast.

In general, you can implement either a caching or recursive server with the choice between speed vs privacy.

Types of DNS Servers

A caching server accepts and caches queries, but doesn’t actually do the lookup itself. It forwards the request on to another DNS server and waits for the answer. If you have a lot of clients configured to use it, chances are someone else has already asked for what you want and it can supply the answer quickly from cache.

A recursive server does more than just cache answers. It knows how to connect to the root of the internet and find out itself. If you need to find some.test.com, it will connect to the .com server, ask where test.com is, then connect to test.com and ask it for some.

Comparison

Between the two, the caching server will generally be faster. If you connect to a large DNS service they will almost always have things cached. You will also get geographically relevant results as content providers work with DNS providers to direct you to the closest content cache.

With a recursive server, you do the lookup yourself and no single entity is able to monitor your DNS queries. You also aren’t dependant upon any upstream provider. But you make every lookup ’the long way’, and that can take many hundreds of milliseconds on some cases, a large part of a page load time.

Testing

In an ad hoc test on a live network with about 5,000 residential user devices, about half the queries were cached. The other half were sent to either quad 9 or a local resolver. Quad 9 took about half the time that the local resolver did.

Here are the numbers - with Steve Gibson’s DNS benchmarker against pi-hole forwarding to a local resolver vs pi-hole forwarding to quad 9. Cached results excluded.

    Forwarder     |  Min  |  Avg  |  Max  |Std.Dev|Reliab%|
  ----------------+-------+-------+-------+-------+-------+
  - Uncached Name | 0.015 | 0.045 | 0.214 | 0.046 | 100.0 |
  - DotCom Lookup | 0.015 | 0.019 | 0.034 | 0.005 | 100.0 |
  ---<O-OO---->---+-------+-------+-------+-------+-------+

    Resolver      |  Min  |  Avg  |  Max  |Std.Dev|Reliab%|
  ----------------+-------+-------+-------+-------+-------+
  - Uncached Name | 0.016 | 0.078 | 0.268 | 0.079 | 100.0 |
  - DotCom Lookup | 0.018 | 0.035 | 0.078 | 0.017 | 100.0 |
  ---<O-OO---->---+-------+-------+-------+-------+-------+

Selection

This test is interesting, but not definitive. While the DNS benchmark shows that the uncached average is better, page load perception is different than the sum of DNS queries. A page metric test would be good, but in general, faster is better.

Use a caching server.

One last point: use your ISP’s name server when possible. They will direct you to their local content caching systems for Netflix, Google (YouTube) and Akamai. If you use quad 9 like I did, you may get to a regional content location, but you miss out on things optimized specifically for your geographic location.

They are (probably) capturing all your queries for monetization, and possibly directing you to their own their own advertising server when you mis-key in a domain name. So you’ll need to decide;

Speed vs privacy.

  1. Informal personal checking of random popular sites. ↩︎

1.1.1 - Pi-hole

Pi-hole is reasonable choice for DNS service, especially if you want metrics and reporting. A single instance will scale to 1000 active clients with just 1 core and 500M RAM and do a good job showing what’s going on.

Performance is limited with version 5, however. When you get past 1000 active clients you can take some mitigating steps, but the main process is single-threaded so you’re unlikely to get past 1500.

But for smaller deployments, it’s hard to beat.

Preparation

Prepare and secure a Debian system

Set a Static Address

sudo vi /etc/network/interfaces

Change

# The primary network interface
allow-hotplug eth0
iface eth0 inet dhcp

to

auto eth0
iface eth0 inet static
    address 192.168.0.2/24
    gateway 192.168.0.1

Secure Access with Nftables

Nftables is the modern way to set netfilter (localhost firewall) rules.

sudo apt install nftables
sudo systemctl enable nftables
sudo vi /etc/nftables.conf

#!/usr/sbin/nft -f

flush ruleset

table inet filter {
        chain input {
                type filter hook input priority 0;

                # accept any localhost traffic
                iif lo accept

                # accept already allowed and related traffic
                ct state established,related accept

                # accept DNS and DHCP traffic from internal only
                define RFC1918 = { 192.168.0.0/16, 10.0.0.0/8, 172.16.0.0/12 }
                udp dport { domain, bootps } ip saddr $RFC1918 ct state new accept
                tcp dport { domain, bootps } ip saddr $RFC1918 ct state new accept

                # accept web and ssh traffic on the first interface or from an addr range
                iifname eth0 tcp dport { ssh, http } ct state new accept
                 # or 
                ip saddr 192.168.0.1/24 ct state new accept

                # Accept pings
                icmp type { echo-request } ct state new accept

                # accept neighbor discovery otherwise IPv6 connectivity breaks.
                ip6 nexthdr icmpv6 icmpv6 type { nd-neighbor-solicit,  nd-router-advert, nd-neighbor-advert } accept

                # count other traffic that does match the above that's dropped
                counter drop
        }
}

sudo nft -f /etc/nftables.conf
sudo systemctl start nftables.service

Add Unattended Updates

This an optional, but useful service.

apt install unattended-upgrades

sudo sed -i 's/\/\/\(.*origin=Debian.*\)/  \1/' /etc/apt/apt.conf.d/50unattended-upgrades
sudo sed -i 's/\/\/\(Unattended-Upgrade::Remove-Unused-Kernel-Packages "true";\)/  \1/' /etc/apt/apt.conf.d/50unattended-upgrades
sudo sed -i 's/\/\/\(Unattended-Upgrade::Remove-Unused-Dependencies\) "false";/  \1 "true";/' /etc/apt/apt.conf.d/50unattended-upgrades
sudo sed -i 's/\/\/\(Unattended-Upgrade::Automatic-Reboot\) "false";/  \1 "true";/' /etc/apt/apt.conf.d/50unattended-upgrades

Installation

sudo apt install curl
curl -sSL https://install.pi-hole.net | bash

Configuration

Upstream Provider

Pi-hole is a DNS Forwarder, meaning it asks someone else on your behalf and caches the results for other people when they ask too.

Assuming the installation messages indicate success, the only thing you need do is pick that upstream DNS provider. In many cases your ISP is the fastest, but you may want to run a [DNS benchmark] with those added to find the best.

  • Settings -> DNS -> Upstream DNS Servers

Also, if you have more than one network, you may need to allow the other LANs to connect.

  • Interface settings -> Permit all origins (needed if you have multiple networks)

Block Lists

One of the main features of Pi-hole is that it blocks adds. But you may need to disable this for google or facebook search results to work as expected. The top search results are often ads and don’t work as expected when pi-hole is blocking them.

  • Admin Panel -> Ad Lists -> Status Column

You might consider adding security only lists instead, such as Intel’s below

Search the web for other examples.

Operation

DNS Cache Size

When Pi-hole looks up a name, it get’s back an IP address and what’s called a TTL (Time To Live). The latter tells Pi-hole how long it should keep the results in cache for.

If you’re asking for names faster than they are expiring, Pi-hole will ’evict’ things from cache before they should be. You can check this at:.

settings -> System -> DNS cache evictions:

You’ll notice that insertions keep climbing as things are added to the cache, but the cache number itself represents only those entries that are current. If you do see evictions, edit CACHE_SIZE in /etc/pihole/setupVars.conf

You can also check this at the command line

dig +short chaos txt evictions.bind @localhost

   dig +short chaos txt cachesize.bind
   dig +short chaos txt hits.bind
   dig +short chaos txt misses.bind

Having more than you need is a waste, however, when it could be used for disk buffers, etc. So don’t add more unless it’s needed.

Use Stale Cache

If you have the spare memory, you can tell Pi-hole not to throw away potentially-useful data with the use-stale-cache flag. It will instead wait for someone to ask for it, serve it up immediately even though it’s expired, then go out and check to see if anything has changed. So TTLs are ignored but unused entries are removed after 24 hours to keep the cache tidy.

This increases performance at the expense of more memory use.

# Add this to pi-hole's dnsmasq settings file
echo "use-stale-cache" >> /etc/dnsmasq.d/01-pihole.conf

Local DNS Entries

You can enter local DNS and CNAME entries via the GUI, (Admin Panel -> Local DNS), but you can also edit the config file for bulk entries.

For A records

vim /etc/pihole/custom.list

10.50.85.2 test.some.lan
10.50.85.3 test2.some.lan

For CNAME records

vim /etc/dnsmasq.d/05-pihole-custom-cname.conf

cname=test3.some.lan,test.some.lan

Upgrading

Pi-hole wasn’t installed via a repo, so it won’t be upgraded via the Unattended Upgrades service. It requires a manual command.

sudo pihole -up

Troubleshooting

Dashboard Hangs

Very busy pi-hole installations generate lots of data and (seemingly) hang the dashboard. If that happens, limit the about of data being displayed.

vi /etc/pihole/pihole-FTL.conf 

# Don't import the existing DB into the GUI - it will hang the web page for a long time
DBIMPORT=no

# Don't import more than an hour of logs from the logfile
MAXLOGAGE=1

# Truncate data older than this many days to keep the size of the database down
MAXDBDAYS=1

sudo systemctl restart pihole-FTL.service

Rate Limiting

The system has a default limit of 1000 queries in a 60 seconds window for each client. If your clients are proxied or relayed, you can run into this. This event is displayed in the dashbaord1 and also in the logs2.

sudo grep -i Rate-limiting /var/log/pihole/pihole.log /var/log/pihole/pihole.log

You may find the address 127.0.0.1 being rate limited. This can be due to pi-hole doing a reverse of all client IPs every hour. You can disable this with:

# In the pihole-FTL.conf
REFRESH_HOSTNAMES=NONE

DNS over HTTP

Firefox, if the user has not yet chosen a setting, will query use-application-dns.net. Pi-hole respods with NXDOMAIN3 as a signal to use pi-hole for DNS.

/etc/pihole/pihole-FTL.conf

Apple devices offer a privacy enhancing service4 that you can pay for. Those devices queries mask.icloud.com as a test and Pi-hole blocks that by default. The user will be notified you are blocking it.

# Signal that Apple iCloud Private Relay is allowed 
BLOCK_ICLOUD_PR=false

sudo systemctl reload pihole-FTL.service

Searching The Query Log Hangs DNS

On a very busy server, clicking show-all in the query log panel will hang the server as pihole-FTL works through it’s database. There is no solution, just don’t do it. The best alternative is to ship logs to a Elasticsearch or similar system.

Ask Yourself

The system continues to use whatever DNS resolver was initially configured. You may want it to use itself, instead.

# revert if pi-hole itself needs fixed.
sudo vi /etc/resolv.conf

nameserver 127.0.0.1

Other Settings

Pi-hole can be configured via it’s two main config files, /etc/pihole/setupVars.conf and pihole-FTL.conf, and you may find other useful settings.

In the v6, currently in beta, the settings are all in the .toml file, or via the gui via:

  • All Settings — Resolver – resolver.resolveIPv4 – resolver.resolveIPv6 – resolver.networkNames

  1. https://pi-hole.net/blog/2021/09/11/pi-hole-ftl-v5-9-web-v5-6-and-core-v5-4-released/#page-content ↩︎

  2. https://discourse.pi-hole.net/t/include-log-entry-in-pihole-ftl-log-when-client-hits-rate-limit/46798/12A ↩︎

  3. https://www.reddit.com/r/pihole/comments/113qkp5/i_am_seeing_useapplicationdnsnet_being_blocked/ ↩︎

  4. https://docs.pi-hole.net/ftldns/configfile/?h=mask#icloud_private_relay [DNS benchmark]:https://www.grc.com/dns/benchmark.htm ↩︎

1.1.2 - Pi-hole Unbound

Pi-hole by itself is just a DNS forwarding and caching service. It’s job is to consolidate requests and forward them on to someone else. That someone else is seeing all your requests.

If that concerns you, add your own DNS Resolver like Unbound. It knows how to fetch answers without any one entity seeing all your requests. It’s probably slower1, but possibly more secure.

Installation

sudo apt install unbound

Configuration

Unbound

The pi-hole guide for [unbound]:(https://docs.pi-hole.net/guides/dns/unbound/) includes a config block to copy and paste as directed. You should also add a config file for dnsmasq while you’re at it, to set EDNS packet sizes. (dnsmasq comes as part of pi-hole)

sudo vi /etc/dnsmasq.d/99-edns.conf

edns-packet-max=1232

When you check the status of unbound, you can ignore the warning: subnetcache:... as it’s just reminding you that data in the subnet cache (if you were to use it) can’t pre-fetched. There’s some conversation2 as to why it’s warning us.

The config includes prefetch, but you may also wish to add serve-expired to it, if you’re not already using use-stale-cache in Pi-hole.

# serve old responses from cache while waiting for the actual resolution to finish.
# don't set this if you're already doing it in Pi-hole
serve-expired: yes

sudo systemctl restart unbound.service

No additional setup is needed, but see the unbound page for more info.

Pi-hole

You must tel Pi-hole about the Resolver you’ve just deployed.

  • Settings -> DNS -> Upstream DNS Servers -> Custom 1 (Check and add 127.0.0.1#5335 as shown in the unbound guide linked above)

  1. DNS Speed Testing Recursive vs Forwarding ↩︎

  2. https://www.reddit.com/r/pihole/comments/11xb7pt/unbound_warning_subnetcache_prefetch_and/ ↩︎

1.1.3 - Pi-hole DHCP

Pi-hole can also do DHCP. However, the GUI only allows for a single range. On a large network you’ll need multiple ranges. You do this by editing the config files directly.

Interface-Based Ranges

In this setup, you have a separate interface per LAN. Easy to do in a virtual or VLAN environment, but you’ll have to define each in the /etc/network/interfaces file.

Let’s create a range from 192.168.0.100-200 tied to eth0 and a range of 192.168.1.100-200 tied to eth1. We’ll also specify the router and two DNS servers.

vim /etc/dnsmasq.d/05-dhcp.conf

dhcp-range=eth0,192.168.0.100,192.168.0.200,24h
dhcp-option=ens161,option:router,192.168.0.1
dhcp-option=ens161,option:dns-server,192.168.0.2,192.168.0.3

dhcp-range=eth1,192.168.1.100,192.168.1.200,24h
dhcp-option=ens161,option:router,192.168.1.1
dhcp-option=ens161,option:dns-server,192.168.1.2,192.168.1.3

# Shared by both
dhcp-option=option:netmask,255.255.0.0

# Respond immediately without waiting for other servers 
dhcp-authoritative

# Don't try and ping the address before assigning it
no-ping

dhcp-lease-max=10000
dhcp-leasefile=/etc/pihole/dhcp.leases

domain=home.lan

These settings can be implicit - i.e. we could have left out ethX in the range, but explicit is often better for clarity.

Note - the DHCP server (dnsmasq) is not enabled by default. You can do that in the GUI under Settings –> DHCP

Relay-Based Ranges

In this setup, the router relays DHCP requests to the server. Only one system network interface is required, though you must configure the router(s).

When configured, the relay (router) sets the relay-agent (giaddr) field, sends it to dnsmasq, which (I think) understands it’s a relayed request when it sees that field, and looks at it’s available ranges for a match. It’s also sets a tag that to be used for assigning different options, such as the gateway, per range.

dhcp-range=tag0,192.168.0.100,192.168.0.250,255.255.255.0,8h
dhcp-range=tag1,192.168.1.100,192.168.1.250,255.255.255.0,8h
dhcp-range=tag2,192.168.2.100,192.168.2.250,255.255.255.0,8h

dhcp-options=tag0,3,192.168.0.1
dhcp-options=tag1,3,192.168.1.1
dhcp-options=tag2,3,192.168.2.1

Sources

https://discourse.pi-hole.net/t/more-than-one-conditional-forwarding-entry-in-the-gui/11359

Troubleshooting

It’s possible that the DHCP part of dnsmasq doesn’t scale to many thousands of leases1

  1. https://news.ycombinator.com/item?id=33895187 ↩︎

2 - Routing

2.1 - Linux Router

Creating a Linux router is fairly simple. Some distros like Alpine Linux are well suited for it but any will do. I used Debian in this example.

Install the base OS without a desktop system. Assuming you have two network interfaces, pick one to be the LAN interface (traditionally the first one, eth0 or such) and set the address statically.

Basic Routing

To route, all you really need do is enable forwarding.

# as root

# enable
sysctl -w net.ipv4.ip_forward=1

# and persist
echo "net.ipv4.ip_forward=1" >> /etc/sysctl.conf

Private Range

If one side is a private network, such as in the 192.168, you probably need to masquerade. This assumes you already have nftables installed and it’s default rules in /etc/nftables.conf

# As root

# Add the firewall rules to masquerade
nft flush ruleset
nft add table nat
nft add chain nat postrouting { type nat hook postrouting priority 100\; }
nft add rule nat postrouting masquerade

# Persist the rules and enable the firewall
nft list ruleset >> /etc/nftables.conf
systemctl enable --now  nftables.service

DNS and DHCP

If you want to provide network services such as DHCP and DNS, you can add dnsmasq

sudo apt install dnsmasq

Assuming the LAN interface is named eth0 and set to 192.168.0.1.

vi  /etc/dnsmasq.d/netboot.conf 

interface=eth0
dhcp-range=192.0.1.100,192.0.1.200,12h
dhcp-option=option:router,192.168.0.1
dhcp-authoritative

systemctl enable --now  nftables.service

Firewall

You may want to add some firewall rules too.

# allow SSH from the lan interface
sudo nft add rule inet filter input iifname eth0 tcp dport ssh accept

# allow DNS and DHCP from the lan interface
sudo nft add rule inet filter input iifname eth0 tcp dport domain accept
sudo nft add rule inet filter input iifname eth0 udp dport {domain, bootps} 

# Change the default input policy to drop 
sudo nft add chain inet filter input {type filter hook input priority 0\; policy drop\;}

You can fine-tune these a bit more with the nft example.

2.2 - OPNsense

10G Speeds

When you set an OPNsense system up with supported 10G cards, say the Intel X540-AT2, you can move 6 to 8 Gb a second. Though this is better than in the past, but not line speed.

# iperf between two systems routed through a dial NIC on OPNsense

[ ID] Interval       Transfer     Bandwidth
[  1] 0.0000-10.0040 sec  8.04 GBytes  6.90 Gbits/sec

This is because the packet filter is getting involved. If you disable that you’ll get closer to line speeds

Firewall –> Settings –> Advanced –> Disable Firewall

[ ID] Interval       Transfer     Bandwidth
[  1] 0.0000-10.0067 sec  11.0 GBytes  9.40 Gbits/sec

3 - VPN

3.1 - Wireguard

Wireguard is the best VPN choice for most situations. It’s faster and simpler than its predecessors and what you should be using on the internet.

Key Concepts

  • Wireguard is works at the IP level and is designed for the Internet/WAN. It doesn’t include DHCP, bridging, or other low-level features
  • Participants authenticate using public-key cryptography, use UDP as a transport and do not respond to unauthenticated connection attempts.
  • Peer to Peer by default.

By the last point, we mean there is no central authority required. Each peer defines their own IP address, routing rules, and decides from whom they will accept traffic. Every peer must exchange public keys with every other other peer. Traffic is sent directly between configured peers. You can create that design, but it’s not baked-in.

Design

The way you deploy depends on what you’re doing, but in general you’ll either connect directly point-to-point or create a central server for remote access or management.

  • Hub and Spoke
  • Point to Point

Hub and Spoke

This is the classic setup where clients initiate a connection. Configure a wireguard server and tell your clients about it. This is also useful for remote management when devices are behind NAT. Perfrom the steps in:

And then choose based on if your goal is to:

  • Provide Remote Access - i.e. allow clients to access to your central network and/or the Internet.

or

  • Provide Remote Management - i.e. allow the server (or an admin console) to connect to the clients.

Point to Point

You can also have peers talk directly to each other. This is often used with routers to connect networks across the internet.

3.1.1 - Server

A central server gives remote devices a reachable target, allowing them to traverse firewalls and NAT and connect. Let’s create a server and generate and configure a remote device.

Preparation

You’ll need:

  • Public Domain Name or Static IP
  • Linux Server and the ability to port-forward UDP 51820 to it
  • To choose a routing network IP block

A dynamic domain name will work and it’s reasonably priced (usually free). You just need something for the peers to connect to, though a static IP is best. You can possibly break connectivity if your IP changes while your peers are connected or have the old IP cached.

We use Debian in this example and derivatives should be similar. UDP 51820 is the standard port but you can choose another if desired.

You must also choose a VPN network that doesn’t overlap with your existing networks. We use 192.168.100.0/24 in this example. This is the internal network used inside the VPN to route traffic.

Installation

sudo apt install wireguard-tools

Configuration

The server needs just a single config file, and it will look something like this:

[Interface]
Address = 192.168.100.1/24
ListenPort = 51820
PrivateKey = sGp9lWqfBx+uOZO8V5NPUlHQ4pwbvebg8xnfOgR00Gw=

We choose 192.168.100.0/24 as our VPN internal network and picked .1 as our server address (pretty standard), created a private key with the wg tool, and put that in the file /etc/wireguard/wg0.conf. Here’s the commands to do that.

# As root
cd /etc/wireguard/
umask 077

wg genkey > server_privatekey
wg pubkey < server_privatekey > server_publickey

read PRIV < server_privatekey

# We create the file wg0.conf here
cat << EOF > wg0.conf
[Interface]
Address = 192.168.100.1/24
ListenPort = 51820
PrivateKey = $PRIV
EOF

Operation

The VPN operates by creating network interface and loading a kernel module. You can use the linux ip command to add a network interface of type wireguard (that automatically loads the kernel module) or use the wg-quick command do do it for you.

Test the Interface

# The tool looks for the wg0.conf file you created earlier
wg-quick up wg0

ping 192.168.100.1

wg-quick down wg0

Enable The Service

For normal use, employ systemctl to create a service using the installed service file.

systemctl enable --now wg-quick@wg0

That’s it - add remote clients/peers and they will be able to connect.

Troubleshooting

When something is wrong, you don’t get an error message, you just get nothing. You bring up the client interface but you can’t ping the server. So turn on log messages on the server with this command.

echo module wireguard +p > /sys/kernel/debug/dynamic_debug/control
dmesg

# When done, send a '-p'

Key Errors

wg0: Invalid handshake initiation from 205.133.134.15:18595

In this case, you should check your keys and possibly take the server interface down and up.

Typeos

ifconfig: ioctl 0x8913 failed: No such device

Check your conf is named /etc/wireguard/wg0.conf and look for any mistakes. Replace from scratch if nothing else.

Firewall Issues

If you see no wireguard error messages, suspect your firewall. Since it’s UDP you can’t test the port directly, but you can use netcat.

# On the server
systemctl stop wg-quick@wg0
nc -ulp 51820  

# On the client.
nc -u some.server 51820  

# Type some text and it should be echoed on the server

3.1.2 - Client

In theory, the client is an autonomous entity with whom you negotiate IPs and exchange public keys. In practice, you’ll just create a conf file and distribute it.

Define a Client on The Server

Each participant must have a unique Key-Pair and IP address. You cannot reuse keys as they are hashed and used as for internal routing.

Generate a Key-Pair

# On the 'server'
cd /etc/wireguard
wg genkey > client_privatekey # Generates and saves the client private key
wg pubkey < client_privatekey # Displays the client's public key

Select an IP

Choose an IP for the client and add a block at the bottom of your server’s wg0.conf. It’s fine to just increment the IP as you add clients . Note the /32, meaning on traffic with that specific IP is accepted from this peer - it’s not a router on the other side, after all.

# Add this block to the bottom of your server's wg0.conf file

##  Some Client  ##
[Peer]
PublicKey = XXXXXX
AllowedIPs = 192.168.100.2/32

# Load the new config
wg-quick down wg0 &&  wg-quick up wg0

Create a Client Config File

This is the file that the client needs. It will look similar to this. The [Interface] is about the client and the [Peer] is about the server.

[Interface]
PrivateKey = THE-CLIENT-PRIVATE-KEY
Address = 192.168.100.2/32

[Peer]
PublicKey = YOUR-SERVERS-PUBLIC-KEY
AllowedIPs = 192.168.100.0/24
Endpoint = your.server.org:51820

Put in the keys and domain name, zip it up and send it on to your client as securely as possible. Maybe keep it around for when they loose it. One neat trick is to display a QR code right in the shell. Devices that have a camera can import from that.

qrencode -t ANSIUTF8 < client-wg0.conf

Test On The Client

On Linux

On the client side, install the tools and place the config file.

# Install the wireguard tools
sudo apt install wireguard-tools

# Copy the config file to the wireguard folder
sudo cp /home/you/client-wg0.conf /etc/wireguard/wg0.conf

sudo wg-quick up wg0
ping 192.168.100.1
sudo wg-quick down wg0

# Possibly enable this as a service or import as a network manager profile
systemctl enable --now wg-quick@wg0
## OR ##
# You may want to rename the file as that's used in nm as it's name
nmcli connection import type wireguard file client-wg0.conf
sudo nmcli connection modify client-wg0.conf autoconnect no

On Windows or Mac

You can download the client from here and add the config block

https://www.wireguard.com/install/

Test

You should be able to ping the server from the client and vice versa. If not, take a look at the troubleshooting steps in the Central Server page.

Next Steps

You’re connected to the server - but that’s it. You can’t do anything other than talk to it. The next step depends on if you want to:

3.1.3 - Remote Access

This is the classic road-warrior setup where remote peers initiate a connection to the central peer. That central system forwards their traffic onward to the corporate network.

Traffic Handling

The main choice is route or masquerade.

Routing

If you route, the client’s VPN IP address is what other devices see. This is generally preferred as it allows you to log who was doing what at the individual servers. But you must update your network equipment to treat the central server as a router.

Masquerading

Masquerading causes the server to translate all the traffic. This makes everything look like its coming from the server. It’s less secure, but less complicated and much quicker to implement.

For this example, we will masquerade traffic from the server.

Central Server Config

Enable Masquerade

Use sysctl to enable forwarding on the server and nft to add masquerade.

# as root
sysctl -w net.ipv4.ip_forward=1

nft flush ruleset
nft add table nat
nft add chain nat postrouting { type nat hook postrouting priority 100\; }
nft add rule nat postrouting masquerade

Persist Changes

It’s best if we add our new rules onto the defaults and enable the nftables service.

# as root
echo "net.ipv4.ip_forward=1" >> /etc/sysctl.conf

nft list ruleset >> /etc/nftables.conf

systemctl enable --now  nftables.service

Client Config

Your remote peer - the one you created when setting up the server - needs it’s AllowedIPs adjusted so it knows to send more traffic through the tunnel.

Full Tunnel

This sends all traffic from the client over the VPN.

AllowedIPs = 0.0.0.0/0

Split Tunnel

The most common config is to send specific networks through the tunnel. This keeps netflix and such off the VPN

AllowedIPs = 192.168.100.0/24, 192.168.XXX.XXX, 192.168.XXX.YYY

DNS

In some cases, you’ll need the client to use your internal DNS server to resolve private domain names. Make sure this server is in the AllowedIPs above.

[Interface]
PrivateKey = ....
Address = ...
DNS = 192.168.100.1

Access Control

Limit Peer Access

By default, everything is open and all the peers can talk to each other and the internet at large - even NetFlix! (they can edit their side of the connection at will). So let’s add some rules to the default filter table.

This example prevents peers from from talking to each other but let’s them ping the central server and reach the corporate network.

# Load the base config in case you haven't arleady. This includes the filter table
sudo nft -f /etc/nftables.conf

# Reject any traffic being sent outside the 192.168.100.0/24
sudo nft add rule inet filter forward iifname "wg0" ip daddr != 192.168.100.0/24 reject with icmp type admin-prohibited

# Reject any traffic between peers
sudo nft add rule inet filter forward iifname "wg0" oifname "wg0" reject with icmp type admin-prohibited

Grant Admin Access

You may want to add an exception for one of the addresses so that an administrator can interact with the remote peers. Order matters, so add it before before the other rules above

sudo nft -f /etc/nftables.conf

# Allow an special 'admin' peer full access and others to reply
sudo nft add rule inet filter forward iifname "wg0" ip saddr 192.168.100.2 accept
sudo nft add rule inet filter forward ct state {established, related} accept

# As above
...
...

Save Changes

Since this change is a little more complex, we’ll replace the existing file config file and add notes.

sudo vi /etc/nftables.conf

#!/usr/sbin/nft -f

flush ruleset

table inet filter {
        chain input {
                type filter hook input priority 0
        }
        chain forward {
                type filter hook forward priority 0

                # Accept admin traffic and responses
                iifname "wg0" ip saddr 192.168.100.2 accept
                iifname "wg0" ct state {established, related} accept

                # Reject other traffic between peers
                iifname "wg0" oifname "wg0" reject with icmp type admin-prohibited

                # Reject traffic outside the desired network
                iifname "wg0" ip daddr != 192.168.100.0/24 reject with icmp admin-prohibited
        }
        chain output {
                type filter hook output priority 0
        }
}
table ip nat {
        chain postrouting {
                type nat hook postrouting priority srcnat
                masquerade
        }
}

Note: The syntax of the file is slightly different than the command. You can use nft list ruleset to see how nft config and commands translate into running rules. For example - the policy accept is being appended. You may want to experiment with explicitly adding policy drop.

The forwarding chain is where routing type rules go (the input chain is traffic sent to the host itself). Prerouting might work as well, though it’s less common and not present by default.

Notes

The default nftable config file in Debian is:

#!/usr/sbin/nft -f

flush ruleset

table inet filter {
        chain input {
                type filter hook input priority filter;
        }
        chain forward {
                type filter hook forward priority filter;
        }
        chain output {
                type filter hook output priority filter;
        }
}

If you have old iptables rules you want to translate to nft, you can install iptables and add them (they get translated on the fly into nft) and nft list ruleset to see how to they turn out.

3.1.4 - Remote Mgmt

In this scenario peers initiate connections to the central server, making their way through NAT and Firewalls, but you don’t want to forward their traffic.

Central Server Config

No forwarding or masquerade is desired, so there is no additional configuration to the central server.

Client Config

The remote peer - the one you created when setting up the server - is already set up with one exception; a keep-alive.

When the remote peer establishes it’s connection to the central server, intervening firewalls allow you to talk back as they assume it’s in response. However, the firewall will eventually ‘close’ this window unless the client continues sending traffic occasionally to ‘keep alive’ the connection.

# Add this to the bottom of your client's conf file
PersistentKeepalive = 20

Firewall Rules

You should apply some controls to your clients to prevent them from talking to each other (and the server), and you also need a rule for the admin station. You can do this by adding rules to the forward chain.

# Allow an 'admin' peer at .2 full access to others and accept their replies
sudo nft add rule inet filter forward iifname "wg0" ip saddr 192.168.100.2 accept
sudo nft add rule inet filter forward ct state {established, related} accept
# Reject any other traffic between peers
sudo nft add rule inet filter forward iifname "wg0" oifname "wg0" reject with icmp type admin-prohibited

You can persist this change by editing your /etc/nftables.conf file to look like this.

sudo vi /etc/nftables.conf

#!/usr/sbin/nft -f

flush ruleset

table inet filter {
        chain input {
                type filter hook input priority 0;
        }
        chain forward {
                type filter hook forward priority 0;

                # Accept admin traffic
                iifname "wg0" ip saddr 192.168.100.2 accept
                iifname "wg0" ct state {established, related} accept

                # Reject other traffic between peers
                iifname "wg0" oifname "wg0" reject with icmp type admin-prohibited
        }
        chain output {
                type filter hook output priority 0;
        }
}
table ip nat {
        chain postrouting {
                type nat hook postrouting priority srcnat; policy accept;
                masquerade
        }
}

3.1.5 - Routing

In our remote access example we chose to masquerade. But you can route where your wireguard server forwards traffic with the VPN addresses intact. You must handle that on your network in one of the following ways.

Symmetric Routing

Classically, you’d treat the wireguard server like any other router. You’d create a management interface and/or a routing interface and advertise routes appropriately.

On a small network, you would simply overlay an additional IP range on top of the existing on by adding a second IP address on your router and put your wireguard server on that network. Your local servers will see the VPN addressed clients and send traffic to the router that will pass it to the wireguard server.

Asymmetric Routing

In a small network you might have the central peer on the same network as the other servers. In this case, it will be acting like a router and forwarding traffic, but the other servers won’t know about it and so will send replies back to their default gateway.

To remedy this, add a static route at the gateway for the VPN range that sends traffic back to the central peer. Asymmetry is generally frowned upon, but it gets the job done with one less hop.

Host Static Routing

You can also configure the servers in question with a static route for VPN traffic so they know to send it directly back to the Wireguard server. This is fastest but you have to visit every host. Though you can use DHCP to distribute this route in some cases.

3.1.6 - Point to Point

If both system are listening then either side can initiate a connection. That’s essentially what a Point-to-Point setup is. Simply translate create two ‘servers’ and add a peer block to each one about the other. They will connect as needed.

This is best done with a routed config where clients who know nothing about the VPN use one side as their gateway for a given network range, and the servers act as routers. I don’t have an example config for this, but if you’ve reached this point you can probably handle that yourself.

3.1.7 - TrueNAS Scale

You can directly bring up a Wireguard interface in TrueNAS Scale, and use that to remotely manage it.

Wireguard isn’t exposed in the GUI, so use the command line to create a config file and enable the service. To make it persistent between upgrades, add a cronjob to restore the config.

Configuration

Add a basic peer as when setting up a Central Server and save the file on the client as /etc/wireguard/wg1.conf. It’s rumored that wg0 is reserved for the TrueNAS cloud service. Once the config is in place, use wg-quick up wg1 command to test and enable as below.

nano /etc/wireguard/wg1.conf

systemctl enable --now wg-quick@wg1

If you use a domain name in this conf for the other side, this service will fail at boot because DNS isn’t up and it’s not easy to get it to wait. So add a pre-start to the service file to specifically test name resolution.

vi /lib/systemd/system/[email protected]

[Service] 
...
...
ExecStartPre=/bin/bash -c 'until host google.com; do sleep 1; done'

Note: Don’t include a DNS server in your wireguard settings or everything on the NAS will attempt to use your remote DNS and fail if the link goes down.

Accessing Apps

When deploying an app, click the enable “Host Network” or “Configure Host Network” box in the apps config and you should be able to access via the VPN address. On Cobia (23.10) at least. If that fails, you can add a command like this to a post-start in the wireguard config file.

iptables -t nat -A PREROUTING --dst 192.168.100.2 -p tcp --dport 20910 -j DNAT --to-destination ACTUAL.LAN.IP:20910

Detecting IP Changes

The other side of your connection may dynamic address and wireguard wont know about it. A simple solution is a cron job that pings the other side periodically, and if it fails, restarts the interface. This will lookup the domain name again and hopefully find the new address.

touch /etc/cron.hourly/wg_test
chmod +x /etc/cron.hourly/wg_test
vi /etc/cron.hourly/wg_test

#!/bin/sh
ping -c1 -W5 192.168.100.1 || ( wg-quick down wg1 ; wg-quick up wg1 )

Troubleshooting

Cronjob Fails

cronjob kills interface when it can’t ping

or

/usr/local/bin/wg-quick: line 32: resolvconf: command not found

Calling wg-quick via cron causes a resolvconf issue, even though it works at the command line. One solution is to remove any DNS config from your wg conf file so it doesn’t try to register the remote DNS server.

Nov 08 08:23:59 truenas wg-quick[2668]: Name or service not known: `some.server.org:port' Nov 08 08:23:59 truenas wg-quick[2668]: Configuration parsing error … Nov 08 08:23:59 truenas systemd[1]: Failed to start WireGuard via wg-quick(8) for wg1.

The DNS service isn’t available (yet), despite Requires=network-online.target nss-lookup.target already in the service unit file. One way to solve this is a pre-exec in the Service section of the unit file1. This is hacky, but none of the normal directives work.

The cron job above will bring the service up eventually, but it’s nice to have it at boot.

Upgrade Kills Connection

An upgrade comes with a new OS image and that replaces anything you’ve added, such as wireguard config and cronjobs. The only way to persist your Wireguard connection it to put a script on the pool and add a cronjob via the official interface2.

Add this script and change for your pool location. This is set to run every 5 min, as you probably don’t want to wait after an upgrade very long to see if it’s working. You can also use this to detect IP changes over the cron.hourly above.

# Create the location and prepare the files
mkdir /mnt/pool02/bin/
cp /etc/wireguard/wg1.conf /mnt/pool02/bin/
touch /mnt/pool02/bin/wg_test
chmod +x /mnt/pool02/bin/wg_test

# Edit the script 
vi /mnt/pool02/bin/wg_test

#!/bin/sh
ping -c1 -W5 192.168.100.1 || ( cp /mnt/pool02/bin/wg1.conf /etc/wireguard/ ; wg-quick down wg1 ; wg-quick up wg1 )


# Invoke the TrueNAS CLI and add the job
cli
task cron_job create command="/mnt/pool02/bin/wg_test" enabled=true description="test" user=root schedule={"minute": "*/5", "hour": "*", "dom": "*", "month": "*", "dow": "*"}

Notes

https://www.truenas.com/docs/core/coretutorials/network/wireguard/ https://www.truenas.com/community/threads/no-internet-connection-with-wireguard-on-truenas-scale-21-06-beta-1.94843/#post-693601

  1. https://serverfault.com/questions/867830/systemd-start-service-only-after-dns-is-available ↩︎

  2. https://www.truenas.com/docs/scale/scaletutorials/systemsettings/advanced/managecronjobsscale/ ↩︎

3.1.8 - Proxmox

Proxmox is frequently used in smaller environments for it’s ability to mix Linux Containers and Virtual Machines at very low cost. LCD - Linux Containers - are especially valuable as they give the benefits of virtualization with minimal overhead.

Using wireguard in a container simply requires adding the host’s kernel module interface.

Edit the container’s config

On the pve host, for lxc id 101:

echo "lxc.mount.entry = /dev/net/tun /dev/net/tun none bind create=file" >> /etc/pve/lxc/101.conf

Older Proxmox

In the past you had to install the module, or use the DKMS method. That’s no longer needed as the Wireguard kernel module is now available on proxmox with the standard install. You don’t even need to install the wireguard tools. But if you run into trouble you can go through these steps

apt install wireguard
modprobe wireguard

# The module will load dynamically when a conainter starts, but you can also manually load it
echo "wireguard" >> /etc/modules-load.d/modules.conf

3.1.9 - LibreELEC

LibreELEC and CoreELEC are Linux-based open source software appliances for running the Kodi media player. These can be used as kiosk displays and you can remotely manage them with wireguard.

Create a Wireguard Service

These systems have wireguard support, but use connman that lacks split-tunnel ability1. This forces all traffic through the VPN and so is unsuitable for remote management. To enable split-tunnel, create a wireguard service instead.

Create a service unit file

vi /storage/.config/system.d/wg0.service

[Unit]
Description=start wireguard interface

# The network-online service isn't guaranteed to work on *ELEC
#Requires=network-online.service

After=time-sync.target
Before=kodi.service

[Service]
Type=oneshot
RemainAfterExit=true
StandardOutput=journal

# Need to check DNS is responding before we proceed
ExecStartPre=/bin/bash -c 'until nslookup google.com; do sleep 1; done'

ExecStart=ip link add dev wg0 type wireguard
ExecStart=ip address add dev wg0 10.1.1.3/24
ExecStart=wg setconf wg0 /storage/.config/wireguard/wg0.conf
ExecStart=ip link set up dev wg0
# On the newest version, a manual route addition is needed too
ExecStart=ip route add 10.2.2.0/24 dev wg0 scope link src 10.1.1.3

# Deleting the device seems to remove the address and routes
ExecStop=ip link del dev wg0

[Install]
WantedBy=multi-user.target

Create a Wireguard Config File

Note: This isn’t exactly the same file wg-quick uses, just close enough to confuse.

vi /storage/.config/wireguard/wg0.conf

[Interface]
PrivateKey = XXXXXXXXXXXXXXX

[Peer]
PublicKey = XXXXXXXXXXXXXXX
AllowedIPs = 10.1.1.0/24
Endpoint = endpoint.hostname:31194
PersistentKeepalive = 25

Enable and Test

systemctl enable --now wg0.service
ping 10.1.1.1

Create a Cron Check

When using a DNS name for the endpoint you may become disconnected. To catch this, use a cron job

# Use the internal wireguard IP address of the peer you are connecting to. .1 in this case
crontab -e
*/5 * * * * ping -c1 -W5 10.1.1.1 || ( systemctl stop wg0; sleep 5; systemctl start wg0 )

  1. https://wiki.libreelec.tv/configuration/wireguard#known-issues ↩︎

3.1.10 - OPNsense

The simplest way to deploy Wireguard is to use the built-in feature of your router. For OPNsense, it’s as simple as:

  • Create an Instance
  • Create a Peer and Enable Wireguard
  • Add a WAN Rule
  • Add a Wireguard Interface Rule

Configuration

Create an Instance

This is your server. Even though in wireguard all systems are considered peers, this is the system that is going to stay up all the time and accept connections, so it’s safe to think of it as ’the server'.

Navigate to:

VPN -> Wiregurad -> Instances

Click the + button on the right to add an instance. You can leave everything at the default except for:

  • Name # This can be anything you want, such as ‘Home’ or ‘Instance-1’
  • Public Key # Click the gear icon to generate keys
  • Listen Port # You’ll need to choose one, or it will somewhat unpredictable
  • Tunnel Address # Pick an IP range that you’re not using anywhere else

Save, but don’t click ‘Enable’ on the main screen yet.

Create a Peer

This is your phone or other enpoint that will be initiating the connection to the server. Navigate to:

VPN -> Wiregurad -> Peer Generator

It’s safe to leave everything at default except:

  • Endpoint # This your WAN address or hostname and port. e.g. “my.cool.org:51820”
  • Name # The thing connecting in, like “Allens-Phone”

If this is your first client, you may need to configure an IP. It’s safe to start one up from your server’s internal tunnel address, but don’t click the button for Store and generate next yet.

Copy the config box to a text file and get it to your client, or use the QR if you have a phone handy. Once you’ve saved the info, then click

“Store and generate next”

The GUI has automatically added the client to instance you created earlier, so at the bottom you can:

  • Enable Wiregaurd
  • Apply

(You can enable Wireguard at the bottom of any of these screens)

Add a WAN Rule

Firewall -> Rules -> WAN

Click ‘+’ to add a rule, and add

  • Interface: WAN
  • Protocol: UDP
  • Destination Port Range: (other) 51820

Add a Wireguard Interface Rule

Wireguard works by creating a network interface and Opnsense helpfully adds that alongside the LAN and WAN interfaces. You’ll notice it actually creates a group and if you had other instances they will (probably) be included.

Simply click the ‘+’ button to add a rule and save without changing any of the defaults. This allows you to leave the tunnel and talk to things on the LAN.

Operation

At this point you can connect from the client. If you look in the details it should add a line about ‘Latest handshake’ after a few seconds. If not, you’ll have to troubleshoot as below.

Adding new clients is similar to the first one, just make sure to disable and enable the service or the new clients won’t get picked up.

https://docs.opnsense.org/manual/how-tos/wireguard-client.html#step-4-b-create-an-outbound-nat-rule

Notes

I used the official setup guide at https://docs.opnsense.org/manual/vpnet.html#wireguard and it has a few flaws.

Mostly, it describes a more complex setup than just a remote access. They note two steps:

  • Create the server and peer
  • Create the rules. Under Firewall –> Rules, add one under
    • WAN
    • WireGuard (Group)

The issue is that the second category isn’t visible right away. Once it is, you can use the group, not the IP address. It’s unclear why the docs point you away from that.

Then I had to reboot to get it to work, which is very odd.

This turns out to be a general issue when you add a client and the service is already active. You can’t restart the service, you must disable and re-enable the service from the wireguard sub page