This is the multi-page printable view of this section. Click here to print.
This is the documentation root. Use the left-hand nav bar to descend taxonomically, or use the search to find what you are after.
The thought for the structure is:
There’s really two areas; home and enterprise.
Home networking is what most of us think about. You have one network, eveyone gets an IP from it, and the router your ISP gave you handles things. You might mix it up with your own router, or create Internet Of Things networks, but mostly you don’t need to worry about routing.
Enterprise networking is where you have to think about buildings and routes and creating zones. That’s mostly what I focus on here. Back in the old days you could refer to the Cisco Campus WAN document. But that’s less useful these days and I’d have to point you to the Cisco Software-Defined Access (SD-Access) Solution Design Guide.
Or you can do what I’ve done; flatten everything and don’t let clients talk to each other. Supress all broadcasts except from the core down. Acomplishes much of what you need with 0 complexity.
A mesh usually means anything with more than one way to get somewhere. For purposes of hosts, it often shows up as all the hosts being directly connected to each other.
When you have three hosts, you can do this when each one as an extra dual interface NIC. But you probably shouldn’t. It saves you the cost of a switch but you’ll have to fight with it later on.
Here’s an example from PVE.
The classic example was to put the same IP address on both interfaces and use destination-based routing differently on each host, to make sure they take the right path.
Modern Trixie however, uses systemd-networkd and it precludes putting the same IP address on two interfaces.
One solution is to use split-brain DNS where each node has two different IP addresses, and they have different entires in the /etc/hosts file so that they’ll use the right IP depending on where they are.
Another is to use a bridge address. This is probably better. According to Google AI:
The most straightforward way to use a single IP across multiple physical ports in a mesh is to create a bridge device. This treats all your interfaces as a single logical switch.
Create the Bridge Device (/etc/systemd/network/10-br0.netdev):
[NetDev]
Name=br0
Kind=bridge
Bind Physical Interfaces (e.g., eth0 and eth1 in /etc/systemd/network/20-mesh.network):
[Match]
Name=eth0 eth1
[Network]
Bridge=br0
Assign the IP to the Bridge (/etc/systemd/network/30-br0.network):
[Match]
Name=br0
[Network]
Address=172.31.1.x/24
And Possibly
To prevent ARP Flux (where the wrong interface responds to an ARP request because both interfaces “see” the same subnet), you need to tell the Linux kernel to only respond if the IP is on the specific interface receiving the packet.
Add these to /etc/sysctl.d/99-linstor-mesh.conf:
# Only respond if the target IP is configured on the incoming interface
net.ipv4.conf.all.arp_ignore = 1
net.ipv4.conf.default.arp_ignore = 1
# Only use the address on the outgoing interface for ARP requests
net.ipv4.conf.all.arp_announce = 2
net.ipv4.conf.default.arp_announce = 2
# Enable strict reverse path filtering to prevent spoofing/asymmetric routing
net.ipv4.conf.all.rp_filter = 1
net.ipv4.conf.default.rp_filter = 1
sudo sysctl -p /etc/sysctl.d/99-linstor-mesh.conf
Network Access Control (NAC) is usually done with the 802.1X Network standard and a RADIUS server. At a base level, users identify themselves and are granted or denied access on an individual basis.
Role Based Access Control (RBAC) uses the same setup, but takes it a step further. It examines user attributes, such as job title, physical location, time of day, etc, to decide if they can have access, and even what network to put them on.
Windows and Mac PCs support this over physical ethernet with little or no configuration. But a more frequent use is with Enterprise WiFi, where you supply both a login name or email address and a password.
In our example we’ll use Linux with FreeRADIUS and OpenLDAP. Importantly, we’ll use the MSCHAPv2 password hash so that the user’s plaintext passwords don’t need to be stored and it directly integrates with Windows/Mac/Phones.
Install and configure as described in the following in order.
RADIUS is an Authentication and Authorization protocol and FreeRADIUS is the most widely deployed server. Others are quite good, but FreeRADIUS is, well, free.
Let’s install on Debian and test the EAP-PEAP protocols with MSCHAPv2 hashed passwords so we can integrate easily with Win/Mac. We’ll also include LDAP extensions for later.
# Start with a clean slate
apt remove --purge freeradius*
rm -rf /etc/freeradius/
apt install freeradius freeradius-ldap
The users file is simply a config file that FreeRADIUS processes when it starts up. You can add users directly to it and they’ll be processed accordingly.
vim /etc/freeradius/3.0/users
# Uncomment the line below
bob Cleartext-Password := "hello"
systemctl stop freeradius.service
/etc/init.d/freeradius debug
# In another terminal on the server, test that you can authenticate
radtest -t mschap bob hello localhost 0 testing123
You’re looking for the message Received Access-Accept. Take a look at the debug in the server console if you need to dig into errors.
We prefer not store plain-text passwords, so let’s use the smbcrypt utility to generate a hash and store that instead. The RADIUS server will simply use that directly rather than hashing the plain-text password it has stored now.
smbencrypt hello
# a quick way to extract just the nt hash
smbencrypt hello 2> /dev/null | tail -1 | awk '{print $2}'
# Change the bob entry
vim /etc/freeradius/3.0/users
bob NT-Password := "066DDFD4EF0E9CD7C256FE77191EF43C"
# Test the same as above
radtest -t mschap bob hello localhost 0 testing123
The above testing worked because the localhost is allowed to connect by default. To use this with an actual access point and client (authenticator and supplicant), we’ll need to create a ‘site’ by creating a file with the AP details and the server process that will talk to them.
You need access points that understand 802.1x. In our example we have some Unifi Access Points on a 10. network that can reach the RADIUS server.
Create a new ‘site’ named ‘wifi’ that describes our network.
vim /etc/freeradius/3.0/sites-available/wifi
#
# /etc/freeradius/3.0/sites-available/wifi
#
client UniFi-APs {
shortname = wifi
virtual_server = wifi
secret = someBigLongRandomString
# allow client connections from the 10.* range
ipaddr = 10.0.0.0/8
}
server wifi {
authorize {
# cleans up attributes, required
preprocess
# we use eap authentication, required
eap
}
authenticate {
# mschap authentication
Auth-Type MS-CHAP {
mschap
}
# eap, this is required
eap
}
}
ln -s /etc/freeradius/3.0/sites-available/wifi /etc/freeradius/3.0/sites-enabled/wifi
systemctl stop freeradius.service
/etc/init.d/freeradius debug
# If you have a workstation in the 10. space you can test this before jumping to a WiFi supplicant
radtest -t mschap bob hello someServer 0 someBigLongRandomString
Sometimes, users will enter their email address as their login - i.e. [email protected]. This is pretty normal these days, but you’ll need to tell RADIUS about it.
vi /etc/freeradius/3.0/proxy.conf
# at the bottom, add
realm example.org {
}
Configure your access point and you should be good - as long as you’re Bob and don’t mind editing text files. A better approach is a user database, like OpenLDAP
LDAP is preferred when dealing with Identity Management and while there are many alternatives to OpenLDAP, it is widely used and performs well.
# Start With a Clean Slate
apt remove --yes --purge slapd ldap-utils
rm -rf /var/backups/slapd*
# For a non-interactive install, supply a root password and domain to debconf before starting ap-get
WORD=somePass
DOMAIN=example.org
echo "slapd slapd/root_password password $PASSWORD" | debconf-set-selections &&\
echo "slapd slapd/root_password_again password $PASSWORD" | debconf-set-selections && \
echo "slapd slapd/domain string $DOMAIN" | debconf-set-selections && \
echo "slapd shared/organization string $DOMAIN" | debconf-set-selections && \
DEBIAN_FRONTEND=noninteractive apt-get install -y slapd ldap-utils
Create a text file that describes how we want to organize people, and a user ‘Bob. Notice that we add use the output of smbencrypt hello to the userPassword field, but prepend it with ‘{nthash}’. That’s a standard flag so FreeRADIUS and LDAP know it’s already hashed. It also prevents users from (easily) logging into OpenLDAP as it can’t use that hash itself.
vi people.ldif
dn: ou=people,dc=example,dc=org
objectClass: organizationalUnit
ou: people
vi bob.ldif
dn: cn=bob,ou=people,dc=example,dc=org
objectClass: person
cn: bob
sn: roberts
description: staff
userPassword: {nthash}066DDFD4EF0E9CD7C256FE77191EF43C
Now we’ll add the people organizational unit and the user bob.
ldapadd -x -W -D cn=admin,dc=example,dc=org -f people.ldif
ldapadd -x -W -D cn=admin,dc=example,dc=org -f bob.ldif
ldapsearch -x -LLL -b dc=example,dc=org
When you view your additions the userPassword field may be different than expected. That’s because it’s base64 encoded. You can decode it with echo (contents) | base64 --decode if you need to check.
Note: If you get the LDAP error ldap_bind: Invalid credentials (49) refer below to reset the root password.
We added the FreeRADIUS LDAP extension when we installed FreeRADIUS so all we need to do is make a couple edits.
vim /etc/freeradius/3.0/mods-available/ldap
# At the top, add the admin account credentials from LDAP and org with your own details
identity = 'cn=admin,dc=example,dc=org'
password = somePass
base_dn = 'ou=people,dc=example,dc=org'
# Further down in the 'update' section, comment out the control, request and reply attributes, and add your own reply that returns the LDAP
# attribute 'description' as the RADIUS attribute 'User-Category'
#control: += 'radiusControlAttribute'
#request: += 'radiusRequestAttribute'
#reply: += 'radiusReplyAttribute'
reply:User-Category := 'description'
# In the 'user' section change the filter from `cn=` to `uid=` if it's not already
filter = "(uid=%{%{Stripped-User-Name}:-%{User-Name}})"
# Enable to mod similar to before
ln -s /etc/freeradius/3.0/mods-available/ldap /etc/freeradius/3.0/mods-enabled/ldap
# Restart in debug and test the login
/etc/init.d/freeradius debug
radtest -t mschap bob hello localhost 0 testing123
Once again, you should see the Received Access-Accept message.
At this point, you have a working FreeRADIUS with an OpenLDAP back-end. All you need to do is import your users and you’re ready for production…well, you should secure it first.
Clients expect to see a certificate when connecting. The default Debian Snakeoil certificate causes some clients to show a red warning message (as it should).
Back in the bad old days, you had to purchase a ‘Unified Communication’ cert at a premium. But now you can use the free Let’s Encrypt project. It includes the EKU (Extended Key Usage) attribute “Server Authentication” which is required.
It can secure the inter-server communication channel as well.
There are a couple ways to prove yourself to Let’s Encrypt, but the simplest is let them connect to you at the DNS name you are requesting.
Create the public DNS CNAME “wifi.example.org” for your hopefully-already-existing reverse proxy server. Forward port 80 to your RADIUS server for later use.
# HAProxy Example
frontend http
bind *:80
mode http
acl wifi hdr(host) wifi.example.org
use_backend wifi if wifi
backend wifi
mode http
server wifi radius.private.lan:80
# Caddy Example
http://wifi.example.org {
import logging
reverse_proxy * radius.private.lan
}
The EFF makes a handy utility called certbot that will do the hard work for us. Though one hesitates to allow the internet in, certbot only listens for a few seconds every two months.
apt install certbot
certbot certonly --standalone -d wifi.example.org
ls /etc/letsencrypt/live/wifi.example.org/
The cert files start as root-only but any changes are preserved.
# Create a certs group (if it doesn't already exist) and add the processes to it.
sudo addgroup certs
sudo chgrp -R certs /etc/letsencrypt/live /etc/letsencrypt/archive
sudo chmod 750 /etc/letsencrypt/live /etc/letsencrypt/archive
Grant the freerad process access to the certs
sudo adduser freerad certs
vi /etc/freeradius/3.0/mods-available/eap
...
...
tls-config tls-common {
private_key_file = /etc/letsencrypt/live/wifi.example.org/privkey.pem
#private_key_file = /etc/ssl/private/ssl-cert-snakeoil.key
...
...
certificate_file = /etc/letsencrypt/live/wifi.example.org/fullchain.pem
#certificate_file = /etc/ssl/certs/ssl-cert-snakeoil.pem
systemctl restart freeradius
The best way to test is to grab an IOS device or Apple Laptop. These will display the Cert when you connect.
FreeRADIUS won’t know when a cert gets updated, so you’ll need to trigger a restart (a reload isn’t enough) with a certbot hook.
sudo touch /etc/letsencrypt/renewal-hooks/post/radius
sudo chmod +x /etc/letsencrypt/renewal-hooks/post/radius
sudo vi /etc/letsencrypt/renewal-hooks/post/radius
#!/bin/sh
logger certbot renewel triggered restart of freerad
systemctl restart freeradius.service
In this example everything is running on one host. But if the data was traversing the network you’d encrypt it. The modern way is to enable and enforce TLS using the certificate we just got. In fact, RFC4513 says servers should disallow the use of passwords when TLS is not in use, so let’s get on this.
First, let’s grant OpenLDAP access to the key.
sudo adduser openldap certs
Then, tell slapd about the cert by creating a configuration file in LDIF format and loading it.
vi cert.ldif
dn: cn=config
changetype: modify
replace: olcTLSCertificateKeyFile
olcTLSCertificateKeyFile: /etc/letsencrypt/live/wifi.example.org/privkey.pem
-
replace: olcTLSCertificateFile
olcTLSCertificateFile: /etc/letsencrypt/live/wifi.example.org/cert.pem
ldapmodify -Y EXTERNAL -H ldapi:/// -f cert.ldif
Test TLS is working with the ZZ option. (A single Z says means continue if negotiation fails, ZZ means stop if failed). Also use LDAPTLS_REQCERT=never. This allows us to verify that a cert is in place (ZZ) but don’t bother testing the validity (never).
# Use 'LDAPTLS_REQCERT=never' to trust any cert we get, but don't check the chain root yet
LDAPTLS_REQCERT=never ldapsearch -x -LLL -W -ZZ -D cn=admin,dc=example,dc=org -b dc=example,dc=org
There’s also the gold standard, openssl.
openssl s_client -starttls ldap -connect localhost:389
# Or a one-liner to decode as well
echo | openssl s_client -starttls ldap -connect localhost:389 | openssl x509 -text
Just like before, we need to tel slapd about the new cert.
sudo touch /etc/letsencrypt/renewal-hooks/post/ldap
sudo chmod +x /etc/letsencrypt/renewal-hooks/post/ldap
sudo vi /etc/letsencrypt/renewal-hooks/post/ldap
#!/bin/bash
logger certbot renewel triggered restart of openldap
systemctl restart slapd.service
We can now tell RADIUS to use TLS with LDAP. Use the ‘require_cert’ attribute to ignore the trust chain until later.
vi /etc/freeradius/3.0/mods-available/ldap
tls {
...
...
start_tls = yes
...
...
require_cert = 'never'
systemctl restart freeradius
# You can start in dubug mode like before to troubleshoot if needed
You may have noticed that are using certs now, but the processes themselves don’t care if they are valid. Just that we have one. We also haven’t done much with LDAP security. Let’s do that next.
Let’s take a few steps to block casual access, stop using the admin account, and validate the certificates we’re using.
By default, the LDAP server allows anonymous connections to read all the user entry data. Let’s prevent that by disabling anonymous binds.
Before we make any changes, this anonymous bind and search will return all the entries.
ldapsearch -x -LLL -b dc=example,dc=org
Let’s create a LDIF config file to disable that, load it and test again
vi disable_anon_bind.ldif
dn: cn=config
changetype: modify
add: olcDisallows
olcDisallows: bind_anon
# Modify the config
ldapmodify -H ldapi:/// -Y EXTERNAL -f disable_anon_bind.ldif
# An anonymous bind will now fail
ldapsearch -x -LLL -b dc=example,dc=org
# An authenticated search will succeed
ldapsearch -x -LLL -W -D cn=admin,dc=example,dc=org -b dc=example,dc=org
OpenLDAP accepts TLS connections, but it doesn’t require it. Now that we’re requiring passwords we should also require encryption. NOTE make sure systems talking to your LDAP server can use TLS and ignore/trust the Let’s Encrypt trust chain.
vi require-tls.ldif
dn: olcDatabase={1}mdb,cn=config
changetype: modify
add: olcSecurity
olcSecurity: tls=1
ldapmodify -H ldapi:// -Y EXTERNAL -f require-tls.ldif
See if it requires encryption.
# Test without -ZZ for no TLS
ldapsearch -x -LLL -W -D cn=admin,dc=example,dc=org -b dc=example,dc=org -h wifi.example.org
Enter LDAP Password:
ldap_bind: Confidentiality required (13)
additional info: TLS confidentiality required
Services should have their own accounts, so let’s create one for FreeRADIUS and a theoretical Identity Management Service. These accounts bind directly so we’ll set their passwords traditionally.
vi service_accts.ldif
dn: cn=FreeRADIUS,dc=example,dc=org
objectClass: person
cn: FreeRADIUS
sn: Server
description: service
dn: cn=IDM,dc=example,dc=org
objectClass: person
cn: IDM
sn: Server
description: service
ldapadd -x -W -D cn=admin,dc=example,dc=org -f service_accts.ldif
ldappasswd -x -W -D cn=admin,dc=example,dc=org -S cn=FreeRADIUS,dc=example,dc=org
ldappasswd -x -W -D cn=admin,dc=example,dc=org -S cn=IDM,dc=example,dc=org
The password attribute is protected by default. The FreeRADIUS account must read and the IDM account write it, so lets replace the ACL on that attribute.
vi FreeRADIUS-IDM-password.ldif
dn: olcDatabase={1}mdb,cn=config
changetype: modify
replace: olcAccess
olcAccess: {0}to attrs=userPassword by self write by anonymous auth by dn="cn=FreeRADIUS,dc=example,dc=org" read by dn="cn=IDM,dc=example,dc=org" write by * none
olcAccess: {1}to attrs=shadowLastChange by self write by * read
olcAccess: {2}to * by * read
The IDM account also gets the ability to manage the whole folder, so as to add and delete people.
vi IDM-Access.ldif
dn: olcDatabase={1}mdb,cn=config
changetype: modify
add: olcAccess
olcAccess: {2}to dn.exact="ou=people,dc=example,dc=org" attrs=children by dn="cn=IDM,dc=example,dc=org" manage
olcAccess: {3}to dn.children="ou=people,dc=example,dc=org" by dn="cn=IDM,dc=example,dc=org" manage
ldapmodify -Y EXTERNAL -H ldapi:/// -f FreeRADIUS-IDM-password.ldif
ldapmodify -Y EXTERNAL -H ldapi:/// -f idm-access.ldif
The FreeRADIUS user should now be able to query a user and see their password value
ldapsearch -x -LLL -W -D cn=FreeRADIUS,dc=example,dc=org -b dc=example,dc=org
...
userPassword:: .....
If you mess something up and need to reset ACLs, or understand more about the rules, see the miscellany.
Now that you’ve created an account for FreeRADIUS, you should go back and update it’s connection and test that it still works.
vi /etc/freeradius/3.0/mods-available/ldap
identity = 'cn=FreeRADIUS,dc=example,dc=org'
password = somePasswordYouSet
The server processes themselves are not validating the certs and that’s bad form. We’ve been skipping validation because in the past it was a hurdle. But Debian 12 and later include the ISRG Root X1 that Let’s Encrypt uses. Let’s test that now.
# And test with the CA option.
# You need to make sure the host name matches
echo 127.0.1.1 wifi.example.org >> /etc/hosts
ldapsearch -x -LLL -W -ZZ -D cn=admin,dc=example,dc=org -b dc=example,dc=org -h wifi.example.org
If this works as expected, you can attempt setting validate in the RADIUS settings.
vi /etc/freeradius/3.0/mods-available/ldap
start_tls = yes
ca_file = /etc/ssl/certs/ca-certificates.crt
require_cert = 'demand'
If these don’t work, you can compile a trust chain as described in the miscellany.
main: TLS init def ctx failed: -1
slapd can’t read the cert. Check the groups and unix permissions.
ldap_start_tls: Connect error (-11)
Check that the file you are specifying in ldapsearch, such LDAPTLS_CACERT=/root/r3-x1.pem, is actually available. Make sure you are using the -h and that the host you are connecting to matches the CN in the certificate. If it’s different, you’ll need to use a hosts file entry to trick the client
Nothing useful in logs
You can increase the log level from ‘stats’ to ‘args` with an ldapmodify. Just make sure to put it back when you’re done.
vi loglevel.ldif
dn: cn=config
changetype:modify
replace: olcLoglevel
#olcLoglevel: stats
olcLoglevel: args
ldapmodify -Y EXTERNAL -H ldapi:/// -b cn=config -D cn=config -s base -LLL -W -f loglevel.ldif
You’ve done a lot of work to get to this point where you can finally apply some Role Based Access Controls with Dynamic VLAN assignment.
IEEE 802.1X VLAN Assignment or Role Based Access Control is what it’s often called. It’s the ability to put people on different VLANs based on any imaginable criteria; time of day, type of computer, location connecting from, etc.
But it’s most often based on a role.
When we created users we added a description attribute, and then passed it to RADIUS as the standard User-Category attribute. Let’s use that to assign a VLAN.
This is a simple example that uses the User-Category attribute.
vim /etc/freeradius/3.0/sites-available/wifi
# After the `authenticate` section add a `post-auth` block
...
...
authenticate {
...
...
}
post-auth {
update reply {
&Tunnel-Type = 13,
&Tunnel-Medium-Type = 6
}
if (reply:User-Category == "staff") {
update reply { &Tunnel-Private-Group-Id = "1020" }
}
elsif (reply:User-Category == "student") {
update reply { &Tunnel-Private-Group-Id = "1021" }
}
else {
update reply { &Tunnel-Private-Group-Id = "1022" }
}
}
Make sure to edit the eap file as noted here. It’s a common gotcha.
vim /etc/freeradius/3.0/mods-available/eap
use_tunneled_reply = yes
You must configure your AP as per your vendor’s docs. For Unifi, it means you create a RADIUS profile that with those check-boxes enabled, then create a WiFi network that uses that profile.
You are often asked when the last time or location a user logged in. You can get that from the RADIUS logfile, or from implementing something like the ElasticStack.
You can create a quick script to check for user IDs and MAC addresses in the local log files.
vi search-netid
#!/bin/bash
[ $1 ] || { echo "enter search user ID"; exit 1;}
ID=$1
echo "Recent Logins: "
grep ") Login OK" /var/log/freeradius/radius.log /var/log/freeradius/radius.log.1 | grep $ID | sed 's/.*cli \(.*\)).*/\1/' | sort | uniq | tr '-' ':'| tr '[:upper:]' '[:lower:]'
echo
echo "Older Logins: "
zgrep ") Login OK" /var/log/freeradius/radius.log*.gz | grep $ID | sed 's/.*cli \(.*\)).*/\1/' | sort | uniq | tr '-' ':'| tr '[:upper:]' '[:lower:]'
Update as needed for modern versions of Elastic.
wget https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-7.6.2-amd64.deb
sudo apt install ./filebeat-7.6.2-amd64.deb
sudo vi /etc/filebeat/filebeat.yml
filebeat.inputs:
- type: log
paths:
- /var/log/freeradius/radius.log
include_lines: ['\) Login OK','incorrect']
tags: ["radius"]
processors:
- drop_event:
when:
contains:
message: "previously"
- if:
contains:
message: "Login OK"
then:
- dissect:
tokenizer: "%{key1} [%{source.user.id}/%{key3}cli %{source.mac})"
target_prefix: ""
- drop_fields:
fields: ["key1","key3"]
- script:
lang: javascript
source: >
function process(event) {
var mac = event.Get("source.mac");
if(mac != null) {
mac = mac.toLowerCase();
mac = mac.replace(/-/g,":");
event.Put("source.mac", mac);
}
}
else:
- dissect:
tokenizer: "%{key1} [%{source.user.id}/<via %{key3}"
target_prefix: ""
- drop_fields:
fields: ["key1","key3"]
output.elasticsearch:
hosts: ["http://some.server:9200"]
allow_older_versions: true
setup.ilm.enabled: false
#output.file:
#path: "/tmp/filebeat"
#filename: filebeat
#rotate_every_kb: 10000
#number_of_files: 7
#permissions: 0600
You could possibly skip LDAP and use only a user file.
#! /bin/sh
cd /etc/raddb
if [ ! -e .last-reload ] || [ "`find users -nt .last-reload`" ]; then
if radiusd -C > .last-reload 2>&1; then
kill -1 `cat /var/run/radiusd.pid`
else
mail -s "radius reload failed!" root < .last-reload
fi
fi
touch .last-reload
And it may be of some value to include multiple files with the $INCLUDE directive.
The ‘WiFi’ client group we created in the RADIUS setup stipulated they’d all be on the 10 network. To test from localhost, you’ll need to explicitly add that. A quick way is to edit the default site file.
Find the post-auth section, Add the block after the update block in the existing post-auth block. Here’s a static sample.
vi /etc/freeradius/3.0/sites-available/default
...
...
post-auth {
...
...
update {
&reply: += &session-state:
}
update reply {
&Tunnel-Type = 13,
&Tunnel-Medium-Type = 6,
&Tunnel-Private-Group-Id = "1020"
}
You can also test this with a users file, so as to testing getting the result from a user database.
# Remove the 'update reply' block and any static assignments added above
vi /etc/freeradius/3.0/sites-available/default
# The attributes will be assigned directly from the userdb to the reply
vim /etc/freeradius/3.0/users
bob NT-Password := "066DDFD4EF0E9CD7C256FE77191EF43C"
Tunnel-Type = "VLAN",
Tunnel-Medium-Type = "IEEE-802",
Tunnel-Private-Group-Id = "1020"
radtest should show you something like
radtest -t mschap bob hello localhost 0 testing123
...
...
Received Access-Accept Id 255 from 127.0.0.1:1812 to 127.0.0.1:37509 length 102
MS-CHAP-MPPE-Keys = 0x0000000000000000ac0782e2de2337dee40e54ee732c1af5
MS-MPPE-Encryption-Policy = Encryption-Allowed
MS-MPPE-Encryption-Types = RC4-40or128-bit-Allowed
Tunnel-Type:0 = VLAN
Tunnel-Medium-Type:0 = IEEE-802
Tunnel-Private-Group-Id:0 = "1020"
Note that with RADIUS attribute definitions you can use the numerical representation, 6, or the actual string “IEEE-802”. It’s rumored that the string works best with the user database, but it’s worth testing yourself.
The Let’s Encrypt certs are short-lived. Apple will prompt you again every time it renews, which is currently 2 months. It will show as valid, but they just want to notify you it’s changed. This can be annoying, but since the industry is moving towards shorter validity periods with a goal of 47 days by 2029, we can only hope Apple changes how they handle it.
for large numbers or real-time changes. One can use a relational database like MySQL, but most of the tools in the Identity Management space work better with LDAP. In the past, one would pass-thru the authentication directly to LDAP. But this doesn’t work with MSCHAPv2. Instead, we will treat OpenLDAP as a database as recommended in the FreeRADIUS docs: ‘We generally recommend that LDAP should be used as a database…"
There is an attribute for LDAP Passwords; userPassword. But this isn’t suitable for our use as it’s designed to be plain text string as per RFC4519. The admin password is salted SHA.
You may notice when using radtest that Bob’s password is returned as what looks like a hash. But it’s not. The userPassword attribute is a base64 encoded plain text string as per RFC4519. You can decode it with a echo Ym9ic1Bhc3M= | base64 --decode. This is one of the reasons we used a hash.
It’s counter intuitive, but many documents and even the LDAP RFC4519 prefer passwords stored in plain-text as this ensures compatibility with all protocols. In the above test, the password is encrypted by the client (-t mschap) before it’s sent. The client tells the server what algorithm it used and the server hashes it’s own copy on the fly to compare. If another client used a different protocol, the server could still handle it.
But we’re only interested in one password protocol, MSCHAPv2. That allows us to store just the hash on the server and makes security guys breathe easier. Do this with the smbcrypt utility that’s installed with FreeRADIUS.
You’ll notice that we’re storing the NT Hash with a header in the userPassword attribute. OpenLDAP doesn’t understand nthash as a scheme and so it blocks direct logins with it, but it’s happy to pass the data to RADIUS as a base64 encoded text string.
# Let's take a look at the person object to see we have to work with
ldapsearch -x -s base -b cn=Subschema objectClasses -LLL -o ldif-wrap=no | sed -nr '/person/ p' | sed -r 's/[$()]+/\n /g'
# There are several variants, but lets look at the first one, the basic 'person' object. You'll see something like
objectClasses:
2.5.6.6 NAME 'person' DESC 'RFC2256: a person' SUP top STRUCTURAL MUST
sn
cn
MAY
userPassword
telephoneNumber
seeAlso
description
# Let's take a look at the attributes to make sure we can just put text in them.
ldapsearch -x -o ldif-wrap=no -LLL -b cn=Subschema -s base '(objectClass=subschema)' attributeTypes | grep description
attributeTypes: ( 2.5.4.13 NAME 'description' DESC 'RFC4519: descriptive information' EQUALITY caseIgnoreMatch SUBSTR caseIgnoreSubstringsMatch SYNTAX 1.3.6.1.4.1.1466.115.121.1.15{1024} )
ldapsearch -x -o ldif-wrap=no -LLL -b cn=Subschema -s base '(objectClass=subschema)' ldapSyntaxes | grep 1.3.6.1.4.1.1466.115.121.1.15
ldapSyntaxes: ( 1.3.6.1.4.1.1466.115.121.1.15 DESC 'Directory String' )
# So what's a directory string? One or more aribtray characters. That will work fine for an aribitrary description like 'staff'
ldapsearch -x -o ldif-wrap=no -LLL -b cn=Subschema -s base '(objectClass=subschema)' attributeTypes | grep seeAlso
attributeTypes: ( 2.5.4.34 NAME 'seeAlso' DESC 'RFC4519: DN of related object' SUP distinguishedName )
# That won't work as it must refer to another ldap object. How about telephoneNumber
ldapsearch -x -o ldif-wrap=no -LLL -b cn=Subschema -s base '(objectClass=subschema)' attributeTypes | grep telephoneNumber
attributeTypes: ( 2.5.4.20 NAME 'telephoneNumber' DESC 'RFC2256: Telephone Number' EQUALITY telephoneNumberMatch SUBSTR telephoneNumberSubstringsMatch SYNTAX 1.3.6.1.4.1.1466.115.121.1.50{32} )
ldapsearch -x -o ldif-wrap=no -LLL -b cn=Subschema -s base '(objectClass=subschema)' ldapSyntaxes | grep 1.3.6.1.4.1.1466.115.121.1.50
ldapSyntaxes: ( 1.3.6.1.4.1.1466.115.121.1.50 DESC 'Telephone Number' )
# According to the web, that syntax is 'a string of [printable characters]' and that will work for a date if we needed to mis-use it.
If you have misconfigured your ACLs but can’t determine how, you may benefit from a reset.
# Default ACLs
vi acls-orig.ldif
dn: olcDatabase={1}mdb,cn=config
changetype: modify
replace: olcAccess
olcAccess: {0}to attrs=userPassword by self write by anonymous auth by * none
olcAccess: {1}to attrs=shadowLastChange by self write by * read
olcAccess: {2}to * by * read
ldapmodify -Y EXTERNAL -H ldapi:/// -f idm-replace.ldif
Take a look at the base, or default ACLs.
# su to root so we can use the EXTERNAL auth for a change.
su -
ldapsearch -Q -LLL -Y EXTERNAL -H ldapi:/// -b cn=config '(olcDatabase={1}mdb)' olcAccess
dn: olcDatabase={1}mdb,cn=config
olcAccess: {0}to attrs=userPassword by self write by anonymous auth by * none
olcAccess: {1}to attrs=shadowLastChange by self write by * read
olcAccess: {2}to * by * read
These rules are somewhat cryptic at first but you can puzzle them out once you see that it’s a set of three-word rules. The first line is the DN of rule object itself (which you can ignore) and the successive lines refer to a thing, and then who can do what with it.
In the first ACL, it’s referring to the attribute userPassword and says by self write and then by anon auth and then by * none. The middle one is a puzzler at first, but means anon can go back and try to authenticate, to be reconsidered later.
Importantly, these rules are short-circuit style. The by * none at the end of the first ACL will catch everyone, so even though the last ACL says by or everything by everyone read, the userPassword in the first ACL hit first and overrides for that attribute.
In our case, we need the FreeRADIUS user as a ‘read’ and the IDM user as a ‘write’. This is a modify-replace ldap operation as we need to replace rule 0.
vi FreeRADIUS-IDM-password.ldif
dn: olcDatabase={1}mdb,cn=config
changetype: modify
replace: olcAccess
olcAccess: {0}to attrs=userPassword by self write by anonymous auth by dn="cn=FreeRADIUS,dc=example,dc=org" read by dn="cn=IDM,dc=example,dc=org" write by * none
olcAccess: {1}to attrs=shadowLastChange by self write by * read
olcAccess: {2}to * by * read
ldapmodify -Y EXTERNAL -H ldapi:/// -f FreeRADIUS-IDM-password.ldif
We also need the IDM account to manage the user container. You must grant it specifically to the container itself (so it can add entries) and to the children (so it can delete and modify existing entries). This is a modify-add ldap operation and as we are adding these at rule 2, it will push anything after down automatically.
vi IDM-Access.ldif
dn: olcDatabase={1}mdb,cn=config
changetype: modify
add: olcAccess
olcAccess: {2}to dn.exact="ou=people,dc=example,dc=org" attrs=children by dn="cn=IDM,dc=example,dc=org" manage
olcAccess: {3}to dn.children="ou=people,dc=example,dc=org" by dn="cn=IDM,dc=example,dc=org" manage
ldapmodify -Y EXTERNAL -H ldapi:/// -f idm-access.ldif
That will give us the new ACL list of:
ldapsearch -Q -LLL -Y EXTERNAL -H ldapi:/// -b cn=config '(olcDatabase={1}mdb)' olcAccess
dn: olcDatabase={1}mdb,cn=config
olcAccess: {0}to attrs=userPassword by self write by anonymous auth by dn="cn=FreeRADIUS,dc=example,dc=org" read by dn="cn=IDM,dc=example,dc=org" write by * none
olcAccess: {1}to attrs=shadowLastChange by self write by * read
olcAccess: {2}to dn.exact="ou=people,dc=example,dc=org" attrs=children by dn="cn=IDM,dc=example,dc=org" manage
olcAccess: {3}to dn.children="ou=people,dc=example,dc=org" by dn="cn=IDM,dc=example,dc=org" manage
olcAccess: {4}to * by * read
ldap_bind: Invalid credentials (49)
I see this sometimes and it appears to be a bug with the debconf that pops up everyone so often. You must manually set the password.
# Get the hash for the password you want to use
slappasswd
# This command connects to an interactive ldap session
ldapmodify -Y EXTERNAL -H ldapi://
# Type in the next the lines to set the password
dn: olcDatabase={1}mdb,cn=config
changetype: modify
replace: olcRootPW
olcRootPW: {SSHA}4uDGf5aZylGWXoTde6Mvsp5CIph+XFNo
# Hit enter one more time and you should see the message
modifying entry "olcDatabase={1}mdb,cn=config"
# Hit Ctrl-C to end the interactive session, then restart slapd
Ctrl-C
systemctl restart slapd
LDAP distinguishes between anonymous and unauthenticated. The latter being when a user identifies themselves but doesn’t supply a password. This is disabled by default.
# Unauthenticated binds (user ID but no password) are disabled by default already
ldapsearch -x -LLL -w "" -D cn=admin,dc=example,dc=org -b dc=example,dc=org
If your system doesn’t have the Let’s Encrypt CA, you must compile the CA certs into a chain-file for yourself and any other clients who may not have a modern trust store.
# Who signed our cert?
openssl x509 -in /etc/letsencrypt/live/wifi.example.org/cert.pem -noout -subject -issuer
subject=CN = wifi.example.org
issuer=C = US, O = Let's Encrypt, CN = R3
# Where is their cert?
openssl x509 -in /etc/letsencrypt/live/wifi.example.org/cert.pem -text | grep "CA Issuers"
CA Issuers - URI:http://r3.i.lencr.org/
# Let's get that cert and see if it was in turned signed by someone else and where to get that cert
wget http://r3.i.lencr.org/ -O r3.der
openssl x509 -inform DER -in r3.der -noout -subject -issuer
subject=C = US, O = Let's Encrypt, CN = R3
issuer=C = US, O = Internet Security Research Group, CN = ISRG Root X1
openssl x509 -inform DER -in r3.der -text | grep "CA Issuers"
CA Issuers - URI:http://x1.i.lencr.org/
wget http://x1.i.lencr.org/ -O x1.der
# OK, if this is the 'root' signer, it will have itself as it's issuer
openssl x509 -inform DER -in x1.der -noout -subject -issuer
subject=C = US, O = Internet Security Research Group, CN = ISRG Root X1
issuer=C = US, O = Internet Security Research Group, CN = ISRG Root X1
# Success!
# Now lets put both of those in a .pem file for the apps to use
openssl x509 -inform DER -in x1.der > x1-r3.pem
openssl x509 -inform DER -in r3.der >> x1-r3.pem
# And test with the CA option.
# You need to make sure the host name matches
echo 127.0.1.1 wifi.example.org >> /etc/hosts
LDAPTLS_CACERT=./x1-r3.pem ldapsearch -x -LLL -W -ZZ -D cn=admin,dc=example,dc=org -b dc=example,dc=org -h wifi.example.org
# If you get unexpected results, you can use the -d -1 options for testing
LDAPTLS_CACERT=./x1-r3.pem ldapsearch -d -1 -x -LLL -W -ZZ -D cn=admin,dc=example,dc=org -b dc=example,dc=org -h wifi.example.org
# And let's add it to our general conf. Check /etc/ldap/ldap.conf for the right crt
su -c "cat x1-r3.pem >> /etc/ssl/certs/ca-certificates.crt"
# This will now work as expected
ldapsearch -x -LLL -W -ZZ -D cn=admin,dc=example,dc=org -b dc=example,dc=org -h wifi.example.org
Checking the schema LDAP object plain text show Resetting the ldap admin password http://deployingradius.com/documents/configuration/pap.html https://xenomorph.net/linux/ubuntu/misc/radius-unifi/ https://clintonmetu.com/2018/05/setting-up-freeradius-openldap-on-a-raspberry-pi-for-network-device-authentication/ https://wiki.debian.org/LDAP/OpenLDAPSetup
As a commercial alternative to FreeRADIUS, Microsoft does a decent RADIUS implementation as part of their Network Policy Server. It scales well and you can install and configure it with a few simple commands or use the GUI. It’s much simpler than FreeRADIUS.
I’ve scaled to to thousands of requests per second and it’s proven to work well with Aruba APs. But I’ve also had issues with Unifi Access Points. So it’s not always the best answer. Though that wa circa 2018 and it’s probably better now.
Here, we’ll configure it with the minimum needed to process RADIUS requests from the VPN server.
If you haven’t already, log into your Windows server, add the NPS role and register it with the domain1, The Windows firewall will auto-config2.
In an administrative powershell window:
Install-WindowsFeature NPAS -IncludeManagementTools
netsh nps add registeredserver YOURDOMAIN YOURNPSSERVER
NPS configuration has three parts; Who are my clients? When are they allowed to connect? and Should we give the end-user the thumbs up? This feels a little strange at times, but you get used to it.
The first step is to add the vpn server as a client3. The RADIUS server won’t even acknowledge your connection attempt unless it knows about you. In an admin powershell window:
# Add the client. Use the actual IP address to avoid Event ID 13 Errors
netsh nps add client name=<client name> address=<ip address> sharedsecret=<password>
The next component is a Connection Policy4. In practice, this is seldom used as anything other than a catch-all-allow-connect as the default policy shows. Though when in a federation, it determines how to route your connection. I.e. should your request be processed here, or should I proxy your request to an external server based on your ID.
Here, we add a very simple policy that looks at the IP and by default accepts the connection.
# Add a Connection Policy
# The default connection policy is sufficient but let's add an explicit one.
# Processing order doesn't work as documented. You must clear existing ones or
# explicitly specify processing order.
netsh nps reset crp
netsh nps add crp name = "VPN Connection" conditionid = "0x100c" conditiondata = "SomeIpAddress"
The last step is to add a Network Policy5. This determines if you are both Authentic and Allowed to connect. That’s an important point: the authorization decision is made at the RADIUS server, not the VPN server. You could in theory return a set of attributes to the VPN server. But in practice that’s not the approach that’s used6.
In this example we assume you have a group in AD to control access and looked up it’s SID.
# Add a Network Policy
netsh nps reset np
netsh nps add np
name = "Allow VPN Access"
conditionid = "0x1fb5" conditiondata = "S-1-5-21-1566005089-1651370962-3825839635-513" `
profileid = "0x100f" profiledata = "True"
profileid = "0x100a" profiledata = "1A000000000000000000000000000000"
profileid = "0x1009" profiledata = "0x5"
The Condition ID above isn’t documented but adding the condition from the GUI and issuing netsh nps show np reveals it. To see what attributes these profileid’s are setting, do a show np to see the desc.
The traditional way of providing a name and password is with 802.1x. It’s done by chaining together a few protocols to get “PEAP-MS-CHAPv2”. This allows you transmit a name and password from the client to the RADIUS server without the AP in middle seeing it.
One of the things you need is a certificate with an extension that specifically allows for server authentication.
Let’s Encrypt works fine.
If you want to go a different route, MS documentation identify that as Server Authentication purpose in Extended Key Usage (EKU) extensions. (The object identifier for Server Authentication is 1.3.6.1.5.5.7.3.1.) And the server certificate is configured with a required algorithm min value of RSA 2048. The Subject Alternative Name (SubjectAltName) extension, if used, must contain the DNS name of the server.
You can also do it the old way with MS’s cert req.
# Don't add the SAN as the provider will do tha in a web form usually
certreq -new radius.inf radius.csr
certreq –accept radius.cer
;----------------- request.inf -----------------
[Version]
Signature="$Windows NT$"
[NewRequest]
;Change to your,country code, company name and common name
Subject = "CN=radius.marietta.edu,OU=Information Technology,O=Marietta College,L=Marietta,S=Ohio,C=US,[email protected]"
KeySpec = 1
KeyLength = 2048
Exportable = TRUE
MachineKeySet = TRUE
SMIME = False
PrivateKeyArchive = FALSE
UserProtected = FALSE
UseExistingKeySet = FALSE
ProviderName = "Microsoft RSA SChannel Cryptographic Provider"
ProviderType = 12
RequestType = PKCS10
KeyUsage = 0xa0
[EnhancedKeyUsageExtension]
OID=1.3.6.1.5.5.7.3.1 ; this is for Server Authentication / Token Signing
;-----------------------------------------------
If you expect a lot of traffic you should increase the number of concurrent connection to AD so you don’t bottle-neck7. This is critical on a large install when you have more than a hundred authentications per second. (say, 10,000+ devices). This is unlikely for a VPN but wireless transaction can easily hit the limit.
# Edit the registry to increase the number of concurrent connections to AD
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Netlogon\Parameters 5
When using with Unifi, if the communication between the authenticator (the AP) and the RADIUS server has any issues, the AP will give up after 3 tries and won’t try again. Packet captures make it seem like the they disagree on how to implement the protocol in some cases. This happened to about 1% of our APs a day.
In these cases - a reboot of the AP was the quick fix.
No such problems with FreeRADIUS.
You may notice we’re referring to older docs here. The most current don’t have the details from the older docs when it comes to the netsh commands.
https://docs.microsoft.com/en-us/windows-server/networking/technologies/nps/nps-manage-install ↩︎
https://docs.microsoft.com/en-us/windows-server/remote/remote-access/vpn/always-on-vpn/deploy/vpn-deploy-dns-firewall#radius-traffic-ports-on-the-vpn-server-and-nps-server ↩︎
https://docs.microsoft.com/en-us/previous-versions/windows/it-pro/windows-server-2008-R2-and-2008/cc731207(v%3dws.10) ↩︎
https://docs.microsoft.com/en-us/previous-versions/windows/it-pro/windows-server-2008-R2-and-2008/cc732464(v%3dws.10)#connection-request-policies-commands ↩︎
https://docs.microsoft.com/en-us/previous-versions/windows/it-pro/windows-server-2008-R2-and-2008/cc755242(v%3dws.10)#network-policy-commands-1 ↩︎
Assume this to concentrate controls, but not sure. Need a citation. ↩︎
https://docs.microsoft.com/en-us/windows-server/networking/technologies/nps/nps-concurrent-auth ↩︎
Web pages today are complex. Your browser will make on average 401 DNS queries to find the various parts of an average web page, so implementing a local DNS system is key to keeping things fast.
In general, you can implement either a caching or recursive server with the choice between speed vs privacy.
A caching server accepts and caches queries, but doesn’t actually do the lookup itself. It forwards the request on to another DNS server and waits for the answer. If you have a lot of clients configured to use it, chances are someone else has already asked for what you want and it can supply the answer quickly from cache.
A recursive server does more than just cache answers. It knows how to connect to the root of the internet and find out itself. If you need to find some.test.com, it will connect to the .com server, ask where test.com is, then connect to test.com and ask it for some.
Between the two, the caching server will generally be faster. If you connect to a large DNS service they will almost always have things cached. You will also get geographically relevant results as content providers work with DNS providers to direct you to the closest content cache.
With a recursive server, you do the lookup yourself and no single entity is able to monitor your DNS queries. You also aren’t dependant upon any upstream provider. But you make every lookup ’the long way’, and that can take many hundreds of milliseconds on some cases, a large part of a page load time.
In an ad hoc test on a live network with about 5,000 residential user devices, about half the queries were cached. The other half were sent to either quad 9 or a local resolver. Quad 9 took about half the time that the local resolver did.
Here are the numbers - with Steve Gibson’s DNS benchmarker against pi-hole forwarding to a local resolver vs pi-hole forwarding to quad 9. Cached results excluded.
Forwarder | Min | Avg | Max |Std.Dev|Reliab%|
----------------+-------+-------+-------+-------+-------+
- Uncached Name | 0.015 | 0.045 | 0.214 | 0.046 | 100.0 |
- DotCom Lookup | 0.015 | 0.019 | 0.034 | 0.005 | 100.0 |
---<O-OO---->---+-------+-------+-------+-------+-------+
Resolver | Min | Avg | Max |Std.Dev|Reliab%|
----------------+-------+-------+-------+-------+-------+
- Uncached Name | 0.016 | 0.078 | 0.268 | 0.079 | 100.0 |
- DotCom Lookup | 0.018 | 0.035 | 0.078 | 0.017 | 100.0 |
---<O-OO---->---+-------+-------+-------+-------+-------+
This test is interesting, but not definitive. While the DNS benchmark shows that the uncached average is better, page load perception is different than the sum of DNS queries. A page metric test would be good, but in general, faster is better.
Use a caching server.
One last point: use your ISP’s name server when possible. They will direct you to their local content caching systems for Netflix, Google (YouTube) and Akamai. If you use quad 9 like I did, you may get to a regional content location, but you miss out on things optimized specifically for your geographic location.
They are (probably) capturing all your queries for monetization, and possibly directing you to their own their own advertising server when you mis-key in a domain name. So you’ll need to decide;
Speed vs privacy.
Informal personal checking of random popular sites. ↩︎
Pi-hole is reasonable choice for DNS service, especially if you want metrics and reporting. A single instance will scale to 1000 active clients with just 1 core and 500M RAM and do a good job showing what’s going on.
Performance is limited with version 5, however. When you get past 1000 active clients you can take some mitigating steps, but the main process is single-threaded so you’re unlikely to get past 1500.
But for smaller deployments, it’s hard to beat.
Prepare and secure a Debian system
sudo vi /etc/network/interfaces
Change
# The primary network interface
allow-hotplug eth0
iface eth0 inet dhcp
to
auto eth0
iface eth0 inet static
address 192.168.0.2/24
gateway 192.168.0.1
Nftables is the modern way to set netfilter (localhost firewall) rules.
sudo apt install nftables
sudo systemctl enable nftables
sudo vi /etc/nftables.conf
#!/usr/sbin/nft -f
flush ruleset
table inet filter {
chain input {
type filter hook input priority 0;
# accept any localhost traffic
iif lo accept
# accept already allowed and related traffic
ct state established,related accept
# accept DNS and DHCP traffic from internal only
define RFC1918 = { 192.168.0.0/16, 10.0.0.0/8, 172.16.0.0/12 }
udp dport { domain, bootps } ip saddr $RFC1918 ct state new accept
tcp dport { domain, bootps } ip saddr $RFC1918 ct state new accept
# accept web and ssh traffic on the first interface or from an addr range
iifname eth0 tcp dport { ssh, http } ct state new accept
# or
ip saddr 192.168.0.1/24 ct state new accept
# Accept pings
icmp type { echo-request } ct state new accept
# accept neighbor discovery otherwise IPv6 connectivity breaks.
ip6 nexthdr icmpv6 icmpv6 type { nd-neighbor-solicit, nd-router-advert, nd-neighbor-advert } accept
# count other traffic that does match the above that's dropped
counter drop
}
}
sudo nft -f /etc/nftables.conf
sudo systemctl start nftables.service
This an optional, but useful service.
apt install unattended-upgrades
sudo sed -i 's/\/\/\(.*origin=Debian.*\)/ \1/' /etc/apt/apt.conf.d/50unattended-upgrades
sudo sed -i 's/\/\/\(Unattended-Upgrade::Remove-Unused-Kernel-Packages "true";\)/ \1/' /etc/apt/apt.conf.d/50unattended-upgrades
sudo sed -i 's/\/\/\(Unattended-Upgrade::Remove-Unused-Dependencies\) "false";/ \1 "true";/' /etc/apt/apt.conf.d/50unattended-upgrades
sudo sed -i 's/\/\/\(Unattended-Upgrade::Automatic-Reboot\) "false";/ \1 "true";/' /etc/apt/apt.conf.d/50unattended-upgrades
sudo apt install curl
curl -sSL https://install.pi-hole.net | bash
Pi-hole is a DNS Forwarder, meaning it asks someone else on your behalf and caches the results for other people when they ask too.
Assuming the installation messages indicate success, the only thing you need do is pick that upstream DNS provider. In many cases your ISP is the fastest, but you may want to run a [DNS benchmark] with those added to find the best.
Also, if you have more than one network, you may need to allow the other LANs to connect.
One of the main features of Pi-hole is that it blocks adds. But you may need to disable this for google or facebook search results to work as expected. The top search results are often ads and don’t work as expected when pi-hole is blocking them.
You might consider adding security only lists instead, such as Intel’s below
Search the web for other examples.
When Pi-hole looks up a name, it get’s back an IP address and what’s called a TTL (Time To Live). The latter tells Pi-hole how long it should keep the results in cache for.
If you’re asking for names faster than they are expiring, Pi-hole will ’evict’ things from cache before they should be. You can check this at:.
settings -> System -> DNS cache evictions:
You’ll notice that insertions keep climbing as things are added to the cache, but the cache number itself represents only those entries that are current. If you do see evictions, edit CACHE_SIZE in /etc/pihole/setupVars.conf
You can also check this at the command line
dig +short chaos txt evictions.bind @localhost
dig +short chaos txt cachesize.bind
dig +short chaos txt hits.bind
dig +short chaos txt misses.bind
Having more than you need is a waste, however, when it could be used for disk buffers, etc. So don’t add more unless it’s needed.
If you have the spare memory, you can tell Pi-hole not to throw away potentially-useful data with the use-stale-cache flag. It will instead wait for someone to ask for it, serve it up immediately even though it’s expired, then go out and check to see if anything has changed. So TTLs are ignored but unused entries are removed after 24 hours to keep the cache tidy.
This increases performance at the expense of more memory use.
# Add this to pi-hole's dnsmasq settings file
echo "use-stale-cache" >> /etc/dnsmasq.d/01-pihole.conf
You can enter local DNS and CNAME entries via the GUI, (Admin Panel -> Local DNS), but you can also edit the config file for bulk entries.
For A records
vim /etc/pihole/custom.list
10.50.85.2 test.some.lan
10.50.85.3 test2.some.lan
For CNAME records
vim /etc/dnsmasq.d/05-pihole-custom-cname.conf
cname=test3.some.lan,test.some.lan
Pi-hole wasn’t installed via a repo, so it won’t be upgraded via the Unattended Upgrades service. It requires a manual command.
sudo pihole -up
Very busy pi-hole installations generate lots of data and (seemingly) hang the dashboard. If that happens, limit the about of data being displayed.
vi /etc/pihole/pihole-FTL.conf
# Don't import the existing DB into the GUI - it will hang the web page for a long time
DBIMPORT=no
# Don't import more than an hour of logs from the logfile
MAXLOGAGE=1
# Truncate data older than this many days to keep the size of the database down
MAXDBDAYS=1
sudo systemctl restart pihole-FTL.service
The system has a default limit of 1000 queries in a 60 seconds window for each client. If your clients are proxied or relayed, you can run into this. This event is displayed in the dashbaord1 and also in the logs2.
sudo grep -i Rate-limiting /var/log/pihole/pihole.log /var/log/pihole/pihole.log
You may find the address 127.0.0.1 being rate limited. This can be due to pi-hole doing a reverse of all client IPs every hour. You can disable this with:
# In the pihole-FTL.conf
REFRESH_HOSTNAMES=NONE
Firefox, if the user has not yet chosen a setting, will query use-application-dns.net. Pi-hole respods with NXDOMAIN3 as a signal to use pi-hole for DNS.
/etc/pihole/pihole-FTL.conf
Apple devices offer a privacy enhancing service4 that you can pay for. Those devices queries mask.icloud.com as a test and Pi-hole blocks that by default. The user will be notified you are blocking it.
# Signal that Apple iCloud Private Relay is allowed
BLOCK_ICLOUD_PR=false
sudo systemctl reload pihole-FTL.service
On a very busy server, clicking show-all in the query log panel will hang the server as pihole-FTL works through it’s database. There is no solution, just don’t do it. The best alternative is to ship logs to a Elasticsearch or similar system.
The system continues to use whatever DNS resolver was initially configured. You may want it to use itself, instead.
# revert if pi-hole itself needs fixed.
sudo vi /etc/resolv.conf
nameserver 127.0.0.1
Pi-hole can be configured via it’s two main config files, /etc/pihole/setupVars.conf and pihole-FTL.conf, and you may find other useful settings.
In the v6, currently in beta, the settings are all in the .toml file, or via the gui via:
https://pi-hole.net/blog/2021/09/11/pi-hole-ftl-v5-9-web-v5-6-and-core-v5-4-released/#page-content ↩︎
https://discourse.pi-hole.net/t/include-log-entry-in-pihole-ftl-log-when-client-hits-rate-limit/46798/12A ↩︎
https://www.reddit.com/r/pihole/comments/113qkp5/i_am_seeing_useapplicationdnsnet_being_blocked/ ↩︎
https://docs.pi-hole.net/ftldns/configfile/?h=mask#icloud_private_relay [DNS benchmark]:https://www.grc.com/dns/benchmark.htm ↩︎
Pi-hole by itself is just a DNS forwarding and caching service. It’s job is to consolidate requests and forward them on to someone else. That someone else is seeing all your requests.
If that concerns you, add your own DNS Resolver like Unbound. It knows how to fetch answers without any one entity seeing all your requests. It’s probably slower1, but possibly more secure.
sudo apt install unbound
The pi-hole guide for [unbound]:(https://docs.pi-hole.net/guides/dns/unbound/) includes a config block to copy and paste as directed. You should also add a config file for dnsmasq while you’re at it, to set EDNS packet sizes. (dnsmasq comes as part of pi-hole)
sudo vi /etc/dnsmasq.d/99-edns.conf
edns-packet-max=1232
When you check the status of unbound, you can ignore the warning: subnetcache:... as it’s just reminding you that data in the subnet cache (if you were to use it) can’t pre-fetched. There’s some conversation2 as to why it’s warning us.
The config includes prefetch, but you may also wish to add serve-expired to it, if you’re not already using use-stale-cache in Pi-hole.
# serve old responses from cache while waiting for the actual resolution to finish.
# don't set this if you're already doing it in Pi-hole
serve-expired: yes
sudo systemctl restart unbound.service
No additional setup is needed, but see the unbound page for more info.
You must tel Pi-hole about the Resolver you’ve just deployed.
Pi-hole can also do DHCP. However, the GUI only allows for a single range. On a large network you’ll need multiple ranges. You do this by editing the config files directly.
In this setup, you have a separate interface per LAN. Easy to do in a virtual or VLAN environment, but you’ll have to define each in the /etc/network/interfaces file.
Let’s create a range from 192.168.0.100-200 tied to eth0 and a range of 192.168.1.100-200 tied to eth1. We’ll also specify the router and two DNS servers.
vim /etc/dnsmasq.d/05-dhcp.conf
dhcp-range=eth0,192.168.0.100,192.168.0.200,24h
dhcp-option=ens161,option:router,192.168.0.1
dhcp-option=ens161,option:dns-server,192.168.0.2,192.168.0.3
dhcp-range=eth1,192.168.1.100,192.168.1.200,24h
dhcp-option=ens161,option:router,192.168.1.1
dhcp-option=ens161,option:dns-server,192.168.1.2,192.168.1.3
# Shared by both
dhcp-option=option:netmask,255.255.0.0
# Respond immediately without waiting for other servers
dhcp-authoritative
# Don't try and ping the address before assigning it
no-ping
dhcp-lease-max=10000
dhcp-leasefile=/etc/pihole/dhcp.leases
domain=home.lan
These settings can be implicit - i.e. we could have left out ethX in the range, but explicit is often better for clarity.
Note - the DHCP server (dnsmasq) is not enabled by default. You can do that in the GUI under Settings –> DHCP
In this setup, the router relays DHCP requests to the server. Only one system network interface is required, though you must configure the router(s).
When configured, the relay (router) sets the relay-agent (giaddr) field, sends it to dnsmasq, which (I think) understands it’s a relayed request when it sees that field, and looks at it’s available ranges for a match. It’s also sets a tag that to be used for assigning different options, such as the gateway, per range.
dhcp-range=tag0,192.168.0.100,192.168.0.250,255.255.255.0,8h
dhcp-range=tag1,192.168.1.100,192.168.1.250,255.255.255.0,8h
dhcp-range=tag2,192.168.2.100,192.168.2.250,255.255.255.0,8h
dhcp-options=tag0,3,192.168.0.1
dhcp-options=tag1,3,192.168.1.1
dhcp-options=tag2,3,192.168.2.1
https://discourse.pi-hole.net/t/more-than-one-conditional-forwarding-entry-in-the-gui/11359
It’s possible that the DHCP part of dnsmasq doesn’t scale to many thousands of leases1
If you host something out in residential land you’ll need this. Your IP changes periodically and you don’t want to have track that manually.
CF provides an excellent DDNS service and all you need is curl. Pretty simple, right?
ZONE_ID=
RECORD_ID=
TOKEN=
IP=$(curl -4 --silent --show-error --fail --max-time 10 https://api.ipify.org)
curl --silent --show-error --fail --max-time 10 \
-X PATCH \
"https://api.cloudflare.com/client/v4/zones/$ZONE_ID/dns_records/$RECORD_ID" \
-H "Authorization: Bearer $TOKEN" \
-H "Content-Type: application/json" \
--data "{\"content\":\"$IP\"}")
Who needs ddclient!
Noice! Done!
OK - well, you need to know your IDs, and you have to set up a token that only has DNS rights to the specific zone in question. And you need to handle errors, so I guess you need something more like…
vi ~/ddns-update
#!/bin/bash
set -euo pipefail
# Configuration
ZONE="your.org"
RECORD="server.your.org"
TOKEN="XXXXXXX"
# State dirs (systemd uses /var/lib/SERVICENAME)
STATE_DIR="/var/lib/cloudflare-ddns"
ZONE_ID_FILE="$STATE_DIR/zone_id"
RECORD_ID_FILE="$STATE_DIR/record_id"
LAST_IP_FILE="$STATE_DIR/last_ip"
# Load or fetch zone and record IDs needed for CF's API
if [ -f "$ZONE_ID_FILE" ]; then
ZONE_ID=$(<"$ZONE_ID_FILE")
else
ZONE_ID=$(curl --silent --show-error --fail --max-time 10 \
-H "Authorization: Bearer $TOKEN" \
-H "Content-Type: application/json" \
"https://api.cloudflare.com/client/v4/zones?name=$ZONE" \
| jq -r '.result[0].id')
echo "$ZONE_ID" > "$ZONE_ID_FILE"
fi
[ -n "$ZONE_ID" ] && [ "$ZONE_ID" != "null" ] || {
echo "Failed to discover Cloudflare Zone ID"
exit 1
}
if [ -f "$RECORD_ID_FILE" ]; then
RECORD_ID=$(<"$RECORD_ID_FILE")
else
RECORD_ID=$(curl --silent --show-error --fail --max-time 10 \
-H "Authorization: Bearer $TOKEN" \
-H "Content-Type: application/json" \
"https://api.cloudflare.com/client/v4/zones/$ZONE_ID/dns_records?type=A&name=$RECORD" \
| jq -r '.result[0].id')
echo "$RECORD_ID" > "$RECORD_ID_FILE"
fi
[ -n "$RECORD_ID" ] && [ "$RECORD_ID" != "null" ] || {
echo "Failed to discover Cloudflare Record ID"
exit 1
}
# Load the last IP.
LAST_IP=""
if [ -f "$LAST_IP_FILE" ]; then
LAST_IP=$(<"$LAST_IP_FILE")
fi
# Get the current IP
IP=$(curl -4 --silent --show-error --fail --max-time 10 https://api.ipify.org)
[[ $IP =~ ^([0-9]{1,3}\.){3}[0-9]{1,3}$ ]] || {
echo "Failed validation check for data: $IP"
exit 1
}
# If they aren't the same
if [ "$IP" != "$LAST_IP" ]; then
echo "Updating IP to $IP"
RESPONSE=$(curl --silent --show-error --fail --max-time 10 \
-X PATCH \
"https://api.cloudflare.com/client/v4/zones/$ZONE_ID/dns_records/$RECORD_ID" \
-H "Authorization: Bearer $TOKEN" \
-H "Content-Type: application/json" \
--data "{\"content\":\"$IP\"}")
RESULT=$(echo "$RESPONSE" | jq -r '.success')
if [ "$RESULT" = "true" ]; then
echo "$IP" > "$LAST_IP_FILE"
echo "Update successful"
else
echo "Cloudflare update failed"
echo "$RESPONSE" | jq -r '.errors[]?.message'
exit 1
fi
fi
Whew, OK. Now you need to get this running somehow. Cron? Oh, right. Not in Debian 13 by default. Systemd timer it is, then.
sudo install -m 700 ddns-update /usr/local/sbin/ddns-update
sudo vi /etc/systemd/system/cloudflare-ddns.service
[Unit]
Description=Cloudflare DDNS Update
Wants=network-online.target
After=network-online.target
[Service]
Type=oneshot
StateDirectory=cloudflare-ddns
StateDirectoryMode=0700
ExecStart=/usr/local/sbin/ddns-update
sudo vi /etc/systemd/system/cloudflare-ddns.timer
[Unit]
Description=Run Cloudflare DDNS updater every 15 minutes
LogLevelMax=warning
[Timer]
OnBootSec=1min
OnUnitActiveSec=15min
[Install]
WantedBy=timers.target
sudo systemctl daemon-reload
sudo systemctl enable --now cloudflare-ddns.timer
Queue Spongebob: Two hours later…and we’ve just made a (worse) copy of ddclient that does (a lot) less.
If less is more, maybe nothing is the most1. Since most appliances are just linux with a wrapper, let’s get rid of that wrapper. We’ll be left with more.
All humor aside, having the simplest thing possible that works is almost always the best solution. I’ve found after several years in production that a simple Debian router/traffic shaper is equal or better than any appliance.
The first problem is how to mock-up the WAN/LAN networks for this new router.
For the outside interface, let’s pretend that your existing LAN is the internet. Just plug eth0 in to whatever you’ve already got.
For the inside network, we will overlay a new LAN on top of your existing one. This is analogous to having different computers use the same physical wires but configuring different IP networks. They may see each others traffic at the physical layer, but will ignore each other logically.
Create a new Debian instance with two interfaces, both on the existing LAN. On bare-metal, install a minimal Debian instance by using the netinst image and selecting only ‘common system tools’ near the bottom.
In a LAN/WAN situation eth0 is traditionally WAN2. So leave the first interface with it’s default DHCP settings and configure the second one with a static address.
Assuming that your existing LAN is 192.168.1.* we’ll overlay our new LAN as 192.168.2.*
sudo vi /etc/systemd/network/eth1.network
[Match]
Name=eth1
[Network]
Address=192.168.2.1/24
sudo systemctl reload systemd-networkd
We’ll need the nft tools to do the basics and make sure curl is handy as well.
sudo apt install nftables curl
The first step is to enable forwarding, the basic job of all routers.
# as root
echo "net.ipv4.ip_forward=1" >> /etc/sysctl.conf
sysctl -p
It’s worth knowing that the system (as is) doesn’t spend a lot of time thinking about where a packet came from. If someone gives it a packet, it consults the route table and sends it on. It doesn’t really care where it came from. This sounds dangerous, but in practice it’s rarely a problem. Though you can look into policy based routing if things are complicated.
If one side is a private network (such as 192.168.*) you probably need to masquerade. This is different than the type attended by the Opera Ghost.
sudo vi /etc/nftables.conf
This is the default debian file. Add WAN and LAN at the top and a table at the bottom with the masquerade details.
It’s traditional to name this table ’nat’ and the chain ‘postrouting’. Those names are arbitrary but make sense as that’s its type and where it’s hooked.
#!/usr/sbin/nft -f
flush ruleset
# Define WAN and LAN
define WAN = eth0
define LAN = eth1
table inet filter {
chain input {
type filter hook input priority filter; policy accept;
}
chain forward {
type filter hook forward priority filter; policy accept;
}
chain output {
type filter hook output priority filter; policy accept;
}
}
table ip nat {
chain postrouting {
type nat hook postrouting priority srcnat; policy accept;
# if the output interface is WAN, masquerade the traffic.
oif $WAN masquerade
}
}
systemctl enable --now nftables.service
To see if things are working so far, take another system and replace it’s .1 LAN address with something on the .2 and add the new router as it’s gateway.
# Replace your existing IP
sudo ip addr replace 192.168.2.50/24 dev eth0
# Replace your existing route
ip route replace default via 192.168.2.1 dev eth0
ping 8.8.8.8
Right now, you’re letting everything in and out, shouting ‘packets want to be free!’. But this is the real world so let’s add firewall to this router’s list of jobs. This is still a router, though. So allow controlled ping and traceroute. This helps avoid fragmentation and provides diagnostic benefits that outweigh attempts at obscurity.
We are going to add a chain and edit two others. Create the chain early_drop at the top to get rid of any obvious garbage in the early prerouting stage. Then edit the input (packets destined to the router itself), and the forward chains (packets being sent on).
table inet filter {
chain early_drop {
type filter hook prerouting priority filter; policy accept;
ct state invalid drop
}
chain input {
# Change the default policy to drop
type filter hook input priority 0; policy drop;
# Allow local and already established connections
iif lo accept
ct state established,related accept
# Allow standard ping with rate limiting and traceroute
icmp type echo-request limit rate 5/second accept
icmp type { destination-unreachable, time-exceeded } accept
# Respond to Linux traceroute
iif $WAN udp dport 33434-33534 reject with icmp type port-unreachable
# IPv6 equivalents
icmpv6 type { echo-request, nd-neighbor-solicit, nd-neighbor-advert, packet-too-big, time-exceeded } accept
}
chain forward {
# Change the default policy to drop
type filter hook forward priority filter; policy drop;
# Accept local and already established connections
ct state established,related accept
# Accept connections from LAN to WAN
iif $LAN oif $WAN accept
}
nft -f /etc/nftables.conf
Congrats! you’ve just secured things and possibly locked yourself out! (don’t close your ssh session yet)
To allow remote management, append a SSH rule to the bottom if your input chain. It’s best to start with LAN for this
chain input {
...
...
# Accept from any source to SSH
iif $LAN tcp dport "ssh" accept
}
You may want to limit access based on IPs. The simplest and more performant way is with a set. That’s basically a list of IPs and networks to compare incoming packets against. We’ll put the details in it’s own file to keep the main config clean.
sudo mkdir -p /etc/nftables/sets
sudo vi /etc/nftables/sets/work_ips.nft
table inet filter {
set work_ips {
type ipv4_addr
flags interval
elements = {
1.2.3.4, # Hot in Cleveland
5.6.7.8, # WKRP in Cincinnati
10.11.12.0/24, # Buzz Beer headquarters
}
}
}
Put an include right after the definitions and use it in a rule like this.
vi /etc/nftables.conf
#!/usr/sbin/nft -f
flush ruleset
# Define WAN and LAN
define WAN = eth0
define LAN = eth1
# Include all the config files in the folder
include "/etc/nftables/sets/*.nft"
table inet filter {
chain input {
...
...
# Accept from any source to SSH
iif $LAN tcp dport "ssh" accept
# Accept from work to SSH
iif $WAN ip saddr @work_ips tcp dport "ssh" accept
}
Reload the rules and take a look. You’ll see your new set displayed.
nft -f /etc/nftables.conf
sudo nft list ruleset
Note: You may have noticed that both the main and included files contain a table inet filter { block. These don’t replace each other, rather they are additive. That’s a nft thing.
Sometimes you don’t know what your IP will be, but you’re pretty sure you’re not leaving the country. In addition, the vast majority of attacks come from just a few countries. The wirefalls/geo-nft script will help by downloading a map of IP’s to countries.
sudo mkdir -p /etc/nftables/geo-nft
cd /etc/nftables/geo-nft
curl -LO https://raw.githubusercontent.com/wirefalls/geo-nft/master/geo-nft.sh
chmod +x geo-nft.sh
sudo ./geo-nft.sh
Create a us.nft that incorporates the ipv4 data like this.
sudo vi /etc/nftables/sets/us.nft
include "/etc/nftables/geo-nft/countrysets/US.ipv4"
table inet filter {
set US_ipv4 {
type ipv4_addr
flags interval
auto-merge
elements = $US.ipv4
}
}
Make use of it just like any set.
...
...
# Accept connections from US to SSH
iif $WAN ip saddr @US_ipv4 tcp dport "ssh" accept
...
...
Reload and observe the now much larger set. But don’t let the size scare you. I’m told speed is the same for ten or ten thousand as they are hashed.
nft -f /etc/nftables.conf
nft list ruleset | more
If you see an error about the Message size like this:
netlink: Error: Could not process rule: Message too long Please, rise /proc/sys/net/core/wmem_max on the host namespace. Hint: 4194304 bytes
You’ll need to increase your netlink buffer size. Do this on the host if running in an instance or container. Jumping up to a 4M buffer should insulate you from needing to change it again even as your set sizes change.
sudo vi /etc/sysctl.d/99-nftables-netlink.conf
net.core.wmem_max=4194304
net.core.wmem_default=524288
sudo sysctl -p /etc/sysctl.d/99-nftables-netlink.conf
Sometimes you want to combine sets, like the worst 5 countries for cyber-attacks. The unfab 5. Just include them all with a semicolon after each include, and you can flatten them into a super-set. (That’s a hidden gotcha - the files don’t end with a newline and using ‘;’ isn’t well known)
sudo vi /etc/nftables/sets/unfab.nft
include "/etc/nftables/geo-nft/countrysets/CN.ipv4";
include "/etc/nftables/geo-nft/countrysets/IN.ipv4";
include "/etc/nftables/geo-nft/countrysets/NE.ipv4";
include "/etc/nftables/geo-nft/countrysets/NG.ipv4";
include "/etc/nftables/geo-nft/countrysets/RU.ipv4";
define UNFAB = { $CN.ipv4, $IN.ipv4, $NE.ipv4, $NG.ipv4, $RU.ipv4 }
table inet filter {
set UNFAB_ipv4 {
type ipv4_addr
flags interval
auto-merge
elements = $UNFAB
}
}
When you want to accept connections from everyplace and just exclude a few countries, it’s a lot easier to just list the exclusions. You do that with an inverse meaning accept any connection not on this list. If you’ve created the unfab set above, you’d use it inversely like this:
# Accept everyone but the unfab to connect to SSH
ip saddr != @UNFAB_ipv4 tcp dport "ssh" accept
Let’s say we are going to forward web traffic to a back-end server. This actually requires two things; an new prerouting chain and entry to catch incoming traffic to those ports and rewrite it with the back-end server’s address, and a forward chain entry where you accept the traffic.
sudo mkdir /etc/nftables/forwards
sudo vi /etc/nftables/forwards/web_server.nft
define WEB_SRV = 192.168.2.10
table ip nat {
chain prerouting {
type nat hook prerouting priority dstnat;
# Rewrite any incoming traffic so they have the web servers address instead
iif $WAN tcp dport { "http", "https" } dnat to $WEB_SRV
}
}
table inet filter {
chain forward {
type filter hook forward priority filter;
# Accept the web server traffic so it can be forwarded on.
iif $WAN ip daddr $WEB_SRV tcp dport { "http", "https" } accept
# Or maybe only accept traffic from the US
iif $WAN ip saddr @US_ipv4 ip daddr $WEB_SRV tcp dport { "http", "https" } accept
}
}
Add an include in the main file after the sets and reload.
...
...
# Include any sets
include "/etc/nftables/sets/*.nft"
# Include any port forwards
include "/etc/nftables/forwards/*.nft"
...
...
nft -f /etc/nftables.conf
Change the IP and gateway on your web server temporarily and it should test correctly.
Note: It’s tempting to add more rules to the prerouting section. Resist this urge, however. According to the docs, some parts of a flow skip prerouting so you should “…never use this chain for filtering”3. The only exception being to drop obvious junk as we did at the beginning.
Say you’re forwarding the external port 222 to an internal server on port 22. It’s just a matter of adding that to the rewrite. The forward chain stays the same.
redefine EXTERNAL_PORT = 222
redefine INTERNAL_IP = 192.168.2.4
redefine INTERNAL_PORT = 22
table ip nat {
chain prerouting {
type nat hook prerouting priority dstnat;
iif $WAN tcp dport $EXTERNAL_PORT dnat to $INTERNAL_IP:$INTERNAL_PORT
}
}
table inet filter {
chain forward {
type filter hook forward priority filter;
iif $WAN ip daddr $INTERNAL_IP tcp dport $INTERNAL_PORT accept
}
}
Note: Last time we defined the (probably) unique name for the variable of WEB_SRV, so that it wouldn’t conflict with other variables in other includes. But you can also just use redefine in a template and save a lot of editing.
Linux can bypass most of the nftables/conntrack path after a flow is established. If you have a decent ethernet card you can even get hardware acceleration.
A flowtable is kernel-managed acceleration table you can add to your normal inet fitler table. It’s a “fastpath” that you can toss flows to that you’ve already looked at. This saves you from checking every packet when it’s already part of an established flow.
This will mostly offload web browsing/streaming and other normal TCP/UDP NAT traffic.
Add this at the top of the table inet filter.
table inet filter {
flowtable ft {
hook ingress priority filter;
devices = { $LAN, $WAN };
}
...
...
Then modify the forward chain to use it.
chain forward {
type filter hook forward priority filter; policy drop;
# Offload established/related flows
ct state established,related flow add @ft accept
# This line is now redundant but usually left in so you can
# comment the above hardware accel in and out easily
ct state established,related accept
# Accept connections from LAN to WAN
iif $LAN oif $WAN accept
}
If you want to dig in more, you can add a counter to the rules above. Comment out the Offload rule and the flow_normal counter should increase steadily. Swap and the flow_offload should increase much more slowly as only the first packets get counted. The rest of the low bypasses.
ct state established,related flow add @ft counter name flow_offload_counter accept
ct state established,related accept counter name flow_normal_counter accept
Pro cards from Intel, Mellinox, Chelsio and others make drivers that provide hardware acceleration to the kernel. Check with this comand.
ethtool -k eth0 | grep offload
You’re mostly looking for hw-tc-offload: on as nftables flow offload can sometimes leverage TC hardware offload underneath. Enable the hardware offload with the flag offload.
flowtable ft {
hook ingress priority filter;
devices = { $LAN, $WAN };
flags offload;
}
Note: if you saw l2-fwd-offload: on then you have a NIC that does autonomous Layer-2 forwarding/switching and you either don’t need to read this page, or need to follow up with another more advanced page!
If this isn’t fast enough, you can use eBGP to take action on packets before the kernel starts working on them. If your card supports XDP, you can do that while they are still in the network card!
This is mostly useful for dropping trash and mitigating denial of service attacks without bothering the CPU. There’s also a case where you move packets around in a kubernetes cluster without looking at them closely for the sake of speed. Though Cloudflare utilizes the xt_bpf extension to embed eBPF programs with nftables, it’s not easy.
It’s not generally useful as in most cases you’ll have to look at the details of the packet anyway. In this case the best you can do is a flowtable you just created
Though if you’re under DDOS attacks consider xdp-filter and for large-scale load balancing you might look the other eBFP solutions.
If you’re running in a container of some kind, you can pass a card directly to the instance (SR-IOV or full PCI) to take advantage of it.
This best done with a pair of routers that share a virtual IP managed by keepalived. It’s fairly simple to deploy as you just:
Your router is still has a DHCP WAN address and that’s fine. But you’ll need to select something static for the virtual WAN ip address you’re about to create. I’ve used 192.168.1.2, but adjust as needed.
# Just install the core server, without perl and ipvsadm
sudo apt install --no-install-recommends keepalived
sudo vi /etc/keepalived/keepalived.conf
You’ll note that we’ve set the state to “BACKUP” and we’ll use that on both. This keeps them from flapping during any intermittent issues.
vrrp_instance WAN {
state BACKUP
interface eth0
virtual_router_id 51
virtual_ipaddress { 192.168.1.2/24 }
}
vrrp_instance LAN {
state BACKUP
interface eth1
virtual_router_id 52
virtual_ipaddress { 192.168.2.1/24 }
}
Add this at the bottom of your chain input section so you accept vrrp traffic. 224.0.0.18 is a reserved IPv4 multicast address specifically designated for the Virtual Router Redundancy Protocol
...
...
table inet filter {
chain input {
...
...
# Allow VRRP on both relevant interfaces. The 224 is
iif { $WAN, $LAN } ip protocol vrrp accept
iif { $WAN, $LAN } ip daddr 224.0.0.18 accept
}
}
Start the service and observe the log to see the status messages.
sudo systemctl start keepalived.service
sudo journalctl -u keepalived*
If things go well, you’ll see it start as backup, then quickly enter master when it doesn’t find any peers to say otherwise. An ip a will show the virtual IPs in place.
Clone the box (or the config), change the LAN IP so it doesn’t conflict, and bring both up. You can experiment by taking the service up and down to watch the IP shift.
This is an Active/Passive arrangement. For the most part, there’s no advantage to Active/Active. Either system can move traffic faster than the uplink permits. If you do have more traffic than either one can move on it’s own, you can’t take one down.
The main benefit of Active/Active is that you know it works. You don’t want to be surprised during an outage. It allows you to gracefully drain traffic for controlled maintenance. Also, you can’t always get a 200G system.
To setup Active/Active for capacity you create a team of three or more. Each is MASTER of a separate VIP and they all back each other up. DHCP passes out the gateway addresses round-robin to balance load. Keep in mind that traffic shaping with tc happens after nftables, so you can’t shape your way out of a capacity issue.
How do you keep them in sync? You can probably just remember to update rules only on router-1 and then run rsync. Though you’ll need to grant yourself (or a service account) permissions.
# On both routers
## Change the owner of the nftable data
sudo chown -R ${USER} /etc/nftables*
## Allow yourself to run nft without a password prompt so you can automate the reload
echo "$USER ALL=(root) NOPASSWD: /usr/sbin/nft" | sudo tee /etc/sudoers.d/nft
sudo chmod 440 /etc/sudoers.d/nft
# On your primary router
## Generate a key and copy it to the other router (make sure dns resolves it correctly)
ssh-keygen -t ed25519 -f ~/.ssh/id_ed25519 -N ''
ssh-copy-id router-2
## Run a sync
rsync --archive --delete --inplace --verbose /etc/nftables.conf /etc/nftables router-2:/etc/
## Trigger a reload on the other router
ssh router-2 sudo nft -f /etc/nftables.conf
You can probably put those last two commands into the bash file go-go-geo-nft to save typing them out. Even better, create a systemd watcher that detects changes and syncs automatically.
The IP will only move when one system goes offline. If both come back up perfectly at once, they’ll have an election and the highest management IP wins. You can override this with an explicit priority or script it if you have reason to.
You can also change one to state MASTER and it will take over the role whenever it’s up. This is useful when you have a hardware preference or are running additional software that can’t do multi-master.
Elections are held via broadcast (the 224 address) but unicast is also supported if you’re in a cloud environment that blocks broadcast
If you have other software running and want to bind it to the virtual ip before it becomes instantiated, look at adding net.ipv4.ip_nonlocal_bind=1 to the /etc/sysctl.conf. Though for things like dnsmasq it’s not needed.
The only other issue with failing over is connection state. When you fail over, any session dependant applications (like SSH) or games get dropped. This isn’t a large issue as the web is (mostly) stateless. But you can deploy conntrackd if desired.
These days, you can Expect 100–400 concurrent connections per device at peak so you outpace a single device faster than you think. On a single device, you can optimize some kernetl settings.
# Create a drop-in configuration file in /etc/sysctl.d for these settings
net.core.netdev_max_backlog=250000
net.core.somaxconn=65535
net.ipv4.ip_local_port_range=1024 65535
net.ipv4.tcp_tw_reuse=1
net.netfilter.nf_conntrack_max=2000000
net.netfilter.nf_conntrack_tcp_timeout_established=86400
net.netfilter.nf_conntrack_generic_timeout=120
On a old-style campus WAN, you have (expensive) layer 3 switches and routers that use the ECMP protocol to use multiple paths. This has wider compatibility than agregating ports with LAG. You’d think you could just distrubute multiple gateways as part of DHCP, but clients handle that randomly or not at all.
A more modern and cost effective strategy is to move from an Active/Passive HA pair, to an Active/Active/Active set of peers.
If you can handle the new however, use and IPv6‑first design.
DHCP and DNS are optional, but sometimes expected in small environments and frequently under other duties as assigned for such systems. Dnsmasq is the go-to for most. I’ve found that particular software doesn’t scale well beyond a few thousand clients, but if you’ve gotten to that point, you’d want a dedicated system anyway. So it’s fine for small scale.
This is covered in many other locations but I’ll toss a sample config here just in case.
# Remove systemd-resolved if installed by the default debian instance
sudo systemctl stop systemd-resolved.service
sudo apt install dnsmasq
You’ll need to accept DNS requests by adding another accept at the bottom. UDP is traditional, but TCP is also used these days. Add this at the bottom of your input chain.
table inet filter {
chain input {
...
...
# Allow DNS queries
iif $LAN udp dport "domain" accept
iif $LAN tcp dport "domain" accept
}
}
By default, dnsmasq just fires up and starts work by caching DNS queries. It looks at your /etc/resolv/conf to see where to forward requests and your /etc/hosts for known hosts. You may want to tighten things up a bit with these.
sudo vi /etc/dnsmasq.conf
Just add these to the bottom.
# Don't forward plain names or non-routable IPs
domain-needed
bogus-priv
# Listen only on the LAN
interface=lo
interface=eth1
# Upstream DNS servers to forward requests to
server=8.8.8.8
server=8.8.4.4
# Cache size (1000 is a good start)
cache-size=1000
# Don't use the hosts file for entries
no-hosts
Create a seperate file for host entries as well. This will let you sync more easily later. Since debian’s packaging of dnsmasq makes use of a drop folder, let’s go ahead and embrace that and add entries in the native format.
sudo vi /etc/dnsmasq.d/hosts.conf
host-record=gateway,192.168.2.1
host-record=router-1,192.168.2.2
host-record=router-2,192.168.2.3
host-record=some-host,192.168.2.10
host-record=some-other-host,192.168.2.11
...
...
# Restart so dnsmasq reads the new config file.
sudo systemctl dnsmasq restart
# Tell the system to use itself for resolution as well.
echo "nameserver 127.0.0.1" | sudo tee /etc/resolv.conf
If you went to the trouble to set up sync above for nft, you can piggyback on that for dns.
# On both boxes
sudo test -f /etc/dnsmasq.d/hosts.conf || sudo touch /etc/dnsmasq.d/hosts.conf
sudo chgrp -R ${USER} /etc/dnsmasq.d/hosts.conf
sudo chmod -R 775 /etc/dnsmasq.d/hosts.conf
echo "$USER ALL=(root) NOPASSWD: /bin/systemctl restart dnsmasq, /bin/systemctl reload dnsmasq" | sudo tee /etc/sudoers.d/dnsmasq
sudo chmod 440 /etc/sudoers.d/dnsmasq
# On router-1, you can send the file and restart (reload doesn't load date from host.conf)
rsync --archive --delete --inplace --no-times --verbose /etc/dnsmasq.d/hosts.conf router-2:/etc/dnsmasq.d/
ssh router-2 sudo systemctl restart dnsmasq
### DHCP
You can pass out addresses as well. Just follow up the last bit with this. Though you may want to leave these commented out until your ready, to avoid confusing the overlay networks.
```bash
dhcp-range=192.168.2.100,192.168.2.200,12h
dhcp-option=option:router,192.168.2.1
dhcp-authoritative
You have a couple of things worth automating. Changes to nft rules and updates to the GeoIP database.
Those IP sets are refreshed the 1st of every month on db-ip.com. We should refresh accordingly. On Debian 13, systemd timers are preferred (cron isn’t even installed).
sudo vi /etc/systemd/system/geo-nft-update.service
[Unit]
Description=Update geo-nft IP datasets
Wants=network-online.target
After=network-online.target
[Service]
Type=oneshot
ExecStart=/etc/nftables/geo-nft/geo-nft.sh
sudo nano /etc/systemd/system/geo-nft-update.timer
[Unit]
Description=Monthly geo-nft update (3rd day)
[Timer]
OnCalendar=*-*-03 03:30:00
Persistent=true
RandomizedDelaySec=30m
[Install]
WantedBy=timers.target
# Just enable the timer. We don't want the service to run unless the timer starts it.
sudo systemctl daemon-reload
sudo systemctl enable --now geo-nft-update.timer
You can also use systemd to watch for file changes and trigget a sync and reload. Note the User= setting. This must be your account (or whatever account you used in setting up the sync and ssh above) to work as the manual process did. Alternatly, create a service account.
sudo vi /etc/systemd/system/router-sync.service
[Unit]
Description=Sync router configuration to router-2
After=network-online.target
Wants=network-online.target
[Service]
Type=oneshot
User=YOU
ExecStart=/usr/bin/sudo /usr/sbin/nft -f /etc/nftables.conf
ExecStart=/usr/bin/rsync --archive --delete --inplace --no-times /etc/nftables.conf /etc/nftables router-2:/etc/
ExecStart=/usr/bin/ssh router-2 sudo nft -f /etc/nftables.conf
ExecStart=/usr/bin/sudo systemctl reload dnsmasq
ExecStart=/usr/bin/rsync --archive --delete --inplace --no-times /etc/dnsmasq.d/hosts.conf router-2:/etc/dnsmasq.d/
ExecStart=/usr/bin/ssh router-2 sudo systemctl reload dnsmasq
sudo vi /etc/systemd/system/router-sync.path
[Unit]
Description=Watch router config files
[Path]
PathChanged=/etc/nftables.conf
PathModified=/etc/nftables
PathChanged=/etc/dnsmasq.d/hosts.conf
Unit=router-sync.service
[Install]
WantedBy=multi-user.target
sudo systemctl daemon-reload
sudo systemctl enable --now router-sync.path
I put both services in this watcher as an example. A a better design is a sperate watch for each to reduce curn if you have frequent changes.
The tool we’re using downloads the full set of countries as a .csv and then converts them into nft definitions. It’s a bit slow and heavy. You can replace it with something like this and get a more compact (and prehapse better) CIDR format like this.
COUNTRY=ad
OUT=/etc/nftables/sets/${COUNTRY}.nft
tmp=$(mktemp)
{
echo "set ${COUNTRY}_ipv4 {"
echo " type ipv4_addr"
echo " flags interval"
echo " elements = {"
curl -s https://www.ipdeny.com/ipblocks/data/countries/${COUNTRY}.zone | \
awk '{print " "$1","}' | sed '$ s/,$//'
echo " }"
echo "}"
} > "$tmp"
mv "$tmp" "$OUT"
In a container, forwarding just stops. Restarting nftables has no affect but restarting systemd-networkd fixes it.
Incus especially likes to replace the bridge interfaces periodically.
sudo journalctl -u systemd-networkd -u NetworkManager --since "1 day ago"
May 28 12:55:50 shire1 systemd-networkd[486]: physSf5Ar4: Interface name change detected, renamed to vethe8067f8a.
May 28 12:55:50 shire1 systemd-networkd[486]: physnLFcl3: Interface name change detected, renamed to vethd880fdd5.
May 28 12:55:50 shire1 systemd-networkd[486]: vethc3f8a16b: Link UP
May 28 12:55:50 shire1 systemd-networkd[486]: vethc3f8a16b: Link DOWN
May 28 12:55:50 shire1 systemd-networkd[486]: veth243c40db: Link UP
May 28 12:55:50 shire1 systemd-networkd[486]: veth243c40db: Link DOWN
May 28 12:55:51 shire1 systemd-networkd[486]: veth24f2e389: Link UP
May 28 12:55:51 shire1 systemd-networkd[486]: vethff99fb0d: Link UP
May 28 12:55:51 shire1 systemd-networkd[486]: veth24f2e389: Gained carrier
May 28 12:55:51 shire1 systemd-networkd[486]: vethff99fb0d: Gained carrier
This causes a loss of static mappings to the interfaces and this breaks how iff and off in your rules work. You must instead us iffname which is dynamically mapped. It’s slightly slower, but needed in some containers.
sudo sed -i.bak -E \
-e 's/\biif (\$WAN|\$LAN|\{ \$WAN)/iifname \1/g' \
-e 's/\boif (\$WAN|\$LAN|\{ \$WAN)/oifname \1/g' \
/etc/nftables.conf
I’ve read Rem Koolhaas meant this as mockey, so let’s think of it as “Simplify, then add lightness” as Chapman would say ↩︎
I don’t have a good attribution for this other than the many articles that assert this too ↩︎
https://wiki.nftables.org/wiki-nftables/index.php/Configuring_chains#Base_chain_types ↩︎
I once read a comparison of router distros (that I can no longer find) and was surprised to see OpenWrt score near the top in terms of speed and reliability. I’d thought it just a solution for small appliances. Turns out the developers spent significant time optimizing it to be a worthy alternative for large scale routing and firewall. And you can run in a container as opposed to needing a full VM like OPNsense.
The official docs seem complicated, but it’s fairly easy.
Go to System -> Software, Click the Update Lists button then search for “ddns” and simply click on the DDNS script for the service you have, then do the same for “luci-app-ddns” so it adds the menu item to the web GUI.
In the Web GUI, select Services -> Dynamic DNS and delete the demo entries at the bottom. Add a new service, choose IPv4 and and select the service provider you added the script for above. In the subsequent screen, you’ll need to know
Domain: Similar, just in the form of "[email protected]"
Username: "Bearer"
Password: This is an API key you generated
If you’re behind NAT, you’ll also need to adjust the Advanced Settings
URL to detect: https://cloudflare.com/cdn-cgi/trace
Save and Apply, and then Reload. The Last Update/Next Update should change to a date.
OpenWrt doesn’t ship with GeoIP capabilities, but you can add it with the IP set extras script. This is a somewhat legacy approach, but the GUI requires it, and it’s translated to modern nft named sets under the hood.
IP Sets don’t work on OpenWrt release 25.12. You’ll have to use the prior stable to use it.
https://forum.openwrt.org/t/ipset-extras-issues-in-25-12-0/247519
# At the command line
opkg update
opkg install ipset resolveip
# Install ipset-extras via their install script
wget -U "" -O ipset-extras.sh "https://openwrt.org/_export/code/docs/guide-user/advanced/ipset_extras?codeblock=0"
chmod +x ipset-extras.sh
./ipset-extras.sh
# Logout and back in to enable the extension from the /etc/profile.d folder
exit
# Configure an IP set for the US
uci set dhcp.us="ipset"
uci add_list dhcp.us.name="US"
uci add_list dhcp.us.name="US6"
uci add_list dhcp.us.geoip="us"
uci commit dhcp
# Populate IP sets
ipset setup
# Check set creation worked
nft list sets
# Install hotplug-extras for persistence
wget -U "" -O hotplug-extras.sh "https://openwrt.org/_export/code/docs/guide-user/advanced/hotplug_extras?codeblock=0"
chmod +x ./hotplug-extras.sh
./hotplug-extras.sh
When adding a port forward, the Advanced tab will now have the “Use ipset” populated and you can select “US”
You can also invert the rule by typing in “! US” - an important feature that doesn’t jump out at you.
This inversion is best with with a list of who you don’t want. Here’s an example of a set for the worst 5 countries for probes.
# Configure an IP set for the worst countries for probes and hacks - the axis of hacks
uci set dhcp.axis="ipset"
uci add_list dhcp.axis.name="axis"
uci add_list dhcp.axis.name="axis6"
uci add_list dhcp.axis.geoip="cn"
uci add_list dhcp.axis.geoip="in"
uci add_list dhcp.axis.geoip="ne"
uci add_list dhcp.axis.geoip="ng"
uci add_list dhcp.axis.geoip="ru"
uci commit dhcp
ipset setup
No NFT Sets Generated:
The script that generates the sets is sensitive to the value you put in `dhcp.axis.name=“value”. Try avoiding spaces and numbers and that ‘6’ is at the end of the second value.
Why are we adding these to the DHCP config section?:
I don’t know. The documentation adds them there. In the code, the script loads them from there. I tried adding them to the Firewall section just for fun and it didn’t work. I suspect if I knew more it would make sense.
The installation uses <www.ipdeny.com> and adds a cronjob that updates the list daily at 3 AM.
There are other tools, such as geopip-shell or ban ip, but these are more of an all-or-nothing solution and can’t be used with individual firewall rules. There is also the python utility from the netfilter team and misc bash scripts, but these lack easy OpenWrt integration.
The authors of nftables have a page[^5] on GeoIP, but it’s about tagging packets. OpenWrt expects named sets and you can’t easily construct that from a map.
[^5] https://wiki.nftables.org/wiki-nftables/index.php/GeoIP_matching
The simplest way to make routing highly-available is to have two routers and automate fail-over. The conventional way is with keepalived. It allows the gateway address to automatically move between your routers so clients can always connect.
You need at least a pair of routers. You probably already have one running, so it’s easier (in a virtual world) to spin up two new ones and pull in the config later. If you can’t, it’s easy enough to translate the instructions here to add one.
Spin up two routers virtually or physically as needed. Name them gateway-1 and gateway-2. If your ISP only permits a single WAN address, you’ll need to start with them behind your existing router.
opkg update && opkg install keepalived
# The optional package keepalived-sync doesn't do anything useful in our case.
# Nor does the LuCI web interface for it.
We are going to use a MASTER and BACKUP config as it’s easier to work with when getting started.
They need to stand on their own so assuming your existing router’s LAN gateway is .1 you’ll address the new routers as .2 and .3 in the /etc/config/network file. Once you’ve done that and restarted networking, you should test to ensure they work by setting a gateway manually on a test client.
The first step is to create the floating LAN interface. The configuration in OpenWrt is done with via the UCI files, not keepalived’s .conf files.
We’ll test with .4 before going live and potentially clobbering the existing gateway.
This is our primary router.
# You don't really need to keep the old file, but it's interesting to look at.
mv /etc/config/keepalived ~/keepalived.orig
vi /etc/config/keepalived
config globals 'globals'
option enabled '1'
option router_id 'gateway-1'
config ipaddress
option name 'vip_lan'
option address '192.168.10.4/24'
option device 'eth1'
option scope 'global'
config vrrp_instance
option name 'VI_LAN'
option state 'MASTER'
option interface 'eth1'
option virtual_router_id '51'
option advert_int '1'
list virtual_ipaddress 'vip_lan'
This is our backup router
rm /etc/config/keepalived
vi /etc/config/keepalived
# Copy and paste the config from above but change two lines
...
...
option router_id 'gateway-2'
...
...
option state 'BACKUP'
...
...
Now restart keepalived on both systems and you’ll see your new address appear on the master. Stop keepalived on the master and you’ll see it appear on your backup system.
When you’re ready to move off your old router you can power it down, connect to the new and change the keepalived config to the virtual gateway. Doing this remotely is a fun magic trick to impress your friends and probably involves tmux and chained incus stop old router sleep incus reboot newrouter commands.
There can be some issues when ISPs only allow one WAN address. Sometimes, it’s the first system that requests an address through the modem. Other times there’s double NAT and you need to ensure the WAN address stays the same for incoming port forwards.
If you have a static WAN address with double NAT or the ability to have multiple WAN addresses - i.e. your OpenWrt routers are behind another router - here’s how to handle it.
# Check DHCP info if needed and set a static address for the main OpenWrt interface.
vi /etc/config/network
# Set this to something OTHER than your main wan address.
config interface 'wan'
option device 'eth0'
option proto 'static'
option ipaddr 'some.external.address'
option netmask 'some.255.netmask'
option gateway 'some.external.gateway'
/etc/init.d/network restart
Create a keepalived config
vi /etc/config/keepalived
# Add to the bottom of your existing file
config ipaddress
option name 'vip_wan'
option address 'the.main.wan.address/24'
option device 'eth0'
option scope 'global'
config vrrp_instance
option name 'VI_WAN'
option state 'MASTER'
option interface 'eth0'
option virtual_router_id '52'
option advert_int '1'
option garp_master_delay '1'
option garp_master_repeat '5'
list virtual_ipaddress 'vip_wan'
config vrrp_sync_group
option name 'VG_1'
list group 'VI_LAN'
list group 'VI_WAN'
You’ll notice the sync_group we added at the bottom. That ensures that if either interface fails (like a cable failure) both are moved to the backup router.
Similar to before, copy the config to the backup router and change the option state to BACKUP.
Restart keepalived and you should see the WAN address move in tandem with the LAN address
This is a simple disaster recovery solution. We are not keeping connection state synced so at fail-over some connections will be interrupted. Though in testing most seem unaffected. A more enterprise solution is to set all nodes as backup and nopreempt, add conntrackd, and allow the addresses to move as part of maintenance or failure and minimize disruption moving them back.
This doesn’t keep your other config data in sync. There is a keepalived-sync package which installs rsync and some other utilities, but it seems to be aimed at keeping the keepalived config the same, not the rest of the router. To keep your settings consistent, you may want to setup ssh keys and rsync specific files from the master to the backup.
You don’t need a router with a Web GUI, but it’s convenient and Incus makes it easy by providing a base image you can spin up in a few seconds. This will already contain the incus-agent for integration with incus management commands.
Incus provides a private network incusbr0 that handles the basics. But to to manage port forwarding, DNS and the like, you’ll need another. We created such a network when configuring Incus. Refer to that if you haven’t created one yet.
Just create a new instance and search for OpenWRT. The newest stable (non RC or snapshot) version is preferred. When creating, add a second network interface and change the name to eth1 before starting. Make sure both are connected to the bridge or vlan network you added when configuring Incus.
The first step is to get access. Use the Incus Web interface to inter the console and edit the config files.
# Configure the LAN interface
vi /etc/config/network
# Add this after the WAN entries
config interface 'lan'
option device 'eth1'
option proto 'static'
option ipaddr '192.168.10.1'
option netmask '255.255.255.0'
Then restart the network service to enable
/etc/init.d/network restart
You need to be on the ‘inside’ to access the web interface. Since this is an overlay network where both interfaces are connected to the same LAN, you can simply change your workstation’s IP address manually to an address in that space.
If that’s not possible, you can stop the firewall service /etc/init.d/network restart to connect from the WAN side.
Access the web interface at http://192.168.10.1 and login as root (no password). Answer yes if prompted about checking for updates, and click Network -> Interfaces and answer yes again when prompted to allow it to update network names. Save and apply after accepting.
You may find that you’re running a release candidate despite not picking one. The image builds are not always tagged with ‘RC’ so you may check the latest actual release before grabbing one.
When running lots of guests it helps to but them behind a virtual router. If you’re keeping things lean by using LXC containers you can put your router in a container too with OpenWRT.
The process in PVE is to:
You’re going to create a LAN inside of Proxmox and you can do it a couple of different ways;
The simplest thing to do is nothing. You just manually assign IPs and a gateway in a different range than your existing router and have two networks operating on the same physical LAN. The main downside is you can’t take advantage of DHCP because it would conflict with the original LAN.
You can also install a second network card. This of course has a cost, though if you only have one PVE host you can cheat by just creating a new bridge interface that goes nowhere. But this isn’t helpful in a cluster.
The best way is to add a Virtual LAN. Simply edit the config for vmbr0 and enable the VLAN aware checkbox. Then add an interface to the container and specify a VLAN Tag, such as “2”. Most network equipment is happy to pass it along to other cluster members so it just works.
You want just the root file system, not the full image that includes the kernel. Happily, OpenWRT makes this available. Navigate to their releases, find the most recent, and drill down to targets / x86 / 64 / rootfs.tar.gz. It will save along the lines of “openwrt-24.10.1-x86-64-rootfs.tar.gz”.
Next, upload it to PVE with a secure copy to the root home folder like scp openwrt* root@pve01:
What we uploaded earlier isn’t actually a template, but it’s close enough as along as we create the container at PVE’s command line1. The key here is that we provide an archive and set the OS type to unmanaged.
pct create \
201 \
./openwrt* \
--rootfs local-lvm:0.4 \
--ostype unmanaged \
--hostname openwrt \
--arch amd64 \
--cores 2 \
--memory 256 \
--swap 0 \
--features nesting=1 \
--net0 name=eth0,bridge=vmbr0,tag=2 \
--net1 name=eth1,bridge=vmbr0
Also of note, we enable nesting so that dnsmasq will start2 and set the VLAN tag on eth0 which comes up as the LAN interface by default on this image of OpenWRT. The container’s disk uses the rootfs syntax of STORAGE_ID:SIZE_IN_GiB, here being .4 Gigs.
When creating guests, make sure to change their network settings in PVE to have a VLAN tag of ‘2’ (or whatever you’re using).
In OpenWRT, add rules Network -> Firewall -> Port Forwards. There are no WAN rules discrete from port forwarding.
You should update by downloading new firmware, not by using the package manger. In fact: “Generally speaking, the use of opkg upgrade is very highly discouraged. It should be avoided in almost all circumstances3.”
But if you must;
opkg update
opkg list-upgradable | cut -f 1 -d ' ' | xargs opkg upgrade
A default install of PVE creates a single Linux Bridge, usually named vmbr0. Think of this as a virtual switch. The management interface is on that bridge, as well as any containers or guests. Most things just need one interface, but OpenWRT expects two. It is a router, after all.
In most cases, adding a VLAN is best, but there are other options. You can see and make changes in the Proxmox web GUI by changing to Server View, selecting a ProxMox Host, then going to System -> Network.
create a new bridge. Select new and allow it to select the name (which should be vmbr1). Leave the rest at the defaults (all blank with autostart checked). Important If you have a cluster you must actually connect this new bridge to a network adapter in the “Bridge ports” setting. Otherwise, it won’t be able to talk beyond the host it’s currently on. If you don’t have a second NIC, then this probably won’t do what you want.
OpenWrt doesn’t cluster. But you can get close by just syncing the configs. Assuming you’ve already deployed keepalived and contrackd of course.
Most of OpenWrt’s configuration is saved in a few text files in the /etc/config directory. It’s generally safe to sync the whole folder with the exception of the network file or others that have some instance-specific IP or names.
The simplest way is to generate SSH keys and use rsync on a schedule. If you want more control you can setup a realtime sync service via procd. But keep things simple if you can.
This works well if you have MASTER and BACKUP routers, ala keepalived.
Let’s use SSH keys to send changes on the other host and trigger a re-load.
# On the master router:
# Generate a key with modern cipher and no passphrase
mkdir .ssh
ssh-keygen -t ed25519 -f ~/.ssh/id_dropbear -N ""
# Copy that key to the other router. Manually, because ssh-copy-id isn't available.
cat ~/.ssh/id_dropbear.pub | ssh [email protected] "cat >> /etc/dropbear/authorized_keys"
# Verify it worked
ssh [email protected]
/etc/init.d/cron enable
/etc/init.d/cron start
crontab -e
*/5 * * * * /usr/bin/rsync -avz --delete --exclude='keepalived' --exclude='network' /etc/config/ 192.168.1.2:/etc/config/ && ssh 192.168.1.2 "/sbin/reload_config"
Most changes will be replicated but it does leave out WireGuard clients. These are in the network file we are excluding. To tackle that you’ll want to perhaps use a template file. I’ll leave that to to you. Or you can tackle a realtime solution like below.
If need realtime sync, you can tap into OpenWrt’s init and daemon management system, procd. You can couple this with diff and patch to send just changes to things like the network config that you can’t copy wholesale.
You’ll need the diff and patch programs.
opkg install diffutils patch
You also need to configure what you want to keep in sync. This is probably the:
dhcp: hostnames and IP sets you create
The OpenWrt way is to do this with a uci config file. This will let you change values from the command line or even a custom LuCI page later, should you aspire to that level of perfection.
# This file will be automatically detected.
vi /etc/config/syncer
config settings 'main'
option remote_ip '192.168.1.2'
list configs 'firewall'
list configs 'dhcp'
list configs 'network'
Create service for procd to interact with. This service will be a run-and-exit approach as we have procd to keep an eye on things for us.
We’ll ask procd to call our reload function after specific config changes. We’ll to the actual work outside the service so we don’t become a blocker if something goes wrong (preventing a spinning Save and Apply action in the web interface).
vi /etc/init.d/syncer
#!/bin/sh /etc/rc.common
USE_PROCD=1
START=99 # Start last
service_triggers() {
local CONFIGS=$(uci -q get syncer.main.configs)
for CFG in $CONFIGS; do
procd_add_config_trigger "config.change" "$CFG" /etc/init.d/config_syncer reload
done
}
reload_service() {
/usr/bin/syncer.sh &
}
start_service() {
local CONFIGS=$(uci -q get syncer.main.configs)
for CFG in $CONFIGS; do
[ -f "/etc/config/$CFG" ] && cp "/etc/config/$CFG" "/tmp/$CFG.bak"
done
}
Enable the service
chmod +x /etc/init.d/syncer
/etc/init.d/syncer enable
/etc/init.d/syncer start
You should now see a few .bak files in your /tmp directory.
Now we need a simple script to create and send the patch files, then trigger a reload on the other router.
touch /usr/bin/syncer.sh
chmod +x /usr/bin/syncer.sh
vi /usr/bin/syncer.sh
#!/bin/sh
# Load settings using the 'uci' command
REMOTE_IP=$(uci -q get syncer.main.remote_ip)
CONFIGS=$(uci -q get syncer.main.configs)
# Exit if no IP is set
[ -z "$REMOTE_IP" ] && exit 1
for CFG in $CONFIGS; do
# If the backup file exists create a patch against it
if [ -f /tmp/$CFG.bak ]; then
diff -u /tmp/$CFG.bak /etc/config/$CFG > /tmp/$CFG.patch
# If the patch has data send it and patch
if [ -s /tmp/$CFG.patch ]; then
scp /tmp/$CFG.patch $REMOTE_IP:/tmp/ && ssh $REMOTE_IP "patch /etc/config/$CFG < /tmp/$CFG.patch"
fi
fi
# Update backup file for next time.
cp /etc/config/$CFG /tmp/$CFG.bak
done
# Ask the other router to reload it's changed configs
ssh $REMOTE_IP "/sbin/reload_config"
Make sure to install diff and patch on the other side.
This on-demand approach works in my testing, but I haven’t run it for a long time. I can imagine the configs getting out of sync should one system be off-line when a change is made as there is no catch-up. It might benefit from error handling when the scp fails.
As an alternative to diff and patch, I tried out the message bus ubus. You can hook into that, but it turns out it doesn’t capture events from the web GUI.
For the GUI, you can find the staged files in /tmp and transmit them. This would work well, but there’s not a great way to know exactly when staged changes are being applied, other than to use ionotify on the config files and suck in the staged changes before they are erased. That’s just too hacky.
DHCP - On larger deployments, clients can step on each other after a fail-over as the second server can lease IPs to new clients that are already in use. Dnsmasq keeps the leases in memory and writes them to /tmp/dhcp.leases whenever a change occurs. So you’d periodically rsync the /tmp/dhcp.leases file to the BACKUP and then have keepalived issue a killall -HUP dnsmasq" when a fail-over event happens. Then handle it moving back.
Contrackd - clients will lose their stateful-ness during a fail over. While this is fine for some web traffic, games and some streaming will die.
Active/Active - Some sources suggest splitting the roles with one router having the internal gateway and the other having the main WAN address. But this causes asymmetric routing. A packet might go out through Router A but the reply comes back through Router B. If you have a firewall (like OpenWrt’s fw4/nftables) enabled, Router B will see a “reply” packet for a connection it never saw start. It will flag this as “Invalid” and drop the packet. Though If you’re using NAT you can simply pass out internal gateways round-robin for both, and move VIPs should one fail.
The official docs work well and are summarized below.
In the GUI, go to System -> Software, click the Update Lists button and search for “luci-proto-wireguard”. Installing that will pull in the needed dependency. Restart the network services via System → Startup → Initscripts -> network → Restart.
Select “Network → Interfaces → Add new interface” Input the name wg0 and select WireGuard VPN.
In the subsequent screen, find and click the “Generate new key pair” button and enter 51820 for the listen port.
For IP addresses enter an address and network that will encompass your VPN, such as 10.0.0.1/24
You can add peers in this interface as well.
You’ll need to create a new zone that allows forwarding to the LAN, and a rule to allow the WireGuard traffic in. Refer to section 6 in the docs for that.
If you want to do it at the command line, edit your /etc/config/firewall file and add
config zone
option name 'WireGuardVPN'
option input 'ACCEPT'
option output 'ACCEPT'
option forward 'ACCEPT'
option mtu_fix '1'
list network 'wg0'
config rule
option src 'wan'
option name 'Wireguard-incoming'
list proto 'udp'
option dest_port '51820'
option target 'ACCEPT'
And for the Interface, to the /etc/config/interfaces add:
config interface 'wg0'
option proto 'wireguard'
option private_key 'xxxxx'
option listen_port '51820'
list addresses '10.0.0.1/24'
When you set an OPNsense system up with supported 10G cards, say the Intel X540-AT2, you can move 6 to 8 Gb a second. Though this is better than in the past, but not line speed.
# iperf between two systems routed through a dial NIC on OPNsense
[ ID] Interval Transfer Bandwidth
[ 1] 0.0000-10.0040 sec 8.04 GBytes 6.90 Gbits/sec
This is because the packet filter is getting involved. If you disable that you’ll get closer to line speeds
Firewall –> Settings –> Advanced –> Disable Firewall
[ ID] Interval Transfer Bandwidth
[ 1] 0.0000-10.0067 sec 11.0 GBytes 9.40 Gbits/sec
VyOS gets a lot of respect as a network appliance. It’s a Debian-based router/firewall descended from the Vyatta project and has a command line config similar to JUNOS. It scores well in speed and reliability tests, a free version is available and commercial support is easy to get.
They do steer you toward the rather expensive commercial option by limiting access to the LTS versions, but you can always download the rolling release, the beta, or build it from source with a fairly straight-forward docker build process.
Downloading the beta, or stream as they call it, can be done with:
wget https://community-downloads.vyos.dev/stream/1.5-stream-2025-Q1/vyos-1.5-stream-2025-Q1-generic-amd64.iso
And a creation something like
qm create 200 \
--name vyos \
--memory 2048 \
--net0 virtio,bridge=vmbr0 \
--net1 virtio,bridge=vmbr0,tag=2 \
--ide2 media=cdrom,file=local:iso/live-image-amd64.hybrid.iso \
--virtio0 local-lvm:15
Then it’s just a matter of booting from the iso and running the very simple [install] and [quick-start] process. Assuming you’re going with the normal setup, hit the console and enter something like this.
Step 1 - Configure the interfaces and enable remote access.
# Enter 'configure' mode
configure
# For the address space 192.168.1.0/24
# Configure the LAN and WAN ports, with eth0 being the WAN
set interfaces ethernet eth0 address dhcp
set interfaces ethernet eth0 description 'OUTSIDE'
set interfaces ethernet eth1 address '192.168.1.1/24'
set interfaces ethernet eth1 description 'LAN'
# Enable remote login
# set service ssh listen-address '192.168.1.1' # Possibly don't listen on WAN if you don't need it
set service ssh port '22'
# Commit the changes and save if they work
commit
save
Step 2 - configure DNS/DHCP. There’s a lot of text, so we usually to SSH in to continue.
# Configure LAN DHCP services
ssh [email protected]
configure
set service dhcp-server shared-network-name LAN subnet 192.168.1.0/24 option default-router '192.168.1.1'
set service dhcp-server shared-network-name LAN subnet 192.168.1.0/24 option name-server '192.168.1.1'
set service dhcp-server shared-network-name LAN subnet 192.168.1.0/24 option domain-name 'vyos.net'
set service dhcp-server shared-network-name LAN subnet 192.168.1.0/24 lease '86400'
set service dhcp-server shared-network-name LAN subnet 192.168.1.0/24 range 0 start '192.168.1.9'
set service dhcp-server shared-network-name LAN subnet 192.168.1.0/24 range 0 stop '192.168.1.254'
set service dhcp-server shared-network-name LAN subnet 192.168.1.0/24 subnet-id '1'
# Configure DNS services
set service dns forwarding cache-size '0'
set service dns forwarding listen-address '192.168.1.1'
set service dns forwarding allow-from '192.168.1.0/24'
commit
save
Step 3 - enable NAT
# Enable masquerade - assuming you need it.
set nat source rule 100 outbound-interface name 'eth0'
set nat source rule 100 source address '192.168.1.0/24'
set nat source rule 100 translation address masquerade
commit
save
Step 4 - Set Names and Enable basic firewall rules
# Use the 'group' feature to give the interfaces more readable names than ethX
set firewall group interface-group WAN interface eth0
set firewall group interface-group LAN interface eth1
set firewall group network-group NET-INSIDE-v4 network '192.168.1.0/24'
# Typically you want a default drop as a global rule.
set firewall global-options state-policy established action accept
set firewall global-options state-policy related action accept
set firewall global-options state-policy invalid action drop
[install]:https://docs.vyos.io/en/latest/installation/install.html#permanent-installation
[quick-start]:https://docs.vyos.io/en/latest/quick-start.html
An old boss of mine said ’network demand is unbounded’. I suppose that’s the ‘if you build it, they will come’ traffic theory of induced demand. I’ve not observed that in practice. When we upgrade capacity traffic doesn’t grow to meet it. But there are peaks involving Apple updates and PlayStation launches you should manage. And if you don’t have enough to meet normal demand as is, you should at least manage what you can’t do.
When you don’t do anything, things still work. TCP for example recognizes when packets are being lost and backs-off the speed. But the side effect is that meanwhile, packets are queued up in the router’s buffers before ultimately being dropped. When lots of users are involved they are all ‘detecting’ congestion on their own, leading to a lot of latency. On top of this, UDP packets have no built-in congestion control and may do who-knows-what. Basically, everything gets slow. Slower than it needs to be.
If you shape the traffic, usually by controlling what packets drop, you can minimize latency, make sure important traffic gets through, and apply some fairness between users.
The old days you would look at protocol and port to classify traffic, and choose how much to let though of any given class. Sometimes dynamically allocating bandwidth depending on the technology on-hand.
Modern methods use flow identification and heuristics. This prevents a bad actor from trying to mimic DNS requests to get around the rules.
You don’t have any control over what people are giving you, other than sending Explicit Congestion Notification (ECN) back to the sender. You can only applying shaping to packets leaving your queue. But on a router this is fine because packets are leaving your queue all time time as you forward them to their destination.
Linux uses the Traffic Control (TC) subsystem to manage traffic by assigning queues to the network interfaces. You choose the algorithm the queue uses. So the network packets get forwarded (or dropped) based on what you want.
The Common Applications Kept Enhanced, or CAKE algorithm is a good choice. It prioritizes “sparse” flows over bulk downloads. The thought being that sparse flows represent gaming and other latency sensitive traffic. It also enforces fairness between hosts and sends congestion signals upstream. And it does all this with minimal configuration.
To use it, you simply set a threshold. No one is limited until you get close, where it starts dropping packets to keep you within that limit.
By default it puts traffic in three “tins”; Bulk, Best Effort, and Voice. It looks for packets that are explicitly marked ‘EF’ (expedited forwarding) by the PC for the Voice tin, but not all games do this, so many fall into Best Effort along with general web traffic. Continuous flows move down from that into the Bulk tin.
Optimal latency is automatically detected and when not being met, packets are dropped algorithmically. This reduces traffic until optimal latency is restored.
Debian makes a good platform and includes tc-cake by default.
# Given that you have two interfaces, enable forwarding.
sysctl -w net.ipv4.ip_forward=1
# and persist
echo "net.ipv4.ip_forward=1" >> /etc/sysctl.conf
# Apply cake to one of the interfaces - preferably the one
# closest to the bottleneck
tc qdisc delete root dev eth0
tc qdisc add root dev eth0 cake bandwidth 1000Mbit ethernet
# See how the traffic looks
tc -s qdisc show dev eth0
This is all you need. Though if you might want a HA pair.
The thought is you have a virtual IP address and if one of the boxes goes down the other will take it’s place.
You can use ucarp, keepalived or pacemaker. Since we only need a floating IP, let’s use ucarp as it’s the simplest.
Do the following on both servers
# Install ucarp
sudo apt -y install ucarp
# Add an interface on both servers to the bottom
vi /etc/network/interfaces
...
iface eth0:ucarp inet static
address 10.1.1.30
netmask 255.255.255.0
# Create a config file. Correct SOURCE_ADDRESS as needed.
# On the second server, increase ADVSKEW by 1.
vi /etc/ucarp/vip-common.conf
BIND_INTERFACE="eth0"
PASSWORD="SomePassLessThan20Chars"
VIP_ADDRESS="10.1.1.30"
SOURCE_ADDRESS="10.1.1.4"
VHID="1"
ADVBASE="1"
ADVSKEW="10"
OPTIONS="-P -z"
UPSCRIPT="/usr/share/ucarp/vip-up"
DOWNSCRIPT="/usr/share/ucarp/vip-down"
# Create a service
vi /etc/systemd/system/ucarp.service
[Unit]
Description=UCARP virtual interface %I
After=network.target
[Service]
EnvironmentFile=-/etc/ucarp/vip-common.conf
ExecStart=/usr/sbin/ucarp $OPTIONS -i $BIND_INTERFACE -p $PASSWORD -v $VHID -a $VIP_ADDRESS -s $SOURCE_ADDRESS -b $ADVBASE -k $ADVSKEW -u $UPSCRIPT -d $DOWNSCRIPT
KillMode=control-group
[Install]
WantedBy=multi-user.target
# And enable
systemctl enable --now ucarp
https://ndk.sytes.net/wordpress/?p=1296 https://blog.no42.org/article/ucarp-high-availability/
This page is mostly is for historical info should I ever need to deploy an IPsec VPN again. Consider it highly suspect.
Most PCs and Handhelds come with an IPsec client built in. This removes the need to download and install software and is especially handy on mobile and other low-power devices where IPsec is often hardware accelerated1. A lower barrier to entry and better speed is always good.
Let’s use pfSense as an example. It (or OPNsense) is available in an appliance model and Netgate maintains reasonable documentation, so we can focus on implementation details. The main steps are;
Use the installation ISO and configure with just a WAN interface.
https://docs.netgate.com/pfsense/en/latest/install/install-walkthrough.html
If you want to use an existing password database you must employ a RADIUS server as the middleman. The VPN server used by pfSense (Strongswan) doesn’t support checking passwords directly via LDAP. Your only choices are to store a password on the pfSense box itself (aka pre-shared-key), or use RADIUS.
In situations where entering pre-shared keys on the firewall in plain text is undesirable, a RADIUS server may be used instead…[^4]
https://docs.netgate.com/pfsense/en/latest/packages/freeradius.html
A VPN uses a different type of cert than a web server so you probably can’t use one you already have. The good news is you can take advantage of the Let’s Encrypt project at no cost and this is better than managing a private CA and wrangling bring-your-own devices into installing certificates. If you want to know more about the details, check out the certificate background page.
Using the stand-alone, disabled the redirect, allow port 80 to this firewall
https://docs.netgate.com/pfsense/en/latest/packages/acme/index.html
System -> User Manager -> Authentication Servers,
Click the “Add” button and configure a source as follows
Server Settings
Descriptive name: (Your RADIUS Server's name)
Type: RADIUS
RADIUS Server Settings
Protocol: MS-CHAPv2
Hostname or IP address: (Your RADIUS IP)
Shared Secret: (From the NPS client setup, above)
Services offered: Authentication and Accounting
Authentication port: 1812
Accounting port: 1813
Authentication Timeout:
RADIUS NAS IP Attribute: WAN
Now test under the diagnostic menu
Diagnostics -> Authentication
If you have trouble, head to the Event Viewer’s Server Roles Network Policy log to see what’s up with NPS or other locally with the FreeRADIUS plugin
(Not sure if this is needed)
Create Matching Groups in AD + Local pfSense
pfSense Side Group Configuration:
System>User Manager> Groups: Scope: Remote, Assigned Privileges: “User - VPN: IPsec xauth Dialin”.
You must enable the IKE extensions and select the RADIUS server as a first step. This is under the Mobile Clients tab
VPN -> IPsec
Select the Mobile Clients (tab) and configure as follows
Enable IPsec Mobile Client Support
IKE Extension: x (check this)
Extended Authentication (Xauth)
User Authentication: RADIUS
Group Authentication: none
Client Configuration (mode-cfg)
Virtual Address Pool: x (check this)
192.168.1.0 24
(Leave the rest un-checked)
At the top, you’ll have an alert prompting you to Create Phase 1. Select that to create a phase 1 tunnel and configure as bellow. You won’t have an option to select EAP-RADIUS as a tunnel option if you tried to navigate to Tunnels othewise.
General Information
Disabled: (Un-checked)
Key Exchange version: IKEv2 **Non Default**
Internet Protocol: IPv4
Interface: WAN
Remote Gateway:
Description:
Phase 1 Proposal (Authentication)
Authentication Method: EAP-RADIUS **Non Default**
My Identifier: My IP Address **possibly actual address, non default**
Peer Identifier: Any **non default**
My Certificate: (Your Cert from above)
Phase 1 Proposal (Encryption Algorithm)
Encryption Algorithm: 3DES SHA1 2 (1024 bit)
(leave the rest a default)
Save and apply
Encryption Algorithm: The following settings were chosen for MAC Client compatibility, based on guides; other settings might be better or possible with further testing on MAC clients.
Phase 2: Select “+ Add P2” a. Mode: Tunnel IPv4 b. Local Network: Test Environment Selected LAN Subnet- LAN Specified in the next step c. Phase 2 Proposal: i. AES, 3DES d. Hash Algorithms: i. SHA1, SHA256
ONLY the CA Cert needs to be installed on clients: a. Double-click to open this certificate.
i. Windows: Install>Local Machine> Trusted Certificate Authorities ii. MAC: Open With> Keychain.app> Login> Select CA Cert> Get Info> Dropdown “Trust”> When Using this Certificate Always Trust.
Windows Setup: Start>Settings>Network & Internet>VPN>Add a VPN Connection> Provider: Windows Built-in Connection Name: VPN Test Server Name or Address: WAN Address of Server OR FQDN, either worked for testing (a DNS A Record is set for the test external IP, so it does resolve an address) VPN Type: IKEv2 Type of Sign-in Info: Username & Password Recommend Uncheck “Remember my Sign-In Info
(As per above) Mac Client Test Add the Cert (testing only)
Added the VPN
To use
Client Config
I read that Windows 10 gets a default route to devices on the same network as its virtual IP but otherwise you must use the command lets to add the routes and that it doesn’t accept route pushdown. Of course I read that in random forums so I take it with a grain of salt
In the strongswan documentation they say that “…windows sends DHCP request upon connection and add routes supplied in option 249 of DHCP reply.”
https://wiki.strongswan.org/projects/strongswan/wiki/WindowsClients
https://forum.netgate.com/topic/67684/explained-example-dhcp-option-121-249
using local dhcp, but not internal https://www.reddit.com/r/PFSENSE/comments/3hql33/configuring_openvpn_bridge_with_local_dhcp/
We may be able to hand off to a lan network can configure the pfSense dhcp server on the same box.
It also appears there is a newer windows VPN client or something that takes XML files. But this might just be their MDM
https://docs.microsoft.com/en-us/windows/security/identity-protection/vpn/vpn-routing
And here is a cool doc that details some of the config options we needs if we had to do it by hand.
https://blog.arrogantrabbit.com/vpn/IKEv2-VPN-setup-on-Apline-Linux/#configuring-client-devices
unknown - probably for the cert mac footnote https://www.personalvpn.com/support/mac/ikev2
need a citation here ↩︎
Certificates have many fields but when it comes to VPNs using IPsec IKEv2 (Internet Key Exchange v2) there are three that are important to us.
These are usually abbreviated KU, EKU and SAN. The IETF publishes guidelines1 on how they should be used in this context, but vendor implementation2 has specific requirements you must accept and a certificate that you’d use on a web server may not work on a VPN.
For a web server KU is all that is needed. As long as that field has the values of digitalSignature and keyEncipherment3 your web client will interact with it.
An IPSec VPN server however, requires the additional fields of EKU and SAN. Specifically, the EKU must contain the value serverAuth and the SAN must contain the DNS name of the server4. For compatibility with older Macs, you should also add the EKU value “IP Security IKE Intermediate5” even though it’s currently deprecated6.
A RADIUS server also requires EKU and SAN values. You may have multuple RADIUS servers in play, e.g. rad1.gattis.org,rad2.gattis.org, and all must be added. Interestingly, KU values are optional.
If you were generating them via openssl and a conf file per the MS reqs, it would look like this7
openssl genrsa -out ./ucm.key 2048
openssl req -new -key ucm.key -config ucm.conf -verbose -out ucm.csr
# Here's the contents of the file
[req]
distinguished_name = req_distinguished_name
req_extensions = v3_req
prompt = no
[req_distinguished_name]
commonName = Secure Communications Server
countryName = US
stateOrProvinceName = State
localityName = Town
organizationName = Company
organizationalUnitName = Department
emailAddress = [email protected]
[v3_req]
keyUsage = digitalSignature,keyEncipherment
extendedKeyUsage = serverAuth
subjectAltName = @alt_names
[alt_names]
DNS.1 = rad1.gattis.org
DNS.2 = rad2.gattis.org
DNS.3 = vpn.gattis.org
The CA that signs it will by default remove the all the things except the distinguished name. If you’re signing it yourself you must create a file similar to above.
vi radius.ext
authorityKeyIdentifier=keyid,issuer
basicConstraints=CA:FALSE
keyUsage = digitalSignature, keyEncipherment
extendedKeyUsage = serverAuth
subjectAltName = @alt_names
[alt_names]
DNS.1 = rad1.gattis.org
DNS.2 = rad2.gattis.org
DNS.3 = vpn.gattis.org
openssl x509 -req -in radius.csr -CA myCA.pem -CAkey myCA.key -CAcreateserial -out radius.crt -days 1825 -sha256 -extfile radius.ext
# View the results. If you don't see the "X509v3 extensions" for SAN and such, check the ext file you used.
openssl x509 -in radius.crt -text -noout
When you get a signed cert back from the provider, it should look something like this. Specifically you should see the line X509v3 Extended Key Usage: as below. Otherwise it won’t work.
Importat - if you get it back in windows format, it won’t show you the EKUs. You must convert from pkcs7 to x509
# Convert if needed
openssl pkcs7 -print_certs -in 366713309.cer -out certificate.pem
# Otherwise, check the extentions
openssl x509 -in certificate.pem -text -noout
...
...
X509v3 extensions:
X509v3 Key Usage: critical
Digital Signature, Key Encipherment
X509v3 Extended Key Usage:
TLS Web Server Authentication, TLS Web Client Authentication
...
...
You’ll notice that since this cert has everything that web servers, VPNs and RADIUS servers need, you can use it in all three locations. Indeed, these certs are sometimes sold as a “Unified Communications Certificate” for significantly more than a normal web server cert.
Interestingly, Let’s Encrypt (the free service) seems to offer certs that have almost exactly this. The only caveat is that all the SAN values must resolve to the verifying server and you may not support older macs.
https://support.microsoft.com/en-us/help/814394/certificate-requirements-when-you-use-eap-tls-or-peap-with-eap-tls ↩︎
https://security.stackexchange.com/questions/24106/which-key-usages-are-required-by-each-key-exchange-method/24107#24107 ↩︎
https://libreswan.org/wiki/VPN_server_for_remote_clients_using_IKEv2 ↩︎
https://wiki.strongswan.org/projects/strongswan/wiki/Win7CertReq ↩︎
https://security.stackexchange.com/questions/34603/how-do-i-make-or-get-computer-certificates-to-use-with-ipsec-user-authentication ↩︎
https://stackoverflow.com/questions/33781051/openssl-csr-using-multiple-san-and-extended-key-usage ↩︎
Wireguard is the best VPN choice for most situations. It’s faster and simpler than its predecessors and what you should be using on the internet.
Check out the deployment if you need to create a network with it, or look for a specific type of peer if you have something unusual.
By the last point, we mean there is no central authority required. Each peer defines their own IP address, routing rules, and decides from whom they will accept traffic. Every peer must exchange public keys with every other other peer. Traffic is sent directly between configured peers. You can create that design, but it’s not baked-in.
The way you deploy depends on what you’re doing, but in general you’ll either connect directly point-to-point or create a central server for remote access or management.
This is the classic setup where clients initiate a connection. Configure a wireguard server and tell your clients about it. This is also useful for remote management when devices are behind NAT. Perform the steps in:
And then choose based on if your goal is to:
or
You can also have peers talk directly to each other. This is often used with routers to connect networks across the internet.
A central server gives remote devices a reachable target, allowing them to traverse firewalls and NAT and connect. Let’s create a server and generate and configure a remote device.
You’ll need:
A dynamic domain name will work and it’s reasonably priced (usually free). You just need something for the peers to connect to, though a static IP is best. You can possibly break connectivity if your IP changes while your peers are connected or have the old IP cached.
We use Debian in this example and derivatives should be similar. UDP 51820 is the standard port but you can choose another if desired.
You must also choose a VPN network that doesn’t overlap with your existing networks. We use 192.168.100.0/24 in this example. This is the internal network used inside the VPN to route traffic.
sudo apt install wireguard-tools
The server needs just a single config file, and it will look something like this:
[Interface]
Address = 192.168.100.1/24
ListenPort = 51820
PrivateKey = sGp9lWqfBx+uOZO8V5NPUlHQ4pwbvebg8xnfOgR00Gw=
We choose 192.168.100.0/24 as our VPN internal network and picked .1 as our server address (pretty standard), created a private key with the wg tool, and put that in the file /etc/wireguard/wg0.conf. Here’s the commands to do that.
# As root
cd /etc/wireguard/
umask 077
wg genkey > server_privatekey
wg pubkey < server_privatekey > server_publickey
read PRIV < server_privatekey
# We create the file wg0.conf here
cat << EOF > wg0.conf
[Interface]
Address = 192.168.100.1/24
ListenPort = 51820
PrivateKey = $PRIV
EOF
The VPN operates by creating network interface and loading a kernel module. You can use the linux ip command to add a network interface of type wireguard (that automatically loads the kernel module) or use the wg-quick command do do it for you.
# The tool looks for the wg0.conf file you created earlier
wg-quick up wg0
ping 192.168.100.1
wg-quick down wg0
For normal use, employ systemctl to create a service using the installed service file.
systemctl enable --now wg-quick@wg0
That’s it - add remote clients/peers and they will be able to connect.
When something is wrong, you don’t get an error message, you just get nothing. You bring up the client interface but you can’t ping the server. So turn on log messages on the server with this command.
echo module wireguard +p > /sys/kernel/debug/dynamic_debug/control
dmesg
# When done, send a '-p'
wg0: Invalid handshake initiation from 205.133.134.15:18595
In this case, you should check your keys and possibly take the server interface down and up.
ifconfig: ioctl 0x8913 failed: No such device
Check your conf is named /etc/wireguard/wg0.conf and look for any mistakes. Replace from scratch if nothing else.
If you see no wireguard error messages, suspect your firewall. Since it’s UDP you can’t test the port directly, but you can use netcat.
# On the server
systemctl stop wg-quick@wg0
nc -ulp 51820
# On the client.
nc -u some.server 51820
# Type some text and it should be echoed on the server
In theory, the client is an autonomous entity with whom you negotiate IPs and exchange public keys. In practice, you’ll just create a conf file and distribute it.
Each participant must have a unique Key-Pair and IP address. You cannot reuse keys as they are hashed and used as for internal routing.
# On the 'server'
cd /etc/wireguard
wg genkey > client_privatekey # Generates and saves the client private key
wg pubkey < client_privatekey # Displays the client's public key
Choose an IP for the client and add a block at the bottom of your server’s wg0.conf. It’s fine to just increment the IP as you add clients . Note the /32, meaning on traffic with that specific IP is accepted from this peer - it’s not a router on the other side, after all.
# Add this block to the bottom of your server's wg0.conf file
## Some Client ##
[Peer]
PublicKey = XXXXXX
AllowedIPs = 192.168.100.2/32
# Load the new config
wg-quick down wg0 && wg-quick up wg0
This is the file that the client needs. It will look similar to this. The [Interface] is about the client and the [Peer] is about the server.
[Interface]
PrivateKey = THE-CLIENT-PRIVATE-KEY
Address = 192.168.100.2/32
[Peer]
PublicKey = YOUR-SERVERS-PUBLIC-KEY
AllowedIPs = 192.168.100.0/24
Endpoint = your.server.org:51820
Put in the keys and domain name, zip it up and send it on to your client as securely as possible. Maybe keep it around for when they loose it. One neat trick is to display a QR code right in the shell. Devices that have a camera can import from that.
qrencode -t ANSIUTF8 < client-wg0.conf
On the client side, install the tools and place the config file.
# Install the wireguard tools
sudo apt install wireguard-tools
# Copy the config file to the wireguard folder
sudo cp /home/you/client-wg0.conf /etc/wireguard/wg0.conf
sudo wg-quick up wg0
ping 192.168.100.1
sudo wg-quick down wg0
# Possibly enable this as a service or import as a network manager profile
systemctl enable --now wg-quick@wg0
## OR ##
# You may want to rename the file as that's used in nm as it's name
nmcli connection import type wireguard file client-wg0.conf
sudo nmcli connection modify client-wg0.conf autoconnect no
You can download the client from here and add the config block
https://www.wireguard.com/install/
You should be able to ping the server from the client and vice versa. If not, take a look at the troubleshooting steps in the Central Server page.
You’re connected to the server - but that’s it. You can’t do anything other than talk to it. The next step depends on if you want to:
This is the classic road-warrior setup where remote peers initiate a connection to the central peer. That central system forwards their traffic onward to the corporate network.
The main choice is route or masquerade.
If you route, the client’s VPN IP address is what other devices see. This is generally preferred as it allows you to log who was doing what at the individual servers. But you must update your network equipment to treat the central server as a router.
Masquerading causes the server to translate all the traffic. This makes everything look like its coming from the server. It’s less secure, but less complicated and much quicker to implement.
For this example, we will masquerade traffic from the server.
Use sysctl to enable forwarding on the server and nft to add masquerade.
# as root
sysctl -w net.ipv4.ip_forward=1
nft flush ruleset
nft add table nat
nft add chain nat postrouting { type nat hook postrouting priority 100\; }
nft add rule nat postrouting masquerade
It’s best if we add our new rules onto the defaults and enable the nftables service.
# as root
echo "net.ipv4.ip_forward=1" >> /etc/sysctl.conf
nft list ruleset >> /etc/nftables.conf
systemctl enable --now nftables.service
Your remote peer - the one you created when setting up the server - needs it’s AllowedIPs adjusted so it knows to send more traffic through the tunnel.
This sends all traffic from the client over the VPN.
AllowedIPs = 0.0.0.0/0
The most common config is to send specific networks through the tunnel. This keeps netflix and such off the VPN
AllowedIPs = 192.168.100.0/24, 192.168.XXX.XXX, 192.168.XXX.YYY
In some cases, you’ll need the client to use your internal DNS server to resolve private domain names. Make sure this server is in the AllowedIPs above.
[Interface]
PrivateKey = ....
Address = ...
DNS = 192.168.100.1
By default, everything is open and all the peers can talk to each other and the internet at large - even NetFlix! (they can edit their side of the connection at will). So let’s add some rules to the default filter table.
This example prevents peers from from talking to each other but let’s them ping the central server and reach the corporate network.
# Load the base config in case you haven't arleady. This includes the filter table
sudo nft -f /etc/nftables.conf
# Reject any traffic being sent outside the 192.168.100.0/24
sudo nft add rule inet filter forward iifname "wg0" ip daddr != 192.168.100.0/24 reject with icmp type admin-prohibited
# Reject any traffic between peers
sudo nft add rule inet filter forward iifname "wg0" oifname "wg0" reject with icmp type admin-prohibited
You may want to add an exception for one of the addresses so that an administrator can interact with the remote peers. Order matters, so add it before before the other rules above
sudo nft -f /etc/nftables.conf
# Allow an special 'admin' peer full access and others to reply
sudo nft add rule inet filter forward iifname "wg0" ip saddr 192.168.100.2 accept
sudo nft add rule inet filter forward ct state {established, related} accept
# As above
...
...
Since this change is a little more complex, we’ll replace the existing file config file and add notes.
sudo vi /etc/nftables.conf
#!/usr/sbin/nft -f
flush ruleset
table inet filter {
chain input {
type filter hook input priority 0
}
chain forward {
type filter hook forward priority 0
# Accept admin traffic and responses
iifname "wg0" ip saddr 192.168.100.2 accept
iifname "wg0" ct state {established, related} accept
# Reject other traffic between peers
iifname "wg0" oifname "wg0" reject with icmp type admin-prohibited
# Reject traffic outside the desired network
iifname "wg0" ip daddr != 192.168.100.0/24 reject with icmp admin-prohibited
}
chain output {
type filter hook output priority 0
}
}
table ip nat {
chain postrouting {
type nat hook postrouting priority srcnat
masquerade
}
}
Note: The syntax of the file is slightly different than the command. You can use nft list ruleset to see how nft config and commands translate into running rules. For example - the policy accept is being appended. You may want to experiment with explicitly adding policy drop.
The forwarding chain is where routing type rules go (the input chain is traffic sent to the host itself). Prerouting might work as well, though it’s less common and not present by default.
The default nftable config file in Debian is:
#!/usr/sbin/nft -f
flush ruleset
table inet filter {
chain input {
type filter hook input priority filter;
}
chain forward {
type filter hook forward priority filter;
}
chain output {
type filter hook output priority filter;
}
}
If you have old iptables rules you want to translate to nft, you can install iptables and add them (they get translated on the fly into nft) and nft list ruleset to see how to they turn out.
In this scenario peers initiate connections to the central server, making their way through NAT and Firewalls, but you don’t want to forward their traffic.
No forwarding or masquerade is desired, so there is no additional configuration to the central server.
The remote peer - the one you created when setting up the server - is already set up with one exception; a keep-alive.
When the remote peer establishes it’s connection to the central server, intervening firewalls allow you to talk back as they assume it’s in response. However, the firewall will eventually ‘close’ this window unless the client continues sending traffic occasionally to ‘keep alive’ the connection.
# Add this to the bottom of your client's conf file
PersistentKeepalive = 20
You should apply some controls to your clients to prevent them from talking to each other (and the server), and you also need a rule for the admin station. You can do this by adding rules to the forward chain.
# Allow an 'admin' peer at .2 full access to others and accept their replies
sudo nft add rule inet filter forward iifname "wg0" ip saddr 192.168.100.2 accept
sudo nft add rule inet filter forward ct state {established, related} accept
# Reject any other traffic between peers
sudo nft add rule inet filter forward iifname "wg0" oifname "wg0" reject with icmp type admin-prohibited
You can persist this change by editing your /etc/nftables.conf file to look like this.
sudo vi /etc/nftables.conf
#!/usr/sbin/nft -f
flush ruleset
table inet filter {
chain input {
type filter hook input priority 0;
}
chain forward {
type filter hook forward priority 0;
# Accept admin traffic
iifname "wg0" ip saddr 192.168.100.2 accept
iifname "wg0" ct state {established, related} accept
# Reject other traffic between peers
iifname "wg0" oifname "wg0" reject with icmp type admin-prohibited
}
chain output {
type filter hook output priority 0;
}
}
table ip nat {
chain postrouting {
type nat hook postrouting priority srcnat; policy accept;
masquerade
}
}
In our remote access example we chose to masquerade. But you can route where your wireguard server forwards traffic with the VPN addresses intact. You must handle that on your network in one of the following ways.
Classically, you’d treat the wireguard server like any other router. You’d create a management interface and/or a routing interface and advertise routes appropriately.
On a small network, you would simply overlay an additional IP range on top of the existing on by adding a second IP address on your router and put your wireguard server on that network. Your local servers will see the VPN addressed clients and send traffic to the router that will pass it to the wireguard server.
In a small network you might have the central peer on the same network as the other servers. In this case, it will be acting like a router and forwarding traffic, but the other servers won’t know about it and so will send replies back to their default gateway.
To remedy this, add a static route at the gateway for the VPN range that sends traffic back to the central peer. Asymmetry is generally frowned upon, but it gets the job done with one less hop.
You can also configure the servers in question with a static route for VPN traffic so they know to send it directly back to the Wireguard server. This is fastest but you have to visit every host. Though you can use DHCP to distribute this route in some cases.
If both system are listening then either side can initiate a connection. That’s essentially what a Point-to-Point setup is. Simply translate create two ‘servers’ and add a peer block to each one about the other. They will connect as needed.
This is best done with a routed config where clients who know nothing about the VPN use one side as their gateway for a given network range, and the servers act as routers. I don’t have an example config for this, but if you’ve reached this point you can probably handle that yourself.
You can directly bring up a Wireguard interface in TrueNAS Scale, and use that to remotely manage it.
Wireguard isn’t exposed in the GUI, so use the command line to create a config file and enable the service. To make it persistent between upgrades, add a cronjob to restore the config.
Add a basic peer as when setting up a Central Server and save the file on the client as /etc/wireguard/wg1.conf. It’s rumored that wg0 is reserved for the TrueNAS cloud service. Once the config is in place, use wg-quick up wg1 command to test and enable as below.
nano /etc/wireguard/wg1.conf
systemctl enable --now wg-quick@wg1
If you use a domain name in this conf for the other side, this service will fail at boot because DNS isn’t up and it’s not easy to get it to wait. So add a pre-start to the service file to specifically test name resolution.
vi /lib/systemd/system/[email protected]
[Service]
...
...
ExecStartPre=/bin/bash -c 'until host google.com; do sleep 1; done'
Note: Don’t include a DNS server in your wireguard settings or everything on the NAS will attempt to use your remote DNS and fail if the link goes down.
When deploying an app, click the enable “Host Network” or “Configure Host Network” box in the apps config and you should be able to access via the VPN address. On Cobia (23.10) at least. If that fails, you can add a command like this to a post-start in the wireguard config file.
iptables -t nat -A PREROUTING --dst 192.168.100.2 -p tcp --dport 20910 -j DNAT --to-destination ACTUAL.LAN.IP:20910
The other side of your connection may dynamic address and wireguard wont know about it. A simple solution is a cron job that pings the other side periodically, and if it fails, restarts the interface. This will lookup the domain name again and hopefully find the new address.
touch /etc/cron.hourly/wg_test
chmod +x /etc/cron.hourly/wg_test
vi /etc/cron.hourly/wg_test
#!/bin/sh
ping -c1 -W5 192.168.100.1 || ( wg-quick down wg1 ; wg-quick up wg1 )
cronjob kills interface when it can’t ping
or
/usr/local/bin/wg-quick: line 32: resolvconf: command not found
Calling wg-quick via cron causes a resolvconf issue, even though it works at the command line. One solution is to remove any DNS config from your wg conf file so it doesn’t try to register the remote DNS server.
Nov 08 08:23:59 truenas wg-quick[2668]: Name or service not known: `some.server.org:port' Nov 08 08:23:59 truenas wg-quick[2668]: Configuration parsing error … Nov 08 08:23:59 truenas systemd[1]: Failed to start WireGuard via wg-quick(8) for wg1.
The DNS service isn’t available (yet), despite Requires=network-online.target nss-lookup.target already in the service unit file. One way to solve this is a pre-exec in the Service section of the unit file1. This is hacky, but none of the normal directives work.
The cron job above will bring the service up eventually, but it’s nice to have it at boot.
An upgrade comes with a new OS image and that replaces anything you’ve added, such as wireguard config and cronjobs. The only way to persist your Wireguard connection it to put a script on the pool and add a cronjob via the official interface2.
Add this script and change for your pool location. This is set to run every 5 min, as you probably don’t want to wait after an upgrade very long to see if it’s working. You can also use this to detect IP changes over the cron.hourly above.
# Create the location and prepare the files
mkdir /mnt/pool02/bin/
cp /etc/wireguard/wg1.conf /mnt/pool02/bin/
touch /mnt/pool02/bin/wg_test
chmod +x /mnt/pool02/bin/wg_test
# Edit the script
vi /mnt/pool02/bin/wg_test
#!/bin/sh
ping -c1 -W5 192.168.100.1 || ( cp /mnt/pool02/bin/wg1.conf /etc/wireguard/ ; wg-quick down wg1 ; wg-quick up wg1 )
# Invoke the TrueNAS CLI and add the job
cli
task cron_job create command="/mnt/pool02/bin/wg_test" enabled=true description="test" user=root schedule={"minute": "*/5", "hour": "*", "dom": "*", "month": "*", "dow": "*"}
https://www.truenas.com/docs/core/coretutorials/network/wireguard/ https://www.truenas.com/community/threads/no-internet-connection-with-wireguard-on-truenas-scale-21-06-beta-1.94843/#post-693601
Proxmox is frequently used in smaller environments for it’s ability to mix Linux Containers and Virtual Machines at very low cost. LCD - Linux Containers - are especially valuable as they give the benefits of virtualization with minimal overhead.
Using wireguard in a container simply requires adding the host’s kernel module interface.
On the pve host, for lxc id 101:
echo "lxc.mount.entry = /dev/net/tun /dev/net/tun none bind create=file" >> /etc/pve/lxc/101.conf
In the past you had to install the module, or use the DKMS method. That’s no longer needed as the Wireguard kernel module is now available on proxmox with the standard install. You don’t even need to install the wireguard tools. But if you run into trouble you can go through these steps
apt install wireguard
modprobe wireguard
# The module will load dynamically when a conainter starts, but you can also manually load it
echo "wireguard" >> /etc/modules-load.d/modules.conf
LibreELEC and CoreELEC are Linux-based open source software appliances for running the Kodi media player. These can be used as kiosk displays and you can remotely manage them with wireguard.
These systems have wireguard support, but use connman that lacks split-tunnel ability1. This forces all traffic through the VPN and so is unsuitable for remote management. To enable split-tunnel, create a wireguard service instead.
vi /storage/.config/system.d/wg0.service
[Unit]
Description=start wireguard interface
# The network-online service isn't guaranteed to work on *ELEC
#Requires=network-online.service
After=time-sync.target
Before=kodi.service
[Service]
Type=oneshot
RemainAfterExit=true
StandardOutput=journal
# Need to check DNS is responding before we proceed
ExecStartPre=/bin/bash -c 'until nslookup google.com; do sleep 1; done'
ExecStart=ip link add dev wg0 type wireguard
ExecStart=ip address add dev wg0 10.1.1.3/24
ExecStart=wg setconf wg0 /storage/.config/wireguard/wg0.conf
ExecStart=ip link set up dev wg0
# On the newest version, a manual route addition is needed too
ExecStart=ip route add 10.2.2.0/24 dev wg0 scope link src 10.1.1.3
# Deleting the device seems to remove the address and routes
ExecStop=ip link del dev wg0
[Install]
WantedBy=multi-user.target
Note: This isn’t exactly the same file wg-quick uses, just close enough to confuse.
vi /storage/.config/wireguard/wg0.conf
[Interface]
PrivateKey = XXXXXXXXXXXXXXX
[Peer]
PublicKey = XXXXXXXXXXXXXXX
AllowedIPs = 10.1.1.0/24
Endpoint = endpoint.hostname:31194
PersistentKeepalive = 25
systemctl enable --now wg0.service
ping 10.1.1.1
When using a DNS name for the endpoint you may become disconnected. To catch this, use a cron job
# Use the internal wireguard IP address of the peer you are connecting to. .1 in this case
crontab -e
*/5 * * * * ping -c1 -W5 10.1.1.1 || ( systemctl stop wg0; sleep 5; systemctl start wg0 )
The simplest way to deploy Wireguard is to use the built-in feature of your router. For OPNsense, it’s as simple as:
This is your server. Even though in wireguard all systems are considered peers, this is the system that is going to stay up all the time and accept connections, so it’s safe to think of it as ’the server'.
Navigate to:
VPN -> Wiregurad -> Instances
Click the + button on the right to add an instance. You can leave everything at the default except for:
Save, but don’t click ‘Enable’ on the main screen yet.
This is your phone or other enpoint that will be initiating the connection to the server. Navigate to:
VPN -> Wiregurad -> Peer Generator
It’s safe to leave everything at default except:
If this is your first client, you may need to configure an IP. It’s safe to start one up from your server’s internal tunnel address, but don’t click the button for Store and generate next yet.
Copy the config box to a text file and get it to your client, or use the QR if you have a phone handy. Once you’ve saved the info, then click
“Store and generate next”
The GUI has automatically added the client to instance you created earlier, so at the bottom you can:
(You can enable Wireguard at the bottom of any of these screens)
Firewall -> Rules -> WAN
Click ‘+’ to add a rule, and add
Wireguard works by creating a network interface and Opnsense helpfully adds that alongside the LAN and WAN interfaces. You’ll notice it actually creates a group and if you had other instances they will (probably) be included.
Simply click the ‘+’ button to add a rule and save without changing any of the defaults. This allows you to leave the tunnel and talk to things on the LAN.
At this point you can connect from the client. If you look in the details it should add a line about ‘Latest handshake’ after a few seconds. If not, you’ll have to troubleshoot as below.
Adding new clients is similar to the first one, just make sure to disable and enable the service or the new clients won’t get picked up.
https://docs.opnsense.org/manual/how-tos/wireguard-client.html#step-4-b-create-an-outbound-nat-rule
I used the official setup guide at https://docs.opnsense.org/manual/vpnet.html#wireguard and it has a few flaws.
Mostly, it describes a more complex setup than just a remote access. They note two steps:
The issue is that the second category isn’t visible right away. Once it is, you can use the group, not the IP address. It’s unclear why the docs point you away from that.
Then I had to reboot to get it to work, which is very odd.
This turns out to be a general issue when you add a client and the service is already active. You can’t restart the service, you must disable and re-enable the service from the wireguard sub page
PIA is a decent VPN service with reasonable community engagement1. They support wireguard (the best protocol) and have some serviceable command line tools, useful for headless servers.
The latest release (v2.0) is somewhat dated, so you may want to check before proceeding. There are some third-party tools, but they lack port-forwarding.
# Install without recommends if you're in a lxc container
sudo apt install --no-install-recommends ca-certificates curl iptables jq wget wireguard-tools
# Download, extract and move. The scripts expect a home of "/opt/piavpn-manual"
wget -O - https://github.com/pia-foss/manual-connections/archive/refs/tags/v2.0.0.tar.gz | tar xzf -
sudo mv manual-connections-2.0.0 /opt/piavpn-manual
The README details a lot of what you need to know. After reading it, you’ll gleen that you just need to run the run_setup.sh script.
# The scripts expect this working directory.
cd /opt/piavpn-manual
# Run a test with port-forwarding
sudo VPN_PROTOCOL=wireguard DISABLE_IPV6=yes AUTOCONNECT=true PIA_PF=true PIA_DNS=true PIA_USER=pXXXXXXX PIA_PASS=XXXXXXXXXX /opt/piavpn-manual/run_setup.sh
You should see messages like these in the text.
...
The WireGuard interface got created.
...
Trying to bind the port... OK!
...
If so, success!. Stop the process and take down the interface
# Hit Ctrl-C to stop the port forwarding script, then down the interface with:
sudo wg-quick down pia
If the test went well it’s time to create a service unit. This outputs any messages to the system journal and doesn’t request a port-forward.
sudo vi /etc/systemd/system/wg-pia.service
[Unit]
Description=PIA WireGuard Connection
After=network-online.target
Wants=network-online.target
[Service]
SyslogIdentifier=wg-pia
Type=oneshot
Restart=no
RemainAfterExit=true
WorkingDirectory=/opt/piavpn-manual
# Set up the ENVs. You could also use an EnvironmentFile if you prefer
Environment="VPN_PROTOCOL=wireguard"
Environment="DISABLE_IPV6=yes"
Environment="AUTOCONNECT=true"
Environment="PIA_PF=false"
Environment="PIA_DNS=true"
Environment="PIA_USER=XXXXXXX"
Environment="PIA_PASS=XXXXXXX"
ExecStart=/opt/piavpn-manual/run_setup.sh
ExecStopPost=/usr/bin/wg-quick down pia
# Need to check DNS is responding before we proceed
ExecStartPre=/bin/bash -c 'until ping -c 1 google.com &> /dev/null; do sleep 1;done'
ExecStart=/usr/bin/bash -c "VPN_PROTOCOL=wireguard DISABLE_IPV6=yes DIP_TOKEN=no AUTOCONNECT=true PIA_DNS=true PIA_PF=false PIA_USER=XXXXXXX PIA_PASS=XXXXXXX /opt/piavpn-manual/run_setup.sh"
ExecStop=/usr/bin/bash -c "wg-quick down pia"
User=root
[Install]
WantedBy=multi-user.target
If you want to forward a port just change these lines. The script will stay running to keep the forwarded port active.
Type=simple
Restart=always
Environment="PIA_PF=true"
Reload, enable and start, and then keep an eye on how it’s doing by following its journal entries.
sudo systemctl daemon-reload
sudo systemctl enable --now wg-pia.service
sudo journalctl -f -u wg-pia.service
If you forwarded a port you probably need to know what it is. There’s nothing built-in for that, but you can hack PIA’s port_forwarding.sh script to write it out somewhere, or you can look at the log. That’s a bit less hacky.
Tac this onto the bottom of the [Service] section above. It will continue running to watch the log and update /run/pia-port as needed.
ExecStartPre=/bin/sh -c 'journalctl -u pia-wg -f --no-pager \
| grep --line-buffered -oP "Forwarded port\s+\K[0-9]+" > /run/pia-port &'
ExecStopPost=rm /run/pia-port
PIA uses wg-quick in a way that assumes you have iptables installed. Many containers don’t and there’s no easy way to tell PIA that. Installing iptables it takes less than a Meg, so you can do that, or you can just create a dummy command.
echo -e '#!/bin/sh\nexit 0' | sudo tee /sbin/iptables-restore
sudo chmod +x /sbin/iptables-restore
The port detection process leaves grep running, but it’s debatable if you actually need that. PIA says the port request has a two month lifespan. And we are just assuming the pia port forwarding script gets a new one when it expires. Also, anything using that port will need to know it changed. It was actually less code to leave it running, so I left it as such.
Certbot is a useful utility for obtaining certificates from the Let’s Encrypt project. This is a free service and can be used both for web and other services like email and WiFi.
The utility establishes proof-of-ownership a couple different ways. The normal way is by answering a request on port 80. This proves you are at least in control of a server. But you can also use DNS when you don’t want to open a given server up to the internet.
You can also request wildcard certs via the DNS challenge. These are useful both when you’re hosting many sites, but also when you don’t want to advertise to the world the sites you are hosting.
If you use Cloudflare, there’s the Cloudflare DNS plugin from certbot.
Create a token first, as in https://roelofjanelsinga.com/articles/using-caddy-ssl-with-cloudflare/
# Install the module. It will pull in the parts of certbot that are needed
sudo apt install python3-certbot-dns-cloudflare
# Create a credential file. Certbot will save the path for use during renewals
sudo bash -c 'echo "dns_cloudflare_api_token = aLongStringOfChars" > /etc/letsencrypt/cloudflare.ini'
sudo chmod 600 /etc/letsencrypt/cloudflare.ini
DOMAIN=your.org
# We added a hook for mail, but substitute your own as desired
sudo certbot certonly \
--agree-tos \
--dns-cloudflare \
--dns-cloudflare-credentials /etc/letsencrypt/cloudflare.ini \
--domains mail.$DOMAIN \
--email postmaster@$DOMAIN \
--deploy-hook "service postfix reload; service dovecot reload"
You may see a warning about the plugin version not being pinned. That’s safe to ignore. You’re looking for the message that it successfully received a certificate.
This is similar to Cloudflare. This main difference is setting up the IAM user.
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"route53:ListHostedZones",
"route53:GetChange"
],
"Resource": "*"
},
{
"Effect": "Allow",
"Action": [
"route53:ChangeResourceRecordSets"
],
"Resource": "arn:aws:route53:::hostedzone/YOUR_HOSTED_ZONE_ID"
}
]
}
sudo apt install python3-certbot-dns-route53
sudo cat /etc/letsencrypt/route53.ini
[default]
aws_access_key_id=aLongString
aws_secret_access_key=anEvenLongerString
sudo AWS_SHARED_CREDENTIALS_FILE=/etc/letsencrypt/route53.ini certbot certonly --dns-route53 -d wifi.your.org
That should get a cert and create the job as shown in
cat /etc/letsencrypt/renewal/wifi.your.org.conf
Verify with
sudo openssl x509 -in /etc/letsencrypt/live/wifi.your.org/cert.pem -noout -text
GPG is an implementation of the OpenPGP standard (the term ‘PGP’ is trademarked by Symantec).
The best practice, that GPG implements by default, is to create a signing-only primary key with an encryption subkey1. These subkeys expire2 and must be extended or replaced from time to time.
The basics of gpg can be broken down into:
Let’s skip the details of asymmetric key encryption, public private keys, and just know that there are two keys; your private key, and your public key. You encrypt with the public key, and you decrypt with the private key.
The private key is the one that matters. That’s the one you use to decrypt things. Your public key you can recreate, should you lose it, as long as you have your private key.
The public key is the one you pass out to your friends and even put on your web site when you want someone to sen you something that only you can read. It sounds crazy, but through wonders of mathematics, it can only be used to encrypt a file, never to decrypt one. So it doesn’t matter who you give it to. They can encrypt something, send it to you, and you can decrypt it - all without anyone sending a password.
You can also sign things. This is when you want to send something that anyone can read, but just want to be sure it came from you. More on that later. Let’s focus on secrecy.
Note - In my opinion, we can probably skip all the old command line stuff, not that it’s not good to know, it’s just slower to use as a novice.
http://ubuntuforums.org/showthread.php?t=680292
To list keys
# If you don't use this list-option arguement, you won't see all the subkeys
gpg --list-options show-unusable-subkeys --list-keys
gpg --edit-key C621C2A8040C51F5C4AD9D2990A1676C9CB79C5D addkey
This will encrypt the file and apply the default option of appending .gpg on the end of the file
gpg -e -r '[email protected]' /path/to/some/file.txt
This will do the reverse - note you have to specify the output file or you will get to view the decrypted file via stdout, probably not what you wanted
gpg -o /path/to/some/file.txt -d /path/to/some/file.txt.gpg
Active Directory Federation Service (AD FS) is Microsoft’s take on SAML federated IdM. It’s not too bad but I suspect they are steering you to the Azure version. It’s better if you can go that way.
This is mostly from the SimpleSAMLphp notes, circa 2018
And then the Next Steps configuring the IdP
Specifically, one must:
sudo apt install nginx-light
sudo apt install php-cli
Install the required PHP libraries[^1] The dom library is now xml[^2]. Also install ldap and fpm and curl
# Check for any missing
php -m | grep -E 'date|dom|hash|libxml'
php -m | grep -E 'openssl|pcre|SPL|zlib|json|mbstring'
sudo apt install php-mbstring php-xml php-ldap php-fpm php-curl
Download the latest version and extract
# Check the link at https://simplesamlphp.org/download
wget *someLink*
# convention is to place in /var
sudo tar -xzf test.tgz -C /var
sudo mv /var/simplesamlphp* /var/simplesamlphp
Setup nginx
# Remove the default config files and content
sudo rm /etc/nginx/sites-available/*
sudo rm /etc/nginx/sites-enabled/*
sudo rm /var/www/html/*
# Add our own
sudo vi /etc/nginx/sites-available/default
server {
listen 80 default_server;
# Add index.php to the list if you are using PHP
index index.html index.htm index.nginx-debian.html index.php;
location ^~ /simplesaml {
alias /var/simplesamlphp/www;
location ~ \.php$ {
include snippets/fastcgi-php.conf;
fastcgi_pass unix:/var/run/php/php7.3-fpm.sock;
fastcgi_split_path_info ^((?U).+\.php)(/?.+)$;
fastcgi_param SCRIPT_FILENAME $request_filename;
fastcgi_param PATH_INFO $fastcgi_path_info if_not_empty;
}
}
}
Make minimal changes to the config.php to enable testing
sudo vi /var/simplesamlphp/config/config.php
'secretsalt' => 'alongstring',
'auth.adminpassword' => 'aPassword',
'enable.saml20-idp' => true,
'module.enable' => [ 'exampleauth' => true, ],
'technicalcontact_email' => '[email protected]',
Configure the example auth
...
/* De-commenting the below */
'example-userpass' => [
...
'student:XXX'
'employee:XXX'
Generate a cert
sudo openssl req -newkey rsa:3072 -new -x509 -days 3652 -nodes -out idp.your.org.crt -keyout idp.your.org.pem
Country Name (2 letter code) [AU]:US
State or Province Name (full name) [Some-State]:Ohio
Locality Name (eg, city) []:Athens
Organization Name (eg, company) [Internet Widgits Pty Ltd]:
Organizational Unit Name (eg, section) []:Information Technology
Common Name (e.g. server FQDN or YOUR name) []:idp.your.org
Email Address []:[email protected]
mv idm.* /var/simplesamlphp/cert
Edit the idp config
vi /var/simplesamlphp/metadata/saml20-idp-hosted.php
# Should already be uncommented
'auth' => 'example-userpass',
# Needed to be uncommented as per recommendation
/* Uncomment the following to use the uri NameFormat on attributes. */
/* Uncommented */
'attributes.NameFormat'
/* */
And access at http://idp.your.org/simplesaml/
Post Steps
Comment out Languages that are displayed at the top of the page[^5]
/*
'language.available' => [
...
...
*/
//'language.rtl'
1:https://simplesamlphp.org/docs/stable/simplesamlphp-install#section_1 2:https://laracasts.com/discuss/channels/servers/how-do-i-install-the-dom-extension-for-php7 3:https://github.com/simplesamlphp/simplesamlphp/issues/751 4:https://simplesamlphp.org/docs/stable/simplesamlphp-install#section_8 5:https://www.howtodojo.com/2012/12/how-to-remove-languages-on-simplesamlphp/
Back in the dawn of time, we called it ‘Central Logging’ and it looked kind of like this:
# The classical way you'd implement this is via a tiered system.
Log Shipper --\ /--> Log Parser --\
Log Shipper ---+--> Log Broker --+---> Log Parser ---+--> Log Storage --> Log Visualizer
Log Shipper --/ \--> Log Parser --/
# The modern way is more distributed. The clients are more powerful so you spread the load out and they can connect to distributed storage directly.
Log Parser Shipper --\ /-- Log Storage <-\
Log Parser Shipper ---+--- Log Storage <--+- Visualizer
Log Parser Shipper --/ \-- Log Storage <-/
# ELK (Elasticsearch Logstash and Kibana) is a good example.
Logstash --\ /-- Elasticsearch <-\
Logstash ---+--- Elasticsearch <--+--> Kibana
Logstash --/ \-- Elasticsearch <-/
More recently, there’s a move toward shippers like NXLog and Elasticsearch’s beats client. A native client saves you from deploying Java and is better suited for thin or micro instances.
# NXLog has an output module for Elasticsearch now. Beats is an Elasticsearch product.
nxlog --\
nxlog ---+--> Elasticsearch <-- Kibana
beats --/
Windows has it’s own log forwarding technology. You can put it to work without installing anything on the clients. This makes Windows admins a lot happier.
# It's built-in and fine for windows events - just doesn't do text files. Beats can read the events and push to elasticsearch.
Windows Event Forwarding --\
Windows Event Forwarding ---+--> Central Windows Event Manager -> Beats/Elasticsearch --> Kibana
Windows Event Forwarding --/
Unix has several ways to do it, but the most modern/least-overhead way is to use the native journald system.
# Built-in to systemd
journald send --> central journald receive --> Beats/Elasticsearch --> Kibana
The original answer used to be ‘reporting’. It was easier to get all the data together and do an analysis in one place.
Now the answer is ‘correlation’. If someone is probing your systems, they’ll do it very slowly and from multiple IPs to evade thresholds if they can, trying to break up patterns of attack. These patterns can become clear however, when you have a complete picture in one place.
This is also referred to ELK, and is an acronym that stands for Elasticsearch, Logstash and Kibana
This is a trio of tools that <www.elasticsearch.org> has packaged up into a simple and flexible way to handle, store and visualize data. Logstash collects the logs, parses them and stores them in Elasticsearch. Kibana is a web application that knows how to to talk to Elasticsearch and visualizes the data.
Quite simple and powerful
To make use of this tio, start by deploying in this order:
More recently, you can use the Elasticsearch Beats client in place of Logstash. These are natively compiled clients that have less capability, but are easier on the infrastructure than Logstash, a Java application.
This is circa 2014 - use with a grain of salt.
This is generally the first step, as you need a place to collect your logs. Elasticsearch itself is a NoSQL database and well suited for pure-web style integrations.
Java is required, and you may wish to deploy Oracle’s java per the Elasticsearch team’s recommendation. You may also want to dedicate a data partition. By default, data is stored in /var/lib/elasticsearch and that can fill up. We will also install the ‘kopf’ plugin that makes it easier to manage your data.
# (add a java repo)
sudo yum install java
# (add the elasticsearch repo)
sudo yum install elasticsearch
# Change the storage location
sudo mkdir /opt/elasticsearch
sudo chown elasticsearch:elasticsearch /opt/elasticsearch
sudo vim /etc/elasticsearch/elasticsearch.yml
...
path.data: /opt/elasticsearch/data
...
# Allow connections on ports 9200, 9300-9400 and set the cluster IP
# By design, Elasticsearch is open so control access with care
sudo iptables --insert INPUT --protocol tcp --source 10.18.0.0/16 --dport 9200 --jump ACCEPT
sudo iptables --insert INPUT --protocol tcp --source 10.18.0.0/16 --dport 9300:9300 --jump ACCEPT
sudo vim /etc/elasticsearch/elasticsearch.yml
...
# Failing to set the 'publish_host can result in the cluster auto-detecting an interface clients or other
# nodes can't reach. If you only have one interface you can leave commented out.
network.publish_host: 10.18.3.1
...
# Increase the heap size
sudo vim /etc/sysconfig/elasticsearch
# Heap size defaults to 256m min, 1g max
# Set ES_HEAP_SIZE to 50% of available RAM, but no more than 31g
ES_HEAP_SIZE=2g
# Install the kopf plugin and access it via your browser
sudo /usr/share/elasticsearch/bin/plugin -install lmenezes/elasticsearch-kopf
sudo service elasticsearch restart
In your browser, navigate to
http://10.18.3.1:9200/_plugin/kopf/
If everything is working correctly you should see a web page with KOPF at the top.
You may need to install on windows to ensure the ‘maximum amount of service ability with existing support staff’. I’ve used it on both Windows and Linux and it’s fine either way. Windows just requires a few more steps.
The current version of Elasticsearch at time of writing these notes is 7.6. It requires an OS and Java. The latest of those supported are:
The installation instructions are at https://www.elastic.co/guide/en/elastic-stack-get-started/current/get-started-elastic-stack.html
Note: Elasicsearch has both an zip and a MSI. The former comes with a java distro but the MSI includes a service installer.
The OpenJDK 13 GA Releases at https://jdk.java.net/13/ no longer include installers or the JRE. But you can install via a MSI from https://github.com/ojdkbuild/ojdkbuild
Download the latest java-13-openjdk-jre-13.X and execute. Use the advanced settings to include the configuration of the JAVA_HOME and other useful variables.
To test the install, open a command prompt and check the version
C:\Users\allen>java --version
openjdk 13.0.2 2020-01-14
OpenJDK Runtime Environment 19.9 (build 13.0.2+8)
OpenJDK 64-Bit Server VM 19.9 (build 13.0.2+8, mixed mode, sharing)
Download the MSI installer from https://www.elastic.co/downloads/elasticsearch. It may be tagged as beta, but it installs the GA product well. Importantly, it also installs a windows service for Elasticsearch.
Verify the installation by checking your services for ‘Elasticsearch’, which should be running.
By default, this is the case. You must edit the config file.
# In an elevated command prompt
notepad C:\ProgramDaata\Elastic\Elasticsearach\config\elasticsearch.yml
# add
discovery.type: single-node
network.host: 0.0.0.0
You can’t run newer versions of the filebeat with older versions of elasticsearch. Download the old deb and sudo apt install ./some.deb
https://discuss.elastic.co/t/filebeat-receives-http-405-from-elasticsearch-after-7-x-8-1-upgrade/303821 https://discuss.elastic.co/t/cant-start-filebeat/181050
This is circa 2014 - use with a grain of salt.
Configuration of elasticsearch itself is seldom needed. You will have to maintain the data in your indexes however. This is done by either using the kopf tool, or at the command line.
After you have some data in elasticsearch, you’ll see that your ‘documents’ are organized into ‘indexes’. This is a simply a container for your data that was specified when logstash originally sent it, and the naming is arbitrarily defined by the client.
The first thing you’re likely to need is to delete some badly-parsed data from your testing.
Delete all indexes with the name test*
curl -XDELETE http://localhost:9200/test*
Delete from all indexes documents of type ‘WindowsEvent’
curl -XDELETE http://localhost:9200/_all/WindowsEvent
Delete from all indexes documents have the attribute ‘path’ equal to ‘/var/log/httpd/ssl_request.log’
curl -XDELETE 'http://localhost:9200/_all/_query?q=path:/var/log/https/ssl_request.log'
Delete from the index ’logstash-2014.10.29’ documents of type ‘shib-access’
curl -XDELETE http://localhost:9200/logstash-2014.10.29/shib-access
http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/docs-delete-by-query.html
All the maintenance by hand has to stop at some point and Curator is a good tool to automate some of it. This is a script that will do some curls for you, so to speak.
Install
wget https://bootstrap.pypa.io/get-pip.py
sudo python get-pip.py
sudo pip install elasticsearch-curator
sudo pip install argparse
Use
curator --help
curator delete --help
And in your crontab
# Note: you must escape % characters with a \ in crontabs
20 0 * * * curator delete indices --time-unit days --older-than 14 --timestring '\%Y.\%m.\%d' --regex '^logstash-bb-.*'
20 0 * * * curator delete indices --time-unit days --older-than 14 --timestring '\%Y.\%m.\%d' --regex '^logstash-adfsv2-.*'
20 0 * * * curator delete indices --time-unit days --older-than 14 --timestring '\%Y.\%m.\%d' --regex '^logstash-20.*'
Sometimes you’ll need to do an inverse match.
0 20 * * * curator delete indices --regex '^((?!logstash).)*$'
A good way to test your regex is by using the show indices method
curator show indices --regex '^((?!logstash).)*$'
Here’s some OLD posts and links, but be aware the syntax had changed and it’s been several versions since these
http://www.ragingcomputer.com/2014/02/removing-old-records-for-logstash-elasticsearch-kibana http://www.elasticsearch.org/blog/curator-tending-your-time-series-indices/ http://stackoverflow.com/questions/406230/regular-expression-to-match-line-that-doesnt-contain-a-word
By default, elasticsearch assumes you want to have two nodes and replicate your data and the default for new indexes is to have 1 replica. You may not want to do that to start with however, so you change the default and change the replica settings on your existing data in-bulk with:
Set all existing replica requirements to just one copy
curl -XPUT 'localhost:9200/_settings' -d '
{
"index" : { "number_of_replicas" : 0 }
}'
Change the default settings for new indexes to have just one copy
curl -XPUT 'localhost:9200/_template/logstash_template' -d '
{
"template" : "*",
"settings" : {"number_of_replicas" : 0 }
} '
You will occasionally have a hiccup where you run out of disk space or something similar and be left with indexes that have no data in them or have shards unassigned. Generally, you will have to delete them but you can also manually reassign them.
http://stackoverflow.com/questions/19967472/elasticsearch-unassigned-shards-how-to-fix
Listing Index Info
You can get a decent human readable list of your indexes using the cat api
curl localhost:9200/_cat/indices
If you wanted to list by size, they use the example
curl localhost:9200/_cat/indices?bytes=b | sort -rnk8
Kibana is a Node.js app using the Express Web framework - meaning to us it looks like a web server running on port 5601. If you’re running elasticsearch on the same box, it will connect with the defaults.
https://www.elastic.co/guide/en/kibana/current/windows.html
No MSI or installer is available for windows so you must download the .zip from https://www.elastic.co/downloads/kibana. Uncompress (this will take a while), rename it to ‘Kibana’ and move it to Program Files.
So that you may access it later, edit the config file at {location}/config/kibana.yaml with wordpad and set the server.host entry to:
server.host: "0.0.0.0"
Download the service manager NSSM from https://nssm.cc/download and extract. Start an admin powershell, navigate to the extracted location and run the installation command like so:
C:\Users\alleng\Downloads\nssm-2.24\nssm-2.24\win64> .\nssm.exe install Kibana
In the Pop-Up, set the application path to the below. The start up path will auto populate.
C:\Program Files\Kibana\kibana-7.6.2-windows-x86_64\bin\kibana.bat
Click ‘Install service’ and it should indicate success. Go to the service manager to find and start it. After a minute (Check process manager for the CPU to drop) You should be able to access it at:
Kibana rounds to 16 significant digits
Turns out, if you have a value of type integer, that’s just the limit. While elasticsearch shows you this:
curl http://localhost:9200/logstash-db-2016/isim-process/8163783564660983218?pretty
{
"_index" : "logstash-db-2016",
"_type" : "isim-process",
"_id" : "8163783564660983218",
"_version" : 1,
"found" : true,
"_source":{"requester_name":"8163783564660983218","request_num":8163783618037078861,"started":"2016-04-07 15:16:16:139 GMT","completed":"2016-04-07 15:16:16:282 GMT","subject_service":"Service","request_type":"EP","result_summary":"AA","requestee_name":"Mr. Requester","subject":"mrRequest","@version":"1","@timestamp":"2016-04-07T15:16:16.282Z"}
}
Kibana shows you this
View: Table / JSON / Raw
Field Action Value
request_num 8163783618037079000
Looking at the JSON will give you the clue - it’s being treated as an integer and not a string.
"_source": {
"requester_name": "8163783564660983218",
"request_num": 8163783618037079000,
"started": "2016-04-07 15:16:16:139 GMT",
"completed": "2016-04-07 15:16:16:282 GMT",
Mutate it to string in logstash to get your precision back.
Logstash is a parser and shipper. It reads from (usually) a file, parses the data into JSON, then connects to something else and send the data. That something else can be Elasticsearch, a systlog server, and others.
But for most things these days, Beats is a better choice. Give that a look fist.
Note: Before you install logstash, take a look at Elasticsearch’s Beats. It’s lighter-weight for most tasks.
This is a summary of the current install page. Visit and adjust versions as needed.
# Install java
apt install default-jre-headless
apt-get install apt-transport-https
apt install gnupg2
wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | apt-key add -
# Check for the current version - 7 is no longer the current version by now
echo "deb https://artifacts.elastic.co/packages/7.x/apt stable main" | tee -a /etc/apt/sources.list.d/elastic-7.x.list
apt update
apt-get install logstash
Logstash has a NetFlow module, but it has been deprecated2. One should instead use the Filebeat Netflow Module.3
The rest of this page is circa 2014 - use with a grain of salt.
If you don’t already have it, install it. You’ll need at least 1.7 and Oracle is recommended. However, with older systems do yourself a favor and use the OpenJDK as older versions of Sun and IBM do things with cryptography leading to strange bugs in recent releases of logstash.
# On RedHat flavors, install the OpenJDK and select it for use (in case there are others) with the system alternatives utility
sudo yum install java-1.7.0-openjdk
sudo /usr/sbin/alternatives --config java
This is essentially:
( Look at https://www.elastic.co/downloads/logstash to get the lastest version or add the repo)
wget (some link from the above page)
sudo yum --nogpgcheck localinstall logstash*
# You may want to grab a plugin, like the syslog output, though elasticsearch installs by default
cd /opt/logstash/
sudo bin/plugin install logstash-output-syslog
# If you're ready to configure the service
sudo vim /etc/logstash/conf.d/logstash.conf
sudo service logstash start
https://www.elastic.co/guide/en/logstash/current/index.html
The most common use of logstash is to tail and parse log files. You do this by specifying a file and filter like so
[gattis@someHost ~]$ vim /etc/logstash/conf.d/logstash.conf
input {
file {
path => "/var/log/httpd/request.log"
}
}
filter {
grok {
match => [ "message", "%{COMBINEDAPACHELOG}"]
}
}
output {
stdout {
codec => rubydebug
}
}
There are many different types of filters, but the main one you’ll be using is grok. It’s all about parsing the message into fields. Without this, you just have a bunch of un-indexed text in your database. It ships with some handy macros such as %{COMBINEDAPACHELOG} that takes this:
10.138.120.138 - schmoej [01/Apr/2016:09:39:04 -0400] "GET /some/url.do?action=start HTTP/1.1" 200 10680 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.87 Safari/537.36"
And turns it into
agent "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.87 Safari/537.36"
auth schmoej
bytes 10680
clientip 10.138.120.138
httpversion 1.1
path /var/pdweb/www-default/log/request.log
referrer "-"
request /some/url.do?action=start
response 200
timestamp 01/Apr/2016:09:39:04 -0400
verb GET
See the grok’ing for more details
We’re outputting to the console so we can see what’s going on with our config. If you get some output, but it’s not parsed fully because of an error in the parsing, you’ll see something like the below with a “_grokparsefailure” tag. That means you have to dig into a custom pattern as in described in grok’ing.
Note: by default, logstash is ’tailing’ your logs, so you’ll only see new entries. If you’ve got no traffic you’ll have to generate some
{
"message" => "test message",
"@version" => "1",
"@timestamp" => "2014-10-31T17:39:28.925Z",
"host" => "some.app.private",
"tags" => [
[0] "_grokparsefailure"
]
}
If it looks good, you’ll want to send it on to your database. Change your output to look like so which will put your data in a default index that kibana (the visualizer) can show by default.
output {
elasticsearch {
hosts => ["10.17.153.1:9200"]
}
}
If you don’t get any output at all, check that the logstash user can actually read the file in question. Check your log files and try running logstash as yourself with the output going to the console.
cat /var/log/logstash/*
/opt/logstash/bin/logstash -f /etc/logstash/conf.d/logstash.conf
Generally, you create a config with 3 sections;
This example uses the grok filter to parse the message.
sudo vi /etc/logstash/conf.d/logstash.conf
input {
file {
path => "/var/pdweb/www-default/log/request.log"
}
}
filter {
grok {
match => [ "message", "%{COMBINEDAPACHELOG}"]
}
}
output {
stdout { }
}
Then you test it at the command line
# Test the config file itself
/opt/logstash/bin/logstash -f /etc/logstash/conf.d/logstash.conf --configtest
# Test the parsing of data
/opt/logstash/bin/logstash -e -f /etc/logstash/conf.d/logstash.conf
You should get some nicely parsed lines. If that’s the case, you can edit your config to add a sincedb and an actual destination.
input {
file {
path => "/var/pdweb/www-default/log/request.log"
sincedb_path => "/opt/logstash/sincedb"
}
}
filter {
grok {
match => [ "message", "%{COMBINEDAPACHELOG}"]
}
}
output {
elasticsearch {
host => "some.server.private"
protocol => "http"
}
}
If instead you see output with a _grokparsefailure like below, you need to change the filter. Take a look at the common gotchas, then the parse failure section below it.
{
"message" => "test message",
"@version" => "1",
"@timestamp" => "2014-10-31T17:39:28.925Z",
"host" => "some.app.private",
"tags" => [
[0] "_grokparsefailure"
]
}
Logstash reads new lines by default. If you don’t have anyone actually hitting your webserver, but you do have some log entries in the file itself, you can tell logstash to process the exiting entries and not save it’s place in the file.
file {
path => "/var/log/httpd/request.log"
start_position => "beginning"
sincedb_path => "/dev/null"
}
Logstash uses all the files in the conf.d directory - even if they don’t end in .conf. Make sure to remove any you don’t want as they can conflict.
Logstash creates Elasticsearch indexes that look like:
logstash-%{+YYYY.MM.dd}
The logstash folks have some great material on how to get started. Really top notch.
http://logstash.net/docs/1.4.2/configuration#fieldreferences
The best way to start is to change your match to a simple pattern and work out from there. Try the ‘GREEDYDATA’ pattern and assign it to a field named ‘Test’. This takes the form of:
%{GREEDYDATA:Test}
And it looks like:
filter {
grok {
match => [ "message" => "%{GREEDYDATA:Test}" ]
}
}
"message" => "test message",
"@version" => "1",
"@timestamp" => "2014-10-31T17:39:28.925Z",
"host" => "some.app.private",
"Test" => "The rest of your message
That should give you some output. You can then start cutting it up with the patterns (also called macros) found here;
You can also use the online grok debugger and the list of default patterns.
There may not be a standard pattern for what you want, but it’s easy to pull together several existing ones. Here’s an example that pulls in a custom timestamp.
Example:
Sun Oct 26 22:20:55 2014 File does not exist: /var/www/html/favicon.ico
Pattern:
match => { "message" => "(?<timestamp>%{DAY} %{MONTH} %{MONTHDAY} %{TIME} %{YEAR})"}
Notice the ‘?’ at the beginning of the parenthetical enclosure. That tells the pattern matching engine not to bother capturing that for later use. Like opting out of a ( ) and \1 in sed.
Some log formats simply skip columns when they don’t have data. This will cause your parse to fail unless you make some fields optional with a ‘?’, like this:
match => [ "message", "%{HOSTNAME:VHost}? %{COMBINEDAPACHELOG} %{IP:XForwardedFor}?"]
http://joda-time.sourceforge.net/apidocs/org/joda/time/format/DateTimeFormat.html
Oftentimes, you’ll have messages that you don’t care about and you’ll want to drop those. Best practice is to do coarse actions first, so you’ll want to compare and drop with a general conditional like:
filter {
if [message] =~ /File does not exist/ {
drop { }
}
grok {
...
...
You can also directly reference fields once you have grok’d the message
filter {
grok {
match => { "message" => "%{HOSTNAME:VHost}? %{COMBINEDAPACHELOG} %{IP:XForwardedFor}?"}
}
if [request] == "/status" {
drop { }
}
}
http://logstash.net/docs/1.4.2/configuration#conditionals
By default, logstash date stamps the message when it sees them. However, there can be a delay between when an action happens and when it gets logged to a file. To remedy this - and allow you to suck in old files without the date on every event being the same - you add a date filter.
Note - you actually have to grok out the date into it’s own variable, you can’t just attempt to match on the whole message. The combined apache macro below does this for us.
filter { grok { match => { “message” => “%{HOSTNAME:VHost}? %{COMBINEDAPACHELOG} %{IP:XForwardedFor}?”} } date { match => [ “timestamp” , “dd/MMM/yyyy:HH:mm:ss Z” ] } }
In the above case, ’timestamp’ is a parsed field and you’re using the date language to tell it what the component parts are
http://joda-time.sourceforge.net/apidocs/org/joda/time/format/DateTimeFormat.html
In addition to an elasticsearch server, you may want to send it to a syslog server at the same time.
input {
file {
path => "/var/pdweb/www-default/log/request.log"
sincedb_path => "/opt/logstash/sincedb"
}
}
filter {
grok {
match => [ "message", "%{HOSTNAME:VHost}? %{COMBINEDAPACHELOG} %{IP:XForwardedFor}?"]
}
date {
match => [ "timestamp" , "dd/MMM/yyyy:HH:mm:ss Z" ]
}
}
output {
elasticsearch {
host => "some.server.private"
protocol => "http"
}
syslog {
host => "some.syslog.server"
port => "514"
severity => "notice"
facility => "daemon"
}
}
Sometimes you’ll accidentally send a bunch of event to the server and need to delete and resend corrected versions.
http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/indices-delete-mapping.html
curl -XDELETE <http://localhost:9200/_all/SOMEINDEX>
curl -XDELETE <http://localhost:9200/_all/SOMEINDEX?q=path:"/var/log/httpd/ssl_request_log>"
When using logstash as a broker, you will want to route events to different indexes according to their type. You have two basic ways to do this;
The latter is significantly better for performance. The less you touch the event, the better it seems. When testing these two different configs in the lab, the multiple output method was about 40% faster when under CPU constraint. (i.e. you can always add more CPU if you want to mutate the events.)
input {
...
...
}
filter {
...
...
}
output {
if [type] == "RADIUS" {
elasticsearch {
hosts => ["localhost:9200"]
index => "logstash-radius-%{+YYYY.MM.dd}"
}
}
else if [type] == "RADIUSAccounting" {
elasticsearch {
hosts => ["localhost:9200"]
index => "logstash-radius-accounting-%{+YYYY.MM.dd}"
}
}
else {
elasticsearch {
hosts => ["localhost:9200"]
index => "logstash-test-%{+YYYY.MM.dd}"
}
}
}
If your source system includes a field to tell you want index to place it in, you might be able to skip mutating altogether, but often you must look at the contents to make that determination. Doing so does reduce performance.
input {
...
...
}
filter {
...
...
# Add a metadata field with the destination index based on the type of event this was
if [type] == "RADIUS" {
mutate { add_field => { "[@metadata][index-name]" => "logstash-radius" } }
}
else if [type] == "RADIUSAccounting" {
mutate { add_field => { "[@metadata][index-name]" => "logstash-radius-accounting" } }
}
else {
mutate { add_field => { "[@metadata][index-name]" => "logstash-test" } }
}
}
output {
elasticsearch {
hosts => ["localhost:9200"]
index => "%{[@metadata][index-name]}-%{+YYYY.MM.dd}"
}
}
https://www.elastic.co/guide/en/logstash/current/event-dependent-configuration.html#metadata
You can connect Logstash to a database to poll events almost as easily as tailing a log file.
The JDBC plug-in ships with logstash so no installation of that is needed. However, you do need the JDBC driver for the DB in question.
Here’s an example for DB2, for which you can get the jar from either the server itself or the DB2 fix-pack associated with the DB Version you’re running. The elasticsearch docs say to just put it in your path. I’ve put it in the logstash folder (based on some old examples) and we’ll see if it survives upgrades.
sudo mkdir /opt/logstash/vendor/jars
sudo cp /home/gattis/db2jcc4.jar /opt/logstash/vendor/jars
sudo chown -R logstash:logstash /opt/logstash/vendor/jars
Configuring the input
Edit the config file like so
sudo vim /etc/logstash/conf.d/logstash.conf
input {
jdbc {
jdbc_driver_library => "/opt/logstash/vendor/jars/db2jcc4.jar"
jdbc_driver_class => "com.ibm.db2.jcc.DB2Driver"
jdbc_connection_string => "jdbc:db2://db1.tim.private:50000/itimdb"
jdbc_user => "itimuser"
jdbc_password => "somePassword"
statement => "select * from someTable"
}
}
You don’t need to do any pattern matching, as the input emits the event pre-parsed based on the DB columns. You may however, want to match a timestamp in the database.
# A sample value in the 'completed' column is 2016-04-07 00:41:03:291 GMT
filter {
date {
match => [ "completed" , "yyyy-MM-dd HH:mm:ss:SSS zzz" ]
}
}
One recommended trick is to link the primary keys between the database and kibana. That way, if you run the query again you update the existing elasticsearch records rather than create duplicates ones. Simply tell the output plugin to use the existing primary key from the database for the document_id when it sends it to elasticsearch.
# Database key is the column 'id'
output {
elasticsearch {
hosts => ["10.17.153.1:9200"]
index => "logstash-db-%{+YYYY}"
document_id => "${id}"
type => "isim-process"
}
}
If any of your columns are non-string type, logstash and elasticsearch will happily store them as such. But be warned that kibana will round them to 16 digits due to a limitation of javascript.
https://github.com/elastic/kibana/issues/4356
https://www.elastic.co/blog/logstash-jdbc-input-plugin https://www.elastic.co/guide/en/logstash/current/plugins-inputs-jdbc.html https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html
Here’s an example that uses the multiline codec (preferred over the multiline filter, as it’s more appropriate when you might have more than one input)
input {
file {
path => "/opt/IBM/tivoli/common/CTGIM/logs/access.log"
type => "itim-access"
start_position => "beginning"
sincedb_path => "/dev/null"
codec => multiline {
pattern => "^<Message Id"
negate => true
what => previous
}
}
}
Getting a match can be difficult, as grok by default does not match against multiple lines. You can mutate to remove all the new lines, or use a seemingly secret preface, the ‘(?m)’ directive as shown below
filter {
grok {
match => { "message" => "(?m)(?<timestamp>%{YEAR}.%{MONTHNUM}.%{MONTHDAY} %{HOUR}:%{MINUTE}:%{SECOND}%{ISO8601_TIMEZONE})%{DATA}com.ibm.itim.security.%{WORD:catagory}%{DATA}CDATA\[%{DATA:auth}\]%{DATA}CDATA\[%{DATA:clientip}\]"}
}
Beats are a family of lightweight shippers that you should consider as a first-solution for sending data to Elasticsearch. The two most common ones to use are:
Filebeat is used both for files, and for other general types, like syslog and NetFlow data.
Winlogbeat is used to load Windows events into Elasticsearch and works well with Windows Event Forwarding.
A summary from the general docs. View and adjust versions as needed.
If you haven’t already added the repo:
apt-get install apt-transport-https
apt install gnupg2
wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | apt-key add -
echo "deb https://artifacts.elastic.co/packages/7.x/apt stable main" | tee -a /etc/apt/sources.list.d/elastic-7.x.list
apt update
apt install filebeat
systemctl enable filebeat
Filebeat uses a default config file at /etc/filebeat/filebeat.yml. If you don’t want to edit that, you can use the ‘modules’ to configure it for you. That command will also load dashboard elements into Kibana, so you must have that already installed Kibana to make use of it.
Here’s a simple test
mv /etc/filebeat/filebeat.yml /etc/filebeat/filebeat.yml.orig
vi /etc/filebeat/filebeat.yml
filebeat.inputs:
- type: log
enabled: true
paths:
- /var/log/*.log
output.file:
path: "/tmp/filebeat"
filename: filebeat
#rotate_every_kb: 10000
#number_of_files: 7
#permissions: 0600
Download the .zip version (the msi doesn’t include the server install script) from the URL below. Extract, rename to Filebeat and move it the to the c:\Program Files directory.
https://www.elastic.co/downloads/beats/filebeat
Start an admin powershell, change to that directory and run the service install command. (Keep the shell up for later when done)
PowerShell.exe -ExecutionPolicy UnRestricted -File .\install-service-filebeat.ps1
Edit the filebeat config file.
write.exe filebeat.yml
You need to configure the input and output sections. The output is already set to elasticsearch localhost so you only have to change the input from the unix to the windows style.
paths:
#- /var/log/*.log
- c:\programdata\elasticsearch\logs\*
Test as per normal
./filebeat test config -e
Filebeat specific dashboards must be added to Kibana. Do that with the setup argument:
.\filebeat.exe setup --dashboards
To start Filebeat in the forrgound (to see any interesting messages)
.\filebeat.exe -e
If you’re happy with the results, you can stop the application then start the service
Ctrl-C
Start-Service filebeat
Adapted from the guide at
https://www.elastic.co/guide/en/beats/filebeat/7.6/filebeat-getting-started.html
The NetFlow protocol is now implemented in Filebeat1. Assuming you’ve installed Filebeat and configured Elasticsearch and Kibana, you can use this input module to auto configure the inputs, indexes and dashboards.
./filebeat modules enable netflow
filebeat setup -e
If you are just testing and don’t want to add the full stack, you can set up the netflow input2 which the module is a wrapper for.
filebeat.inputs:
- type: netflow
max_message_size: 10KiB
host: "0.0.0.0:2055"
protocols: [ v5, v9, ipfix ]
expiration_timeout: 30m
queue_size: 8192
output.file:
path: "/tmp/filebeat"
filename: filebeat
filebeat test config -e
Consider dropping all the fields you don’t care about as there are a lot of them. Use the include_fields processor to limit what you take in
- include_fields:
fields: ["destination.port", "destination.ip", "source.port", "source.mac", "source.ip"]
We will use the Beat’s Filebeat syslog module. There is a Palo Alo module, but it includes a lot more data than we need and is best reserved for a SIEM project. We did however, steal the a few field mapping details from it for this config.
# This filebeat config accepts TRAFFIC and SYSTEM syslog messages from a Palo Alto,
# tags and parses them
filebeat.inputs:
- type: syslog
protocol.udp:
host: ':9000' # This is an arbitrary port. The normal port for syslog is UDP 512
processors:
# The message field will have "TRAFFIC" for netflow logs and we can
# extract the details with a CSV decoder and array extractor
- if:
contains:
message: ',TRAFFIC,'
then:
- add_tags:
tags: netflow
- decode_csv_fields:
fields:
message: csv
- extract_array:
field: csv
overwrite_keys: true
omit_empty: true
fail_on_error: false
mappings:
source.ip: 7
destination.ip: 8
source.nat.ip: 9
network.application: 14
source.port: 24
destination.port: 25
source.nat.port: 26
# Drop the original fields now that we've parsed them out
- drop_fields:
fields:
- csv
- message
# The message field will have "SYSTEM,dhcp" for dhcp logs and we can
# do a similar process to above
else:
- if:
contains:
message: ',SYSTEM,dhcp'
then:
- add_tags:
tags: dhcp
- decode_csv_fields:
fields:
message: csv
- extract_array:
field: csv
overwrite_keys: true
omit_empty: true
fail_on_error: false
mappings:
message: 14
# The DHCP info can be further pulled apart using space as a delimiter
- decode_csv_fields:
fields:
message: csv2
separator: ' '
- extract_array:
field: csv2
overwrite_keys: true
omit_empty: true
fail_on_error: false
mappings:
source.ip: 4
source.mac: 7
hostname: 10
# After parsing we can drop the original fields
- drop_fields:
fields:
- csv
- csv2
- drop_fields:
fields:
- agent.ephemeral_id
- agent.hostname
- agent.id
- agent.type
- agent.version
- ecs.version
- host.name
- event.severity
- input.type
- hostname
- log.source.address
- syslog.facility
- syslog.facility_label
- syslog.priority
- syslog.priority_label
- syslog.severity_label
ignore_missing: true
filebeat.config.modules:
path: '${path.config}/modules.d/*.yml'
reload.enabled: false
setup.template.settings:
index.number_of_shards: 1
output.elasticsearch:
hosts:
- 'localhost:9200'
Note: This is cleaned up with a YAML pretty-fier. It’s possible that it’s done something that needs reverted in practice
We will also use Beat’s Filebeat Palo Alto module. They have already done the hard work of figuring out how to parse the data.
On the Beats server, start an admin powershell session, change to the Filebeat directory, list the available modules, and enable PAN and the dashboard.
cd "C:\Program Files\Filebeat"
.\filebeat.exe modules list
.\filebeat.exe modules enable panw
.\filebeat.exe setup -e
# Or is it .\filebeat.exe setup --dashboards -e
Edit the module’s top-level config so that it listens on all addresses (note: we are using write.exe as the files come with unix-style newlines)
write.exe .\modules.d\panw.yml
panos:
enabled: tru
var.syslog_host: 0.0.0.0
Here’s an example of sending FreeRADIUS logs to Elasticsearch.
cat /etc/filebeat/filebeat.yml
filebeat.inputs:
- type: log
paths:
- /var/log/freeradius/radius.log
include_lines: ['\) Login OK','incorrect']
tags: ["radius"]
processors:
- drop_event:
when:
contains:
message: "previously"
- if:
contains:
message: "Login OK"
then:
- dissect:
tokenizer: "%{key1} [%{source.user.id}/%{key3}cli %{source.mac})"
target_prefix: ""
- drop_fields:
fields: ["key1","key3"]
- script:
lang: javascript
source: >
function process(event) {
var mac = event.Get("source.mac");
if(mac != null) {
mac = mac.toLowerCase();
mac = mac.replace(/-/g,":");
event.Put("source.mac", mac);
}
}
else:
- dissect:
tokenizer: "%{key1} [%{source.user.id}/<via %{key3}"
target_prefix: ""
- drop_fields:
fields: ["key1","key3"]
output.elasticsearch:
hosts: ["http://logcollector.yourorg.local:9200"]
allow_older_versions: true
setup.ilm.enabled: false
You may have an older system or appliance that can transmit syslog data. You can use filebeat to accept that data and store it in Elasticsearch.
Install filebeat and test the reception the /tmp.
vi /etc/filebeat/filebeat.yml
filebeat.inputs:
- type: syslog
protocol.udp:
host: ":9000"
output.file:
path: "/tmp"
filename: filebeat
sudo systemctl filebeat restart
The instructions are NetGate’s remote logging example.
Status -> System Logs -> Settings
Enable and configure. Internet rumor has it that it’s UDP only so the config above reflects that. Interpreting the output requires parsing the message section detailed in the filter log format docs.
'5,,,1000000103,bge1.1099,match,block,in,4,0x0,,64,0,0,DF,17,udp,338,10.99.147.15,255.255.255.255,2048,30003,318'
'5,,,1000000103,bge2,match,block,in,4,0x0,,84,1,0,DF,17,udp,77,157.240.18.15,205.133.125.165,443,61343,57'
'222,,,1000029965,bge2,match,pass,out,4,0x0,,128,27169,0,DF,6,tcp,52,205.133.125.142,205.133.125.106,5225,445,0,S,1248570004,,8192,,mss;nop;wscale;nop;nop;sackOK'
'222,,,1000029965,bge2,match,pass,out,4,0x0,,128,11613,0,DF,6,tcp,52,205.133.125.142,211.24.111.75,15305,445,0,S,2205942835,,8192,,mss;nop;wscale;nop;nop;sackOK'
Loki is a system for handling logs (unstructured data) but is lighter-weight than Elasticsearch. It also has fewer add-ons. But if you’re already using Prometheus and Grafana and you want to do it yourself, it can be a better solution.
Install Loki and Promtail together. These are available in the debian stable repos at current version. No need to go to backports or testing
sudo apt install loki promtail
curl localhost:3100/metrics
Default config files are create in /etc/loki and /etc/promtail. Promtail is tailing /var/log/*log file, pushing them to localhost loki on the default port (3100) and loki is saving data in the /tmp directory. This is fine for testing.
Promtail runs as the promtail user (not root) and can’t read anything useful, so add them to the adm group.
sudo usermod -a -G adm promtail
sudo systemctl restart promtail
In grafana, add a datasource.
Configuration –> Add new data source –> Loki
Set the URL to http://localhost:3100
Then view the logs
Explore –> Select label (filename) –> Select value (daemon)
error notifying frontend about finished query
Edit the timeout setting in your loki datasource. The default may be too short so set it to 30s or some such
Failed to load log volume for this query
If you added a logfmt parser like the gui suggested, you may find not all your entries can be parsed, leading to this error.:w
Recoding traffic on the network is critical for troubleshooting and compliance. For the latter, the most common strategy is to record the “flows”. These are the connections each host makes or accepts, and how much data is involved.
You can collect this information at the LAN on individual switches, but the WAN (at the router) is usually more important. And if the router is performing NAT, it’s the only place to record the mappings of internal to external IPs and ports.
A network flow log system usually has three main parts.
Exporter --> Collector --> Analyzer
The Exporter, which records the data, the Collector, which is where the data is stored, and the Analyzer which makes the data more human-readable.
We’ll use a Palo Alto NG Firewall as our exporter, and an Elasticsearch back-end. The data we are collecting is essentially log data, and Elasticsearch is probably the best at handling unstructured information.
At small scale, you can combine all of the the collection and analysis parts on a single system. We’ll use windows servers in our example as well.
graph LR A(Palo) B(Beats) C(ElasticSearch) D(Kibana) subgraph Exporter A end subgraph Collector and Analyzer B --> C --> D end A --> B
Start with Elasticsearch and Kibana, then install Beats.
Beats and Palo have a couple of protocols in common. NetFlow is the traditional protocol, but when you’re using NAT the best choice is the syslog protocol as the Palo will directly tell you the NAT info all in one record and you don’t have to correlate multiple interface flows to see who did what.
On the Beats server, start an admin powershell session, change to the Filebeat directory, edit the config file and restart the server.
There is a bunch of example text in the config so tread carefully and keep in mind that indentation matters. Stick this block right under the filebeat.inputs: line and you should be OK.
This config stanza has a processor block that decodes the CVS content sent over in the message field, extracts a few select fields, then discards the rest. There’s quite a bit left over though, so see tuning below if you’d like to reduce the data load even more.
cd "C:\Program Files\Filebeat"
write.exe filebeat.yml
filebeat.inputs:
- type: syslog
protocol.udp:
host: ":9000"
processors:
- decode_csv_fields:
fields:
message: csv
- extract_array:
field: csv
overwrite_keys: true
omit_empty: true
fail_on_error: false
mappings:
source.ip: 7
destination.ip: 8
source.nat.ip: 9
network.application: 14
source.port: 24
destination.port: 25
source.nat.port: 26
- drop_fields:
fields: ["csv", "message"]
A larger is example is under the beats documentation.
Perform steps 1 and 2 of the Palo setup guide with the notes below.
You can reduce the amount of data even more by adding a few more Beats directives.
# At the very top level of the file, you can add this processor to affect global fields
processors:
- drop_fields:
fields: ["agent.ephemeral_id","agent.id","agent.hostname","agent.type","agent.version","ecs.version","host.name"]
# You can also drop syslog fields that aren't that useful (you may need to put this under the syslog input)
- drop_fields:
fields: ["event.severity","input.type","hostname","syslog.facility", "syslog.facility_label", "syslog.priority", "syslog.priority_label","syslog.severity_label"]
You may want even more data. See the Full Palo Syslog data to see what’s available. An example
At this point you can navigate to the Kibana web console and explore the logs. There is no dashboard as this is just for log retention and covers the minimum required. If you’re interested in more, check out the SIEM and Netflow dashboards Elasticsearch offers.
Palo Shipping
https://docs.logz.io/shipping/security-sources/palo-alto-networks.html
This info on NXLog is circa 2014 - use with caution.
NXLog is best used when Windows Event Forwarding can’t be and filebeats isn’t sufficient.
There are several solutions for capturing logs in Windows, but NXLog has some advantages;
You could just run logstash everywhere. But in practice, Logstash’s memory requirements are several times NXLog and not everyone likes to install java everywhere.
Download from http://nxlog.org/download. This will take you to the sourceforge site and the MSI you can install from. This installation is clean and the service installs automatically.
NXLog uses a config file with blocks in the basic pattern of:
The latter being what ties together your inputs and outputs. You start out with one variable, called the $raw_event with everything in it. As you call modules, that variable gets parsed out to more useful individual variables.
Here’s an example of invoking the module that pulls in data from the windows event log entries associated.
# Set the ROOT to the folder your nxlog was installed into
define ROOT C:\Program Files (x86)\nxlog
## Default required locations based on the above
Moduledir %ROOT%\modules
CacheDir %ROOT%\data
Pidfile %ROOT%\data\nxlog.pid
SpoolDir %ROOT%\data
LogFile %ROOT%\data\nxlog.log
# Increase to DEBUG if needed for diagnosis
LogLevel INFO
# Input the windows event logs
<Input in>
Module im_msvistalog
</Input>
# Output the logs to a file for testing
<Output out>
Module om_file
File "C:/Program Files (x86)/nxlog/data/log-test-output.txt"
</Output>
# Define the route by mapping the input to an output
<Route 1>
Path in => out
</Route>
With any luck, you’ve now got some lines in your output file.
# Set the ROOT to the folder your nxlog was installed into
define ROOT C:\Program Files (x86)\nxlog
## Default required locations based on the above
Moduledir %ROOT%\modules
CacheDir %ROOT%\data
Pidfile %ROOT%\data\nxlog.pid
SpoolDir %ROOT%\data
LogFile %ROOT%\data\nxlog.log
# Increase to DEBUG if needed for diagnosis
LogLevel INFO
# Input a test file
<Input in>
Module im_file
File ""C:/Program Files (x86)/nxlog/data/test-in.txt"
SavePos FALSE
ReadFromLast FALSE
</Input>
# Output the logs to a file for testing
<Output out>
Module om_file
File "C:/Program Files (x86)/nxlog/data/log-test-output.txt"
</Output>
# Define the route by mapping the input to an output
<Route 1>
Path in => out
</Route>
To be useful, you need to send your logs somewhere. Here’s an example of sending them to a Logstash receiver.
# Set the ROOT to the folder your nxlog was installed into
define ROOT C:\Program Files (x86)\nxlog
## Default required locations based on the above
Moduledir %ROOT%\modules
CacheDir %ROOT%\data
Pidfile %ROOT%\data\nxlog.pid
SpoolDir %ROOT%\data
LogFile %ROOT%\data\nxlog.log
# Increase to DEBUG if needed for diagnosis
LogLevel INFO
# Load the JSON module needed by the output module
<Extension json>
Module xm_json
</Extension>
# Input the windows event logs
<Input in>
Module im_msvistalog
</Input>
# Output the logs out using the TCP module, convert to JSON format (important)
<Output out>
Module om_tcp
Host some.server
Port 6379
Exec to_json();
</Output>
# Define the route by mapping the input to an output
<Route 1>
Path in => out
</Route>
Restart the service in the windows services, and you are in business.
You’re probably shipping logs to a logstash broker (or similar json based tcp receiver). In that case, make sure to specify JSON on the way out, as in the example above or you’ll spend hours trying to figure out why you’re getting a glob of plain txt and loose all the pre-parsed windows event messages which are nearly impossible to parse back from plain text.
Using that to_json() will replace the contents. The variable we mentioned earlier, $raw_event, with all of the already parsed fields. If you hand’t invoked a module to parse that data out, you’d just get a bunch of empty events as the data was replaced with a bunch of nothing.
You can use the ‘Exec’ statement in any block and some pattern matching to drop events you don’t care about.
<Input in>
Module im_file
File "E:/Imports/get_accessplans/log-test.txt"
Exec if $raw_event =~ /someThing/ drop();
</Input>
Or the inverse, with the operator !~
The alternative is the patternDB approach as it has some parallelization advantages you’ll read about in the docs should you dig into it further. This matters when you have lots of patterns to check against.
# Set the ROOT to the folder your nxlog was installed into
define ROOT C:\Program Files (x86)\nxlog
## Default required locations based on the above
Moduledir %ROOT%\modules
CacheDir %ROOT%\data
Pidfile %ROOT%\data\nxlog.pid
SpoolDir %ROOT%\data
LogFile %ROOT%\data\nxlog.log
# Increase to DEBUG if needed for diagnosis
LogLevel INFO
# Load the JSON module needed by the output module
<Extension json>
Module xm_json
</Extension>
# Input the windows event logs
<Input in>
Module im_msvistalog
</Input>
# Process log events
<Processor pattern>
Module pm_pattern
PatternFile %ROOT%/conf/patterndb.xml
</Processor>
# Output the logs out using the TCP module, convert to JSON format (important)
<Output out>
Module om_tcp
Host some.server
Port 6379
Exec to_json();
</Output>
# Define the route by mapping the input to an output
<Route 1>
Path in => pattern => out
</Route>
And create an XML file like so:
<?xml version="1.0" encoding="UTF-8"?>
<patterndb>
<group>
<name>eventlog</name>
<id>1</id>
<pattern>
<id>2</id>
<name>500s not needed</name>
<matchfield>
<name>EventID</name>
<type>exact</type>
<value>500</value>
</matchfield>
<exec>drop();</exec>
</pattern>
</group>
</patterndb>
You may not want ALL the event logs. You can add a query to that module however, and limit logs to the security logs, like so
<Input in>
Module im_msvistalog
Query <QueryList><Query Id="0" Path="Security"><Select Path="Security">*</Select></Query></QueryList>
</Input>
You can break that into multiple lines for easier reading by escaping the returns. Here’s an example that ships the ADFS Admin logs.
<Input in>
Module im_msvistalog
Query <QueryList>\
<Query Id="0">\
<Select Path='AD FS 2.0/Admin'>*</Select>\
</Query>\
</QueryList>
</Input>
If you’re interested in very specific logs, you can create a custom view in the windows event viewer, and after selecting the criteria in with the graphical tool, click on the XML tab to see what the query is. For example, to ship all the ADFS 2 logs (assuming you’ve turned on auditing) Take the output of the XML tab (shown below) and modify to be compliant with the nxlog format.
<QueryList>
<Query Id="0" Path="AD FS 2.0 Tracing/Debug">
<Select Path="AD FS 2.0 Tracing/Debug">*[System[Provider[@Name='AD FS 2.0' or @Name='AD FS 2.0 Auditing' or @Name='AD FS 2.0 Tracing']]]</Select>
<Select Path="AD FS 2.0/Admin">*[System[Provider[@Name='AD FS 2.0' or @Name='AD FS 2.0 Auditing' or @Name='AD FS 2.0 Tracing']]]</Select>
<Select Path="Security">*[System[Provider[@Name='AD FS 2.0' or @Name='AD FS 2.0 Auditing' or @Name='AD FS 2.0 Tracing']]]</Select>
</Query>
</QueryList
Here’s the query from a MS NPS
<QueryList>
<Query Id="0" Path="System">
<Select Path="System">*[System[Provider[@Name='NPS']]]</Select>
<Select Path="System">*[System[Provider[@Name='HRA']]]</Select>
<Select Path="System">*[System[Provider[@Name='Microsoft-Windows-HCAP']]]</Select>
<Select Path="System">*[System[Provider[@Name='RemoteAccess']]]</Select>
<Select Path="Security">*[System[Provider[@Name='Microsoft-Windows-Security-Auditing'] and Task = 12552]]</Select>
</Query>
</QueryList>
NXLog has decent ability to rotate it’s own output files, but it’s doesn’t come with a lot of methods to rotate input files - i.e. your reading in Accounting logs from a windows RADIUS and it would be nice to archive those with NXLog, because Windows won’t do it. You could bust out some perl (if you’re on unix) and use the xm_perl module, but there’s a simpler way.
On windows, the solution is to use an exec block with a scheduled command. The forfiles executable is already present in windows and does the trick. The only gotcha is that ALL the parameters must be delimited like below.
So the command
forfiles /P "E:\IAS_Logs" /D -1 /C "cmd /c move @file \\server\share"
Becomes
<Extension exec>
Module xm_exec
<Schedule>
When @daily
Exec exec('C:\Windows\System32\forfiles.exe','/P','"E:\IAS_Logs"','/D','-1','/C','"cmd','/c','move','@file','\\server\share"');
</Schedule>
</Extension>
A slightly more complex example with added compression and removal of old files (there isn’t a great command line zip utility for windows in advance of powershell 5)
# Add log rotation for the windows input files
<Extension exec>
Module xm_exec
<Schedule>
When @daily
# Make a compressed copy of .log files older than 1 day
Exec exec('C:\Windows\System32\forfiles.exe','/P','"E:\IAS_Logs"','/M','*.log','/D','-1','/C','"cmd','/c','makecab','@file"')
# Delete original files after 2 days, leaving the compressed copies
Exec exec('C:\Windows\System32\forfiles.exe','/P','"E:\IAS_Logs"','/M','*.log','/D','-2','/C','"cmd','/c','del','@file"')
# Move compressed files to the depot after 2 days
Exec exec('C:\Windows\System32\forfiles.exe','/P','"E:\IAS_Logs"','/M','*.lo_','/D','-2','/C','"cmd','/c','move','@file','\\shared.ohio.edu\appshare\radius\logs\radius1.oit.ohio.edu"');
</Schedule>
</Extension>
The @daily runs right at 0 0 0 0 0 (midnight every night). Check the manual for more precise cron controls
You can use the ‘Exec’ statement to match inverse like so
<Input in>
Module im_file
File "E:/Imports/get_accessplans/log-test.txt"
Exec if $raw_event !~ /someThing/ drop();
</Input>
However, when you’re using a pattern db this is harder as the REGEXP doesn’t seem to honor inverses like you’d expect. Instead, you must look for matches in your pattern db like normal;
<?xml version="1.0" encoding="UTF-8"?>
<patterndb>
<group>
<name>eventlog</name>
<id>1</id>
<pattern>
<id>2</id>
<name>Identify user login success usernames</name>
<matchfield>
<name>EventID</name>
<type>exact</type>
<value>501</value>
</matchfield>
<matchfield>
<name>Message</name>
<type>REGEXP</type>
<value>windowsaccountname \r\n(\S+)</value>
<capturedfield>
<name>ADFSLoginSuccessID</name>
<type>STRING</type>
</capturedfield>
</matchfield
</pattern>
</group>
</patterndb>
Then, add a section to your nxlog.conf to take action when the above capture field doesn’t existing (meaning there wasn’t a regexp match).
...
# Process log events
<Processor pattern>
Module pm_pattern
PatternFile %ROOT%/conf/patterndb.xml
</Processor>
# Using a null processor just to have a place to put the exec statement
<Processor filter>
Module pm_null
Exec if (($EventID == 501) and ($ADFSLoginSucccessID == undef)) drop();
</Processor>
# Output the logs out using the TCP module, convert to JSON format (important)
<Output out>
Module om_tcp
Host some.server
Port 6379
Exec to_json();
</Output>
# Define the route by mapping the input to an output
<Route 1>
Path in => pattern => filter => out
</Route>
When using logstash as a Broker/Parser to receive events from nxlog, you’ll need to explicitly tell it that the message is in json format with a filter, like so:
input {
tcp {
port => 6379
type => "WindowsEventLog"
}
}
filter {
json {
source => message
}
}
output {
stdout { codec => rubydebug }
}
NXLog makes and handful of attributes about the event available to you. Some of these are from the ‘core’ module
$raw_event
$EventReceivedTime
$SourceModuleName
$SourceModuleType
These are always present and added to by the input module or processing module you use. For example, the mseventlog module adds all the attributes from the windows event logs as attributes to the nxlog event. So your event contains:
$raw_event
$EventReceivedTime
$SourceModuleName
$SourceModuleType
$Message
$EventTime
$Hostname
$SourceName
$EventID
...
You can also create new attributes by using a processing module, such as parsing an input file’s XML. This will translate all the tags (within limites) into attributes.
<Extension xml>
Module xm_xml
</Extension>
<Input IAS_Accounting_Logs>
Module im_file
File "E:\IAS_Logs\IN*.log"
Exec parse_xml();
</Input>
And you can also add an Exec at any point to create or replace new attribute as desired
<Input IAS_Accounting_Logs>
Module im_file
File "E:\IAS_Logs\IN*.log"
Exec $type = "RADIUSAccounting";
</Input>
Rather than manipulate everything in the input and output modules, use the pm_null module to group a block together.
<Processor rewrite>
Module pm_null
Exec parse_syslog_bsd();\
if $Message =~ /error/ \
{\
$SeverityValue = syslog_severity_value("error");\
to_syslog_bsd(); \
}
</Processor>
<Route 1>
Path in => rewrite => fileout
</Route>
define ROOT C:\Program Files (x86)\nxlog
Moduledir %ROOT%\modules
CacheDir %ROOT%\data
Pidfile %ROOT%\data\nxlog.pid
SpoolDir %ROOT%\data
LogFile %ROOT%\data\nxlog.log
# Load the modules needed by the outputs
<Extension json>
Module xm_json
</Extension>
<Extension xml>
Module xm_xml
</Extension>
# Inputs. Add the field '$type' so the receiver can easily tell what type they are.
<Input IAS_Event_Logs>
Module im_msvistalog
Query \
<QueryList>\
<Query Id="0" Path="System">\
<Select Path="System">*[System[Provider[@Name='NPS']]]</Select>\
<Select Path="System">*[System[Provider[@Name='HRA']]]</Select>\
<Select Path="System">*[System[Provider[@Name='Microsoft-Windows-HCAP']]]</Select>\
<Select Path="System">*[System[Provider[@Name='RemoteAccess']]]</Select>\
<Select Path="Security">*[System[Provider[@Name='Microsoft-Windows-Security-Auditing'] and Task = 12552]]</Select>\
</Query>\
</QueryList>
Exec $type = "RADIUS";
</Input>
<Input IAS_Accounting_Logs>
Module im_file
File "E:\IAS_Logs\IN*.log"
Exec parse_xml();
Exec $type = "RADIUSAccounting";
</Input>
# Output the logs out using the TCP module, convert to JSON format (important)
<Output broker>
Module om_tcp
Host 192.168.1.1
Port 8899
Exec to_json();
</Output>
# Routes
<Route 1>
Path IAS_Event_Logs,IAS_Accounting_Logs => broker
</Route>
# Rotate the input logs while we're at it, so we don't need a separate tool
<Extension exec>
Module xm_exec
<Schedule>
When @daily
#Note - the Exec statement is one line but may appear wrapped
Exec exec('C:\Windows\System32\forfiles.exe','/P','"E:\IAS_Logs"','/D','-1','/C','"cmd','/c','move','@file','\\some.windows.server\share\logs\radius1"');
</Schedule>
</Extension>
You can also extract and set values with a pattern_db, like this; (Note, nxlog uses perl pattern matching syntax if you need to look things up)
<?xml version="1.0" encoding="UTF-8"?>
<patterndb>
<group>
<name>ADFS Logs</name>
<id>1</id>
<pattern>
<id>2</id>
<name>Identify user login fails</name>
<matchfield>
<name>EventID</name>
<type>exact</type>
<value>111</value>
</matchfield>
<matchfield>
<name>Message</name>
<type>REGEXP</type>
<value>LogonUser failed for the '(\S+)'</value>
<capturedfield>
<name>ADFSLoginFailUsername</name>
<type>STRING</type>
</capturedfield>
</matchfield>
<set>
<field>
<name>ADFSLoginFail</name>
<value>failure</value>
<type>string</type>
</field>
</set>
</pattern>
And a more complex example, where we’re matching against a sting like:
2015-03-03T19:45:03 get_records 58 DailyAddAcct completed (Success) with: 15 Records Processed 0 adds 0 removes 0 modified 15 unchanged
<?xml version="1.0" encoding="UTF-8"?>
<patterndb>
<group>
<name>Bbts Logs</name>
<id>1</id>
<pattern>
<id>2</id>
<name>Get TS Records</name>
<matchfield>
<name>raw_event</name>
<type>REGEXP</type>
<value>^(\S+) get_record (\S+)\s+(\S+) completed \((\S+)\) with: (\S+) Records Processed (\S+) adds (\S+) removes (\S+) modified (\S+) unchanged</value>
<capturedfield>
<name>timestamp</name>
<type>STRING</type>
</capturedfield>
<capturedfield>
<name>Transaction_ID</name>
<type>STRING</type>
</capturedfield>
<capturedfield>
<name>Job_Subtype</name>
<type>STRING</type>
</capturedfield>
<capturedfield>
<name>Job_Status</name>
<type>STRING</type>
</capturedfield>
<capturedfield>
<name>Record_Total</name>
<type>STRING</type>
</capturedfield>
<capturedfield>
<name>Record_Add</name>
<type>STRING</type>
</capturedfield>
<capturedfield>
<name>Record_Remove</name>
<type>STRING</type>
</capturedfield>
<capturedfield>
<name>Record_Mod</name>
<type>STRING</type>
</capturedfield>
<capturedfield>
<name>Record_NoChange</name>
<type>STRING</type>
</capturedfield>
</matchfield>
<set>
<field>
<name>Job_Type</name>
<value>Get_Records</value>
<type>string</type>
</field>
</set>
</pattern>
</group>
</patterndb>
Sometimes you have a parse error when you’re testing and you need to feed all your source files back in. Problem is you’re usually saving position and reading only new entries by default.
Defeat this by adding a line to the nxlog config so it starts reading files at the beginning and deleting the ConfigCache file (so there’s no last position to start from).
<Input IAS_Accounting_Logs>
Module im_file
ReadFromLast FALSE
File "E:\IAS_Logs\IN*.log"
Exec parse_xml();
Exec $type = "RADIUSAccounting";
</Input>
del C:\Program Files (x86)\nxlog\data\configcache.dat
Restart and it will begin reprocessing all the data. When you’re done, remove the ReadFromLast line and restart.
Note: If you had just deleted the cache file, nxlog would have resumed at the tail of the file. You could have told it not to save position, but you actually do want that for when you’re ready to resume normal operation.
https://www.mail-archive.com/[email protected]/msg00158.html
There are two components; adding the syslog module and adding the export path.
<Extension syslog>
Module xm_syslog
</Extension>
<Input IAS_Accounting_Logs>
Module im_file
File "E:\IAS_Logs\IN*.log"
Exec $type = "RADIUSAccounting";
</Input>
<Output siem>
Module om_udp
Host 192.168.1.1
Port 514
Exec to_syslog_ietf();
</Output>
<Route 1>
Path IAS_Accounting_Logs => siem
</Route>
If you see this error message from nxlog:
ERROR Couldn't read next event, corrupted eventlog?; The data is invalid.
Congrats - you’ve hit a bug.
https://nxlog.org/support-tickets/immsvistalog-maximum-event-log-count-support
The work-around is to limit your log event subscriptions on the input side by using a query. Example:
<Input in>
Module im_msvistalog
Query <QueryList><Query Id="0" Path="Microsoft-Windows-PrintService/Operational"><Select Path="Microsoft-Windows-PrintService/Operational">*</Select></Query></QueryList>
Exec if $EventID != 307 drop();
Exec $type = "IDWorks";
</Input>
We found that nxlog made for the best windows log-shipper. But it didn’t seem to parse the events in the event log. Output to logstash seemed not to be in json format, and we confirmed this by writing directly to disk. This happens even though the event log input module explicitly emits the log attributes atomically.
Turns out you have to explicitly tell the output module to use json. This isn’t well documented.
When using Windows UNC paths, don’t forget that the backslash is also used for escaping characters, so the path
\\server\radius
looks like
\\server;adius
in your error log message. You’ll want to escape your back slashes like this;
\\\\server\\radius\\file.log
Files you’re reading may be any character set and this can cause strange things when you modify or pass the data on, as an example at stack exchange shows. This isn’t a problem with windows event logs, but windows applications use several different types of charsets.
Best practice is to convert everything to UTF-8. This is especially true when invoking modules such as json, that don’t handle other codes well.
NXLog has the ability to convert and can even to this automatically. However, there is some room for error. If you can, identity what the encoding is by looking at it in a hex editor and comparing to MS’s identification chart.
Here’s an snippet of a manual conversion of a powershell generated log. Having looked at the first part and identified it as UTF-16LE
...
<Extension charconv>
Module xm_charconv
AutodetectCharsets utf-8, utf-16, utf-32, iso8859-2, ucs-2le
</Extension>
<Input in1>
Module im_file
File "E:/Imports/log.txt"
Exec $raw_event = convert($raw_event,"UTF-16LE","UTF-8");
</Input>
...
Notice however that the charconv module has an automatic directive. You can use that as long as what you have is included as marked in bold here.
<Extension charconv>
Module xm_charconv
AutodetectCharsets utf-8, utf-16, utf-16le, utf-32, iso8859-2
</Extension>
<Input sql-ERlogs>
Module im_file
File 'C:\Program Files\Microsoft SQL Server\MSSQL11.SQL\MSSQL\Log\ER*'
ReadFromLast TRUE
Exec convert_fields("AUTO", "utf-8");
</Input>
If you’re curious what charsets are supported, you can type this command in any unix system to see the names.
iconv -i
Windows uses UTF-16 by default. Other services may use derivations thereof. In any event, it’s recommended to normalize things to UTF-8. Here’s a good example of what will happen if you don’t;
<http://stackoverflow.com/questions/27596676/nxlog-logs-are-in-unicode-charecters>
The answer to that question is to use the specific code field, as “AUTO” doesn’t seem to detect properly.
<Input in>
Module im_file
File "E:/Imports/get_accessplans/log-test.txt"
Exec if $raw_event == '' drop();
Exec $Event = convert($raw_event,"UCS-2LE","UTF-8"); to_json();
SavePos FALSE
ReadFromLast FALSE
</Input>
From the manual on SQL Server
Microsoft SQL Server
Microsoft SQL Server stores its logs in UTF-16 encoding using a line-based format.
It is recommended to normalize the encoding to UTF-8. The following config snipped
will do that.
<Extension _charconv>
Module xm_charconv
</Extension>
<Input in>
Module im_file
File "C:\\MSSQL\\ERRORLOG"
Exec convert_fields('UCS-2LE','UTF-8'); if $raw_event == '' drop();
</Input>
As of this writing, the LineBased parser, the default InputType for im_file is not able to properly read the double-byte UTF-16 encoded files and will read
an additional empty line (because of the double-byte CRLF). The above drop() call is intended to fix this.
convert_fields('UTF-16','UTF-8'); might also work instead of UCS-2LE.
If you’re in a Windows shop, this is the best way to keep the Windows admins happy. No installation of extra tools. ‘Keeps it in the MS family’ so to speak.
Configure your servers to push1 logs to a cental location and use a client there, to send it on. Beats works well for this.
The key seems to be
check the runtime status on the collector
For printing, in Event Viewer navigate to Microsoft-Windows-PrintService/Operational and enable it as its not on by default.
Make sure to enable for latency or you’ll spend a long time wondering why there is no data.
https://hackernoon.com/the-windows-event-forwarding-survival-guide-2010db7a68c4 https://www.ibm.com/docs/en/netcoolomnibus/8?topic=acquisition-forwarded-event-log https://www.youtube.com/watch?v=oyPuRE51k3o&t=158s
It’s one thing to know what’s going on, it’s another to do something about it.
CrowdSec is a three-part system; detection, coordination and mitigation.
The detection agent runs wherever you have log files worth looking at. It compares entries to known patterns of intrusion attempts and reports any events to the local hub. You can can also create your own patterns for custom services.
This local hub coordinates these security events into a list that all participants can see. It submits these and pulls other known offenders from CrowdSec central. This keeps the local block list in sync with greater community. Putting the ‘Crowd’ in CrowdSec.
The mitigation agent is the final part of the solution. It runs on firewalls or servers and blocks any IPs associated with an event.
A basic deploy of CrowdSec is fairly straight-forward.
With Debian, you can simply add the repo via their script and install with a couple lines.
curl -s https://install.crowdsec.net | sudo sh
sudo apt install crowdsec
sudo apt install crowdsec-firewall-bouncer-nftables
This installs all three parts. The detection and coordination part (crowdsec), and the mitigation (crowdsec-firewall-bouncer) part.
The crowdsec binary will check in with the cloud, download a baseline list of known bad-actors, and begin watching the syslog for known patterns (like ssh login failures). The firewall-bouncer will create a nft drop table and keep it up to date with any changes from the hub.
# Check out the very long drop list
sudo nft list ruleset | less
Note - if the CrowdSec tables are empty, you may need to sudo systemctl restart nftables.service or possibly reboot (as I’ve found in testing)
CrowdSec comes pre-configured to watch for the basics and seems to do some inspection of your system to add appropriate log monitoring. You can see what’s it monitoring with the command:
sudo cscli collections list
Sometimes, that’s all you need. But take a look at operations for more options.
Deploy a single hub that all agents connect to. This allows the whole network to share events.
Start with just the main CrowdSec binary.
curl -s https://install.crowdsec.net | sudo sh
sudo apt install crowdsec
Configure the hub on to listen on the network
sudo sed -i 's/listen_uri: 127.0.0.1:8080/listen_uri: 0.0.0.0:8080/' /etc/crowdsec/config.yaml
sudo systemctl restart crowdsec.service
To connect, clients should use individual API keys.
# For a host named 'www' in this case, change as desired.
sudo cscli machine add www --auto -f -
Machine 'www' successfully added to the local API.
url: http://0.0.0.0:8080
login: www
password: sadkljfhaslkdjhfalkwsuehfaliseudhbf00987
Install CrowdSec on the client.
curl -s https://install.crowdsec.net | sudo sh
sudo apt install crowdsec
Comment out the server block to prevent starting a hub here too.
sudo vi /etc/crowdsec/config.yaml
...
...
user: nobody # plugin process would be ran on behalf of this user
group: nogroup # plugin process would be ran on behalf of this group
api:
client:
insecure_skip_verify: false
credentials_path: /etc/crowdsec/local_api_credentials.yaml
# server:
# log_level: info
# listen_uri: 127.0.0.1:8080
# profiles_path: /etc/crowdsec/profiles.yaml
# console_path: /etc/crowdsec/console.yaml
# online_client: # Central API credentials (to push signals and receive bad IPs)
# credentials_path: /etc/crowdsec/online_api_credentials.yaml
# trusted_ips: # IP ranges, or IPs which can have admin API access
# - 127.0.0.1
# - ::1
# tls:
# cert_file: /etc/crowdsec/ssl/cert.pem
# key_file: /etc/crowdsec/ssl/key.pem
prometheus:
enabled: true
level: full
listen_addr: 127.0.0.1
listen_port: 6060
Supply the Sec Hub address and password.
sudo vi /etc/crowdsec/local_api_credentials.yaml
url: http://sechub.lan:8080
login: www
password: sadkljfhaslkdjhfalkwsuehfaliseudhbf00987
sudo systemctl restart crowdsec.service
sudo systemctl status crowdsec.service
The best place to stop attacks is at the border. If you’re running OpenWRT or a Linux Firewall there’s a handy netfilter ‘bouncer’ you can add.
Back on your SecHub, generate an API key for it.
sudo cscli bouncers add router-1-bouncer
API key for 'router-1-bouncer':
XXXXXXXXXX
Over on your firewall, install the bouncer and add the API details
curl -s https://install.crowdsec.net | sudo sh
sudo apt install crowdsec-firewall-bouncer-nftables
# Adjust 'SecHub' as needed and past in your key (or directly edit the file)
sudo sed -i 's#api_url: http://127.0.0.1:8080/#api_url: http://SecHub:8080/#' /etc/crowdsec/bouncers/crowdsec-firewall-bouncer.yaml
sudo sed -i 's#api_key: <API_KEY>#api_key: XXXXXX#' /etc/crowdsec/bouncers/crowdsec-firewall-bouncer.yaml
sudo systemctl restart crowdsec-firewall-bouncer.service
You can install the bouncer in other places as well if you need to establish defense in depth. Though you may need to look into how private ranges are whitelisted to protect different segments.
Is it working? Take a look at the decision list.
# On the hub, this will show the community block list.
sudo cscli decisions list --origin CAPI
# On the firewall, this will show the corresponding table contents from the list
sudo nft list table ip crowdsec
table ip crowdsec { … }
In some cases, the hub server will not download the community block list after installation without a reboot. Everything will seem to be working but no default list until you reboot.
CrowdSec is most useful when you integrate local data sources. The easiest way to add local collections is to enable new collections.
A collection is a bundle of config files that detail what to look for, and where to look. The install selects a few to start with based on it’s inspection of already already installed software.
sudo cscli collections list
If you want something else, take a look at CrowdSec’s collections to see what’s available and the more info link will show you how to install it. You can also do a quick look at the command line as well.
sudo cscli collections list -a
For example, to configure it to watch caddy:
sudo cscli collections install crowdsecurity/caddy
# It should add a file to the drop folder
cat /etc/crowdsec/acquis.d/caddy.yaml
# Reload crowdsec for these changes to take effect
sudo systemctl reload crowdsec
Inspecting the collection will tell you what parsers and scenarios it bundles together. As well as some metrics.
sudo cscli collections list
sudo cscli collections inspect crowdsecurity/linux
sudo cscli collections inspect crowdsecurity/sshd
To learn more a collection and it’s components, you can check out their page:
https://hub.crowdsec.net/author/crowdsecurity/collections/linux
The metrics are a bit confusing until you learn that the ‘Unparsed’ column doesn’t mean unparsed so much as it means a non-event. These are just normal log lines that don’t have a pattern a parser was looking for.
A collection starts by putting a config file in the acquis.d folder. This is the acquisition drop folder and tells crowdsec where to acquire data - i.e. what log files to look at. If it’s a non-standard format it will also add a new parser under parsers and any patterns under scenarios. These are things like “5 failed login attempts in less than 30 seconds”.
The important thing to know is parsers are only successful on interesting events. A normal login is just recorded as a “failed” parse because it didn’t match anything interesting. The scenario then only considers actual events, reducing the contextual overhead.
Matching a scenario triggers an alert which is sent to the hub. These alerts usually (unless whitelisted) become decisions. The default decision is to add the IP added to the ban list. These have a configurable expiration, so that if you really guess wrong 10 times in a row, you’re not banned forever.
Is anyone currently attacking you? The decisions list shows you any current bad actors and the alerts list shows you a summary of past decisions. If you are just getting started this is probably none, but if you’re open to the internet this will grow quickly.
sudo cscli decisions list
sudo cscli alerts list
You’re also getting events from the cloud and you can check those with the -a option. You’ll notice that every 2 hours the community-blocklist is updated.
sudo cscli alerts list -a
After a while of this collection running, you’ll start to see these kinds of alerts
sudo cscli alerts list
╭────┬───────────────────┬───────────────────────────────────────────┬─────────┬────────────────────────┬───────────┬─────────────────────────────────────────╮
│ ID │ value │ reason │ country │ as │ decisions │ created_at │
├────┼───────────────────┼───────────────────────────────────────────┼─────────┼────────────────────────┼───────────┼─────────────────────────────────────────┤
│ 27 │ Ip:18.220.128.229 │ crowdsecurity/http-bad-user-agent │ US │ 16509 AMAZON-02 │ ban:1 │ 2023-03-02 13:12:27.948429492 +0000 UTC │
│ 26 │ Ip:18.220.128.229 │ crowdsecurity/http-path-traversal-probing │ US │ 16509 AMAZON-02 │ ban:1 │ 2023-03-02 13:12:27.979479713 +0000 UTC │
│ 25 │ Ip:18.220.128.229 │ crowdsecurity/http-probing │ US │ 16509 AMAZON-02 │ ban:1 │ 2023-03-02 13:12:27.9460075 +0000 UTC │
│ 24 │ Ip:18.220.128.229 │ crowdsecurity/http-sensitive-files │ US │ 16509 AMAZON-02 │ ban:1 │ 2023-03-02 13:12:27.945759433 +0000 UTC │
│ 16 │ Ip:159.223.78.147 │ crowdsecurity/http-probing │ SG │ 14061 DIGITALOCEAN-ASN │ ban:1 │ 2023-03-01 23:03:06.818512212 +0000 UTC │
│ 15 │ Ip:159.223.78.147 │ crowdsecurity/http-sensitive-files │ SG │ 14061 DIGITALOCEAN-ASN │ ban:1 │ 2023-03-01 23:03:05.814690037 +0000 UTC │
╰────┴───────────────────┴───────────────────────────────────────────┴─────────┴────────────────────────┴───────────┴─────────────────────────────────────────╯
You may even need to unblock yourself
sudo cscli decisions list
sudo cscli decision delete --id XXXXXXX
Data comes in through the parsers. To see what they are doing, let’s take a look at the Acquisition and Parser metrics.
sudo cscli metrics
Most of the ‘Acquisition Metrics’ lines will be read and parsed, but not all. Some entries don’t have anything a scenario can use. For example, when reading the SSH logs, the parser quits early if there’s not even an IP address.
sudo cscli metrics show acquisition
╭──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ Acquisition Metrics │
├────────────────────────────────────────┬────────────┬──────────────┬────────────────┬────────────────────────┬───────────────────┤
│ Source │ Lines read │ Lines parsed │ Lines unparsed │ Lines poured to bucket │ Lines whitelisted │
├────────────────────────────────────────┼────────────┼──────────────┼────────────────┼────────────────────────┼───────────────────┤
│ file:/var/log/caddy/access.log │ 1 │ 1 │ - │ 1 │ - │
│ file:/var/log/caddy/www.some.org.log │ 15 │ 15 │ - │ 18 │ - │
╰────────────────────────────────────────┴────────────┴──────────────┴────────────────┴────────────────────────┴───────────────────╯
“Lines poured to bucket” is the interesting column. If it matches a scenario, the parser ‘pours’ the event into a scenario’s ‘bucket’. Some get poured into multiple scenarios. Tha can make the “Lines poured” larger than the number of lines, like above.
Let’s take a look at what scenarios were matched.
sudo cscli metrics show scenarios
╭────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ Scenario Metrics │
├──────────────────────────────────────┬───────────────┬───────────┬──────────────┬────────┬─────────┤
│ Scenario │ Current Count │ Overflows │ Instantiated │ Poured │ Expired │
├──────────────────────────────────────┼───────────────┼───────────┼──────────────┼────────┼─────────┤
│ crowdsecurity/http-crawl-non_statics │ 2 │ - │ 13 │ 14 │ 11 │
│ crowdsecurity/http-probing │ 2 │ - │ 5 │ 5 │ 3 │
╰──────────────────────────────────────┴───────────────┴───────────┴──────────────┴────────┴─────────╯
There are two scenarios matched. But while lines were poured in, nothing ‘Overflowed’. Sometimes people just type their password wrong or there’s a bad link in a web page. So the scenarios have threshold before action is taken.
You can take a look at the scenario to see what it’s looking for as well. Here’s the key lines:
# Let's look at the definition for http-probing
sudo cscli hub list
sudo cat /etc/crowdsec/hub/scenarios/crowdsecurity/http-probing.yaml
...
...
description: "Detect site scanning/probing from a single ip"
filter: "evt.Meta.service == 'http' && evt.Meta.http_status in ['404', '403', '400'] && evt.Parsed.static_ressource == 'false'"
...
capacity: 10
leakspeed: "10s"
...
It’s looking for HTTP events with 400’s that are not static resources. The capacity 10 means more than 10 events from a single IP triggers an alert. A leakspeed of 10s means to expire events that many seconds after they enter the bucket.
After a while, you’ll see something overflow and generate an alert and a decision.
╭────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ Scenario Metrics │
├──────────────────────────────────────┬───────────────┬───────────┬──────────────┬────────┬─────────┤
│ Scenario │ Current Count │ Overflows │ Instantiated │ Poured │ Expired │
├──────────────────────────────────────┼───────────────┼───────────┼──────────────┼────────┼─────────┤
│ crowdsecurity/http-crawl-non_statics │ 2 │ - │ 65 │ 78 │ 63 │
│ crowdsecurity/http-generic-bf │ - │ 3 │ 5 │ 24 │ 2 │
│ crowdsecurity/http-probing │ 2 │ - │ 26 │ 26 │ 24 │
╰──────────────────────────────────────┴───────────────┴───────────┴──────────────┴────────┴─────────╯
sudo cscli alerts list
╭─────┬───────────────────┬───────────────────────────────┬─────────┬──────────────────────────────────┬───────────┬──────────────────────╮
│ ID │ value │ reason │ country │ as │ decisions │ created_at │
├─────┼───────────────────┼───────────────────────────────┼─────────┼──────────────────────────────────┼───────────┼──────────────────────┤
│ 117 │ Ip:20.114.185.119 │ crowdsecurity/http-generic-bf │ US │ 8075 MICROSOFT-CORP-MSN-AS-BLOCK │ ban:1 │ 2026-02-06T18:11:51Z │
╰─────┴───────────────────┴───────────────────────────────┴─────────┴──────────────────────────────────┴───────────┴──────────────────────╯
sudo cscli decision list
╭────────┬──────────┬───────────────────┬───────────────────────────────┬────────┬─────────┬──────────────────────────────────┬────────┬────────────┬──────────╮
│ ID │ Source │ Scope:Value │ Reason │ Action │ Country │ AS │ Events │ expiration │ Alert ID │
├────────┼──────────┼───────────────────┼───────────────────────────────┼────────┼─────────┼──────────────────────────────────┼────────┼────────────┼──────────┤
│ 904849 │ crowdsec │ Ip:20.114.185.119 │ crowdsecurity/http-generic-bf │ ban │ US │ 8075 MICROSOFT-CORP-MSN-AS-BLOCK │ 6 │ 3h39m30s │ 117 │
╰────────┴──────────┴───────────────────┴───────────────────────────────┴────────┴─────────┴──────────────────────────────────┴────────┴────────────┴──────────╯
An overflow generates an alert and a decision. That decision (to block) will stay in place for 4 hours until it expires but the alert will remain as a historical record.
Your bouncers will enforce the decisions and you can check them as well.
# Run this wherever your firewall bouncer is running
nft list ruleset | grep 20.114.185.119
elements = { 20.114.185.119 timeout 3h59m57s expires 3h33m25s132ms,
# If you're using the caddy bouncer
caddy crowdsec check 20.114.185.119
blocked: true
The reason why things parse or don’t parse is sometimes worth getting into and you can have Crowdsec explain it to you.
SSH authentication is interesting, because not all lines parse.
sudo cscli metrics show acquisition
╭───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ Acquisition Metrics │
├─────────────────────────────────────────────────┬────────────┬──────────────┬────────────────┬────────────────────────┬───────────────────┤
│ Source │ Lines read │ Lines parsed │ Lines unparsed │ Lines poured to bucket │ Lines whitelisted │
├─────────────────────────────────────────────────┼────────────┼──────────────┼────────────────┼────────────────────────┼───────────────────┤
│ journalctl:journalctl-_SYSTEMD_UNIT=ssh.service │ 9 │ 5 │ 4 │ - │ 5 │
╰─────────────────────────────────────────────────┴────────────┴──────────────┴────────────────┴────────────────────────┴───────────────────╯
sudo cscli explain --dsn "journalctl://filters=_SYSTEMD_UNIT=ssh.service" --type syslog
# This line is just the server starting - it says 'success' but in the sense of 'successfully bailed out'
# before it completed parsing because it detected the 0.0.0.0 address. It will be counted as unparsed in the metrics.
line: Feb 02 15:04:22 www sshd[180]: Server listening on 0.0.0.0 port 22.
├ s00-raw
| ├ 🟢 crowdsecurity/syslog-logs (+12 ~8)
| └ 🔴 crowdsecurity/non-syslog
├ s01-parse
| ├ 🔴 crowdsecurity/caddy-logs
| ├ 🔴 crowdsecurity/sshd-logs
| └ 🔴 crowdsecurity/sshd-success-logs
└-------- parser success, ignored by whitelist (private ipv4/ipv6 ip/ranges) 🟢
# Here's me failing to log in. This was successfully parsed and made it to step 2. That's where crowdsec
# starts pulling out bits of the log message for further inspection.
#
# The numbers at the end of the line such as (+9 ~1) signify that the parser added 9 attributes
# (pieces of data) to the event, parsed from the raw log message, and updated 1.
#
# Now that it has more data, the step 2 'whitelists' get another go and see that I'm attacking
# from a private IP range. They flag it as such and it stops processing it before it gets further.
# Turns out private IPs are whitelisted by default so you can't lock yourself out.
#
# This entry is counted as parsed and whitelisted
line: Feb 06 18:12:30 www sshd-session[12643]: Failed password for allen from ::1 port 36082 ssh2
├ s00-raw
| ├ 🟢 crowdsecurity/syslog-logs (+12 ~8)
| └ 🔴 crowdsecurity/non-syslog
├ s01-parse
| ├ 🔴 crowdsecurity/caddy-logs
| ├ 🟢 crowdsecurity/sshd-logs (+9 ~1)
| └ 🔴 crowdsecurity/sshd-success-logs
├ s02-enrich
| ├ 🟢 crowdsecurity/dateparse-enrich (+2 ~1)
| ├ 🔴 crowdsecurity/geoip-enrich
| ├ 🔴 crowdsecurity/http-logs
| ├ 🟢 crowdsecurity/public-dns-allowlist (unchanged)
| └ 🟢 crowdsecurity/whitelists (~2 [whitelisted])
└-------- parser success, ignored by whitelist (private ipv4/ipv6 ip/ranges) 🟢
If that attempt was from the outside, a few of them would lead to an alert.
Almost all HTTP access logs are parsed as they almost always have data interesting to CrowdSec. Take a look to see what a full parsing and scenario progression looks like.
sudo cscli metrics show acquisition
╭───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ Acquisition Metrics │
├─────────────────────────────────────────────────┬────────────┬──────────────┬────────────────┬────────────────────────┬───────────────────┤
│ Source │ Lines read │ Lines parsed │ Lines unparsed │ Lines poured to bucket │ Lines whitelisted │
├─────────────────────────────────────────────────┼────────────┼──────────────┼────────────────┼────────────────────────┼───────────────────┤
│ file:/var/log/caddy/access.log │ 71 │ 71 │ - │ 63 │ - │
╰─────────────────────────────────────────────────┴────────────┴──────────────┴────────────────┴────────────────────────┴───────────────────╯
sudo tail -1 /var/log/caddy/access.log > ./logs
sudo cscli explain -v --file ./logs --type caddy
...
...
├ Scenarios
├ 🟢 crowdsecurity/http-crawl-non_statics
└ 🟢 crowdsecurity/http-generic-bf
# At the very bottom you'll see this line was feed to two different scenarios
You may find entries aren’t being parsed even though they should be. Maybe the log format has changed or you’re logging additional data the author didn’t anticipate.
You can modify the existing parser, but best-practice it to add a custom parser.
There are several type of parsers and they are used in stages. Some are designed to work with the raw log entries while others are designed to take pre-parsed data and add or enrich it. This way you can do branching and not every parser needs to now how to read a syslog message.
Their Local Path will tell you what stage they kick in at. Use sudo cscli parsers list to display the details. s00-raw works with the ‘raw’ files while s01 and s02 work further down the pipeline. Currently, you can only create s00 and s01 level parsers.
Useful parsers supply data that Scenarios are interested in. You can create a parser that watches the system logs for ‘FOOBAR’ entries, extracts the ‘FOOBAR-LEVEL`, and passes it on. But if nothing is looking for ‘FOOBARs’ then nothing will happen.
Let’s say you’ve added the Caddy collection. It’s pulled in a bunch of Scenarios you can view with sudo cscli scenarios list. If you look at one of the assicated files you’ll see a filter section where they look for ’evt.Meta.http_path’ and ’evt.Parsed.verb’. They are all different though, so how do you know what data to supply?
Your best bet is to take an existing parser and modify it.
Note - CrowdSec is pretty awesome and after talking in the discord they handled it. So these two examples are solved. I’m sure you’ll find new ones, though ;-)
It’s now also become an example of a prior version of the parser. I’ll update it as time permits.
Let’s say that you’ve installed the Caddy collection, but you’ve noticed basic auth login failures don’t trigger the parser. So let’s add a new file and edit it.
sudo cp /etc/crowdsec/parsers/s01-parse/caddy-logs.yaml /etc/crowdsec/parsers/s01-parse/caddy-logs-custom.yaml
You’ll notice two top level sections where the parsing happens; nodes and statics and some grok pattern matching going on.
Nodes allow you try multiple patterns and if any match, the whole section is considered successful. I.e. if the log could have either the standard HTTPDATE or a CUSTOMDATE, as long as it has one it’s good and the matching can move on. Statics just goes down the list extracting data. If any fail the whole event is considered a fail and dropped as unparseable.
All the pasrsed data gets attached to event as ’evt.Parsed.something’ and some of the statics are moving it to evt values the Senarios will be looking for Caddy logs are JSON formatted and so basically already parsed and this example makes use of the JsonExtract method quite a bit.
# We added the caddy logs in the acquis.yaml file with the label 'caddy' and so we use that as our filter here
filter: "evt.Parsed.program startsWith 'caddy'"
onsuccess: next_stage
# debug: true
name: caddy-logs-custom
description: "Parse custom caddy logs"
pattern_syntax:
CUSTOMDATE: '%{DAY:day}, %{MONTHDAY:monthday} %{MONTH:month} %{YEAR:year} %{TIME:time} %{WORD:tz}'
nodes:
- nodes:
- grok:
pattern: '%{NOTSPACE} %{NOTSPACE} %{NOTSPACE} \[%{HTTPDATE:timestamp}\]%{DATA}'
expression: JsonExtract(evt.Line.Raw, "common_log")
statics:
- target: evt.StrTime
expression: evt.Parsed.timestamp
- grok:
pattern: "%{CUSTOMDATE:timestamp}"
expression: JsonExtract(evt.Line.Raw, "resp_headers.Date[0]")
statics:
- target: evt.StrTime
expression: evt.Parsed.day + " " + evt.Parsed.month + " " + evt.Parsed.monthday + " " + evt.Parsed.time + ".000000" + " " + evt.Parsed.year
- grok:
pattern: '%{IPORHOST:remote_addr}:%{NUMBER}'
expression: JsonExtract(evt.Line.Raw, "request.remote_addr")
- grok:
pattern: '%{IPORHOST:remote_ip}'
expression: JsonExtract(evt.Line.Raw, "request.remote_ip")
- grok:
pattern: '\["%{NOTDQUOTE:http_user_agent}\"]'
expression: JsonExtract(evt.Line.Raw, "request.headers.User-Agent")
statics:
- meta: log_type
value: http_access-log
- meta: service
value: http
- meta: source_ip
expression: evt.Parsed.remote_addr
- meta: source_ip
expression: evt.Parsed.remote_ip
- meta: http_status
expression: JsonExtract(evt.Line.Raw, "status")
- meta: http_path
expression: JsonExtract(evt.Line.Raw, "request.uri")
- target: evt.Parsed.request #Add for http-logs enricher
expression: JsonExtract(evt.Line.Raw, "request.uri")
- parsed: verb
expression: JsonExtract(evt.Line.Raw, "request.method")
- meta: http_verb
expression: JsonExtract(evt.Line.Raw, "request.method")
- meta: http_user_agent
expression: evt.Parsed.http_user_agent
- meta: target_fqdn
expression: JsonExtract(evt.Line.Raw, "request.host")
- meta: sub_type
expression: "JsonExtract(evt.Line.Raw, 'status') == '401' && JsonExtract(evt.Line.Raw, 'request.headers.Authorization[0]') startsWith 'Basic ' ? 'auth_fail' : ''"
The very last line is where a status 401 is checked. It looks for a 401 and a request for Basic auth. However, this misses events where someone asks for a resource that is protected and the server responds telling you Basic is needed. I.e. when a bot is poking at URLs on your server ignoring the prompts to login. You can look at the log entries more easily with this command to follow the log and decode it while you recreate failed attempts.
sudo tail -f /var/log/caddy/access.log | jq
To change this, update the expression to also check the response header with an additional ? (or) condition.
expression: "JsonExtract(evt.Line.Raw, 'status') == '401' && JsonExtract(evt.Line.Raw, 'request.headers.Authorization[0]') startsWith 'Basic ' ? 'auth_fail' : ''"xtract(evt.Line.Raw, 'status') == '401' && JsonExtract(evt.Line.Raw, 'resp_headers.Www-Authenticate[0]') startsWith 'Basic ' ? 'auth_fail' : ''"
Let’s say you’re using dropbear and failed logins are not being picked up by the ssh parser
To see what’s going on, you use the crowdsec command line interface. The shell command is cscli and you can ask it about it’s metrics to see how many lines it’s parsed and if any of them are suspicious. Since we just restarted, you may not have any syslog lines yet, so let’s add some and check.
ssh [email protected]
logger "This is an innocuous message"
cscli metrics
INFO[28-06-2022 02:41:33 PM] Acquisition Metrics:
+------------------------+------------+--------------+----------------+------------------------+
| SOURCE | LINES READ | LINES PARSED | LINES UNPARSED | LINES POURED TO BUCKET |
+------------------------+------------+--------------+----------------+------------------------+
| file:/var/log/messages | 1 | - | 1 | - |
+------------------------+------------+--------------+----------------+------------------------+
Notice that the line we just read is unparsed and that’s OK. That just means it wasn’t an entry the parser cared about. Let’s see if it responds to an actual failed login.
dbclient some.remote.host
# Enter some bad passwords and then exit with a Ctrl-C. Remember, localhost attempts are whitelisted so you must be remote.
[email protected]'s password:
[email protected]'s password:
cscli metrics
INFO[28-06-2022 02:49:51 PM] Acquisition Metrics:
+------------------------+------------+--------------+----------------+------------------------+
| SOURCE | LINES READ | LINES PARSED | LINES UNPARSED | LINES POURED TO BUCKET |
+------------------------+------------+--------------+----------------+------------------------+
| file:/var/log/messages | 7 | - | 7 | - |
+------------------------+------------+--------------+----------------+------------------------+
Well, no luck. We will need to adjust the parser
sudo cp /etc/crowdsec/parsers/s01-parse/sshd-logs.yaml /etc/crowdsec/parsers/s01-parse/sshd-logs-custom.yaml
Take a look at the logfile and copy an example line over to https://grokdebugger.com/. Use a pattern like
Bad PAM password attempt for '%{DATA:user}' from %{IP:source_ip}:%{INT:port}
Assuming you get the pattern worked out, you can then add a section to the bottom of the custom log file you created.
- grok:
name: "SSHD_AUTH_FAIL"
pattern: "Login attempt for nonexistent user from %{IP:source_ip}:%{INT:port}"
apply_on: message
In the previous examples we’ve looked at the metrics and details of internal facing service like failed SSH logins. Those types aren’t prone to a lot of false positives. But other sources, like web access logs, can be.
You’ll recall that when looking at metrics that a high number of ‘Lines unparsed’ is normal. They were simply entries that didn’t match any specific events the parser was looking for. Parsed lines however, are ‘poured’ to a bucket. A bucket being a potential attack type.
sudo cscli metrics
Acquisition Metrics:
╭────────────────────────────────┬────────────┬──────────────┬────────────────┬────────────────────────╮
│ Source │ Lines read │ Lines parsed │ Lines unparsed │ Lines poured to bucket │
├────────────────────────────────┼────────────┼──────────────┼────────────────┼────────────────────────┤
│ file:/var/log/auth.log │ 69 │ - │ 69 │ - │
│ file:/var/log/caddy/access.log │ 21 │ 21 │ - │ 32 │
│ file:/var/log/syslog │ 2 │ - │ 2 │ - │
╰────────────────────────────────┴────────────┴──────────────┴────────────────┴────────────────────────╯
Scenario Metrics:
╭──────────────────────────────────────┬───────────────┬───────────┬──────────────┬────────┬─────────╮
│ Scenario │ Current Count │ Overflows │ Instantiated │ Poured │ Expired │
├──────────────────────────────────────┼───────────────┼───────────┼──────────────┼────────┼─────────┤
│ crowdsecurity/http-crawl-non_statics │ - │ - │ 2 │ 17 │ 2 │
│ crowdsecurity/http-probing │ - │ 1 │ 2 │ 15 │ 1 │
╰──────────────────────────────────────┴───────────────┴───────────┴──────────────┴────────┴─────────╯
The 21 web requests where interesting to a couple scenarios. With 32 pours some where interesting to both.
The scenario http-crawl-non_statics is designed to allow some light web-crawling but http-probing is more sensitive and it overflowed into an alert.
Assuming this is related to a web application you’re trying to use, you just got blocked. Let’s take a look at the alert
allen@www:~$ sudo cscli alert list
╭─────┬────────────────────┬────────────────────────────────────────────┬─────────┬──────────────────────────────────┬───────────┬──────────────────────╮
│ ID │ value │ reason │ country │ as │ decisions │ created_at │
├─────┼────────────────────┼────────────────────────────────────────────┼─────────┼──────────────────────────────────┼───────────┼──────────────────────┤
│ 129 │ Ip:20.42.209.0 │ crowdsecurity/http-probing │ AU │ 8075 MICROSOFT-CORP-MSN-AS-BLOCK │ ban:1 │ 2026-02-06T20:33:26Z │
sudo cscli alerts inspect -d 129
...
...
- Context :
╭────────────┬───────────────────────────────────────────────────╮
│ Key │ Value │
├────────────┼───────────────────────────────────────────────────┤
│ method │ GET │
│ status │ 404 │
│ target_uri │ /0/icon/John │
│ target_uri │ /0/icon/Smith │
│ target_uri │ /0/icon/Adam │
│ target_uri │ /0/icon/Smith │
╰────────────┴───────────────────────────────────────────────────╯
...
...
target_fqdn │ app.some.org
You can see the application in question is trying to load some missing icons and just fails silently, That may be fine for the app, but it looks like someone crawling or probing the server. To fix it we’ll need to create a whitelist definition for the app.
To whitelist an app, we create a file with an expression that matches the behavior we see above, such as the apps attempts to load a file that doesn’t exist, and exempts it. You can only add these to the s02 stage folder and the name element but be unique for each.
sudo vi /etc/crowdsec/parsers/s02-enrich/some-app-whitelist.yaml
This example uses the startsWith expression and assumes that all requests start the same
name: you/some-app
description: "Whitelist 404s for icon requests"
whitelist:
reason: "icon request"
expression:
- evt.Parsed.request startsWith '/0/icon/'
If it’s less predictable, you can use a regular expression instead and combine with other expressions like a site match. In general, the more specific the better.
name: you/some-app-whitelist
description: "Whitelist 404s for icon requests"
whitelist:
reason: "icon request"
expression:
- evt.Parsed.request matches '^/[0-9]/icon/.*' && evt.Meta.target_fqdn == "some-app.you.org"
Now you can reload crowdsec and test
sudo systemctl restart crowdsec.service
sudo cscli explain -v --file ./test.log --type caddy
├ s00-raw
| ├ 🔴 crowdsecurity/syslog-logs
| └ 🟢 crowdsecurity/non-syslog (+5 ~8)
| └ update evt.ExpectMode : %!s(int=0) -> 1
| └ update evt.Stage : -> s01-parse
...
├ s02-enrich
| ├ 🟢 you/some-app-whitelist (~2 [whitelisted])
| ├ update evt.Whitelisted : %!s(bool=false) -> true
| ├ update evt.WhitelistReason : -> some icon request
| ├ 🟢 crowdsecurity/dateparse-enrich (+2 ~2)
...
...
| ├ 🟢 crowdsecurity/http-logs (+7)
| └ 🟢 crowdsecurity/whitelists (unchanged)
└-------- parser success, ignored by whitelist (icon request) 🟢
You’ll see in the above example, we successfully parsed the entry, but it was ‘ignored’ and didn’t go on to the Scenario section.
If you have more than one whitelist, check the name you gave it on the first line. If that’s not unique, the whole thing will be silently ignored.
CrowdSec uses the go-centric expr-lang. You may be used to unix regex where you’d escape slashes, for example. A tool like https://www.akto.io/tools/regex-tester is helpful.
The parsing is almost always right, but you may need to add a scenario that is specific to your environment. A great example is when you need to pay attention to other HTTP Status codes.
Take a look at the On Caddy example for that.
There are two things you need to look at; detecting aborted connections and trusted proxies. Connections that you abort aren’t an established HTTP Status code, so you’ll need to tell CrowdSec about it. And trusted proxies require blocking at a higher level in the stack.
If you’ve secured caddy by aborting invalid requests they will be recorded in the log as “status 0”.
grep '"status":0' /var/log/caddy/access.log | tail -1 > aborts.log
sudo cscli explain -v --file ./aborts.log --type caddy
├ s01-parse
| └ 🟢 crowdsecurity/caddy-logs (+14 ~2)
| └ update evt.Stage : s01-parse -> s02-enrich
| └ ...
| └ ...
| └ create evt.Meta.http_status : 0
Crowdsec is parsing correctly, but none of the scenarios are looking for that status code.
sudo grep evt.Meta.http_status /etc/crowdsec/scenarios/*
...
...
# a bunch of thing looking for 401 and such
We just need to let CrowdSec know we’re doing this by adding a scenario.
sudo vi /etc/crowdsec/scenarios/custom-http-probe-aborted.yaml
type: leaky
name: custom/http-probe-aborted
description: "Detect http probes that caddy aborted"
filter: "evt.Meta.log_type == 'http_access-log' && evt.Meta.http_status == '0'"
groupby: evt.Meta.source_ip
leakspeed: 10s
capacity: 5
blackhole: 1m
labels:
remediation: true
classification:
- attack.T1595
behavior: "http:scan"
label: "HTTP Probing"
service: http
sudo systemctl restart crowdsec
Make a request and check metrics to see it.
# Make a request for just the IP address so it doesn't match a domain
curl -D - http://123.123.123.123
curl: (52) Empty reply from server
# Check metrics to see if it poured (matched a scenario)
sudo cscli metrics
...
...
╭────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ Scenario Metrics │
├──────────────────────────────────────┬───────────────┬───────────┬──────────────┬────────┬─────────┤
│ Scenario │ Current Count │ Overflows │ Instantiated │ Poured │ Expired │
├──────────────────────────────────────┼───────────────┼───────────┼──────────────┼────────┼─────────┤
│ crowdsecurity/http-crawl-non_statics │ 1 │ - │ 1 │ 1 │ - │
│ custom/http-probe-aborted │ 1 │ - │ 1 │ 1 │ - │
╰──────────────────────────────────────┴───────────────┴───────────┴──────────────┴────────┴─────────╯
# Check alerts - there won't be any with just one hit so far
sudo cscli alerts list
No active alerts
# Run a series of probes
for X in {1..10}; do curl -D - http://123.123.123.123/$X;done
sudo cscli alert list
╭────┬──────────────────┬───────────────────────────┬─────────┬────────────────────┬───────────┬──────────────────────╮
│ ID │ value │ reason │ country │ as │ decisions │ created_at │
├────┼──────────────────┼───────────────────────────┼─────────┼────────────────────┼───────────┼──────────────────────┤
│ 64 │ Ip:some.ip.addr │ custom/http-probe-aborted │ US │ 1234 SOME NAME │ ban:1 │ 2026-02-05T15:04:11Z │
╰────┴──────────────────┴───────────────────────────┴─────────┴────────────────────┴───────────┴──────────────────────╯
sudo cscli decision list
╭────────┬──────────┬──────────────────┬───────────────────────────┬────────┬─────────┬────────────────────┬────────┬────────────┬──────────╮
│ ID │ Source │ Scope:Value │ Reason │ Action │ Country │ AS │ Events │ expiration │ Alert ID │
├────────┼──────────┼──────────────────┼───────────────────────────┼────────┼─────────┼────────────────────┼────────┼────────────┼──────────┤
│ 709809 │ crowdsec │ Ip:some.ip.addr │ custom/http-probe-aborted │ ban │ US │ 1234 SOME NAME │ 6 │ 3h59m40s │ 64 │
╰────────┴──────────┴──────────────────┴───────────────────────────┴────────┴─────────┴────────────────────┴────────┴────────────┴──────────╯
# And un-ban yourself
sudo cscli decision delete --id 709809
If you’re using a proxy, you’ve probably already configured caddy to log the correct end-user IP address. This will show up in your logs as the difference between remote_ip and client_ip. This is a good time to do that if you haven’t already.
sudo tail -1 /var/log/caddy/site.some.org.log | jq '.request'
# Just the relevant bits pasted in - the first two should be different if your trusted proxy setup is working
"remote_ip": "172.68.70.128",
"client_ip": "74.7.175.138",
"X-Forwarded-For": [
"74.7.175.138"
],
"Cf-Connecting-Ip": [
"74.7.175.138"
],
Cloudflare filters most bad actors for you, but many make it though. Here’s a sample of what it looks like if you haven’t set up trusted proxies. You can see that CrowdSec took action, but it’s a broad brush. It’s blocking the Cloudflare exit node and so removed everyone’s access, not just the attacker.
sudo cscli alert list
╭─────┬────────────────────┬───────────────────────────────────┬─────────┬────────────────────────┬───────────┬─────────────────────────────────────────╮
│ ID │ value │ reason │ country │ as │ decisions │ created_at │
├─────┼────────────────────┼───────────────────────────────────┼─────────┼────────────────────────┼───────────┼─────────────────────────────────────────┤
│ 221 │ Ip:162.158.49.136 │ crowdsecurity/jira_cve-2021-26086 │ IE │ 13335 CLOUDFLARENET │ ban:1 │ 2025-01-22 15:14:34.554328601 +0000 UTC │
│ 187 │ Ip:128.199.182.152 │ crowdsecurity/jira_cve-2021-26086 │ SG │ 14061 DIGITALOCEAN-ASN │ ban:1 │ 2025-01-19 20:50:45.822199509 +0000 UTC │
│ 186 │ Ip:46.101.1.225 │ crowdsecurity/jira_cve-2021-26086 │ GB │ 14061 DIGITALOCEAN-ASN │ ban:1 │ 2025-01-19 20:50:41.699518104 +0000 UTC │
│ 181 │ Ip:162.158.108.104 │ crowdsecurity/http-bad-user-agent │ SG │ 13335 CLOUDFLARENET │ ban:1 │ 2025-01-19 12:39:20.468268327 +0000 UTC │
│ 180 │ Ip:172.70.208.61 │ crowdsecurity/http-bad-user-agent │ SG │ 13335 CLOUDFLARENET │ ban:1 │ 2025-01-19 12:38:36.664997131 +0000 UTC │
╰─────┴────────────────────┴───────────────────────────────────┴─────────┴────────────────────────┴───────────┴─────────────────────────────────────────╯
But even though caddy identifies the correct client_ip, still have a problem. Crowdsec dutifully adds the client_ip to the block list, but their attacks continue. The bouncer can’t separate the good traffic from the bad because it’s layer 4 and only sees the Cloudflare IP.
for X in {1..100}; do curl -D - https://proxied.your.org/$X; done
allen@www:~$ sudo cscli alert list
# Even though you're listed, you can continue probing
╭────┬───────────────────┬───────────────────────────────────┬─────────┬──────────────────────────────┬───────────┬──────────────────────╮
│ ID │ value │ reason │ country │ as │ decisions │ created_at │
├────┼───────────────────┼───────────────────────────────────┼─────────┼──────────────────────────────┼───────────┼──────────────────────┤
│ 66 │ Ip:212.56.42.20 │ crowdsecurity/http-cve-2021-41773 │ US │ 40021 CONTABO-40021 │ ban:1 │ 2026-02-05T16:02:43Z │
│ 65 │ Ip:34.133.180.181 │ crowdsecurity/http-probing │ US │ 396982 GOOGLE-CLOUD-PLATFORM │ ban:1 │ 2026-02-05T15:38:37Z │
│ 64 │ Ip:some.ip.addr │ custom/http-probe-aborted │ US │ 1234 SOME NAME │ ban:1 │ 2026-02-05T15:04:11Z │
╰────┴───────────────────┴───────────────────────────────────┴─────────┴──────────────────────────────┴───────────┴──────────────────────╯
The ideal approach is to tell Cloudflare to stop forwarding traffic from the bad actors. There is a cloudflare-bouncer to do just that. It’s rate limited however, and only suitable for premium clients. There is also the CrowdSec Cloudflare Worker. It’s better, but still suffers from limits for non-premium clients.
Instead, we’ll use the caddy-crowdsec-bouncer. This is a layer 7 (protocol level) bouncer. It works inside Caddy and will block IPs based on the client_ip from the HTTP request.
The fist step is to tell the Local API that we’re going to add a bouncer and generate a key for it. If you’re running the LAPI on your router, you’d do this there.
# I named this after the hostname 'www' but you can use anything you like
sudo cscli bouncers add www-bouncer
Add the module to Caddy
sudo caddy add-package github.com/hslatman/caddy-crowdsec-bouncer
Configure Caddy
#
# Global Options Block
#
{
# Format like this or the syntax will fail.
crowdsec {
api_key 2WXxY440sLxjapyQKlvp6ILdLA/hn/dhoG1ML82+xiw
# Only needed if your LAPI is remote, like on your router
api_url http://192.168.1.1:8080
}
order crowdsec first
}
www.some.org {
crowdsec
root * /var/www/www.some.org
file_server
}
And restart. Make sure you restart as a reload will fail.
sudo systemctl restart caddy.service
#You should see something like this is your system journal
sudo journalctl | grep caddy | grep crowdsec
Feb 05 16:42:23 www caddy[10724]: {"level":"info","ts":1770309743.418141,"logger":"crowdsec","msg":"started","instance_id":"155c21c7"}
Let’s test that probe again. Initially, you’ll get a 404 (not found) but after while of that, it should switch to 403 (access denied)
for X in {1..100}; do curl -D - --silent https://proxied.your.org/$X | grep HTTP; done
HTTP/2 404
HTTP/2 404
...
...
HTTP/2 403
HTTP/2 403
Congrats! after much work you’ve traded 404s for 403s. Was it worth it? Probably. If an adversary’s probe had a chance to find something, it has less of a chance now.
Back when we set up a trusted proxy for caddy we explicitly used the CF-Connecting-IP header. This is because the X-Forwarded-For header is easily spoofed. If you want to give that a test, comment out that line in your caddy config and reload.
sudo vi /etc/caddy/Caddyfile
# Comment out 'client_ip_headers CF-Connecting-IP' in the servers block
sudo systemctl reload caddy
for X in {1..100}; do curl -D - --silent "X-Forwarded-For: 192.168.0.2" https://www.some.org/$X | grep HTTP;done
HTTP/2 404
HTTP/2 404
...
...
HTTP/2 404
HTTP/2 404
No remediation happens. Turns out Cloudflare appends by default, giving you:
sudo tail -f /var/log/caddy/www.some.org.log | jq
"client_ip": "192.168.0.2",
"X-Forwarded-For": [
"192.168.0.2,109.206.128.45"
],
Caddy only uses the X-Forwarded-For when it trusts the upstream proxy, but it takes the first value, which is rather trusting but canonically correct. That’s why we use the CF-Connecting-IP header as francislavoie suggests, as it will always be replaced and not appended.
You don’t need to, but you can configure Cloudflare to “Remove visitor IP headers”. This is counterintuitive, but the notes say “…Cloudflare will only keep the IP address of the last proxy”. In testing, it keeps the last value in the X-Forwarded-For string, and that’s what we’re after. It works for normal and forged headers.
This adds an extra layer of protection - but it’s not clear if it’s worth it. It may add to the proxy time, so use at your own discretion.
The Overview page may look different depending on your plan, so you may have to hunt around for this setting.
Now when you test, you’ll get access denied regardless of your header
for X in {1..100}; do curl -D - --silent "X-Forwarded-For: 192.168.0.2" https://www.some.org/$X | grep HTTP;done
HTTP/2 404
HTTP/2 404
...
...
HTTP/2 403
HTTP/2 403
If you’re generating lots of overflows but no alerts, check your caddy logs for an error like this
level=error msg=“while pushing to api : failed sending alert to LAPI: API error: invalid character ‘\x1f’ looking for beginning of value”
This happens when your client is 1.7 but your LAPI is 1.6, as is the situation right now with OpenWrt if that’s what you’re using as the local hub. You can reverse the roles and use just a bouncer on the router, or install a true multi-server setup with a dedicated hub. You can also just copy the binaries around.
# On OpenWrt
mv /usr/bin/crowdsec /usr/bin/crowdsec.orig
mv /usr/bin/crowdsec-cli /usr/bin/crowdsec-cli
scp -T [email protected]:'/usr/bin/crowdsec /usr/bin/cscli' /usr/bin/
service crowdsec restart
It’s better to block bad-guys at the boarder and CrowdSec on OpenWrt allows you do do that. It’s also more efficient to do it there than on every instance you want to protect.
There’s two parts; the ‘hub’ that downloads and maintains the list of baddies, and the ‘bouncer’ that checks the list every so often and updates firewall tables to block them.
OpenWRT is best served by installing just the bouncer. But if your install is very small, you can run the hub (crowdsec) there as well.
# At the OpenWrt command line
apk update
apk install crowdsec crowdsec-firewall-bouncer luci-app-crowdsec-firewall-bouncer
There will be some error messages as the packages disagree on file ownership, but this seems OK as they don’t use the .yaml files you expect.
The bouncer won’t be running after installation like it would with a normal linux install. You must register the bouncer with the hub and start it like so:
# Note the API key this command generates
cscli bouncers add openwrt
# Edit OpenWRT's config file for the service and configure the bouncer section.
vi /etc/config/crowdsec
config bouncer
option enabled '1'
option ipv4 '1'
option ipv6 '1'
option api_url 'http://localhost:8080/'
option api_key 'XXXXXXXXXXXXXXX'
option deny_action 'drop'
option deny_log '0'
option log_prefix 'crowdsec: '
option log_level 'info'
option filter_input '1'
option filter_forward '1'
list interface 'eth0'
And then restart and check.
service crowdsec restart
service crowdsec-firewall-bouncer restart
# This should now have a long list of rules at the end under "crowdsec-blacklists-CAPI"
nft list ruleset
We installed the luci-gui package that lets you access the bouncer config in the GUI at network -> firewall -> crowdsec-firewall-bouncer. Though beyond being a curiosity, I’ve not found a use for it.
https://docs.crowdsec.net/u/user_guides/multiserver_setup/ https://kroon.email/site/en/posts/2025/10/openwrt-crowdsec/ https://openwrt.org/docs/guide-user/services/crowdsec
These notes are very old at this point, but I kept them in case I ever need to refer back to them.
There are some packages available, but (as of 2022) they are a bit behind and don’t include the config and service files. So let’s download the latest binaries from Crowsec and create our own.
Note: Download the static versions. Alpine uses a differnt libc than other distros.
cd /tmp
wget https://github.com/crowdsecurity/crowdsec/releases/latest/download/crowdsec-release-static.tgz
wget https://github.com/crowdsecurity/cs-firewall-bouncer/releases/latest/download/crowdsec-firewall-bouncer.tgz
tar xzf crowdsec-firewall*
tar xzf crowdsec-release*
rm *.tgz
You cannot use the wizard as it expects systemd and doesn’t support OpenRC. Follow the Binary Install steps from CrowdSec’s binary instrcutions.
sudo apk add bash newt envsubst
cd /tmp/crowdsec-v*
# Docker mode skips configuring systemd
sudo ./wizard.sh --docker-mode
sudo cscli hub update
sudo cscli machines add -a
sudo cscli capi register
# A collection is just a bunch of parsers and scenarios bundled together for convienence
sudo cscli collections install crowdsecurity/linux
We need a netfilter tool so install nftables. If you already have iptables installed you can skip this step and set FW_BACKEND to that below when generating the API keys.
sudo apk add nftables
Now we install the firewall bouncer. There is no static build of the firewall bouncer yet from CrowdSec, but you can get one from Alpine testing (if you don’t want to compile it yourself)
# Change from 'edge' to other versions a needed
echo "http://dl-cdn.alpinelinux.org/alpine/edge/testing" >> /etc/apk/repositories
apk update
apk add cs-firewall-bouncer
Now configure the bouncer. We will once again do this manually becase there is not support for non-systemd linuxes with the install script. But cribbing from their install script, we see we can:
cd /tmp/crowdsec-firewall*
BIN_PATH_INSTALLED="/usr/local/bin/crowdsec-firewall-bouncer"
BIN_PATH="./crowdsec-firewall-bouncer"
sudo install -v -m 755 -D "${BIN_PATH}" "${BIN_PATH_INSTALLED}"
CONFIG_DIR="/etc/crowdsec/bouncers/"
sudo mkdir -p "${CONFIG_DIR}"
sudo install -m 0600 "./config/crowdsec-firewall-bouncer.yaml" "${CONFIG_DIR}crowdsec-firewall-bouncer.yaml"
Note: If you used the APK, just do the first two lines to get the API_KEY (echo $API_KEY) and manually edit the file (vim /etc/crowdsec/bouncers/crowdsec-firewall-bouncer.yaml)
cd /tmp/crowdsec-firewall*
CONFIG_DIR="/etc/crowdsec/bouncers/"
SUFFIX=`tr -dc A-Za-z0-9 </dev/urandom | head -c 8`
API_KEY=`sudo cscli bouncers add cs-firewall-bouncer-${SUFFIX} -o raw`
FW_BACKEND="nftables"
API_KEY=${API_KEY} BACKEND=${FW_BACKEND} envsubst < ./config/crowdsec-firewall-bouncer.yaml | sudo install -m 0600 /dev/stdin "${CONFIG_DIR}crowdsec-firewall-bouncer.yaml"
sudo touch /etc/init.d/crowdsec
sudo chmod +x /etc/init.d/crowdsec
sudo rc-update add crowdsec
sudo vim /etc/init.d/crowdsec
#!/sbin/openrc-run
command=/usr/local/bin/crowdsec
command_background=true
pidfile="/run/${RC_SVCNAME}.pid"
depend() {
need localmount
need net
}
Note: If you used the package from Alpine testing above it came with a service file. Just rc-update add cs-firewall-bouncer and skip this next step.
sudo touch /etc/init.d/cs-firewall-bouncer
sudo chmod +x /etc/init.d/cs-firewall-bouncer
sudo rc-update add cs-firewall-bouncer
sudo vim /etc/init.d/cs-firewall-bouncer
#!/sbin/openrc-run
command=/usr/local/bin/crowdsec-firewall-bouncer
command_args="-c /etc/crowdsec/bouncers/crowdsec-firewall-bouncer.yaml"
pidfile="/run/${RC_SVCNAME}.pid"
command_background=true
depend() {
after firewall
}
Start up the services and view the logs to see that everything started properly
sudo service start crowdsec
sudo service cs-firewall-bouncer status
sudo tail /var/log/crowdsec.log
sudo tail /var/log/crowdsec-firewall-bouncer.log
# The firewall bouncer should tell you about how it's inserting decisions it got from the hub
sudo cat /var/log/crowdsec-firewall-bouncer.log
time="28-06-2022 13:10:05" level=info msg="backend type : nftables"
time="28-06-2022 13:10:05" level=info msg="nftables initiated"
time="28-06-2022 13:10:05" level=info msg="Processing new and deleted decisions . . ."
time="28-06-2022 14:35:35" level=info msg="100 decisions added"
time="28-06-2022 14:35:45" level=info msg="1150 decisions added"
...
...
# If you are curious about what it's blocking
sudo nft list table crowdsec
...
Back in the old days, you’ve have one really big server, EMC appliance or similar. You’d access it over the network and things were fine. There were expensive hardware solutions if you needed more uptime. You went vertical.
This is still the way for most home users. A turn-key NAS, TrueNAS or OMV solution is pretty decent. It’s centralized and reliable. The best of these use ZFS and you can get good results on commodity hardware.
But if you want to reboot the hardware without causing an outage, you’ll have to have more than one box. For file storage, that means a distributed system like Ceph, Gluster, DRBD or one of the many other entrants.
If you want to be cutting edge, ditch the filesystem altogether for object storage like RustFS rsync or minio. Though you’ll need to pair that with rclone or juicefs for programs that don’t’ talk S3 web object language (which is most things).
A modern, non-enterprise, backup solution for an individual client should be:
For built-in solutions, Apple has Time Machine, Windows has File History (and Windows Backup), and Linux has…well, a lot of things.
Rsync is a perennial favorite and a good (short) historical post on the evolution of rsync style backups is here over at Reddit. Though I hesitate to think of it as backup, because it meets none of the above features.
<https://www.reddit.com/r/linux/comments/42feqz/i_asked_here_for_the_optimal_backup_solution_and/czbeuby?
Duplicity is a well established traditional solution that supports Amazon Cloud Drive, but it relies on generational methods meaning a regular full backup is required. That’s resource intensive with large data sets.
Restic is interesting, but doesn’t work with many cloud providers; specifically Amazon Cloud Drive
https://github.com/restic/restic/issues/212
Obnam and Borg are also interesting, but similarly fail with Amazon Cloud Drive.
restic panic: rename
https://www.bountysource.com/issues/23684796-storage-backend-amazon-cloud-drive
Duplicati supports ACD as long as you’re willing to add mono. Though it’s still beta as of this writing.
sudo /usr/lib/duplicati/Duplicati.Server.exe --webservice-port=8200 --webservice-interface=any
And some other background.
<http://silverskysoft.com/open-stack-xwrpr/2015/08/the-search-for-the-ideal-backup-tool-part-1-of-2/>
<http://changelog.complete.org/archives/9353-roundup-of-remote-encrypted-deduplicated-backups-in-linux>
<http://www.acronis.com/en-us/resource/tips-tricks/2004/generational-backup.html>
Not backup, it’s simply coping data between multiple locations. More like mirroring.
This is used enough that it deserves several pages.
If you regularly copy lots of files it’s best to use rsync. It’s efficient, as it only copies what you need, and secure, being able to use SSH. Many other tools such as BackupPC, Duplicity etc. use rsync under the hood, and when you are doing cross-platform data replication it may be the only tool that works, so you’re best to learn it.
Generally, it’s 10% slower than just using cp -a. Sometimes start with that and finish up with this.
rsync \
--archive \
--delete \
--dry-run \
--human-readable \
--inplace \
--itemize-changes \
--progress \
--verbose \
/some/source/Directory \
/some/destination/
The explanations of the more interesting options are:
--archive: Preserves all the metadata, as you'd expect
--delete : Removes extraneous files at the destination that no longer exist at the source (i.e. _not_ a merge)
--dry-run: Makes no changes. This is important for testing. Remove for the actual run
--inplace: This overwrites the file directly, rather than the default behavior that is to build a copy on the other end before moving it into place. This is slightly faster and better when space is limited (I've read)
If you don’t trust the timestamps at your destination, you can add the --checksum option, though when you’re local this may be slower than just recopying the whole thing.
A note about trailing slashes: In the source above, there is no trailing slash. But we could have added one, or even a /*. Here’s what happens when you do that.
This uses SSH for encryption and authentication.
rsync \
--archive \
--delete \
--dry-run \
--human-readable \
--inplace \
--itemize-changes \
--progress \
--verbose \
/srv/Source_Directory/* \
[email protected]:/srv/Destination_Directory
One easy way to do this is to grab a bundled version of rsync and ssh for windows from the cwRsync folks
<https://www.itefix.net/content/cwrsync-free-edition>
Extract the standalone client to a folder and edit the .cmd file to add this at the end (the ^ is the windows CRNL escape)
rsync ^
--archive ^
--delete ^
--dry-run ^
--human-readable ^
--inplace ^
--itemize-changes ^
--no-group ^
--no-owner ^
--progress ^
--verbose ^
--stats ^
[email protected]:/srv/media/video/movies/* /cygdrive/D/Media/Video/Movies/
pause
The version that comes with recent versions of OS X is a 2.6.9 (or so) variant. You can use that, or obtain the more recent 3.0.9 that has some slight speed improvements and features. To get the newest (you have to build it yourself) install brew, then issue the commands:
brew install https://raw.github.com/Homebrew/homebrew-dupes/master/rsync.rb
brew install rsync
One of the issues with syncing between OS X and Linux is the handling of Mac resource forks (file metadata). Lets assume that you are only interested in data files (such as mp4) and are leaving out the extended attributes that apple uses to store icons and other assorted data (replacing the old resource fork).
Since we are going between file systems, rather than use the ‘a’ option that preserves file attributes, we specify only ‘recursive’ and ’times’. We also use some excludes keep mac specific files from tagging along.
/usr/local/bin/rsync
--exclude .DS*
--exclude ._*
--human-readable
--inplace
--progress
--recursive
--times
--verbose
--itemize-changes
--dry-run
"/Volumes/3TB/source/"
[email protected]:"/Volumes/3TB/"
Importantly, we are ‘itemizing’ and doing a ‘dry-run’. When you do, you will see a report like:
skipping non-regular file "Photos/Summer.2004"
skipping non-regular file "Photos/Summer.2005"
.d..t....... Documents/
.d..t....... Documents/Work/
cd++++++++++ ISOs/
<f++++++++++ ISOs/Office.ISO
The line with cd+++ indicate a directory will be created and <f+++ indicate a file is going to be copied. When it says ‘skipping’ a non regular file, that’s (in this case, at least) a symlink. You can include them, but if your paths don’t match up on both systems, these links will fail.
Generally you quote and escape.
rsync
--archive ^
--itemize-changes ^
--progress ^
[email protected]:"/srv/media/audio/Music/Basil\ Poledouris" ^
/cygdrive/c/Users/Allen/Music
Though it’s rumored that you can single quote and escape with the –protect-args option
--protect-args ^
[email protected]:'/srv/media/audio/Music/Basil Poledouris' ^
You may want to combine find and rsync to get files of a specific criteria. Use the --from-file parameter
ssh server.gattis.org find /srv/media/video -type f -mtime -360 > list
rsync --progress --files-from=list server.gattis.org:/ /mnt/media/video/
If you have no data on the destination to begin with, rsync will be somewhat slower than a straight copy. On a local system simply use ‘cp -a’ (to preserve file times). On a remote system, you can use tar to minimize the file overhead.
tar -c /path/to/dir | ssh remote_server 'tar -xvf - -C /absolute/path/to/remotedir'
It is also possible to use rsync with the option --whole-file and this will skip the things that slow rsync down though I have not tested it’s speed
Rsync uses time and size to determine if a file should be updated. If you have already copied files and you are trying to do a sync, you may find your modification times are off. Add the –size-only or the –modify-window=NUM. Even better, correct your times. (this requires on OS X the coreutils to get the GNU ls command and working with the idea here)
http://notemagnet.blogspot.com/2009/10/getting-started-with-rsync-for-paranoid.html http://www.chrissearle.org/blog/technical/mac_homebrew_and_homebrew_alt http://ubuntuforums.org/showthread.php?t=1806213
Some low-power devices, such as the Raspbery Pi, struggle with the encryption overheard of rsync default network transport, ssh.
If you don’t need encryption or authentication, you can significantly speed things up by using rsync in daemon mode.
In this example, we’ll push data from our server to the low-power client.
Create a config file on the sever that we’ll send over to the client later.
nano client-rsyncd.conf
log file = /var/log/rsync.log
pid file = /var/run/rsyncd.pid
lock file = /var/run/rsync.lock
# This is the name you refer to in rsync. The path is where that maps to.
[media]
path = /var/media
comment = Media
read only = false
timeout = 300
uid = you
gid = you
The default port is hi-level and doesn’t require root privileges.
# Send the daemon config over to the home dir
scp client-rsyncd.conf [email protected]:
# Launch rsync in daemon mode
ssh [email protected]: rsync --daemon --config ./client-rsyncd.conf
# Send the data over
rsync \
--archive \
--delete \
--human-readable \
--inplace \
--itemize-changes \
--no-group \
--no-owner \
--no-perms \
--omit-dir-times \
--progress \
--recursive \
--verbose \
--stats \
/mnt/pool01/media/movies rsync://client.some.lan:8730/media
# Terminate the remote instance
ssh [email protected] killall rsync
One common task is to rsync through a bastion host to an internal system. Do it with the rsync shell options
rsync \
--archive \
--delete \
--delete-excluded \
--exclude "lost+found" \
--human-readable \
--inplace \
--progress \
--rsh='ssh -o "ProxyCommand ssh [email protected] -W %h:%p"' \
--verbose \
[email protected]:/srv/plex/* \
/data/
There is a -J or ProxyJUmp option on new versions of SSH as well.
https://superuser.com/questions/964244/rsyncing-directories-through-ssh-tunnel https://unix.stackexchange.com/questions/183951/what-do-the-h-and-p-do-in-this-command https://superuser.com/questions/1115715/rsync-files-via-intermediate-host
It’s usually best to wrap rsync and call it from cron. Preferably with something that doesn’t step on itself for long running syncs, like this:
vi ~/bin/schedule-rsync
#!/bin/bash
THE_USER="remote-account-1"
THE_KEY="remote-account-1"
SCRIPT_NAME=$(basename "$0")
PIDOF=$(pidof -x $SCRIPT_NAME)
for PID in $PIDOF; do
if [ $PID != $$ ]; then
echo "[$(date)] : $SCRIPT_NAME : Process is already running with PID $PID"
exit 1
fi
done
# Change to working directory. Assume running as non-root user per cronfile config below
cd ~/bin
rsync \
--archive \
--bwlimit=5m \
--delete \
--delete-excluded \
--exclude .DS* \
--exclude ._* \
--human-readable \
--inplace \
--itemize-changes \
--no-group \
--no-owner \
--no-perms \
--progress \
--recursive \
--rsh "ssh -i ${THE_KEY}" \
--verbose \
--stats \
${THE_USER}@some.server.org\
:/mnt/pool01/folder.1 \
:/mnt/pool01/folder.2 \
:/mnt/pool01/folder.2 \
/mnt/pool02/
Then, call it from a file in the cron drop folder.
echo "0 1 * * * $USER /home/$USER/schedule-rsync >> /home/$USER/bin/rsync-video.log 2>&1" > /etc/cron.d/schedule-rsync
You’d like to use rsync, but ensure users can only use rsync and can’t login with a shell, forward sessions, or other shenanigans. Do this with ssh keys and ForceCommand.
# On your server, add a central location for keys
sudo mkdir /etc/ssh/authorized_keys
# Configure SSH to look for user public keys in that spot - the %u is the variable for user ID
echo "AuthorizedKeysFile /etc/ssh/authorized_keys/%u.pub" > /etc/ssh/sshd_config.d/authorized_users.conf
# Create a script that checks incoming ssh commands to make sure they are for rsync
sudo tee /etc/ssh/authorized_keys/checkssh.sh << "EOF"
#!/bin/bash
if [ -n "$SSH_ORIGINAL_COMMAND" ]; then
if [[ "$SSH_ORIGINAL_COMMAND" =~ ^rsync\ ]]; then
echo $SSH_ORIGINAL_COMMAND | systemd-cat -t rsync
exec $SSH_ORIGINAL_COMMAND
else
echo DENIED $SSH_ORIGINAL_COMMAND | systemd-cat -t rsync
fi
fi
EOF
chmod +x /etc/ssh/authorized_keys/checkssh.sh
systemctl restart ssh.service
Now that we have the SSH server configured, let’s create a system user (required, unfortunately) and a key. We’ll limit the account as much as possible, though you can’t use /usr/sbin/nologin shell, as rsync requires something to run in.
THE_USER="remote-account-1"
sudo adduser --no-create-home --home /nonexistent --disabled-password --gecos "" ${THE_USER}
# Its easiest to create the key yourself, but a .pub from them is also fine.
# Send out the private key from the folder (it's the one without the .pub on the end) to the remote system.
ssh-keygen -f /etc/ssh/authorized_keys/${THE_USER} -q -N "" -C "${THE_USER}"
Let’s add a ForcedCommand to the key so that it can only be used with the features we allow.
vi /etc/ssh/authorized_keys/${THE_USER}
# Paste this command="... string in front of the existing key
command="/etc/ssh/authorized_keys/checkssh.sh",no-port-forwarding,no-X11-forwarding,no-agent-forwarding,no-pty ssh-rsa AAAAB3NzaC1...
This remote user can now use rsync, but can’t login or do other activities. Their command would look something like this (using the private key you created above)
rsync \
--rsh "ssh -i /where/your/private/key/is/remote-account-1" \
[email protected]:/some/folder /some/local/place/
Why not use rrsync?
The rrsync script is similar to the script we use, but is distributed and maintained as part of the rsync package. It’s arguably a better choice. I like the checkssh.sh approach as it’s more flexible, allows for things other than rsync, and doesn’t force relative paths. But if you’re only doing rsync, consider using rrsync like this;
# Paste this command="... string in front of the existing key
command="rrsync -ro /some/folder/share",no-port-forwarding,no-X11-forwarding,no-agent-forwarding,no-pty ssh-rsa AAAAB3NzaC1...
In your client’s rsync command, make the paths relative to path rrsync expects above.
rsync [email protected]:folder.1 /destination/folder/
If you see the client-side error message:
rrsync error: option -L has been disabled on this server
You discovered that following symlinks has been disabled by default in rrsync. You can enable with an edit to the script.
sudo sed -i 's/KLk//' /usr/bin/rrsync
# This changes
# short_disabled_subdir = 'KLk'
to
# short_disabled_subdir = ''
bash: line 1: /usr/bin/rrsync: cannot execute: required file not found
Python3 is needed for later versions of the script. 26M more in your container, but hey, everyone loves python.
apt install python3
If you need do it more than once, it might look something like this.
#!/bin/bash
HELP_MESSAGE="Usage: $0 <user> \n\nThis script requires a username to be specified.\n"
if [ "$#" -eq 0 ]; then
echo -e "$HELP_MESSAGE"
exit 1
fi
if [ id username &>/dev/null ]; then
echo "User already exists."
exit 1
fi
if [ "$EUID" -ne 0 ]; then
echo "This script must be run with sudo."
exit 1
fi
THE_USER=$1
THE_COMMAND="\
command=\
\"/etc/ssh/authorized_keys/checkssh.sh\",\
no-port-forwarding,\
no-X11-forwarding,\
no-agent-forwarding,\
no-pty "
useradd --home-dir /nonexistent ${THE_USER}
mkdir -p /etc/ssh/authorized_keys
ssh-keygen -f /etc/ssh/authorized_keys/${THE_USER} -q -N "" -C "${THE_USER}"
sed -i "1s|^|$THE_COMMAND|" /etc/ssh/authorized_keys/${THE_USER}.pub
https://peterbabic.dev/blog/transfer-files-between-servers-using-rrsync/ http://gergap.de/restrict-ssh-to-rsync.html https://superuser.com/questions/641275/make-linux-server-allow-rsync-scp-sftp-but-not-a-terminal-login
Syncthing synchronizes files between any number of computers using peer-to-peer technology and encryption.
When deploying in a container, using apt-based install will pull in 400+ Mb of x11 and such libraries. Download the latest binary release from github instead.
https://github.com/syncthing/syncthing/releases
Extract and mv to /usr/bin, chown to yourself so it can update, copy a service unit file to /etc/systemd/system, and enable
[Unit]
Description=Syncthing - Open Source Continuous File Synchronization for %I
Documentation=man:syncthing(1)
After=network.target
StartLimitIntervalSec=60
StartLimitBurst=4
[Service]
User=%i
Environment="STLOGFORMATTIMESTAMP="
Environment="STLOGFORMATLEVELSTRING=false"
Environment="STLOGFORMATLEVELSYSLOG=true"
ExecStart=/usr/bin/syncthing serve --no-browser --no-restart
Restart=on-failure
RestartSec=1
SuccessExitStatus=3 4
RestartForceExitStatus=3 4
# Hardening
ProtectSystem=full
PrivateTmp=true
SystemCallArchitectures=native
MemoryDenyWriteExecute=true
NoNewPrivileges=true
# Elevated permissions to sync ownership (disabled by default),
# see https://docs.syncthing.net/advanced/folder-sync-ownership
#AmbientCapabilities=CAP_CHOWN CAP_FOWNER
[Install]
WantedBy=multi-user.target
sudo systemctl enable [email protected]
sudo systemctl start [email protected]
AKA - The Fastest Way to Copy Files.
When you don’t want to copy a whole file system, many admins suggest the fastest way is with a ’tar pipe'.
From one disk to another on the same system. This uses pv to buffer.
(cd /src; tar cpf - .) | pv -trab -B 500M | (cd /dst; tar xpf -)
You can add netcat to the mix (as long as you don’t need encryption) to get it across the network.
On the receiver:
(change to the directory you want to receive the files or directories in)
nc -l -p 8989 | tar -xpzf -
On the sender:
(change to the directory that has the file or directory - like ‘pics’ - in it)
tar -czf - pics | nc some.server 8989
This takes the place of pc and nc and is somewhat faster.
On the receiving side
mbuffer -4 -I 9090 | tar -xf -
On the sending side
sudo tar -c plexmediaserver | mbuffer -m 1G -O SOME.IP:9090
You can use ssh when netcat isn’t appropriate or you want to automate with a SSH key and limited interaction with the other side. This examples ‘pulls’ from a remote server.
(ssh [email protected] tar -czf - /srv/http/someSite) | (tar -xzf -)
If you already have a NFS server on one of the systems though, it’s basically just as fast. At least in informal testing, it behaves more steadily as opposed to a tar pipe’s higher peaks and lower troughs. A simple cp -a will suffice though for lots of little files a tar pipe still may be faster.
rsync is generally best if you can or expect the transfer to be interrupted. In my testing, rsync achieved about 15% less throughput with about 10% more processor overhead.
http://serverfault.com/questions/43014/copying-a-large-directory-tree-locally-cp-or-rsync http://unix.stackexchange.com/questions/66647/faster-alternative-to-cp-a http://serverfault.com/questions/18125/how-to-copy-a-large-number-of-files-quickly-between-two-servers
Unison offers several features that make it more useful than rsync;
Rsync is good at one-way synchronization. i.e. one to many. But when you need to sync multiple authoritative systems, i.e. many to many, you want to use unison. It allows you to merge changes.
Another problem with rsync is that when you rename a file, it re-sends it. This is because a re-named file appears ’new’ to the sync utility. Unison however, maintains a hash of every file you’ve synced and if there is already a local copy (i.e. the file before you renamed it), it will use that and do a ’local copy’ rather than sending it. So a rename effectively is a local copy and a delete. Not perfect, but better than sending it across the wire.
Unison uses it’s own implementation of the rsync delta copy algorithm. However, for large files the authors recommend an option that wraps rsync itself as you can optimize it for large files. Use Unison can use config files in your ~/.unison folder. If you type ‘unison’ without any arguments, it will use the ‘default.prf’ file. Here is a sample
# Unison preferences file
# Here are the two server 'roots' i.e., the start of where we will pick out things to sync.
# The first root is local, and the other remote over ssh
root = /mnt/someFolder
root = ssh://[email protected]//mnt/someFolder
# The 'path' is simply the name of a folder or file you want to sync. Notice the spaces are preserved. No not excape them.
path = A Folder Inside someFolder
# We're 'forcing' the first root to win all conflicts. This sort of negates the multi-way
# sync feature but it's just an example
force = /mnt/someFolder
# This instructs unison to copy the contents of sym links, rather than the link itself
follow = Regex .*
# You can also ignore files and paths explicitly or pattern. See the 'Path specification'
ignore = Name .AppleDouble
ignore = Name .DS_Store
ignore = Name .Parent
ignore = Name ._*
# Here we are invoking an external engine (rsync) when a file is over 10M, and passing it some arguments
copythreshold = 10000
copyprog = rsync --inplace
copyprogrest = rsync --partial --inplace
Hostname is important. Unison builds a hash of all the files to determine what’s changed (similar to md5sum with rsync, but faster). If you get repeated messages about ‘…first time being run…’ you may have an error in your path
http://www.cis.upenn.edu/~bcpierce/unison/download/releases/stable/unison-manual.html
You man want to use newer BTRFS features on an older OS.
With debian your choices are:
It’s often recommended to install from backports. These are newer versions of apps that have been explicitly taken out of testing and packaged for the stable release. i.e. if you’re running ‘buster’ you would install from buster-backports.
It’s possible that one would use the identifiers ‘stable’ and ’testing’ which pegs you to whatever is current, rather than to a specific release.
echo deb http://deb.debian.org/debian buster-backports main | sudo tee /etc/apt/sources.list.d/buster-backports.list
sudo apt update
# seach for the most recent amd64 image.
sudo apt search -t buster-backports linux-image-5
sudo apt install -t buster-backports linux-image-5.2.0-0.bpo.3-amd64-unsigned
sudo apt install -t buster-backports btrfs-progs
If there’s no backport, you can install from the release candidate. These are simply upcoming versions of debian that haven’t been released yet
To install the kernel from the experimental version of debian, add the repo to your sources and explicitly add the kernel (this is safe to add to your repos because experimental packages aren’t installed by default)
sudo su -c "echo deb http://deb.debian.org/debian experimental main > /etc/apt/sources.list.d/experimental.list"
sudo apt update
sudo apt -t experimental search linux-image
sudo apt -t experimental install linux-image-XXX
sudo apt -t experimental install btrfs-progs
You can also download the packages and manually install.
Navigate to
And download, similar to this (from a very long time ago :-)
wget https://kernel.ubuntu.com/~kernel-ppa/mainline/v5.0/linux-headers-5.0.0-050000_5.0.0-050000.201903032031_all.deb
wget https://kernel.ubuntu.com/~kernel-ppa/mainline/v5.0/linux-headers-5.0.0-050000-generic_5.0.0-050000.201903032031_amd64.deb
wget https://kernel.ubuntu.com/~kernel-ppa/mainline/v5.0/linux-image-unsigned-5.0.0-050000-generic_5.0.0-050000.201903032031_amd64.deb
wget https://kernel.ubuntu.com/~kernel-ppa/mainline/v5.0/linux-modules-5.0.0-050000-generic_5.0.0-050000.201903032031_amd64.deb
The following signatures couldn’t be verified because the public key is not available:
sudo apt-key adv –recv-key –keyserver keyserver.ubuntu.com 648ACFD622F3D138
https://unix.stackexchange.com/questions/432406/install-the-latest-rc-kernel-on-debian https://wiki.debian.org/HowToUpgradeKernel https://www.tecmint.com/upgrade-kernel-in-ubuntu/ https://raspberrypi.stackexchange.com/questions/12258/where-is-the-archive-key-for-backports-debian-org/60051#60051
Ceph is a distributed object storage system that supports both replicated and erasure coding of data. So you can design it to be fast or efficient. Or even both.
It’s complex compared to other solutions, but it’s also the main focus of Redhat and other development. So it may eclipse other technologies just through adaption.
It also comes ‘baked-in’ with Proxmox. If you’re already using PVE it’s worth deploying over others, as long as you’re willing to devote the time to learning it.
There are two main use-cases;
They both provide High Availability. But VMs need speed whereas Bulk should be economical. What makes Ceph awesome is you can do both - all with the same set of disks.
Put 3 or 4 PCs on a LAN. Have at least a couple HDDs in addition to a boot drive. 1G of RAM per TB of disk is recommended1 and it will use it. You can have less RAM, it will just be slower.
The speed of Ceph is essentially a third or half your network speed. With a 1 Gig NIC you’ll average around 60 MB/Sec for file operations (multiple copies are being saved behind the scenes). This sounds terrible, but in reality its fine for a cluster that serves up small files and/or media streams. But it will take a long time to get data onto the system.
If you need more, install a secondary NIC for Ceph traffic. Do that before you configure the system, if you can. Doing it after is hard. You can use a mesh config via the PVE docs2 or purchase a switch. Reasonable 2.5 Gb switches and NICs can now be had.
Install PVE and cluster the servers.
Double check the Ceph repo is current by comparing what you have enabled:
grep ceph /etc/apt/sources.list.d/*
Against what PVE has available.
curl -s https://enterprise.proxmox.com/debian/ | grep ceph
If you don’t have the latest, take a look at Install PVE to update the Ceph repo.
After that, log into the PVE web GUI, click on each server and in that server’s submenu, click on Ceph. It will prompt for permission to install. When the setup windows appears, select the newest version and the No-Subscription repository. You can also refer to the official notes.
The wizard will ask some additional configuration options for which you can take the defaults and finish. If you have an additional ceph-specific network hardware, set it up with a separate IP range and choose interface for both the public network and cluster network.
Ceph uses multiple daemons on each node.
To configure them, you will:
Ceph recommends at least three monitors3 and manager processes4. To install, click on a server’s Ceph menu and select the Monitor menu entry. Add a monitor and manager to the first three nodes.
The next step is to add disks - aka Object Storage Devices. Select a server from the left-hand menu, and from that server’s Ceph menu, select OSD. Then click Create: OSD, select the disks and click create. If they are missing, enter the server’s shell and issue the command wipefs -a /dev/sdX on the disks in question.
If you have a lot of disks, you can also do this at the shell
# Assuming you have 5 disks, starting at 'b'
for X in {b..f}; do echo pveceph osd create /dev/sd$X; done
To add MDSs, click on the CephFS submenu for each server and click Create under the Metadata Servers section. Don’t create a CephFS yet, however.
We are going to create two pools; a replicated pool for VMs and an erasure coded pool for bulk storage.
If you want to mix SSDs and HDDs see the storage tiering page before creating the pools. You’ll want to set up classes of storage and create pools based on that page.
On this pool we’ll use the default replication level that gives us three copies of everything. These are guaranteed at a per-host level. Loose any one or two hosts, no problem. But loose individual disks from all three hoots at the same time and you’re out of luck.
This is the GUI default so creation is easy. Navigate to a host and select Ceph –> Pools –> Create. The defaults are fine and all you need do is give it a name, like “VMs”. You may notice there is already a .mgr pool. That’s created by the manager service and safe to ignore.
If you only need storage for VMs and containers, you can actually stop here. You can create VMs directly on your new pool, and containers on local storage then migrate (Server –> Container –> Resources –> Root Disk -> Volume Action –> Target Storage)
Erasure coding requires that you determine how many data and parity bits to use, and issue the create command in the terminal. For the first question, it’s pretty simple - if you have three servers you’ll need 2 data and 1 parity. The more systems you have the more efficient you’ll be, though when you get to 6 you should probably increase your parity. Unlike the replicated pool, you can only loose one host with this level of protection.
Here’s the command5 for a 3 node system that can withstand one node loss (2,1). For a 4 node system you’d use (3,1) and so on. Increase the first number as your node count goes up, and the second as you desire more resilience. Ceph doesn’t require a ‘balanced’ cluster in terms of drives, but you’ll loose some capacity if you don’t have roughly the same amount of space on each node.
# k is for data and m is for parity
pveceph pool create POOLNAME --erasure-coding k=2,m=1 --pg_autoscale_mode on --application cephfs
Note that we specified application in the command. If you don’t, you won’t be able to use it for a filesystem later on. We also specified PGs (placement groups) as auto-scaling. This is how Ceph chunks data as it gets sent to storage. If you know how much data you have to load, you can specify the starting number of PGs with the --pg_num parameter. This will make things a little faster for an initial copy. Redhat suggests6 the OSD*100 / K+M. You’ll get a warning7 from Ceph if it’s not a power of 2 (2, 4, 8, 16, 32, 64, 128, 256, 512) so use the closest number, such as --pg_num 512.
If you don’t know how much data you’re going to bing in, but expect it to be a lot, you can turn on the bulk flag, rather than specifying pg_num.
# Make sure to add '-data' at the end
ceph osd pool set POOLNAME-data bulk true
When you look at the pools in the GUI you’ll see it also created two pools, one for data and one for metadata, which isn’t compatible with EC pools yet. You’ll also notice that you can put VMs and containers on this pool just like the replicated pool. It will just be slower.
The GUI won’t allow you to choose the erasure coded pool you just created so you’ll use the command line again. The name you pick for your Filesystem will be how it’s mounted.
ceph osd pool set POOLNAME-data bulk true
ceph fs new FILE-SYSTEM-NAME POOLNAME-metadata POOLNAME-data --force
To mount it cluster-wide, go back to the Web GUI and select Datacenter at the top left, then Storage. Click Add and select CephFS as the type. In the subsequent dialog box, put the name you’d like it mounted as in the ID field, such as “bulk” and leave the rest at their defaults.
You can now find it mounted at /mnt/pve/IDNAME and you can bind mount it to your Containers or setup NFS for your VMs.
Ceph defaults to a failure domain of ‘host’. That means you can loose a whole host with all it’s disks and continue operating. You can also lose individual disks from different hosts, but operations are NOT guaranteed. If you have two copies, as well and continue operating. After a short time, Ceph will re-establish parity as disks fail or hosts remain off-line. Should they come back, it will re-adjust. Though in both cases this can take some time.
Ceph immediately panics and starts reestablishing resilience. When the host comes back up, it starts redoing it back. This is OK, but Redhat suggests to avoid it with a few steps.
On the node you want to reboot:
sudo ceph osd set noout
sudo ceph osd set norebalance
sudo reboot
# Log back in and check that the pgmap reports all pgs as normal (active+clean).
sudo ceph -s
# Continue on to the next node
sudo reboot
sudo ceph -s
# When done
sudo ceph osd unset noout
sudo ceph osd unset norebalance
# Perform a final status check to make sure the cluster reports HEALTH_OK:
sudo ceph status
If you created a pool but left off the –application flag it will be set to RDP by default. You’d have to change it from RDP to CephFS like so, for both the data and metadata
ceph osd pool application enable srv-data cephfs --yes-i-really-mean-it
ceph osd pool application disable srv-data rdb --yes-i-really-mean-it
ceph osd pool application enable srv-metadata cephfs --yes-i-really-mean-it
ceph osd pool application disable srv-metadata rdb --yes-i-really-mean-it
If you want to change your IP addresses, you may be able to just change the public network in the /etc/pve/ceph.conf and then destroy and recreate what it says. This worked. I don’t know if it’s good. I think the OSD cluster network needed changed also.
Based on https://www.reddit.com/r/Proxmox/comments/p7s8ne/change_ceph_network/
https://docs.ceph.com/en/mimic/start/hardware-recommendations/#hard-disk-drives ↩︎
https://pve.proxmox.com/wiki/Full_Mesh_Network_for_Ceph_Server ↩︎
https://docs.ceph.com/en/reef/rados/configuration/mon-config-ref/#initial-members ↩︎
https://docs.ceph.com/en/reef/mgr/administrator/#high-availability ↩︎
https://docs.redhat.com/en/documentation/red_hat_ceph_storage/4/html/storage_strategies_guide/placement_groups_pgs#calculating_pg_count ↩︎
https://docs.ceph.com/en/reef/rados/operations/health-checks/#pool-pg-num-not-power-of-two ↩︎
If you have a mix of workloads you should create a mix of pools. Cache tiering is out1. So use a mix of NVMEs, SSDs, and HDDs with rules to control what pool uses what class of device.
In this example, we’ll create a replicated SSD pool for our VMs, and a erasure coded HDD pool for our content and media files.
When an OSD is added, its device class is automatically assigned. The typical ones are ssd or hdd. But either way, the default config will use them all as soon as you add them. Let’s change that by creating some additional rules
For replicated data, it’s as easy as creating a couple new rules and then migrating data, if you have any.
If you haven’t yet created any pools, great! we can create the rules so they are available when creating pools. Add all your disks as OSDs (visit the PVE ceph menu for each server in the cluster). Then add these rules at the command line of any PVE server to update the global config.
# Add rules for both types
#
# The format is
# ceph osd crush rule create-replicated RULENAME default host CLASSTYPE
#
ceph osd crush rule create-replicated replicated_hdd_rule default host hdd
ceph osd crush rule create-replicated replicated_ssd_rule default host ssd
And you’re done! When you create a pool in the PVE GUI, click the advanced button and choose the appropriate CRUSH rule from the drop-down. Or you can create one now while you’re at the command line.
# Create a pool for your VMs on replicated SSD. Default replication is used (so 3 copies)
# pveceph pool create POOLNAME --crush_rule RULENAME --pg_autoscale_mode on
pveceph pool create VMs --crush_rule replicated_ssd_rule --pg_autoscale_mode on
With an existing system you must migrate your data. If you’ve haven’t added your SSDs yet, do so now. It will start moving data using the default rule, but we’ll apply a new rule that will take over.
# Add rules for both types
ceph osd crush rule create-replicated replicated_hdd_rule default host hdd
ceph osd crush rule create-replicated replicated_ssd_rule default host ssd
# If you've just added SSDs, apply the new rule right away to minimize the time spent waiting for data moves.
# Use the SSD or HDD rule as you prefer. In this example we're moving POOLNAME to SSDs
ceph osd pool set VMs crush_rule replicated_sdd_rule
EC data is a little different. You need a profile to describe the resilience and class, and Ceph manages the CRUSH rule directly. But you can have the pveceph to do this for you.
# Create pool name 'Content' with 2 data and 1 parity. Add --application cephfs as we're using this for file storage. The --crush_rule affects the metadata pool so its on fast storage.
pveceph pool create Content --erasure-coding k=2,m=1,device-class=hdd --crush_rule replicated_ssd_rule --pg_autoscale_mode on --application cephfs
You’ll notice separate pools for data and metadata were automatically created as the latter doesn’t support EC pools yet.
Normally, you set device class as part of creating a profile and you cannot change the profile after creating the pool2. However, you can change the CRUSH rule and that’s all we need for changing the class.
# Create a new profile that to base a CRUSH rule on. This one uses HDD
# ceph osd erasure-code-profile set PROFILENAME crush-device-class=CLASS k=2 m=1
ceph osd erasure-code-profile set ec_hdd_2_1_profile crush-device-class=hdd k=2 m=1
# ceph osd crush rule create-erasure RULENAME PROFILENAME (from above)
ceph osd crush rule create-erasure erasure_hdd_rule ec_hdd_2_1_profile
# ceph osd pool set POOLNAME crush_rule RULENAME
ceph osd pool set Content-data crush_rule erasure_hdd_rule
Don’t forget about the metadata pool and it’s a good time to turn on bulk setting if you’re going to store a lot of data.
# Put the metadata pool on SSD for speed
ceph osd pool set Content-metadata crush_rule replicated_ssd_rule
ceph osd pool set Content-data bulk true
There’s some reports that NVMe aren’t separated from SSDs. You may need to create that class and turn off auto detection, though this is quite old information.
When investigating a system, you may want to drill down with thees commands.
ceph osd lspools
ceph osd pool get VMs crush_rule
ceph osd crush rule dumpreplicated_ssd_rule
# or view rules with
ceph osd getcrushmap | crushtool -d -
The fastest way is to use a Ceph Client at the source of the data, or at least separate the interfaces.
With 1Gb NIC, one of the Ceph storage servers also connected to an external NFS and coping data to a CephFS.
Same, but reversed with NFS server itself running the Ceph client pushing the data.
# Adjust letters as needed
for X in {b..h}; do pveceph osd create /dev/sd${X};done
mkdir -p /var/lib/ceph/mon/
mkdir /var/lib/ceph/osd
# Adjust numbers as needed
for X in {16..23};do systemctl stop ceph-osd@${X}.service;done
for X in {0..7}; do umount /var/lib/ceph/osd/ceph-$X;done
for X in {a..h}; do ceph-volume lvm zap /dev/sd$X --destroy;done
This assumes you already have a working cluster and a ceph file system.
You need the ceph software. You use the cephadm tool, or add the repos and packages manually. You also need to pick what version by it’s release name; ‘Octopus, Nautilus, etc’
sudo apt install software-properties-common gnupg2
wget -q -O- 'https://download.ceph.com/keys/release.asc' | sudo apt-key add -
# Discover the current release. PVE is a good place to check when at the command line
curl -s https://enterprise.proxmox.com/debian/ | grep ceph
# Note the release name after debian, 'debian-squid' in this example.
sudo apt-add-repository 'deb https://download.ceph.com/debian-squid/ bullseye main'
sudo apt update; sudo apt install ceph-common -y
#
# Alternatively
#
curl --silent --remote-name --location https://github.com/ceph/ceph/raw/squid/src/cephadm/cephadm
chmod +x cephadm
./cephadm add-repo --release squid
./cephadm install ceph-common
On a cluster member, generate a basic conf and keyring for the client
# for a client named 'minecraft'
ceph config generate-minimal-conf > /etc/ceph/minimal.ceph.conf
ceph-authtool --create-keyring /etc/ceph/ceph.client.minecraft.keyring --gen-key -n client.minecraft
You must add file system permissions by adding lines to the bottom of the keyring, then import it to the cluster.
nano /etc/ceph/ceph.client.minecraft.keyring
# Allowing the client to read the root and write to the subdirectory '/srv/minecraft'
caps mds = "allow rwps path=/srv/minecraft"
caps mon = "allow r"
caps osd = "allow *"
Import the keyring to the cluster and copy it to the client
ceph auth import -i /etc/ceph/ceph.client.minecraft.keyring
scp minimal.ceph.conf ceph.client.minecraft.keyring [email protected]:
On the client, copy the keyring and rename and move the basic config file.
ssh [email protected]
sudo cp ceph.client.minecraft.keyring /etc/ceph
sudo cp minimal.ceph.conf /etc/ceph/ceph.conf
Now, you may mount the filesystem
# the format is "User ID" @ "Cluster ID" . "Filesystem Name" = "/some/folder/on/the/server" "/some/place/on/the/client"
# You can get the cluster ID from your server's ceph.conf file and the filesystem name
# with a ceph fs ls, if you don't already know it. It will be the part after name, as in "name: XXXX, ..."
sudo mount.ceph [email protected]=/srv/minecraft /mnt
You can and entry to your fstab like so
[email protected]=/srv/minecraft /mnt ceph noatime,_netdev 0 2
source mount path was not specified
unable to parse mount source: -22
You might have accidentally installed the distro’s older version of ceph. The mount notation above is based on “quincy” aka ver 17
ceph --version
ceph version 17.2.3 (dff484dfc9e19a9819f375586300b3b79d80034d) quincy (stable)
Try an apt remove --purge ceph-common and then apt update before trying a apt install ceph-common again.
**unable to get monitor info from DNS SRV with service name: ceph-mon**
Check your client’s ceph.conf. You may not have the right file in place
**mount error: no mds server is up or the cluster is laggy**
This is likely a problem with your client file.
https://docs.ceph.com/en/quincy/install/get-packages/ https://knowledgebase.45drives.com/kb/creating-client-keyrings-for-cephfs/ https://docs.ceph.com/en/nautilus/cephfs/fstab/
Gluster is a distributed file system that supports both replicated and dispersed data.
Supporting dispersed data is a differentiating feature. Only a few can distribute the data in a erasure-coded or RAID-like fashion, making efficient use of space while providing redundancy. Have 5 cluster members? Just add one ‘parity bit’ for just a 20% overhead and you can loose a host. Add more parity if you like with the incremental cost. Other systems require you to duplicate your data for a 50% hit.
It’s also generally perceived as less complex than competitors like Ceph, as it has fewer moving parts and is focused on block storage. And since it uses native filesystems, you can always access your data directly. Redhat has ceased it’s corporate sponsorship, but the project is still quite active.
So you just need file storage and you have a lot of data, use gluster.
Let’s set up a distributed and dispersed example cluster. We’ll XCP-NG for this. This is similar to an erasure-coded ceph pool.
We use three hosts, each connected to a common network. With three we can disperse data enough to take one host at a time out of service. We use 4 disks on each host in this example but any number will work as long as they are all the same.
Gluster requires1 the hosts be resolvable by hostname. Verify all the hosts can ping each other by name. You may want to create a hosts file and copy to all three to help.
If you have free ports on each server, consider using the second interface for storage, or a mesh network for better performance.
# Normal management and or guest network
192.168.1.1 xcp-ng-01.lan
192.168.1.2 xcp-ng-02.lan
192.168.1.3 xcp-ng-03.lan
# Storage network in a different subnet (if you have a second interface)
192.168.10.1 xcp-ng-01.storage.lan
192.168.10.2 xcp-ng-02.storage.lan
192.168.10.3 xcp-ng-03.storage.lan
Gluster requires a few rules; one for the daemon itself and one per ‘brick’ (drive) on the server. You can also just allow the cluster members cart-blanc access. We’ll do both examples here. Add these to all cluster members.
vi /etc/sysconfig/iptables
# Note that the last line in the existing file is a REJECT. Make sure to insert these new rules BEFORE that line.
-A RH-Firewall-1-INPUT -p tcp -s xcp-ng-01.storage.lan -j ACCEPT
-A RH-Firewall-1-INPUT -p tcp -s xcp-ng-02.storage.lan -j ACCEPT
-A RH-Firewall-1-INPUT -p tcp -s xcp-ng-03.storage.lan -j ACCEPT
# Possibly for clients
-A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp --dport 24007:24008 -s client-01.storage.lan -j ACCEPT
-A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp --dport 49152:49156 -s client-01.storage.lan -j ACCEPT
service iptables restart
OR
vi /etc/sysconfig/iptables
# The gluster daemon needs ports 24007 and 24008
# Individual bricks need ports starting at 49152. Add an additional port per brick.
# Here we have 49152-49155 open for 4 brickes.
# TODO - test this command
-A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp -s xcp-ng-01.storage.lan --dport 24007:24008 -j ACCEPT
-A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp -s xcp-ng-01.storage.lan --dport 49152:49155 -j ACCEPT
-A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp -s xcp-ng-02.storage.lan --dport 24007:24008 -j ACCEPT
-A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp -s xcp-ng-02.storage.lan --dport 49152:49155 -j ACCEPT
-A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp -s xcp-ng-03.storage.lan --dport 24007:24008 -j ACCEPT
-A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp -s xcp-ng-03.storage.lan --dport 49152:49155 -j ACCEPT
Gluster works with filesystems. This is convenient because if all else fails, you still have files on disks you can access. XFS is well regarded with gluster admins, so we’ll use that.
# Install the xfs programs
yum install -y xfsprogs
# Wipe the disks before using, then format the whole disk. Repeat for each disk
wipefs -a /dev/sda
mkfs.xfs /dev/sda
Let’s mount those disks. The convention2 is to put them in /data organized by volume. We’ll use ‘volume01’ later in the config, so lets use that here as well.
On each server
# For 4 disks - Note, gluster likes to call them 'bricks'
mkdir -p /data/glusterfs/volume01/brick0{1,2,3,4}
mount /dev/sda /data/glusterfs/volume01/brick01
mount /dev/sdb /data/glusterfs/volume01/brick02
mount /dev/sdc /data/glusterfs/volume01/brick03
mount /dev/sdd /data/glusterfs/volume01/brick04
Add the appropriate config to your /etc/fstab so they mount at boot
XCP-NG is CentOS 7 based and provides GlusterFS v8 in their Repo. This version went EOL in 2021. You can add the CentOS Storage Special Interest group repo to get to v9, but no current version can be installed.
# Not recommended
yum install centos-release-gluster --enablerepo=epel,base,updates,extras
# On each host
yum install -y glusterfs-server
systemctl enable --now glusterd
# On the first host
gluster peer probe xcp-ng-02.storage.lan
gluster peer probe xcp-ng-03.storage.lan
gluster pool list
UUID Hostname State
a103d6a5-367b-4807-be93-497b06cf1614 xcp-ng-02.storage.lan Connected
10bc7918-364d-4e4d-aa16-85c1c879963a xcp-ng-03.storage.lan Connected
d00ea7e3-ed94-49ed-b56d-e9ca4327cb82 localhost Connected
# Note - localhost will always show up for the host you're running the command on
Gluster talks about data as being distributed and dispersed.
# Distribute data amongst 3 servers, each with a single brick
gluster volume create MyVolume server1:/brick1 server2:brick1 server3:brick1
Any time you have more that one drive, it’s distributed. That can be across different disks on the same host, or across different hosts. There is no redundancy, however, and any loss of disk is loss of data.
# Disperse data amongst 3 bricks, each on a different server
gluster volume create MyVolume disperse server1:/brick1 server2:/brick1 server3:/brick1
Dispersed is how you build redundancy across servers. Any one of these servers or bricks can fail and the data is safe.
# Disperse data amongst 6 six bricks, but some on the same server. Problem!
gluster volume create MyVolume disperse \
server1:/brick1 server2:/brick1 server3:/brick1
server1:/brick2 server2:/brick2 server3:/brick2
If you try and disperse your data across multiple bricks on the same server, you’ll run into the problem of sub-optimal parity. You’ll see the error message:
Multiple bricks of a disperse volume are present on the same server. This setup is not >optimal. Bricks should be on different nodes to have best fault tolerant configuration
# Disperse data into 3 brick subvolumes before distributing
gluster volume create MyVolume disperse 3 \
server1:/brick1 server2:/brick1 server3:/brick1
server1:/brick2 server2:/brick2 server3:/brick2
By specifying disperse COUNT you tell gluster that you want to create a subvolumes every COUNT bricks. In the above example, it’s every three bricks, so two subvolumes get created from the six bricks. This ensures the parity is optimally handled as it’s distributed.
You can also take advantage of bash shell expansion like below. Each subvolume is one line, repeated for each of the 4 bricks it will be distributed across.
gluster volume create volume01 disperse 3 \
xcp-ng-0{1..3}.storage.lan:/data/glusterfs/volume01/brick01/brick \
xcp-ng-0{1..3}.storage.lan:/data/glusterfs/volume01/brick02/brick \
xcp-ng-0{1..3}.storage.lan:/data/glusterfs/volume01/brick03/brick \
xcp-ng-0{1..3}.storage.lan:/data/glusterfs/volume01/brick04/brick
mount -t glusterfs xcp-ng-01.storage.lan:/volume01 /mnt
gluster volume set volume01 group metadata-cache
gluster volume set volume01 performance.readdir-ahead on
gluster volume set volume01 performance.parallel-readdir on
gluster volume set volume01 group nl-cache
gluster volume set volume01 nl-cache-positive-entry on
mount -t glusterfs xcp-ng-01.lan:/volume01/media.2 /root/mnt2/
mkdir mnt2/xcp-ng
xe sr-create content-type=user type=glusterfs name-label=GlusterSharedStorage shared=true \
device-config:server=xcp-ng-01.lan:/volume01/xcp-ng \
device-config:backupservers=xcp-ng-02.lan:xcp-ng-03.lan
Scrub is off by default. You can enable scrub at which point the scrub daemon will begin “signing” files3 (by calculating checksum). The file-system parity isn’t used. So if you enable and immediately begin a scrub you will see many “Skipped files” as their checksum hasn’t been calculated yet.
The FUSE client is recommended4. The docs cover a .deb based install, but you can also install from the repo. On Debian:
sudo apt install lsb-release gnupg
OS=$(lsb_release --codename --short)
# Assuming the current version of gluster is 11
wget -O - https://download.gluster.org/pub/gluster/glusterfs/11/rsa.pub | sudo pt-key add -
echo deb [arch=amd64] https://download.gluster.org/pub/gluster/glusterfs/11/LATEST/Debian/${OS}/amd64/apt ${OS} main | sudo tee /etc/apt/sources.list.d/gluster.list
sudo apt update; sudo apt install glusterfs-client
You need quite a few options to use this successfully at boot in the fstab
192.168.3.235:/volume01 /mnt glusterfs nofail,x-systemd.automount,x-systemd.requires=network-online.target,x-systemd.device-timeout=10 0 0
You may find that your filesystem has paused during a reboot. Take a look at your network timeout and see if setting it lower helps.
https://unix.stackexchange.com/questions/452939/glusterfs-graceful-reboot-of-brick
gluster volume set volume01 network.ping-timeout 5
Using notes from https://www.youtube.com/watch?v=TByeZBT4hfQ
https://docs.gluster.org/en/latest/Administrator-Guide/Storage-Pools/ ↩︎
https://docs.gluster.org/en/latest/Administrator-Guide/Brick-Naming-Conventions/ ↩︎
https://docs.google.com/document/d/1OtxINm8s1d9z3E_XYgpcmIUs_XQMCyspp6dWuug3_JY/edit#heading=h.gny8si2k4qu2 ↩︎
https://docs.gluster.org/en/main/Administrator-Guide/Setting-Up-Clients/#installing-on-debian-based-distributions ↩︎
From https://docs.gluster.org/en/main/Administrator-Guide/Performance-Tuning/
sudo gluster volume set media group metadata-cache
sudo gluster volume set media performance.readdir-ahead on
sudo gluster volume set media performance.parallel-readdir on
sudo gluster volume set media group nl-cache
sudo gluster volume set media nl-cache-positive-entry on
sudo gluster volume bitrot media enable
Also turned off atime at the brick and mount level in the etc/fstab
Some other notes in relation to syncthing https://forum.syncthing.net/t/glusterfs-vs-syncthing/24423/21?
File System Table example
cat /etc/fstab
LABEL=BRICK1 /data/brick1 xfs defaults,noatime 0 0
LABEL=BRICK1 /data/brick2 xfs defaults,noatime 0 0
shire1:/media /mnt/gluster/media glusterfs defaults,noatime,_netdev 0 0
Hashdeep is a reasonable way to generate checksums and test integrity on file trees.
On the source side, navigate to the parent directory of the one you want to hash and use the relative paths flag so as to avoid any issue with absolute path differences between the source and destination folders.
cd /srv/some/Parent/Directory
hashdeep -j 1 -c md5 -r -l > ~/hashes DirectoryYouWantToHash
The flags we are using are:
-j 1 # Threads to use. Adjust down if you have slow IO
-c md5 # Algorythem. md5 is the fastet in testing
-r # Recursive
-l # Use relative paths so it's more portable
Copy the hashes text file to the destination side and repeat.
cd /srv/some/Possibly/Different/Parent/Directory
hashdeep -r -a -v -l -k ~/hashes DirectoryToCheck
The flags we are using are:
-r # Recursive, as before
-a # Archive checking mode
-v # Verbose summary
-l # Relative as before
-k # File that has the sums to use for checking
LINSTOR is a management layer, frequently deployed with DRBD which does the actual work of replicating a block device. It’s mostly known for being a very fast solution for virtualization clustering. It focuses on keeping a replica of a VMs disk up to date for quick migration and redundancy, with primary IO operating locally.
https://linuxcontainers.org/incus/docs/main/howto/storage_linstor_setup/
This involves building the kernel modules from source and pulls in all the g++ and libs adding up to 575G of needed space.
Your instructed to use add-apt-repository but that’s an Ubuntu command and you can’t install it via software-properties-common anymore. So you’d next want to install from a repo provided by Linbit, but they only make part of their stack available for Trixie. To get the drbd module, You can bodge it by putting in a link to the proxmox repo that does have it all.
sudo apt update
sudo apt install curl gnupg
curl -fsSL https://packages.linbit.com/package-signing-pubkey.asc | gpg --dearmor | sudo tee /usr/share/keyrings/linbit-keyring.gpg > /dev/null
echo "deb [signed-by=/usr/share/keyrings/linbit-keyring.gpg] http://packages.linbit.com/public/ proxmox-9 drbd-9" | sudo tee /etc/apt/sources.list.d/linbit.list
You can then proceed.
# Note: If you're running the zabbly+ kernel, you'll need to add a .config link
sudo ln -s /boot/config-$(uname -r) /usr/src/linux-headers-$(uname -r)/.config
# In most cases the headers are already installed, but you can check with:
sudo apt install linux-headers-$(uname -r)
# Install the parts that are already build
sudo apt install lvm2 drbd-utils linstor-satellite
# Allow the dynamic kernel module system to do it's thing
sudo apt install drbd-dkms
Start the Satellite service on all nodes
sudo systemctl enable --now linstor-satellite
LINSTOR uses a single controller. On that system, run:
# Install the controller components
sudo apt install linstor-controller linstor-client python3-setuptools
# Enable the controller service
sudo systemctl enable --now linstor-controller
On the controller
TODO: I was able to add the node simply by simply omitting the IP address and entering the command “linstor node interface create shire2” and “linstor node interface create shire3”
# You will see a message about "Unsupported storage providers". This is expected as we have only installed LVM
linstor node create server01 <server_1> --node-type combined
linstor node create server02 <server_2> --node-type satellite
linstor node create server03 <server_3> --node-type satellite
# These command should now show all nodes and relevant features
linstor node list
linstor node info
Add the drives to linstore. You may want to run a wipefs -a /dev/sdX on the nodes to preempt any error messages.
# Change the server names such as "server01" and device names as appropriate.
linstor physical-storage create-device-pool --storage-pool nvme_pool --pool-name nvme_pool lvmthin server01 /dev/nvme1n1
linstor physical-storage create-device-pool --storage-pool nvme_pool --pool-name nvme_pool lvmthin server02 /dev/nvme1n1
linstor physical-storage create-device-pool --storage-pool nvme_pool --pool-name nvme_pool lvmthin server03 /dev/nvme1n1
# Double check that they are all up.
linstor storage-pool list
Tell Incus that you have a linstor volume
# Tell Incus where the linstor controller is
incus config set storage.linstor.controller_connection=http://<server_1>:3370
# Create a storage pool named 'remote' in incus on the individual nodes
incus storage create remote linstor --target server01
incus storage create remote linstor --target server02
incus storage create remote linstor --target server03
# link the incus storeage pool to the linstore storage pool named 'nvme_pool`
incus storage create remote linstor linstor.resource_group.storage_pool=nvme_pool
Launch an instance to test
sudo incus launch images:debian/13 c2 --storage remote
Launching c2
If this command hangs, you must cancel out (the process will keep trying for a while in the background) and check to make sure all the nodes see each other with the command sudo drbdadm status at each node.
The normal thing to do is migrate existing containers. The only gotcha there is size. Incus creates a 10G volume on the remote disk. This is LVM thin provisioned, so it won’t use it unless it needs it, but it does set an upper limit on how big the instance can get. If you’re brining in an instance larger than 10G you’ll need to change the default volume size temporarily or you’ll get a cryptic error message about rsync and other issues.
# Check your size then move with an optional parameter to set the volume size if needed.
sudo du -sh /var/lib/incus/storage-pools/local/containers/
4.5G /var/lib/incus/storage-pools/local/containers/desktop
68G /var/lib/incus/storage-pools/local/containers/database
...
...
# This one is under the default 10GB so nothing extra is needed
incus move desktop --storage remote
# Since this on is 68G, set the size up, move it, then unset it
incus storage set remote volume.size 100Gib
incus move database --storage remote
incus storage unset remote volume.size
# Some resouces suggesting setting this option, but it fails
#incus move database --storage remote -d root,size=100GiB
When you move your instance it lands on two nodes. The nodes are chosen by linstore’s auto-place logic and incus doesn’t get to pick. So you’ll want to check instance is running on one of those two nodes for the best speed.
Let’s first look at a case where it lined up.
Take a look in incus to see where your instance is running. Here we can see it’s running on the server “shire2”
sudo incus list name=something
+-----------+---------+-----------------------+---------------------------------------------+-----------+-----------+----------+
| NAME | STATE | IPV4 | IPV6 | TYPE | SNAPSHOTS | LOCATION |
+-----------+---------+-----------------------+---------------------------------------------+-----------+-----------+----------+
| something | RUNNING | 192.168.250.16 (eth0) | fd96:bd12:59eb:1:1266:6aff:fea3:b529 (eth0) | CONTAINER | 0 | shire2 |
+-----------+---------+-----------------------+---------------------------------------------+-----------+-----------+----------+
Now let’s ask linstore where it’s disk is at.
# Get the volume UUID
sudo linstor resource-definition list --show-props Aux/Incus/name | grep something
| incus-volume-38a5a7036bdf4001a073bc559ec06212 | remote | DRBD,STORAGE | ok | incus-volume-something
# See where its running at
linstor resource list
╭───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
┊ ResourceName ┊ Node ┊ Layers ┊ Usage ┊ Conns ┊ State ┊ CreatedOn ┊
╞═══════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════╡
┊ incus-volume-434bcfb9288d460fbfee6482dbf9670c ┊ shire1 ┊ DRBD,STORAGE ┊ Unused ┊ Ok ┊ UpToDate ┊ 2026-03-21 20:47:02 ┊
┊ incus-volume-434bcfb9288d460fbfee6482dbf9670c ┊ shire2 ┊ DRBD,STORAGE ┊ InUse ┊ Ok ┊ UpToDate ┊ 2026-03-21 20:47:02 ┊
┊ incus-volume-434bcfb9288d460fbfee6482dbf9670c ┊ shire3 ┊ DRBD,STORAGE ┊ Unused ┊ Ok ┊ TieBreaker ┊ 2026-03-21 20:47:02 ┊
╰───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
You can see that it’s InUse and UpToDate on shire2, the same server the instance is running on.
$ linstor resource list
╭───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
┊ ResourceName ┊ Node ┊ Layers ┊ Usage ┊ Conns ┊ State ┊ CreatedOn ┊
╞═══════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════╡
┊ incus-volume-38a5a7036bdf4001a073bc559ec06212 ┊ shire1 ┊ DRBD,STORAGE ┊ Unused ┊ Ok ┊ UpToDate ┊ 2026-03-22 08:53:11 ┊
┊ incus-volume-38a5a7036bdf4001a073bc559ec06212 ┊ shire2 ┊ DRBD,STORAGE ┊ InUse ┊ Ok ┊ Diskless ┊ 2026-03-22 08:53:10 ┊
┊ incus-volume-38a5a7036bdf4001a073bc559ec06212 ┊ shire3 ┊ DRBD,STORAGE ┊ Unused ┊ Ok ┊ UpToDate ┊ 2026-03-22 08:53:11 ┊
╰───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
You’ll notice the state Diskless on shire2…where it’s useage is InUse. That’s not ideal. Let’s fix that.
# Create a replica on the same node it's runing on. It will automatically start using it when it's done
linstor resource create shire2 incus-volume-38a5a7036bdf4001a073bc559ec06212 --storage-pool ssd_pool
linstor resource list
╭──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
┊ ResourceName ┊ Node ┊ Layers ┊ Usage ┊ Conns ┊ State ┊ CreatedOn ┊
╞══════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════╡
┊ incus-volume-38a5a7036bdf4001a073bc559ec06212 ┊ shire1 ┊ DRBD,STORAGE ┊ Unused ┊ Ok ┊ UpToDate ┊ 2026-03-22 08:53:11 ┊
┊ incus-volume-38a5a7036bdf4001a073bc559ec06212 ┊ shire2 ┊ DRBD,STORAGE ┊ InUse ┊ Ok ┊ SyncTarget(0.81%) ┊ 2026-03-22 08:53:10 ┊
┊ incus-volume-38a5a7036bdf4001a073bc559ec06212 ┊ shire3 ┊ DRBD,STORAGE ┊ Unused ┊ Ok ┊ UpToDate ┊ 2026-03-22 08:53:11 ┊
╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
# After a while that will change to UpToDate and you can delete one of the other replicas
linstor resource delete shire3 incus-volume-38a5a7036bdf4001a073bc559ec06212
Building initial module drbd/9.3.0-1 for 6.18.8-zabbly+ grep: /lib/modules/6.18.8-zabbly+/build/.config: No such file or directory
Assuming you’ve already addressed the note about the missing .config file above, make sure you’re running the same version. If you’ve updated but not rebooted, you’ll see this error.
uname -a
If it’s trying to build a version that’s different than your running kernel, like an older version, you see a message something like
Building for 6.18.8-zabbly+ and 6.19.6-zabbly+
It can fail because you don’t have the old headers anymore. Simply purge the old kernel so it doesn’t try and build for it.
sudo apt purge *6.18.8-zabbly+*
DRBD not available.
If you check your note status you may see something like this
╭─────────────────────────────────────────────────────────────────────╮ ┊ Node ┊ DRBD ┊ LUKS ┊ NVMe ┊ Cache ┊ BCache ┊ WriteCache ┊ Storage ┊ ╞═════════════════════════════════════════════════════════════════════╡ ┊ shire1 ┊ + ┊ - ┊ - ┊ + ┊ - ┊ + ┊ + ┊ ┊ shire2 ┊ - ┊ - ┊ - ┊ + ┊ - ┊ + ┊ + ┊ ┊ shire3 ┊ - ┊ - ┊ - ┊ + ┊ - ┊ + ┊ + ┊ ╰─────────────────────────────────────────────────────────────────────╯
Only one of the nodes looks up. If you check the node that’s up it will show this:
cat /proc/drbd
version: 9.3.1 (api:2/proto:118-124)
If you’ve built the new DKMS module, but the other nodes show an older version such as version 8, try a reboot. It’s probably running default module. You may also be able to try a `modprobe -r drbd && modprobe drbd’ to load the new one
This is a FUSE utility that merges multiple individual filesystem disks into one large filesystem. It shines when you have a random collection of disks and can’t use RAID. It also tolerates failure as even if you loose a disk, the data on the remaining are still available normally.
When combined with SnapRAID, you can even recover.
The developer recommends users install the most recent version available from the releases page as the Debian repos can be very out of date.
Navigate to https://github.com/trapexit/mergerfs/releases, download, and then:
sudo dpkg -i mergerfs*
# First, format your disks and give each of them their own mount point
wipefs -a /dev/sda
mkfs.ext4 /dev/sda
blkid /dev/sda
... repeat for each disk and note the UUID
# Mount them all permanently
mkdir -v /mnt/disk-{1..4}
echo "UUID=c3659080-ce25-4c13-bc96-016959f1e097 /mnt/disk-1 ext4 defaults,noatime 0 0" >> /etc/fstab
echo "UUID=3cf40be6-647f-4dbe-a617-fb0a2b7b6b9e /mnt/disk-2 ext4 defaults,noatime 0 0" >> /etc/fstab
... and so on
mount --all
# Install MergerFS, create a mount point for it and add a final line to the fstab
sudo apt install mergerfs
mkdir /mnt/pool
echo "/mnt/disk* /mnt/pool fuse.mergerfs cache.files=off,category.create=mfs,func.getattr=newest, 0 0" >> /etc/fstab
According to the developer1, the basic FUSE options above work for most cases if you’re on a recent kernel uname -r > 6.6.
| cache.files | off | Let the OS handle caching, so as to not double-cache things |
| category.create | mfs | Space allocation - see below |
| func.getattr | newest | Make directory tree mod times accurate so changes are detected2 |
Note:
The default space allocation used to keep folder contents together. This is good for some types of content, but deep directory trees could become unbalanced and run out of space early. So we changed to mfs which is simply ‘most free space’.
The default category.create may work better now that it’s changed and the [other policies] are interesting to read. There are also [other options] for fuse.
The docs also recommend starting with dropcacheonclose=false but that defaults to false anyway, so i’ve left it out.
[[other policies]:https://trapexit.github.io/mergerfs/latest/config/functions_categories_policies/#policy-descriptions [other options]:https://trapexit.github.io/mergerfs/latest/config/options/#mount-options
MooseFS is a great distributed filesystem that supports erasure coding for maximum efficiency. It’s also free to use, but there’s one catch. No redundancy. You’re only allowed to run one master node for free. The license is cheaper than enterprise solutions, but far from free.
But if you don’t mind doing a full shutdown when rebooting the master node, it’s a great system.
# Make sure all hosts can find the mfs master
sudo -i
echo "10.96.146.1 mfsmaster" >> /etc/hosts
# Install the repo (for Deb 13)
apt install -y gpg curl
mkdir -p /etc/apt/keyrings
curl https://repository.moosefs.com/moosefs.key | gpg -o /etc/apt/keyrings/moosefs.gpg --dearmor
echo "deb [arch=amd64 signed-by=/etc/apt/keyrings/moosefs.gpg] http://repository.moosefs.com/moosefs-4/apt/debian/trixie trixie main" > /etc/apt/sources.list.d/moosefs.list
apt update
# Install the Master Server on the first instance
apt install moosefs-master moosefs-cgi moosefs-cgiserv moosefs-cli
# Install the chunk master on it as well as secondary servers
## Prepare the disks
umount /data/rustfs*
mkfs.xfs -i size=512 -n ftype=1 -L RUSTFS0 /dev/sda -f
mkfs.xfs -i size=512 -n ftype=1 -L RUSTFS1 /dev/sdc -f
# if needed
sudo mkdir -p /data/rustfs{0..1}
echo "LABEL=RUSTFS0 /data/rustfs0 xfs defaults,noatime,nodiratime 0 0" >> /etc/fstab
echo "LABEL=RUSTFS1 /data/rustfs1 xfs defaults,noatime,nodiratime 0 0" >> /etc/fstab
mount -a
apt install moosefs-chunkserver
# Configure the chunkserver
cd /etc/mfs
cp mfschunkserver.cfg.sample mfschunkserver.cfg
cp mfshdd.cfg.sample mfshdd.cfg
# Note - If there is an error above because these files are missing, you may not have the correct Repo.
# make sure to run `apt update` and that the moosefs repos are listed.
# Of course, they are only full of comments by default, so worst case you can just create them yourself.
touch mfschunkserver.cfg
touch mfshdd.cfg
echo /data/rustfs0/ >> /etc/mfs/mfshdd.cfg
echo /data/rustfs1/ >> /etc/mfs/mfshdd.cfg
chown -R mfs:mfs /data/*
chmod -R 770 /data/*
mfschunkserver start
# Assuming it started, enable it at boot.
systemctl enable moosefs-chunkserver.service
# Install the client and mount
apt install moosefs-client
mkdir /mnt/mfs
mount -t moosefs mfsmaster: /mnt/mfs
sudo chown -R rustfs-user:rustfs-user /data/*
sudo systemctl restart rustfs.service
NFS is the fastest way move files around a small network. It beats both samba and afp in throughput (Circa 2014) in my testing and with a little extra config works well between apple and linux.
The NFS server supports multiple protocol versions, but we’ll focus on the current 4.X version of the protocol. It’s been out since 2010 and simplifies security.
This will install the server and a few requisites.
sudo apt-get install nfs-kernel-server
In order to streamline the ports needed (in case one uses firewalls) and reduce required services, we will limit the server to v41 only.
Edit nfs-common
sudo vi /etc/default/nfs-common
NEED_STATD=“no” NEED_IDMAPD=“yes”
And the defaults
sudo vi /etc/default/nfs-kernel-server
RPCNFSDOPTS="-N 2 -N 3" RPCMOUNTDOPTS="–manage-gids -N 2 -N 3"
Disable rpcbind
sudo systemctl mask rpcbind.service sudo systemctl mask rpcbind.socket
In NFS parlance, you ’export’ a folder when you share it. We’ll use the same location for our exports as suggested in the Debian example.
sudo vim /etc/exports
/srv/nfs4 192.168.1.0/24(rw,async,fsid=0,crossmnt,no_subtree_check,all_squash,anonuid=1000,anongid=1000,insecure)
/srv/nfs4 # This is the actual folder on the server's file system you're sharing
192.168.1.0/24 # This is the network you're sharing with
rw # Read-Write mode
async # Allow cached writes
fsid=0 # This signifies this is the 'root' of the exported file system and that
# clients will mount this share as '/'
crossmnt # Allow subfolders that are seperate filesystem to be accessed also
no_subtree_check # Disable checking for access rights outside the exported file system
all_squash # all user IDs will translated to anonymous
anonuid=1000 # all anonymous connections will be mapped to this user account in /etc/passwd
anongid=1000 # all anonymous connections will be mapped to this group account in /etc/passwd
insecure # Allows macs to mount using source ports from non-root source ports
If you can’t put all your content under this folder, it’s recommended you create pseudo file system for security reasons. See the notes for more info on that, but keep things simple if you can.
If you have a system with ufw you can get this working fairly easily. NFS is already defined as a well-known service.
ufw allow from 192.168.1.0/24 to any port nfs
You don’t actually need to restart. You put your changed into effect by issuing the exportfs command. This is best practice so you don’t to disrupt currently connected clients.
exportfs -rav
Modern Macs support NFSv4 with a couple tweaks
# In a terminal, issue the command
sudo mount -t nfs -o nolocks,resvport,locallocks 192.168.1.2:/srv ./mnt
You can also mount in finder with a version 4 flag. That may help but is somewhat awkward
nfs://vers=4,192.168.1.5/srv/nfs4
You can also edit the mac’s config file. This will allow you to use the finder to mount NFS 4 exports.
sudo vim /etc/nfs.conf
#
# nfs.conf: the NFS configuration file
#
#nfs.client.mount.options = nolock
#
nfs.client.mount.options = vers=4.1,nolocks,resvport,locallocks
You can now hit command-k and enter the string below to connect
nfs://my.server.or.ip/
Some sources suggest editing the autofs.conf file to add ’nolocks,locallocks to the automount options. This may or may not have an effect.
sudo vim /etc/autofs.conf
AUTOMOUNTD_MNTOPTS=nosuid,nodev,nolocks,locallocks
Must use v3
If you must use v3, you can set static ports. Use the internet for this.
lockd: cannot monitor
You may want to check your mac’s nfs options and set ’nolock’ or possibly ‘vers=4’ as above. Don’t set them both on at once as in the next issue.
mount_nfs: can’t mount / from home onto /Volumes/mnt: Invalid argument
You can’t combine -o vers=4 with options like ’nolocks’, presumably because it’s not implemented fully. This may have changed by now.
No Such File or Directory mount.nfs: mounting some.ip:/srv failed, reason given by server: No such file or directory
Version 4 maps directories and starts with ‘/’. Try mounting just the root path as opposed to /srv/nfs4.
mount -o nfsvers=4.1 some.ip:/ /srv
<There was a problem ….
Check that you have ‘insecure’ in your nfs export options on the server
/srv 192.168.1.0/24(rw,async,fsid=0,insecure,crossmnt,no_subtree_check)
Can’t create or see files
Don’t forget that file permissions apply as the user you specified above. Set chown and chmod accordingly
Can Create Files But Not Modify or Delete
Check the parent directory permissions
NFS doesn’t mount at boot
Try adding some mount [options].
some.ip:/ /srv nfs nofail,x-systemd.automount,x-systemd.requires=network-online.target,x-systemd.device-timeout=10,vers=4.1 0 0
mount.nfs: requested NFS version or transport protocol is not supported
Try specifying the nfs version
mount -o nfsvers=4.1 some.ip:/ /srv
This is usually a question of overhead. NFS has less CPU overhead and faster speeds circa 2023, and anecdotal testing showed fewer issues with common clients like VLC, Infuse and Kodi. However, there’s no advertisement1 like SMB has, so you have to pre-configure all clients.
This is the basic config for an anonymous, read-only share.
apt install nfs-kernel-server
echo "/mnt/pool *(fsid=0,ro,all_squash,no_subtree_check)" >> /etc/exports
exportfs -rav
mDNS SRV records has some quasi supports, but not with common clients ↩︎
This is problematic. NFS requires kernel privileges so the usual answer is “don’t”. Client’s also. So from a security and config standpoint, it’s better have PVE act as the NFS client and use bind mounts for the containers. But this can blur the line between services and infrastructure.
Either way, here’s my notes from setting up an Alpine NFS server.
Create privileged container and enable nesting
https://forum.proxmox.com/threads/is-it-possible-to-run-a-nfs-server-within-a-lxc.24403/page-2
Create a privileged container by unchecking “Unprivileged” during creation. May be possible to convert an existing container from unprivileged to privileged by backing-up and restoring. In the container Options -> Features, enable Nesting. (The NFS feature doesn’t seem necessary for running an NFS server. May be required for an NFS client - I haven’t checked
For Alpine, CAP_SETPCAP is also needed
vi /etc/pve/lxc/100.conf
# clear cap.drop
lxc.cap.drop:
# copy drop list from /usr/share/lxc/config/common.conf
lxc.cap.drop = mac_admin mac_override sys_time sys_module sys_rawio
# copy drop list from /usr/share/lxc/config/alpine.common.conf with setpcap commented
lxc.cap.drop = audit_write
lxc.cap.drop = ipc_owner
lxc.cap.drop = mknod
# lxc.cap.drop = setpcap
lxc.cap.drop = sys_nice
lxc.cap.drop = sys_pacct
lxc.cap.drop = sys_ptrace
lxc.cap.drop = sys_rawio
lxc.cap.drop = sys_resource
lxc.cap.drop = sys_tty_config
lxc.cap.drop = syslog
lxc.cap.drop = wake_alarm
Then proceed with https://wiki.alpinelinux.org/wiki/Setting_up_a_nfs-server.
SnapRAID protects against disk failure, but but not in the normal way. Rather than calculating parity with every write, it calculates parity on a schedule. Think of it as off-line RAID that’s like a backup. If you delete a file, you can recover it. If you loose a disk, you can rebuild it. Just not in real-time. All you need is a spare disk to save the parity to.
The natural question is: If you have an extra disk, why wouldn’t you just run RAID to begin with? The answers are usually:
The developer mentions it’s targeted at large media collections, so the first few points are most common.
I say disk a lot, but I should really say filesystem, as it works at that level. If you already have a large mdadm raid0 filesystem, your parity storage will have to be as large as the whole mdadm filesystem. So mostly too late in that case.
One common use is paired with MergerFS. It has individual file systems so you can snap it easily. And MergerFS lets the (remaining) content to continue being served while the missing filesystem is recovered.
You can install from apt but on most systems it’s a major version behind. The developer recommends downloading the current version and building.
https://www.snapraid.it/download
# If you're OK with v12
apt install snapraid
# If you prefer v13 and don't mind all the space from the build tools
sudo apt-get update && sudo apt-get install gcc git make -y
git clone https://github.com/amadvance/snapraid.git
cd snapraid
./autogen.sh
./configure
make
make check
sudo make install
SnapRAID needs a configuration file and a content file. The first is where you tell it what to do, and the second is for it to keep track of what it’s done.
It also needs a place to store parity information. This can grow as large as the largest disk. So if your largest disk is 10TB you’ll need a 10TB parity disk.
For an example, let’s assume we have 4 disks that range from 4 to 8 TB. To that we’ll add an 8TB disk to store parity. The developer’s suggests1 this command to maximize space (though if you go past 16TB use xfs).
mkfs.ext4 -m 0 -T largefile4 DEVICE
That give us:
lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
sdb 8:16 0 4T 0 disk
└─sdb5 8:21 0 3.9T 0 part /mnt/pics
sdc 8:32 0 4T 0 disk /mnt/movies
sdd 8:48 0 6T 0 disk /mnt/movies2
sde 8:64 0 8T 0 disk /mnt/tv.shows
sdf 8:64 0 8T 0 disk /mnt/snapraid # <-- Just added this
So you’d create the config file with the following options:
sudo vi /etc/snapraid.conf
/var/snapraid/snapraid.content
data d1 /mnt/pics/
data d2 /mnt/movies/
data d3 /mnt/movies2/
data d4 /mnt/tv.shows/
parity /mnt/snapraid/snapraid.parity
# This is the 'working' file for snapraid
content /var/snapraid/snapraid.content
# A couple more copies is suggested as backup
content /mnt/pics/.snapraid.content
content /mnt/movies/.snapraid.content
The command to sync up the parity disk is just sync. This will take a long time for it’s initial run.
snapraid sync
You can restore a deleted file or directory with:
snapraid fix -f SOMETHING
If you loose a disk, just add a new one, format and mount it at the same place, and type:
# Assuming 'd2' was the replaced disk
snapraid -d d2 fix
If your disks are suspect, you can run periodic scrubs to check for bit-rot.
snapraid scrub
snapraid status
There are may other options and the manual is worth checking out.
You can mount a remote system via sshfs. It’s slow, but better than nothing.
# Mount a host dir you have ssh access to
sshfs [email protected]:/var/www/html /var/www/html
# Mount a remote system over a proxy jump host
sshfs [email protected]:/var/www/html /var/www/html -o ssh_command="ssh -J [email protected]",allow_other,default_permissions
It’s often handy to add a shortcut to your ssh config so you don’t have to type as much.
cat .ssh/config
Host some.host
ProxyJump some.in.between.host
ZFS: the last word in file systems - at least according to Sun Microsystems back in 2004. But it’s pretty much true for traditional file servers. You add disks to a pool where you decide how much redundancy you want, then create file systems that sit on top. ZFS directly manages it all. No labeling, partitioning or formatting required.
There is error detection and correction, zero-space clones and snapshots, compression and deduplication, and delta-based replication for backup. It’s also highly resistent to corruption from power loss and crashes because it uses Copy-On-Write.
This last feature means that as files are changed, only the changed-bits are written out, and then the metadata updated at the end as a separate and final step to include these changed bits. The original file stays the same until the very end. An interruption in the middle of a write (such as from a crash) leaves the file undamaged.
Lastly, everything is checksummed and automatically repaired should you ever suffer from silent corruption.
Let’s create a pool and mount a file system.
A ‘pool’ is group of disks and that’s where RAID levels are established. You can choose to mirror drives, or use distributed parity. A common choice is RAIDZ1 so that you can sustain the loss of one drive. You can increase the redundancy to RAIDZ2 and RAIDZ3 as desired.
zpool create pool01 raidz1 /dev/sdc /dev/sdd /dev/sde /dev/sdf
zpool list
NAME SIZE ALLOC FREE
pool01 40T 0T 40T
A default root file system is created and mounted automatically when the pool is created. Using this is fine and you can easily change where it’s mounted.
zfs list
NAME USED AVAIL REFER MOUNTPOINT
pool01 0T 40T 0T /pool01
zfs set mountpoint=/mnt pool01
But often, you need more than one file system - say one for holding working files and another for long term storage. This allows you to easily back up things different things, differently.
# Get rid of the initial fs and pool
zpool destroy pool01
# Create it again but leave the root filesystem unmounted with the `-m` option. You can also use drive short-names
zpool create -m none pool01 raidz1 sdc sdd sde sdf
# Add a couple filesystems and mount under the /srv directory
zfs create pool01/working -o mountpoint=/srv/working
zfs create pool01/archive -o mountpoint=/srv/archive
^ ^
/ \
pool name -- filesystem name
Now you can do things like snapshot the archive folder regularly while skipping the working folder. The only downside is that they are separate filessytems. Moving things doesn’t happen instantly.
Compression is on by default and this will save space for things that can benefit from it. It also makes things faster as moving compressed data takes less time. CPU use for the default algorithm lz4, is negligible and it quickly detects files that aren’t compressible and gives up so that CPU time isn’t wasted.
zfs get compression pool01
NAME PROPERTY VALUE SOURCE
pool01 compression lz4 local
Now you’ve got a filesystem, take a look at creating and work with snapshots.
About to do something dangerous? Let’s create a ‘save’ point so you can reload your game, so to speak. They don’t take any space (to start with) and are nearly instant.
# Create a snapshot named `save-1`
zfs snapshot pool01/archive@save-1
The snapshot is a read-only copy of the filesystem at that time. It’s mounted by default in a hidden directory and you can examine and even copy things out of it, if you desire.
ls /srv/archive/.zfs/save-1
While a snapshot doesn’t take up any space to start with, it begins to as you make changes. Anything you delete stays around on the snapshot. Things you edit consume space as well for the changed bits. So when you’re done with a snapshot, it’s easy to remove.
zfs snapshot destroy pool01/archive@save-1
Mess things up in your archive folder after all? Not a problem, just roll back to the same state as your snapshot.
zfs rollback pool01/archive@save-1
Importantly, this is a one-way trip back. You can’t branch and jump around like it was a filesystem multiverse of alternate possibilities. ZFS will warn you about this and if you have more than one snapshot in between you and where you’re going, it will let you know they are about to be deleted.
One of the most useful tools is the zfs-auto-snapshot utility. This will create periodic snapshots of your filesystem and keep them pruned for efficiency. By default, it creates a snapshot every 15 min, and then prunes them down so you have one:
Install with the command:
sudo apt install zfs-auto-snapshot
That’s it. You’ll see new folders based on time created in the hidden .zfs folder at the root of your filesystems. Each filesystem will get it’s own. Anytime you need to look for a file you’ve deleted, you’ll find it there.
# Look at snapshots
ls /srv/archive/.zfs/
# Disable
zfs set com.sun:auto-snapshot=false rpool/export
There are sub-properties you can set under the basic auto-snapshot value
zfs set com.sun:auto-snapshot=true pool02/someDataSet
zfs set com.sun:auto-snapshot:frequent=false pool02/someDataSet
zfs get com.sun:auto-snapshot pool02/someDataSet
zfs get com.sun:auto-snapshot:frequent pool02/someDataSet
# Possibly also the number to keep if other than the default is desired
zfs set com.sun:auto-snapshot:weekly=true,keep=52
# Take only weekly
zfs set com.sun:auto-snapshot:weekly=true rpool/export
You can’t use globbing or similar to mass-delete snapshots, but you can string together a couple commands.
# Disable auto-snap as needed
zfs set com.sun:auto-snapshot=false pool04
zfs list -H -o name -t snapshot | grep auto | xargs -n1 zfs destroy
On some centOS-based systems, like XCP-NG, you will only see frequent snapshots. This is because only the frequent cron job uses the correct path. You must add a PATH statement to the other cron jobs
https://forum.level1techs.com/t/setting-up-zfs-auto-snapshot-on-centos-7/129574/12
Now that you have snapshots, let’s send them somewhere for backup with replication.
https://www.reddit.com/r/zfs/comments/829v5a/zfs_ubuntu_1604_delete_snapshots_with_wildcard/
Replication is how you backup and copy ZFS. It turns a snapshot into a bit-stream that you can pipe to something else. Usually, you pipe it over the network to another system where you connect it to zfs receive.
It is also the only way. The snapshot is what allows point-in-time handling and the receive ensures consistency. And a snapshot is a filesystem, but it’s more than just the files. Two identically named filesystems with the same files you put in place by rsync are not the same filesystem and you can’t jump-start a sync this way.
# On the receiving side, create a pool to hold the filesystem
zpool create -m none pool02 raidz1 sdc sdd sde sdf
# On the sending side, pipe over SSH. The -F forces the filesystem on the receiving side to be replaced
zfs snapshot pool01@snap1
zfs send pool01@snap1 | ssh some.other.server zfs receive -F pool02
Importantly, this replaces the root filesystem on the receiving side. The filesystem you just copied over is accessible when the replication is finished - assuming it’s mounted and your only using the default root. If you’re using multiple filesystems, you’ll want to recursively send things so you can pick up children like the archive filesystem.
# The -r and -R trigger recursive operations
zfs snapshot -r pool01@snap1
zfs send -R pool01@snap1 | ssh some.other.server zfs receive -F pool02
You can also pick a specific filesystem to send. You can name it whatever you like on the other side, or replace something already named.
# Sending just the archive filesystem
zfs snapshot pool01/archive@snap1
zfs send pool01/archive@snap1 | ssh some.other.server zfs receive -F pool02/archive
And of course, you may have two pools on the same system. One line in a terminal is all you need.
zfs send -R pool01@snap1 | zfs receive -F pool02
These are much faster than ssh if you don’t care about someone capturing the traffic. But it does require you to start both ends separately.
# On the receiving side
ssh some.other.system
mbuffer -4 -s 128k -m 1G -I 8990 | zfs receive -F pool02/archive
# On the sending side
zfs send pool01/archive@snap1 | mbuffer -s 128k -m 1G -O some.other.system:8990
You can also use good-ole netcat. It’s a little slower but still faster than SSH. Combine it with pv for some visuals.
# On the receiving end
nc -l 8989 | pv -trab -B 500M | zfs recv -F pool02/archive
# On the sending side
zfs send pool01/archive@snap1 | nc some.other.system 8989
You may want to know how big the transmit is to estimate time or capacity. You can do this with a dry-run.
zfs send -nv pool01/archive@snap1
Any interruptions and you have to start all over again - or do you? If you’re sending a long-running transfer, add a token on the receiving side and you can restart from where it broke, turning a tragedy into just an annoyance.
# On the receiving side, add -s
ssh some.other.system
mbuffer -4 -s 128k -m 1G -I 8990 | zfs receive -s -F pool01/archive
# Send the stream normally
zfs send pool01/archive@snap1 | mbuffer -s 128k -m 1G -O some.other.system:8990
# If you get interrupted, on the receiving side, look up the token
zfs get -H -o value receive_resume_token pool01
# Then use that on the sending side to resume where you left off
zfs send -t 'some_long_key' | mbuffer -s 128k -m 1G -O some.other.system:8990
If you decide you don’t want to resume, clean up with the -A command to release the space consumed by the pending transfer.
# On the receiving side
zfs recv -A pool01/archive
After you’ve sent the initial snapshot, subsequent ones are much smaller. Even very large backups can be kept current up if you ‘pre-seed’ before taking the other pool remote.
# at some point in past
zfs snapshot pool01/archive@snap1
zfs send pool01/archive@snap1 ....
# now we'll snap again and send just the changes between the two using -i
zfs snapshot pool01/archive@snap2
zfs send -i pool01/archive/@snap1 pool01/archive@snap2 ...
If you are jumping more than one snapshot ahead, the intervening ones are skipped. If you want to include them for some reason, use the -I option.
# This will send all the snaps between 1 and 9
zfs send -I pool01/archive/@snap1 pool01/archive@snap9 ...
You’ll often need to use -F to force changes as even though you haven’t used the remote system, it may think you have if it’s mounted and atimes are on.
You need at least 1 snapshot in common at both locations. This must have been sent from one to the other, not just named the same. Say you create snap1 and send it. Later, you create snap2 and, thinking you don’t need it anymore, delete snap1. Even though you have snap1 on the destination you cannot send snap2 as a delta. You need snap1 in both locations to generate the delta. You are out of luck and must send snap2 as a full snapshot.
You can use a zfs feature called a bookmark as an alternative, but that is something you set up in advance and won’t save you from the above scenario.
Take a snapshot, estimate the size with a dry-run, then use a resumable token and force changes
zfs snapshot pool01/archive@12th
zfs send -nv -i pool01/archive@11th pool01/archive@12th
zfs send -i pool01/archive@11th pool01/archive@12th | pv -trab -B 500M | ssh some.other.server zfs recv -F -s pool01/archive
Here’s an example of a recursive snapshot to a file. The snapshot takes a -r for recursive, and the send a -R.
# Take the snapshot
zfs snapshot -r pool01/archive@21
# Estimate the size
zfs send -nv -R -i pool01/archive@20 pool01/archive@21
# Mount a portable drive to /mnt - a traditional ext4 or whatever
zfs send -Ri pool01/archive@20 pool01/archive@21 > /mnt/backupfile
# When you get to the other location, receive from the file
zfs receive -s -F pool02/archive < /mnt/backupfile
This is a common thought when you’re starting, but it ends up being deceptive because pools aren’t things that can be sent. Only filesets. So what you’re actually doing is a recursive send of the root fileset with a implicit snapshot created on the fly. This is find, but you won’t be able to refer to it later for updates, so you’re better off not.
# Unmount -a (all available filesystems) on the given pool
zfs unmount -a pool01
# Send the unmounted filesystem with an implicit snapshot
zfs send -R pool01 ...
You don’t want to do this by hand all the time. One way is with a simple script. If you’ve already installed zfs-auto-snapshot you may have something that looks like this:
# use '-o name' to get just the snapshot name without all the details
# use '-s creation' to sort by creation time
zfs list -t snapshot -o name -s creation pool01/archive
pool01/archive@auto-2024-10-13_00-00
pool01/archive@auto-2024-10-20_00-00
pool01/archive@auto-2024-10-27_00-00
pool01/archive@auto-2024-11-01_00-00
pool01/archive@auto-2024-11-02_00-00
You can get the last two like this, then use send and receive. Adjust the grep to get just the daily as needed.
CURRENT=$(zfs list -t snapshot -o name -s creation pool01/replication | grep auto | tail -1 )
LAST=$(zfs list -t snapshot -o name -s creation pool01/replication | grep auto | tail -2 | head -1)
zfs send -i $LAST $CURRENT | pv -trab -B 500M | ssh some.other.server zfs recv -F -s pool01/archive
This is pretty basic and can fall out of sync. You can bring it up a notch by asking the other side to list it’s snapshots with a zfs list over ssh and comparing against yours to find the most recent match. And add a resume token. But by that point you may consider just using a tool like:
This looks to replace your auto snapshots as well, and that’s probably fine. I have’t used it myself as I scripted back in the day. I will probably start, though.
Sometimes, good disks go bad. Learn how to catch them before they do, and replace them when needed.
Your disks will fail, but you’ll usually get some warnings because ZFS proactively checks every occupied bit to guard against silent corruption. This is normally done every month, but you can launch one manually if you’re suspicious. They take a long time, but operate at low priority.
# Start a scrub
zpool scrub pool01
# Check the status
zpool scrub -s pool01
You can check the status of your pool at anytime with the command zpool status. When there’s a problem, you’ll see this:
zpool status
NAME STATE READ WRITE CKSUM
pool01 DEGRADED 0 0 0
raidz1 DEGRADED 0 0 0
/dev/sda ONLINE 0 0 0
/dev/sdb ONLINE 0 0 0
/dev/sdc ONLINE 0 0 0
/dev/sdd FAULTED 53 0 0 too many errors
Time to replace that last drive before it goes all the way bad
# You don't need to manually offline the drive if it's faulted, but it's good practice to as there's other states it can be in
zpool offline pool01 /dev/sdd
# Physically replace that drive. If you're shutting down to do this, the replacement usually has the same device path
zpool replace pool01 /dev/sdd
There’s a lot of strange things that can happen with drives and depending on your version of ZFS it might be using UUIDs or other drive identification strings. Check the link below for some of those conditions.
You’ve guarded against disk failure this by adding redundancy, but was it enough? There’s a very mathy calculator at https://jro.io/r2c2/ that will allow you chart different parity configs. But a reasonable rule-of-thumb is to devote 20%, or 1 in 5 drives, to parity.
Oracle however, recommends Virtual Devices when you go past 9 disks.
When you get past 15 drives, you can’t increase parity. You can however, create virtual devices. Best practice from Oracle says to do this even earlier as a VDev should be less than 9 disks 1. So given 24 disks, you should have 3 VDevs of 8 each. Here’s an example with 2 parity. Slightly better than 1 in 5 and suitable for older disks.
### Build a 3-Wide RAIDZ2 across 24 disks
zpool create \
pool01 \
-m none \
-f \
raidz2 sdb sdc sdd sde sdf sdg sdh sdi \
raidz2 sdj sdk sdl sdm sdn sdo sdp sdq \
raidz2 sdr sds sdt sdu sdv sdw sdx sdy
Drive letters can be hard to trace back to a physical drive. A a better2 way is the /dev/disk/by-id identifiers.
ls /dev/disk/by-id | grep ata | grep -v part
zpool create -m none -o ashift=12 -O compression=lz4 \
pool04 \
raidz2 \
ata-ST4000NM0035-1V4107_ZC11AHH9 \
ata-ST4000NM0035-1V4107_ZC116F11 \
ata-ST4000NM0035-1V4107_ZC1195V5 \
ata-ST4000NM0035-1V4107_ZC11CDMB \
ata-ST4000NM0035-1V4107_ZC1195PR \
ata-ST4000NM0024-1HT178_Z4F164WG \
ata-ST4000NM0024-1HT178_Z4F17SJK \
ata-ST4000NM0024-1HT178_Z4F17M6B \
ata-ST4000NM0024-1HT178_Z4F18FZE \
ata-ST4000NM0024-1HT178_Z4F18G35
You may not be reach a location quickly when a disk fails. In such a case, is it better to have a Z3 filesystem run in degraded performance mode (i.e. calculating parity the whole time) or a Z2 system that replaces the failed disk automatically?
It’s better to have a more parity until you go past the guidelines of 15 drives in a Z3 config. If you have 16 bays, add a hot spare.
A distributed spare is a newer feature that allows you reserve space on all of your disks, rather than just one. That allows resilvering to go much faster as you’re no longer limited by the speed of one disk. Here’s an example of such a pool that has 16 total devices.
# This pool has 3 parity, 12 data, 16 total count, with 1 spare
```bash
zpool create -f pool02 \
draid3:12d:16c:1s \
ata-ST4000NM000A-2HZ100_WJG04M27 \
ata-ST4000NM000A-2HZ100_WJG09BH7 \
ata-ST4000NM000A-2HZ100_WJG0QJ7X \
ata-ST4000NM000A-2HZ100_WS20ECCD \
ata-ST4000NM000A-2HZ100_WS20ECFH \
ata-ST4000NM000A-2HZ100_WS20JXTA \
ata-ST4000NM0024-1HT178_Z4F14K76 \
ata-ST4000NM0024-1HT178_Z4F17SJK \
ata-ST4000NM0024-1HT178_Z4F17YBP \
ata-ST4000NM0024-1HT178_Z4F1BJR1 \
ata-ST4000NM002A-2HZ101_WJG0GBXB \
ata-ST4000NM002A-2HZ101_WJG11NGC \
ata-ST4000NM0035-1V4107_ZC1168N3 \
ata-ST4000NM0035-1V4107_ZC116F11 \
ata-ST4000NM0035-1V4107_ZC116MSW \
ata-ST4000NM0035-1V4107_ZC116NZM \
Best practice from Oracle says a VDev should be less than 9 disks1. So given 24 disks, you should have 3 VDevs. They further recommend the following amount of parity vs data:
It is not recommended to create a zpool with a single large vdev, say 20 disks, because write IOPS performance will be that of a single disk, which also means that resilver time will be very long (possibly weeks with future large drives).
I interpret this as meaning that when a single IO write operation is given to the VDev, it won’t write anything else until it’s done. But if you have multiple VDevs, you can hand out a writes to other VDevs while you’re waiting on the first. Reading is probably unaffected, but writes will be faster with more VDevs.
Also, when resilvering the array, you have to read from each of the drives in the VDev to calculate the parity bit. If you have 24 drives in a VDev, then you have to read a block of data from all 24 drives to produce the parity bit. If you have only 8, then you have only 1/3 as much data to read. Meanwhile, the rest of the VDevs are available for real work.
Rebuilding the array also introduces stress which can cause other disks to fail, so it’s best to limit that to a smaller set of drives. I’ve heard many times of resilvering causing sister drives that were already on the edge, to go over and fail the array.
You can calculate the failure rates of different configurations with an on-line tool2. The chart scales the X axis by 50, so the differences in failure rates are not as large as it would seem, but if they didn’t you wouldn’t be able to see the lines. But in most cases, there’s not a large difference say between a 4x9 and a 3x12.
Given 9 disks where one fails, is it better to drop from 3 parity to 2 and run in degraded mode, or 2 parity that drops to 1 and a spare that recovers without intervention. The math2 says its better to have parity. But what about speed? When you loose a disk, 1 out of every 9 IOPS requires that you reconstruct it from parity. Anecdotally, observed performance penalties are minor. So the only times to use a hot spare is:
Say you have 16 bays of 4TB Drives. A 2x8 Z2 config gives you 48TB but you only want 32TB. Change that to a 2x8 Z3 and get 40TB. Still only need 32 TB? Change that to a 2x7 Z3 with 2 hot spares. Now you have 32TB with the maximum protection and the insurance of an automatic replacement.
Or maybe you have a 37 bay system. You do something that equals 36 plus a spare.
The other case is when your IOPS demands push past what RAIDZ can do an you must use a mirror pool. A failure there looses all redundancy and a hot spare is your only option.
A distributed spare recovers in half the time3 from a disk loss, and is always better than a dedicated spare - though you should almost never use a spare anyway. The only time to use a normal hot spare is when you have a single global spare.
The speed difference isn’t charted. So let’s test that some.
Given 24 disks, and deciding to live dangerously, should you should have a single, 24 disk vdev with three parity disks, or three VDevs with a single parity disk each? The reason for the 1st case is better resiliency, and the latter better write speed and recovery from disk failures.
Create the pool across 24 disks
zpool create \
-f -m /srv srv \
raidz sdb sdc sdd sde sdf sdg sdh sdi \
raidz sdj sdk sdl sdm sdn sdo sdp sdq \
raidz sdr sds sdt sdu sdv sdw sdx sdy
Now copy a lot of random data to it
#!/bin/bash
no_of_files=1000
counter=0
while [[ $counter -le $no_of_files ]]
do echo Creating file no $counter
touch random-file.$counter
shred -n 1 -s 1G random-file.$counter
let "counter += 1"
done
Now yank (literally) one of the physical disks and replace it
allen@server:~$ sudo zpool status
pool: srv
state: DEGRADED
status: One or more devices could not be used because the label is missing or
invalid. Sufficient replicas exist for the pool to continue
functioning in a degraded state.
action: Replace the device using 'zpool replace'.
see: http://zfsonlinux.org/msg/ZFS-8000-4J
scan: none requested
config:
NAME STATE READ WRITE CKSUM
srv DEGRADED 0 0 0
raidz1-0 DEGRADED 0 0 0
sdb ONLINE 0 0 0
6847353731192779603 UNAVAIL 0 0 0 was /dev/sdc1
sdd ONLINE 0 0 0
sde ONLINE 0 0 0
sdf ONLINE 0 0 0
sdg ONLINE 0 0 0
sdh ONLINE 0 0 0
sdi ONLINE 0 0 0
raidz1-1 ONLINE 0 0 0
sdr ONLINE 0 0 0
sds ONLINE 0 0 0
sdt ONLINE 0 0 0
sdu ONLINE 0 0 0
sdj ONLINE 0 0 0
sdk ONLINE 0 0 0
sdl ONLINE 0 0 0
sdm ONLINE 0 0 0
raidz1-2 ONLINE 0 0 0
sdv ONLINE 0 0 0
sdw ONLINE 0 0 0
sdx ONLINE 0 0 0
sdy ONLINE 0 0 0
sdn ONLINE 0 0 0
sdo ONLINE 0 0 0
sdp ONLINE 0 0 0
sdq ONLINE 0 0 0
errors: No known data errors
allen@server:~$ lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 465.8G 0 disk
├─sda1 8:1 0 449.9G 0 part /
├─sda2 8:2 0 1K 0 part
└─sda5 8:5 0 15.9G 0 part [SWAP]
sdb 8:16 1 931.5G 0 disk
├─sdb1 8:17 1 931.5G 0 part
└─sdb9 8:25 1 8M 0 part
sdc 8:32 1 931.5G 0 disk
sdd 8:48 1 931.5G 0 disk
├─sdd1 8:49 1 931.5G 0 part
└─sdd9 8:57 1 8M 0 part
...
sudo zpool replace srv 6847353731192779603 /dev/sdc -f
allen@server:~$ sudo zpool status
pool: srv
state: DEGRADED
status: One or more devices is currently being resilvered. The pool will
continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
scan: resilver in progress since Fri Mar 22 15:50:21 2019
131G scanned out of 13.5T at 941M/s, 4h7m to go
5.40G resilvered, 0.95% done
config:
NAME STATE READ WRITE CKSUM
srv DEGRADED 0 0 0
raidz1-0 DEGRADED 0 0 0
sdb ONLINE 0 0 0
replacing-1 OFFLINE 0 0 0
6847353731192779603 OFFLINE 0 0 0 was /dev/sdc1/old
sdc ONLINE 0 0 0 (resilvering)
sdd ONLINE 0 0 0
sde ONLINE 0 0 0
sdf ONLINE 0 0 0
...
A few hours later…
$ sudo zpool status
pool: srv
state: DEGRADED
status: One or more devices has experienced an error resulting in data corruption. Applications may be affected.
action: Restore the file in question if possible. Otherwise restore the entire pool from backup.
see: http://zfsonlinux.org/msg/ZFS-8000-8A
scan: resilvered 571G in 5h16m with 2946 errors on Fri Mar 22 21:06:48 2019
config:
NAME STATE READ WRITE CKSUM
srv DEGRADED 208 0 2.67K
raidz1-0 DEGRADED 208 0 5.16K
sdb ONLINE 0 0 0
replacing-1 OFFLINE 0 0 0
6847353731192779603 OFFLINE 0 0 0 was /dev/sdc1/old
sdc ONLINE 0 0 0
sdd ONLINE 208 0 1
sde ONLINE 0 0 0
sdf ONLINE 0 0 0
sdg ONLINE 0 0 0
sdh ONLINE 0 0 0
sdi ONLINE 0 0 0
raidz1-1 ONLINE 0 0 0
sdr ONLINE 0 0 0
sds ONLINE 0 0 0
sdt ONLINE 0 0 0
sdu ONLINE 0 0 0
sdj ONLINE 0 0 1
sdk ONLINE 0 0 1
sdl ONLINE 0 0 0
sdm ONLINE 0 0 0
raidz1-2 ONLINE 0 0 0
sdv ONLINE 0 0 0
sdw ONLINE 0 0 0
sdx ONLINE 0 0 0
sdy ONLINE 0 0 0
sdn ONLINE 0 0 0
sdo ONLINE 0 0 0
sdp ONLINE 0 0 0
sdq ONLINE 0 0 0
The time was 5h16m. But notice the error - during resilvering drive sdd had 208 read errors and data was lost. This is the classic RAID situation where resilvering stresses the drives, another goes bad and you can’t restore.
It’s somewhat questionable if this is a valid test as the affect of the error on resilvering duration is unknown. But on with the test.
Let’s wipe that away and create a raidz3
sudo zpool destroy srv
zpool create \
-f -m /srv srv \
raidz3 \
sdb sdc sdd sde sdf sdg sdh sdi \
sdj sdk sdl sdm sdn sdo sdp sdq \
sdr sds sdt sdu sdv sdw sdx sdy
zdb
zpool offline srv 15700807100581040709
sudo zpool replace srv 15700807100581040709 sdc
allen@server:~$ sudo zpool status
pool: srv
state: DEGRADED
status: One or more devices is currently being resilvered. The pool will continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
scan: resilver in progress since Sun Mar 24 10:07:18 2019
27.9G scanned out of 9.14T at 362M/s, 7h19m to go
1.21G resilvered, 0.30% done
config:
NAME STATE READ WRITE CKSUM
srv DEGRADED 0 0 0
raidz3-0 DEGRADED 0 0 0
sdb ONLINE 0 0 0
replacing-1 OFFLINE 0 0 0
sdd OFFLINE 0 0 0
sdc ONLINE 0 0 0 (resilvering)
sde ONLINE 0 0 0
sdf ONLINE 0 0 0
...
allen@server:~$ sudo zpool status
pool: srv
state: ONLINE
scan: resilvered 405G in 6h58m with 0 errors on Sun Mar 24 17:05:50 2019
config:
NAME STATE READ WRITE CKSUM
srv ONLINE 0 0 0
raidz3-0 ONLINE 0 0 0
sdb ONLINE 0 0 0
sdc ONLINE 0 0 0
sde ONLINE 0 0 0
sdf ONLINE 0 0 0
sdg ONLINE 0 0 0
...
The time? 6h58m. Longer, but safer.
There is a lot out there about ZFS cache config. I’ve found the most significant feature to be putting your metadata on a dedicated NVMe devices. This is noted as a ‘Special’ VDev. Here’s an example of a draid with such a device at the end.
Note: A 2x18 is bad practice - just more fun than a 3x12 with no spares.
zpool create -f pool02 \
draid3:14d:18c:1s \
ata-ST4000NM000A-2HZ100_WJG04M27 \
ata-ST4000NM000A-2HZ100_WJG09BH7 \
ata-ST4000NM000A-2HZ100_WJG0QJ7X \
ata-ST4000NM000A-2HZ100_WS20ECCD \
ata-ST4000NM000A-2HZ100_WS20ECFH \
ata-ST4000NM000A-2HZ100_WS20JXTA \
ata-ST4000NM0024-1HT178_Z4F14K76 \
ata-ST4000NM0024-1HT178_Z4F17SJK \
ata-ST4000NM0024-1HT178_Z4F17YBP \
ata-ST4000NM0024-1HT178_Z4F1BJR1 \
ata-ST4000NM002A-2HZ101_WJG0GBXB \
ata-ST4000NM002A-2HZ101_WJG11NGC \
ata-ST4000NM0035-1V4107_ZC1168N3 \
ata-ST4000NM0035-1V4107_ZC116F11 \
ata-ST4000NM0035-1V4107_ZC116MSW \
ata-ST4000NM0035-1V4107_ZC116NZM \
ata-ST4000NM0035-1V4107_ZC118WV5 \
ata-ST4000NM0035-1V4107_ZC118WW0 \
draid3:14d:18c:1s \
ata-ST4000NM0035-1V4107_ZC118X74 \
ata-ST4000NM0035-1V4107_ZC118X90 \
ata-ST4000NM0035-1V4107_ZC118XBS \
ata-ST4000NM0035-1V4107_ZC118Z23 \
ata-ST4000NM0035-1V4107_ZC11907W \
ata-ST4000NM0035-1V4107_ZC1192GG \
ata-ST4000NM0035-1V4107_ZC1195PR \
ata-ST4000NM0035-1V4107_ZC1195V5 \
ata-ST4000NM0035-1V4107_ZC1195ZJ \
ata-ST4000NM0035-1V4107_ZC11AHH9 \
ata-ST4000NM0035-1V4107_ZC11CDD0 \
ata-ST4000NM0035-1V4107_ZC11CE77 \
ata-ST4000NM0035-1V4107_ZC11CV5E \
ata-ST4000NM0035-1V4107_ZC11D2AQ \
ata-ST4000NM0035-1V4107_ZC11HRGR \
ata-ST4000NM0035-1V4107_ZC1B200R \
ata-ST4000NM0035-1V4107_ZC1CBXEH \
ata-ST4000NM0035-1V4107_ZC1DC98B \
special mirror \
ata-MICRON_M510DC_MTFDDAK960MBP_164614A1DBC4 \
ata-MICRON_M510DC_MTFDDAK960MBP_170615BD4A74
zfs set special_small_blocks=64K pool02
Metadata is stores automatically on the special device but there’s a benefit in also directing the pool to use the special vdev for small files as well.
https://forum.level1techs.com/t/zfs-metadata-special-device-z/159954
You might want to store data such that it’s encrypted at rest. Or replicate data to such as system. ZFS offers this on a per-dataset option.
Let’s assume that you’re at a remote site and want to create an encrypted fileset to receive your replications.
zfs create -o encryption=on -o keylocation=prompt -o keyformat=passphrase pool02/encrypted
This example uses mbuffer and assumes a secure VPN. Replace with SSH as needed.
# On the receiving side
sudo zfs load-key -r pool02/encrypted
mbuffer -4 -s 128k -m 1G -I 8990 | sudo zfs receive -s -F pool02/encrypted
# On the sending side
zfs send -i pool01/archive@snap1 pool01/archived@snap2 | mbuffer -s 128k -m 1G -O some.server:8990
Best practice from Oracle says a VDev should be less than 9 disks1. So given 24 disks you should have 3 VDevs. However, when using RAIDZ, the math shows they should be as large as possible with multiple parity disks2. I.e. with 24 disks you should have a single, 24 disk VDev.
The reason for the best practice seems to be about the speed of writing and recovering from disk failures.
It is not recommended to create a zpool with a single large vdev, say 20 disks, because write IOPS performance will be that of a single disk, which also means that resilver time will be very long (possibly weeks with future large drives).
With a single VDev, you break up the data to send a chunk to each drive, then wait for them all to finish writing before you send the next. With several VDevs, you can move on to the next while you wait for the others to finish.
Create the pool across 24 disks
#
# -O is the pool's root dataset. Lowercase letter -o is for pool properties
# sudo zfs get compression to check. lz4 is now prefered
#
zpool create \
-m /srv srv \
-O compression=lz4 \
raidz sdb sdc sdd sde sdf sdg sdh sdi \
raidz sdj sdk sdl sdm sdn sdo sdp sdq \
raidz sdr sds sdt sdu sdv sdw sdx sdy -f
Copy a lot of random data to it.
#!/bin/bash
no_of_files=1000
counter=0
while [[ $counter -le $no_of_files ]]
do echo Creating file no $counter
touch random-file.$counter
shred -n 1 -s 1G random-file.$counter
let "counter += 1"
done
Yank out (literally) one of the physical disks and replace it.
sudo zpool status [433/433]
pool: srv
state: DEGRADED
status: One or more devices could not be used because the label is missing or
invalid. Sufficient replicas exist for the pool to continue
functioning in a degraded state.
action: Replace the device using 'zpool replace'.
see: http://zfsonlinux.org/msg/ZFS-8000-4J
scan: none requested
config:
NAME STATE READ WRITE CKSUM
srv DEGRADED 0 0 0
raidz1-0 DEGRADED 0 0 0
sdb ONLINE 0 0 0
6847353731192779603 UNAVAIL 0 0 0 was /dev/sdc1
sdd ONLINE 0 0 0
sde ONLINE 0 0 0
sdf ONLINE 0 0 0
sdg ONLINE 0 0 0
sdh ONLINE 0 0 0
sdi ONLINE 0 0 0
raidz1-1 ONLINE 0 0 0
sdr ONLINE 0 0 0
sds ONLINE 0 0 0
sdt ONLINE 0 0 0
sdu ONLINE 0 0 0
sdj ONLINE 0 0 0
sdk ONLINE 0 0 0
sdl ONLINE 0 0 0
sdm ONLINE 0 0 0
raidz1-2 ONLINE 0 0 0
sdv ONLINE 0 0 0
sdw ONLINE 0 0 0
sdx ONLINE 0 0 0
sdy ONLINE 0 0 0
sdn ONLINE 0 0 0
sdo ONLINE 0 0 0
sdp ONLINE 0 0 0
sdq ONLINE 0 0 0
errors: No known data errors
Insert a new disk and replace the missing one.
allen@server:~$ lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 465.8G 0 disk
├─sda1 8:1 0 449.9G 0 part /
├─sda2 8:2 0 1K 0 part
└─sda5 8:5 0 15.9G 0 part [SWAP]
sdb 8:16 1 931.5G 0 disk
├─sdb1 8:17 1 931.5G 0 part
└─sdb9 8:25 1 8M 0 part
sdc 8:32 1 931.5G 0 disk # <-- new disk showed up here
sdd 8:48 1 931.5G 0 disk
├─sdd1 8:49 1 931.5G 0 part
└─sdd9 8:57 1 8M 0 part
...
sudo zpool replace srv 6847353731192779603 /dev/sdc -f
sudo zpool status
pool: srv
state: DEGRADED
status: One or more devices is currently being resilvered. The pool will
continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
scan: resilver in progress since Fri Mar 22 15:50:21 2019
131G scanned out of 13.5T at 941M/s, 4h7m to go
5.40G resilvered, 0.95% done
config:
NAME STATE READ WRITE CKSUM
srv DEGRADED 0 0 0
raidz1-0 DEGRADED 0 0 0
sdb ONLINE 0 0 0
replacing-1 OFFLINE 0 0 0
6847353731192779603 OFFLINE 0 0 0 was /dev/sdc1/old
sdc ONLINE 0 0 0 (resilvering)
sdd ONLINE 0 0 0
sde ONLINE 0 0 0
sdf ONLINE 0 0 0
...
We can see it’s running at 941M/s. Not too bad.
sudo zpool destroy srv
zpool create \
-m /srv srv \
-O compression=lz4 \
raidz3 \
sdb sdc sdd sde sdf sdg sdh sdi \
sdj sdk sdl sdm sdn sdo sdp sdq \
sdr sds sdt sdu sdv sdw sdx sdy -f
Copy a lot of random data to it again (as above)
Replace a disk (as above)
allen@server:~$ sudo zpool status
pool: srv
state: DEGRADED
status: One or more devices is currently being resilvered. The pool will
continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
scan: resilver in progress since Sun Mar 24 10:07:18 2019
27.9G scanned out of 9.14T at 362M/s, 7h19m to go
1.21G resilvered, 0.30% done
config:
NAME STATE READ WRITE CKSUM
srv DEGRADED 0 0 0
raidz3-0 DEGRADED 0 0 0
sdb ONLINE 0 0 0
replacing-1 OFFLINE 0 0 0
sdd OFFLINE 0 0 0
sdc ONLINE 0 0 0 (resilvering)
sde ONLINE 0 0 0
sdf ONLINE 0 0 0
So that’s running quite a bit slower. Not exactly 1/3, but closer to it than not.
That was all about speed. What about reliability?
Our first resilver was going a lot faster, but it ended badly. Other errors popped up, some on the same VDev as was being resilvered, and so it failed.
sudo zpool status
pool: srv
state: DEGRADED
status: One or more devices has experienced an error resulting in data
corruption. Applications may be affected.
action: Restore the file in question if possible. Otherwise restore the
entire pool from backup.
see: http://zfsonlinux.org/msg/ZFS-8000-8A
scan: resilvered 571G in 5h16m with 2946 errors on Fri Mar 22 21:06:48 2019
config:
NAME STATE READ WRITE CKSUM
srv DEGRADED 208 0 2.67K
raidz1-0 DEGRADED 208 0 5.16K
sdb ONLINE 0 0 0
replacing-1 OFFLINE 0 0 0
6847353731192779603 OFFLINE 0 0 0 was /dev/sdc1/old
sdc ONLINE 0 0 0
sdd ONLINE 208 0 1
sde ONLINE 0 0 0
sdf ONLINE 0 0 0
sdg ONLINE 0 0 0
sdh ONLINE 0 0 0
sdi ONLINE 0 0 0
raidz1-1 ONLINE 0 0 0
sdr ONLINE 0 0 0
sds ONLINE 0 0 0
sdt ONLINE 0 0 0
sdu ONLINE 0 0 0
sdj ONLINE 0 0 1
sdk ONLINE 0 0 1
sdl ONLINE 0 0 0
sdm ONLINE 0 0 0
raidz1-2 ONLINE 0 0 0
sdv ONLINE 0 0 0
sdw ONLINE 0 0 0
sdx ONLINE 0 0 0
sdy ONLINE 0 0 0
sdn ONLINE 0 0 0
sdo ONLINE 0 0 0
sdp ONLINE 0 0 0
sdq ONLINE 0 0 0
errors: 2946 data errors, use '-v' for a list
Our second resilver was going very slowly, but did slow and sure when the race? It did, but very very slowly.
allen@server:~$ sudo zpool status
[sudo] password for allen:
pool: srv
state: ONLINE
scan: resilvered 405G in 6h58m with 0 errors on Sun Mar 24 17:05:50 2019
config:
NAME STATE READ WRITE CKSUM
srv ONLINE 0 0 0
raidz3-0 ONLINE 0 0 0
sdb ONLINE 0 0 0
sdc ONLINE 0 0 0
sde ONLINE 0 0 0
sdf ONLINE 0 0 0
sdg ONLINE 0 0 0
...
...
It slowed even further down, as 400G in 7 hours is something like a 16M/s. I didn’t see any checksum errors this time, but that time is abysmal.
Though, to paraphrase Livy, better late than never.
#!/usr/bin/env bash
#
# zfs-pull.sh
#
# Pulls incremental ZFS snapshots from a remote (source) server to the local (destination) server.
# Uses snapshots made by zfs-auto-snapshot. Locates the latest snapshot common to both sides
# to perform an incremental replication; if none is found, it does a full send.
#
# Usage: replicate-zfs-pull.sh <SOURCE_HOST> <SOURCE_DATASET> <DEST_DATASET>
#
# Example:
# ./replicate-zfs-pull.sh mysourcehost tank/mydata tank/backup/mydata
#
# Assumptions/Notes:
# - The local server is the destination. The remote server is the source.
# - We're using "zfs recv -F" locally, which can forcibly roll back the destination
# dataset if it has diverging snapshots. Remove or change -F as desired.
# - This script is minimal and doesn't handle advanced errors or timeouts gracefully.
# - Key-based SSH authentication should be set up so that `ssh <SOURCE_HOST>` doesn't require a password prompt.
#
set -euo pipefail
##############################################################################
# 1. Parse command-line arguments
##############################################################################
if [[ $# -ne 3 ]]; then
echo "Usage: $0 <SOURCE_HOST> <SOURCE_DATASET> <DEST_DATASET>"
exit 1
fi
SOURCE_HOST="$1"
SOURCE_DATASET="$2"
DEST_DATASET="$3"
##############################################################################
# 2. Gather snapshot lists
#
# The command zfs list -H -t snapshot -o name -S creation -d 1
# -H : Output without headers for script-friendliness
# -t snapshot : Only list snapshots
# -o name : Only list the name
# -d 1 : Only descend one level - i.e. don't tree out child datasets
##############################################################################
# - Remote (source) snapshots: via SSH to the remote host
# - Local (destination) snapshots: from the local ZFS
echo "Collecting snapshots from remote source: ${SOURCE_HOST}:${SOURCE_DATASET}..."
REMOTE_SNAPSHOTS=$(ssh "${SOURCE_HOST}" zfs list -H -t snapshot -o name -d 1 "${SOURCE_DATASET}" 2>/dev/null \
| grep "${SOURCE_DATASET}@" \
| awk -F'@' '{print $2}' || true)
echo "Collecting snapshots from local destination: ${DEST_DATASET}..."
LOCAL_SNAPSHOTS=$(zfs list -H -t snapshot -o name -d 1 "${DEST_DATASET}" 2>/dev/null \
| grep "${DEST_DATASET}@" \
| awk -F'@' '{print $2}' || true)
##############################################################################
# 3. Find the latest common snapshot
#
# The snapshots names have prefixes like "zfs-auto-snap_daily" and "zfs-auto-snap_hourly"
# that confuse sorting for the linux comm program, so we strip the prefix with sed before
# using 'comm -12' to find common elements of input 1 and 2, and tail to get the last one.
#
COMMON_SNAPSHOT=$(comm -12 <(echo "$REMOTE_SNAPSHOTS" | sed 's/zfs-auto-snap_\w*-//' | sort) <(echo "$LOCAL_SNAPSHOTS" | sed 's/zfs-auto-snap_\w*-//' | sort) | tail -n 1)
# We need the full name back for the transfer, so grep it out of the local list. Make sure to quote the variable sent to grep or you'll loose the newlines.
COMMON_SNAPSHOT=$(echo "$LOCAL_SNAPSHOTS" | grep $COMMON_SNAPSHOT)
if [[ -n "$COMMON_SNAPSHOT" ]]; then
echo "Found common snapshot: $COMMON_SNAPSHOT"
else
echo "No common snapshot found—will perform a full send."
fi
##############################################################################
# 4. Identify the most recent snapshot on the remote source
#
# This works because we zfs list'ed the snapshots originally in order
# so we can just take the first line with 'head -n 1'
##############################################################################
LATEST_REMOTE_SNAPSHOT=$(echo "$REMOTE_SNAPSHOTS" | head -n 1)
if [[ -z "$LATEST_REMOTE_SNAPSHOT" ]]; then
echo "No snapshots found on the remote source. Check if zfs-auto-snapshot is enabled there."
exit 1
fi
##############################################################################
# 5. Perform replication
##############################################################################
echo "Starting pull-based replication from ${SOURCE_HOST}:${SOURCE_DATASET} to local ${DEST_DATASET}..."
if [[ -n "$COMMON_SNAPSHOT" ]]; then
echo "Performing incremental replication from @$COMMON_SNAPSHOT up to @$LATEST_REMOTE_SNAPSHOT."
ssh "${SOURCE_HOST}" zfs send -I "${SOURCE_DATASET}@${COMMON_SNAPSHOT}" "${SOURCE_DATASET}@${LATEST_REMOTE_SNAPSHOT}" \
| zfs recv -F "${DEST_DATASET}"
else
echo "Performing full replication of @$LATEST_REMOTE_SNAPSHOT."
ssh "${SOURCE_HOST}" zfs send "${SOURCE_DATASET}@${LATEST_REMOTE_SNAPSHOT}" \
| zfs recv -F "${DEST_DATASET}"
fi
echo "Replication completed successfully!"
Here’s the troubleshooting process
Verify DNS Records according to apple’s document.
DOMAIN=gattis.org
dns-sd -q _ldap._tcp.$DOMAIN SRV
dns-sd -q _kerberos._tcp.$DOMAIN SRV
dns-sd -q _kpasswd._tcp.$DOMAIN SRV
dns-sd -q _gc._tcp.$DOMAIN SRV
Ping the results. Then test for ports according the Microsoft’s document.
HOST=dc01.gattis.org
nc -z -v -u $HOST 88
nc -z -v -u $HOST 135
nc -z -v $HOST 135
nc -z -v -u $HOST 389
nc -z -v -u $HOST 445
nc -z -v $HOST 445
nc -z -v -u $HOST 464
nc -z -v $HOST 464
nc -z -v $HOST 3268
nc -z -v $HOST 3269
nc -z -v $HOST 53
nc -z -v -u $HOST 53
nc -z -v -u $HOST 123
A useful script is like so
#!/bin/bash
HOST=dc01.gattis.local
#HOST=dc02.gattis.local
## declare an array of the commands to run
declare -a COMMANDS=(\
"nc -z -u $HOST 88"
"nc -z -u $HOST 135"
"nc -z $HOST 135"
"nc -z -u $HOST 389"
"nc -z -u $HOST 445"
"nc -z $HOST 445"
"nc -z -u $HOST 464"
"nc -z $HOST 464"
"nc -z $HOST 3268"
"nc -z $HOST 3269"
"nc -z $HOST 53"
"nc -z -u $HOST 53"
"nc -z -u $HOST 123")
PIDS=""
for i in "${COMMANDS[@]}";do
$i &
PIDS+="$! "
done
sudo apt-get install libnss-ldap ldap-utils
# To get the attribute 'memberOf'
# Simple Bind with TLS
ldapsearch -v -x -Z -D "[email protected]" -W -H ldap://ad.domain.local -b 'OU=People,DC=domain,DC=local' '(sAMAccountName=someuser)' memberOf
# older style
ldapsearch -v -D "[email protected]" -w Passw0rd -H ldap://ad1.domain.local -b 'OU=People,DC=domain,DC=local' '(sAMAccountName=someuser)' memberOf
# Get all user accounts from AD created since 2007-07.
ldapsearch -v -x -Z -D "[email protected]" -W -H ldap://ad1.domain.local -b 'DC=domain,DC=local' -E pr=1000/noprompt '(&(objectClass=user)(whenCreated>=20100701000000.0Z))' sAMAccountName description whenCreated > all
If you want to be cutting edge, ditch the filesystem altogether for object storage like RustFS rsync or minio. Though you’ll need to pair that with rclone or juicefs for programs that don’t’ talk S3 web object language (which is most things).
JuiceFS presents an object store as a filesystem. It’s useful for clustering and it’s fast and well regarded, but it requires a bit of setup. The only downside (beyond the setup) is that the data is encoded into the object store as blobs and you can’t get it out directly should something fail.
These are just some partial notes from when I tested it.
# Creating and mounting a volume once everything is installed
juicefs format --storage s3 --bucket http://127.0.0.1:9000/myjfs --access-key allen --secret-key SOMEKEY sqlite3://myjfs.db myjfs
juicefs mount sqlite3://myjfs.db ~/jfs
# Creating a JuiceFS Service
[Service]
Type=notify
NotifyAccess=main
User=rustfs-user
Group=rustfs-user
WorkingDirectory=/usr/local
EnvironmentFile=-/etc/default/rustfs
ExecStart=/usr/local/bin/rustfs \$RUSTFS_VOLUMES
LimitNOFILE=1048576
LimitNPROC=32768
TasksMax=infinity
Restart=always
RestartSec=10s
OOMScoreAdjust=-1000
SendSIGKILL=no
TimeoutStartSec=30s
TimeoutStopSec=30s
NoNewPrivileges=true
ProtectHome=true
PrivateTmp=true
PrivateDevices=true
ProtectClock=true
ProtectKernelTunables=true
ProtectKernelModules=true
ProtectControlGroups=true
RestrictSUIDSGID=true
RestrictRealtime=true
# service log configuration
StandardOutput=append:/var/log/rustfs/rustfs.log
StandardError=append:/var/log/rustfs/rustfs-err.log
[Install]
WantedBy=multi-user.target
# You may also want to
sudo apt install minio-client
mc alias set rustfs http://localhost:9000 allen somepassword
Rclone is a versatile tool with many functions, one of which is is mounting a S3 compatible object store as a file system. This lets you run minio or rustfs and still access it with programs that require file access.
You can easily implement a clustered file system with it, but it does not handle concurrent writes as JuiceFS does. But if you’re careful, it works well. Metadata retrieval is slower than JuiceFS as it doesn’t have a dedicated system for that, but it handles some of that with caching.
If file modification time is important to you - say because you’re trying to use rsync or syncthing - be aware that most back-ends don’t support that metadata field and rclone will only present it correctly for a short time while it’s cached. You’ll need to use JuiceFS.
# You may not already have fuse, which rclone needs
sudo apt install -y fuse3 curl
# as per their first (assuming preferred) installation method. Distrod such as
# debian have older versions.
# https://rclone.org/install/#script-installation
sudo -v ; curl https://rclone.org/install.sh | sudo bash
rclone --version
# You can run `rclone config`, but it's easier to place this config for a S3 compatible service
sudo vi /etc/rclone.conf
[rustfs]
type = s3
provider = Other
access_key_id = rustfsadmin
secret_access_key = rustfsadmin
endpoint = http://localhost:9000
EOF
# Assuming you've created a bucket already with a `mc mb rustfs/test` or similar.
# Launch in the foreground and Ctrl-C to exit
sudo mkdir -p /mnt/rclone/
sudo rclone mount rustfs:test /mnt/rclone --config /etc/rclone.conf --log-level INFO
# assuming it worked, in another window:
sudo ls /mnt/rclone
# Create the user and group
sudo useradd \
--system \
--user-group \
--home-dir /var/lib/rclone \
--create-home \
--shell /usr/sbin/nologin \
rclone
# Change fuse settings to allow other
sudo sed -i '/#user_allow_other/ s/^#//' /etc/fuse.conf
# Create Explicit locations for cache and mount
sudo mkdir -p /var/cache/rclone /mnt/rclone
sudo chown -R rclone:rclone /var/cache/rclone /mnt/rclone
sudo chmod -R 750 /var/cache/rclone
# Create a service unit
sudo vi /etc/systemd/system/rclone-mount.service
[Unit]
Description=Rclone Mount for Local RustFS Object Service
After=network-online.target rustfs.service
Wants=network-online.target
Requires=rustfs.service
[Service]
Type=notify
User=rclone
Group=rclone
ExecStart=/usr/bin/rclone mount rustfs:rclone /mnt/rclone \
--allow-other \
--cache-dir /var/cache/rclone \
--config /etc/rclone.conf \
--dir-cache-time 30s \
--log-level INFO \
--log-systemd \
--vfs-cache-mode full
# --log-file /var/log/rclone.log \
# --vfs-cache-max-size 1G
# --vfs-cache-max-age 48h
ExecStop=/bin/fusermount3 -u /mnt/rclone
Restart=on-failure
RestartSec=10
# Hardening (optional but recommended)
#NoNewPrivileges=true
#PrivateTmp=true
#ProtectSystem=full
#ProtectHome=true
[Install]
WantedBy=multi-user.target
sudo systemctl daemon-reload
sudo systemctl enable --now rclone-mount.service
full is the exception, not the default. Start with --vfs-cache-mode writes. Only upgrade to full if something actually breaks.
For torrent clients, the correct VFS cache mode is –vfs-cache-mode=full. Like a database, this is one of the clear-cut cases where full is not overkill.
mc clientcurl –progress-bar -L https://dl.min.io/client/mc/release/linux-amd64/mc –create-dirs -o $HOME/minio-binaries/mc
This program is getting a lot of attention because it’s iterating fast, supports erasure coding, and uses the Apache 2 license so you don’t have to worry about minio rug-pull shenanigans.
I used it for a month and like it - though when they say it’s alpha, they mean it. The devs fix bugs fast, but you will objects wont create, delete, etc for unknown reasons.
Setup a four-node cluster, each with two disks similar to https://docs.rustfs.com/installation/linux/multiple-node-multiple-disk.html
Ensure time services are running
# Make sure NTP service is active and stem clock synchronized
timedatectl status
System clock synchronized: yes
NTP service: active
Add entries to the hosts file for the other cluster members
172.31.1.1 shire1
172.31.1.2 shire2
172.31.1.3 shire3
172.31.1.4 shire4
And set up private addresses if desired
sudo vi /etc/network/interfaces
auto enp1s0f4
iface enp1s0f4 inet static
address 172.31.1.1
netmask 255.255.255.0
Prep and mount the disks (two of them in this example)
# Format the disks
sudo apt install -y xfsprogs
sudo mkfs.xfs -i size=512 -n ftype=1 -L RUSTFS0 /dev/sdc
sudo mkfs.xfs -i size=512 -n ftype=1 -L RUSTFS1 /dev/sdd
# Make the mountpoints
sudo mkdir -p /data/rustfs{0..1}
# Add to /etc/fstab
sudo tee -a /etc/fstab <<EOF
# RustFS Volumes
LABEL=RUSTFS0 /data/rustfs0 xfs defaults,noatime,nodiratime 0 0
LABEL=RUSTFS1 /data/rustfs1 xfs defaults,noatime,nodiratime 0 0
EOF
# And mount
sudo systemctl daemon-reload
sudo mount -a
Add the service user and own the mounts
sudo groupadd -r rustfs-user
sudo useradd -M -r -g rustfs-user rustfs-user
sudo chown rustfs-user:rustfs-user /data/rustfs*
Download and install the binary. I adjusted for the (possibly faster) GNU version over the (more compatible) MUSL
sudo apt install -y unzip
wget https://dl.rustfs.com/artifacts/rustfs/release/rustfs-linux-x86_64-gnu-latest.zip
unzip rustfs-linux-x86_64-gnu-latest.zip
chmod +x rustfs
sudo mv rustfs /usr/local/bin/
Create the config file, adjusted for two disks (0 and 1)
# Multiple node multiple disk mode
sudo tee /etc/default/rustfs <<EOF
RUSTFS_ACCESS_KEY=rustfsadmin
RUSTFS_SECRET_KEY=rustfsadmin
RUSTFS_VOLUMES="http://node{1...4}:9000/data/rustfs{0...1}"
RUSTFS_ADDRESS=":9000"
RUSTFS_CONSOLE_ENABLE=true
RUST_LOG=error
RUSTFS_OBS_LOG_DIRECTORY="/var/logs/rustfs/"
EOF
Adjust for whatever you named your hosts
sudo sed -i 's/node/shire/' /etc/default/rustfs
TODO - Rustfs wants to log to a new folder ’logs’. Let’s change that later
# Note: We already created the mount dirs above, but including here too as it's in the docs
sudo mkdir -p /data/rustfs{0..1} /var/logs/rustfs /opt/tls
sudo chmod -R 750 /data/rustfs* /var/logs/rustfs
Add a service unit
sudo tee /etc/systemd/system/rustfs.service <<EOF
[Unit]
Description=RustFS Object Storage Server
Documentation=https://rustfs.com/docs/
After=network-online.target
Wants=network-online.target
[Service]
Type=notify
NotifyAccess=main
User=root
Group=root
WorkingDirectory=/usr/local
EnvironmentFile=-/etc/default/rustfs
ExecStart=/usr/local/bin/rustfs \$RUSTFS_VOLUMES
LimitNOFILE=1048576
LimitNPROC=32768
TasksMax=infinity
Restart=always
RestartSec=10s
OOMScoreAdjust=-1000
SendSIGKILL=no
TimeoutStartSec=30s
TimeoutStopSec=30s
NoNewPrivileges=true
ProtectHome=true
PrivateTmp=true
PrivateDevices=true
ProtectClock=true
ProtectKernelTunables=true
ProtectKernelModules=true
ProtectControlGroups=true
RestrictSUIDSGID=true
RestrictRealtime=true
# service log configuration
StandardOutput=append:/var/logs/rustfs/rustfs.log
StandardError=append:/var/logs/rustfs/rustfs-err.log
[Install]
WantedBy=multi-user.target
EOF
Update for the dedicated service user
sudo sed -i 's/User=root/User=rustfs-user/' /etc/systemd/system/rustfs.service
sudo sed -i 's/Group=root/Group=rustfs-user/' /etc/systemd/system/rustfs.service
Enable and start the service
sudo systemctl daemon-reload
sudo systemctl enable --now rustfs
sudo systemctl status rustfs
# This seems to have an issue with perms
sudo tail -f /var/logs/rustfs/rustfs*.log
You’re usually looking for something in a file. Some phrase or word, and you don’t know where the file is at or what you named it.
For occasional search through file content at the command line you’d use grep. Something like:
grep --ignore-case --files-with-matches --recursive foo /some/file/path
This can take quite a while. There are some tweaks to grep you can add, but for source code it’s become fashionable to use use ripgrep or ugrep.
sudo apt install ugrep
# a drop-in for grep in most cases
ug --ignore-case --files-with-matches --recursive foo /some/file/path
You can also use ugrep interactively with the -Q option. Though you may prefer a whiptail style menu - see the bottom.
If you know the name of the file you want, it’s usually a matter of using the find utility or locate.
find /some/place -iname "*something*"
On large filesystems, the locate utility is much faster, as it pre-builds a database.
# 'plocate' is the modern replacement for the locate utility
sudo apt install plocate
# a systemd timer updates this database daily, but to use immediately after install you must:
sudo updatedb
sudo locate -i "*something*"
You can also use a utility like fzf which uses some fuzzy name matching and includes shell integration. This has become more popular recently.
This isn’t about search per se, but this is a great place to mention whiptail, a text user interface. This let’s you wrap a TUI around your search regardless what tool you use.
sudo apt instal whiptail
# We'll name our search utility 'f'
vi ~/bin/f
#!/bin/bash
[ -z "$1" ] && { echo "You must supply a search term."; exit; }
# Pipe the output of the search to a while loop that will build two arrays; One is a list
# of files and the other a list of those file's basenames we can use for a pretty menu.
# We'll add numbers to the menu array so that when the user chooses a basename, we will
# know what filename to pull from the other array.
FILENUMBER=0
FILENAME=
LIST=()
MENU=()
LOCATION=/home/allen/projects
ug --ignore-case --files-with-matches --recursive "$1" $LOCATION | \
{
while read FILENAME; do
LIST=( "${LIST[@]}" "$FILENAME" )
DATE=$(stat -c '%.10y' "$FILENAME")
BASENAME=$(basename "$FILENAME")
MENU=( "${MENU[@]}" "$FILENUMBER" "$DATE $BASENAME" )
((FILENUMBER++))
done
# If there are no results
[ -z "${LIST[0]}" ] && { echo "No results"; exit; }
# Return the filenumber from the user's choice so we can access that array element
while [ 1 ]; do
KEY=$( whiptail --title "Search Results" --notags --menu "Search Results" 25 78 16 "${MENU[@]}" 3>&1 1>&2 2>&3 )
[ -z $KEY ] && exit
less "${LIST[$KEY]}"
done
} # The braces are required so that the while loops can modify 'global' variables
Why don’t we index the file contents?
You’d think an index would be great - but it turns out for unstructured text files you’re indexing every word in every file, making the index is larger than the things it’s indexing.
Though lucene/elasticseach and spinx come up in conversation.
SMB is a file sharing protocol and we’re going to handle it in the context of NAS systems. For a linux system, setting it up is very simple for minimal install.
apt install samba
The docs1 say only a minimal config is needed. Here’s an example that’s backwards compatible for anonymous gest access.
vi /etc/samba/smb.conf
[global]
map to guest = bad user
min protocol = NT1
[srv]
path = /srv
read only = yes
guest ok = yes
guest only = yes
Windows clients should not prompt for credentials, but some Linux clients will and depends on the file browser in question. Creds are not checked so supply anything you like.
Possibly, for Windows 11 you must add the browseable = yes directive to [srv] or windows will also prompt you for a password just like Linux above.
You can add hosts allow = some.ip.address to the global settings to restrict access somewhat if needed. If you find the default log file location is in a non-writeable space2 you can specify with log file = /var/log/samba/%m.log
Check the config with testparm, a samba utility for checking your config syntax.
TrueNAS is a storage “appliance” in the sense that it’s a well put together linux system with a web administration layer. If you find yourself digging into the linux internals, you’re doing it wrong.
You may get an alert in the GUI along the lines of
Device: /dev/sda [SAT], 8 Currently unreadable (pending) sectors.
This is one of the predictive failures that Backblaze mentions. You should replace the drive. You may also get an outright failure such as:
Pool pool01 state is DEGRADED: One or more devices has experienced an unrecoverable error.
That’s a drive that has already failed and likewise you must replace it.
On larger systems you’ll need both the GUI and commandline.
In the GUI, examine the failing device. If it hasn’t fully faulted yet, use the /dev/sdX identifier in the alert to find it.
Storage -> (pool name) -> Manage Devices -> (Select the disk) -> Disk Info
You’ll see a serial number similar to “Z4F18FFX” and can copy it to your clipboard and offline the disk if it’s not already faulted.
ZFS Info -> Offline
Large arrays have lights to help locate the disk. Use the tool sas3ircu for this. If it’s not already present you may need to track it down (see troubleshooting below). To find out, enter the CLI by navigating to:
System -> Shell
Issue this command to search for the location of the disk.
sudo sas3ircu 0 display | grep -B 10 ZC11HPN3
Device is a Hard disk
Enclosure # : 3
Slot # : 5
...
...
Use the same utility to turn on the locate LED.
sudo sas3ircu 0 locate 3:5 ON
ASwap the physical device, and back in the GUI select the drive and in the Disk Info menu click “replace”.
Drive -> (Disk Info) -> Replace
The new windows should allow you to pick the ’new’ drive. There should be only one choice. If nothing is listed, you may need to wipe the disk wipefs -a /dev/sdX and repeat.
Don’t forget to turn off the locate LED
sudo sas3ircu 0 locate 3:5 OFF
It’s ‘strongly advised against’ using the CLI to replace the disk. The GUI takes several steps to prepare the disk and adds a partition to the pool, not the whole disk.
Use the gptid from zpool to get the device number, then off-line the disk.
sudo zpool status
raidz3-2 ONLINE 0 0 0
976e8f03-931d-4c9f-873e-048eeef08680 ONLINE 0 0 0
f9384b4f-d94a-43b6-99c4-b8af6702ca42 ONLINE 0 0 0
c5e4f2e5-62f2-41cc-a8de-836ff9683332 ONLINE 0 0 35.4K
sudo zpool offline pool01 c5e4f2e5-62f2-41cc-a8de-836ff9683332
find /dev/disk -name c5e4f2e5-62f2-41cc-a8de-836ff9683332 -exec ls -lah {} ;
lrwxrwxrwx 1 root root 11 Jan 10 08:41 /dev/disk/by-partuuid/c5e4f2e5-62f2-41cc-a8de-836ff9683332 -> ../../sdae1
sudo smartctl -a /dev/sdae1 | grep Serial
Serial Number: WJG1LNP7
sas3ircu list
sas3ircu 0 display
sas3ircu 0 display | grep -B 10 WJG1LNP7
(Inspect the output for Enclosure and Slot to use in that order below)
sudo sas3ircu 0 locate 3:5 ON
This is a physical swap - the indicator will be blinking red. Turn it off when you’re done
sudo sas3ircu 0 locate 3:5 OFF
(It’s probably the same device identifier, but you can tail the message log to make sure)
sudo dmesg | tail
sudo zpool replace pool01 c5e4f2e5-62f2-41cc-a8de-836ff9683332 sdae -f
sudo zpool status
raidz3-2 DEGRADED 0 0 0
976e8f03-931d-4c9f-873e-048eeef08680 ONLINE 0 0 0
f9384b4f-d94a-43b6-99c4-b8af6702ca42 ONLINE 0 0 0
replacing-2 DEGRADED 0 0 0
c5e4f2e5-62f2-41cc-a8de-836ff9683332 REMOVED 0 0 0
sdae ONLINE 0 0 0
If you’re using a hot spare, you may need to detach it after the resilver is finished so status as a spare is returned. Check the spare’s ID at the bottom and then detach it.
zpool status
zpool detach pool01 9d794dfd-2ef6-432d-8252-0c93e79509dc
When working at the command line, you may need download the sas3ircu utility from Broadcom.
wget https://docs.broadcom.com/docs-and-downloads/host-bus-adapters/host-bus-adapters-common-files/sas_sata_12g_p16_point_release/SAS3IRCU_P16.zip
If you forgot what light you turned on, you can turn off all slot lights with something like:
for X in {0..23};do echo sas3ircu 0 locate 2:$X OFF;done
for X in {0..11};do sas3ircu 0 locate 3:$X OFF;done
To recreate the GUI process at the command line, as adapted from https://www.truenas.com/community/resources/creating-a-degraded-pool.100/ use these commands. Though gpart and glable are not present on TrueNAS Scale, so the first set are AI informed.
# On TruNAS Scale
sudo sgdisk --zap-all /dev/sdX
sudo sgdisk -o /dev/sdX
sudo sgdisk -n 1:0:0 -t 1:BF00 /dev/sdX
# Find the UUID and replace
ls -l /dev/disk/by-partuuid/ | grep sdX
sudo zpool replace pool01 /dev/sdad /dev/disk/by-partuuid/4c4f943a-569b-4e54-b97d-9c72adf2ed5a
GoAccess is a lightweight web stats visualizer than can display in a terminal window or in a browser. It supports Caddy’s native JSON log format and can also be run as a real-time service with a little work.
If you have Debian/Ubuntu, you can add the repo as the [official docs] show.
# Note: The GoAccess docs use the `lsb_release -cs` utility that some Debains don't have, so I've substituted the $VERSION_CODENAME variable from the os-release file
wget -O - https://deb.goaccess.io/gnugpg.key | gpg --dearmor | sudo tee /usr/share/keyrings/goaccess.gpg >/dev/null
source /etc/os-release
echo "deb [signed-by=/usr/share/keyrings/goaccess.gpg arch=$(dpkg --print-architecture)] https://deb.goaccess.io/ $VERSION_CODENAME main" | sudo tee /etc/apt/sources.list.d/goaccess.list
sudo apt update
sudo apt install goaccess
No configuration is needed if your webserver is logging in a supported format1. Though you may need to adjust file permissions so the log file can be read by the user running GoAccess.
To use in the terminal, all you need to is invoke it with a couple parameters.
sudo goaccess --log-format CADDY --log-file /var/log/caddy/access.log
To produce a HTML report, just add an output file somewhere your web server can find it.
sudo touch /var/www/some.server.org/report.html
sudo chown $USER /var/www/some.server.org/report.html
goaccess --log-format=CADDY --output /var/www/some.server.org/report.html --log-file /var/log/caddy/access.log
History is useful and GoAccess lets you persist your data and incorporate it on the next run. This updates your report rather than replace it.
# Create a database location
sudo mkdir -p /var/lib/goaccess-db
sudo chown $USER /var/lib/goaccess-db
goaccess \
--persist --restore --db-path /var/lib/goaccess-db \
--log-format CADDY --output /var/www/some.server.org/report.html --log-file /var/log/caddy/access.log
To display country and city you need GeoIP databases, preferably the GeoIP2 format. An easy way to get started is with DB-IP’s Lite GeoIP2 databases that have a permissive license.
# These are for Jan 2025. Check https://db-ip.com/db/lite.php for the updated links
wget --directory-prefix /var/lib/goaccess-db \
https://download.db-ip.com/free/dbip-asn-lite-2025-01.mmdb.gz \
https://download.db-ip.com/free/dbip-city-lite-2025-01.mmdb.gz \
https://download.db-ip.com/free/dbip-country-lite-2025-01.mmdb.gz
gunzip /var/lib/goaccess-db/*.gz
goaccess \
--persist --restore --db-path /var/lib/goaccess-db \
--geoip-database /var/lib/goaccess-db/dbip-asn-lite-2025-01.mmdb \
--geoip-database /var/lib/goaccess-db/dbip-city-lite-2025-01.mmdb \
--geoip-database /var/lib/goaccess-db/dbip-country-lite-2025-01.mmdb \
--log-format CADDY --output /var/www/some.server.org/report.html --log-file /var/log/caddy/access.log
This will add a Country and ASN panel, and populate the city column on the “Visitor Hostnames and IPs” panel.
This one-time download is “good enough” for most purposes. But if you want to automate updates of the GeoIP data, you can create an account with a provider and get an API key. Maxmind offers free accounts. Sign up here and you can sudo apt install geoipupdate to get regular updates.
A reasonable way to automate is with a logrotate hook. Most systems already have this in place to handle their logs so it’s an easy add. If you’re using Apache or nginx you probably already have one that you can just add a prerotate hook to. For Caddy, something like this should be added.
sudo vi /etc/logrotate.d/caddy
/var/log/caddy/access.log {
daily
rotate 7
compress
missingok
prerotate
goaccess \
--persist --restore --db-path /var/lib/goaccess-db \
--geoip-database /var/lib/goaccess-db/dbip-asn-lite-2025-01.mmdb \
--geoip-database /var/lib/goaccess-db/dbip-city-lite-2025-01.mmdb \
--geoip-database /var/lib/goaccess-db/dbip-country-lite-2025-01.mmdb \
--log-format CADDY --output /var/www/some.server.org/report.html --log-file /var/log/caddy/access.log
endscript
postrotate
systemctl restart caddy
endscript
}
You can test this works with force option. If it works, you’ll be left with updated stats, an empty access.log, and a newly minted access.log.1.gz
sudo logrotate --force /etc/logrotate.d/caddy
For Caddy, you’ll also need to disable it’s built-in log rotation
sudo vi /etc/caddy/Caddyfile
log {
output file /path/to/your/log.log {
roll_disabled
}
}
sudo systemctl restart caddy
Of course, this runs as root. You can design that out, but you can also configure systemd to trigger a timed execution and run as non-root. Arnaud Rebillout has some good info on that.
You can also run GoAccess as a service for real-time log analysis. It works in conjunction with your web server by providing a Web Socket to push the latest data to the browser.
In this example, Caddy is logging vhosts to individual files as described in the Caddy logging page. This is convenient as it allows you view vhosts separately which is often desired. Adjust as needed.
# Create a system user with no home
sudo adduser --system --group --no-create-home goaccess
# Add the user to whatever group the logfiles are set to (ls -lah /var/log/caddy/ or whatever)
sudo adduser goaccess caddy
# Put the site in a variable to make edits easier
SITE=www.site.org
# Create a database location
sudo mkdir -p /var/lib/goaccess-db/$SITE
sudo chown goaccess:goaccess /var/lib/goaccess-db/$SITE
# Possibly create the report location
sudo touch /var/www/$SITE/report-$SITE.html
sudo chown goaccess /var/www/$SITE/report-$SITE.html
# Test that the goaccess user can create the report
sudo -u goaccess goaccess \
--log-format CADDY \
--output /var/www/$SITE/report-$SITE.html \
--persist \
--restore \
--db-path /var/lib/goaccess-db/$SITE \
--log-file /var/log/caddy/$SITE.log
sudo vi /etc/systemd/system/goaccess.service
[Unit]
Description=GoAccess Web log report
After=network.target
# Note: Variable braces are required for sysmtemd variable expansion
[Service]
Environment="SITE=www.site.org"
Type=simple
User=goaccess
Group=goaccess
Restart=always
ExecStart=/usr/bin/goaccess \
--log-file /var/log/caddy/${SITE}.log \
--log-format CADDY \
--persist \
--restore \
--db-path /var/lib/goaccess-db/${SITE} \
--output /var/www/${SITE}/report-${SITE}.html \
--real-time-html
StandardOutput=null
StandardError=null
[Install]
WantedBy=multi-user.target
sudo systemctl enable --now goaccess.service
sudo systemctl status goaccess.service
sudo journalctl -u goaccess.service
If everything went well, it should be listening on the default port of 7890
nc -zv localhost 7890
localhost [127.0.0.1] 7890 (?) open
BUT you can’t access that port unless you’re on the same LAN. You can start forwarding that port and even setup SSL in the WS config, but in most cases it’s better to handle it with a proxy.
To avoid adding additional port forwarding you can convert the websocket connection from a high-level port to a proxy path. This works with cloudflare as well.
Edit your GoAccess service unit to indicate the proxied URL.
ExecStart
...
# Note the added escape slash the the formerly last line
--real-time-html \
--ws-url wss://www.site.org:443/ws/goaccess
# If you don't add the port explicitly, GoAccess
# will 'helpfully' add the internal port (which isn't helpful), silently.
Add a proxy line to your web server. If using Caddy, add a path handler and proxy like this, and restart.
some.server.org {
...
...
handle_path /ws/goaccess* {
reverse_proxy * http://localhost:7890
}
}
Take your browser to the URL, and you should see the gear icon top left now has a green dot under it.
If the dot isn’t green you’re not connected so take a look at the troubleshooting section.
When you have lots of vhosts its useful to separate them at the log level and report separately. To do that you can use a systemd template so as to create multiple instances. Arnaud Rebillout has some details on that.
But scaling that becomes onerous. My preference is to automate report generation more frequently and skip the realtime.
Check the permissions on the files. If you accidentally typed caddy start as root it will be running as root and later runs may not be able to write log entries.
Your best bet is to open the developer tools in your browser, check the network tab and refresh. If proxying is wrong it will give you some hints.
The file /etc/goaccess/goaccess.conf can be used, just make sure to remove the arguments from the unit file so there isn’t a conflict.
Some services, such as Cloudflare, support WS but can cause intermittent disconnections. Try separating your stats from your main traffic site.
See https://goaccess.io/man –log-format options ↩︎
There are many ways to manage and produce web content. Traditionally, you’d use a large application with roles and permissions.
A more modern approach is to use a distributed version control system, like git, and a site generator.
Static Site Generators are gaining popularity as they produce static HTML with javascript and CSS that can be deployed to any Content Delivery Network without need for server-side processing.
Astro is great, as is Hugo, with the latter being around longer and having more resources.
Hugo is a Static Site Generator (SSG) that turns Markdown files into static web pages that can be deployed anywhere.
Like WordPress, you apply a ’theme’ to style your content. But rather than use a web-inteface to create content, you directly edit the content in markdown files. This lends itself well tomanaging the content as code and appeals to those who prefer editing text.
However, unlike other SSGs, you don’t have to be a front-end developer to get great results and you can jump in with a minimal investment of time.
I use Debian in this example, but any apt-based distro will be similar.
Enable and pin the Debian Backports and Testing repos so you can get recent versions of Hugo and needed tools.
Hugo requires git and go
# Assuming you have enable backports as per above
sudo apt install -t bullseye-backports git
sudo apt install -t bullseye-backports golang-go
For a recent version of Hugo you’ll need to go to the testing repo. The extended version is recommended by Hugo and it’s chosen by default.
# This pulls in a number of other required packages, so take a close look at the messages for any conflicts. It's normally fine, though.
sudo apt install -t testing hugo
In some cases, you can just install from the Debian package with a lot less effort. Take a look at latest and copy the URL into a wget.
https://github.com/gohugoio/hugo/releases/latest
wget https://github.com/gohugoio/hugo/releases/download/v0.124.1/hugo_extended_0.124.1_linux-amd64.deb
A quick test right from the quickstart page to make sure everything works
hugo new site quickstart
cd quickstart
git init
git submodule add https://github.com/theNewDynamic/gohugo-theme-ananke themes/ananke
echo "theme = 'ananke'" >> config.toml
hugo server
Open up a browser to http://localhost:1313/ and you you’ll see the default ananke-themed site.
The ananke theme you just deployed is nice, but a much better theme is Docsy. Go give that a try.
Docsy is a good-looking Hugo theme that provides a landing page, blog, and a documentation sub-sites using bootstrap CSS.
The documentation site in particular let’s you turn a directory of text files into a documentation tree with relative ease. It even has a collapsible left nav bar. That is harder to find than you’d think.
Docsy requires Hugo. Install that if you haven’t already. It also needs a few other things; postcss, postcss-cli, and autoprefixer from the Node.JS ecosystem. These should be installed in the project directory as version requirements change per theme.
mkdir some.site.org
cd some.site.org
sudo apt install -t testing nodejs npm
npm install -D autoprefixer
npm install -D postcss
npm install -D postcss-cli
Deploy Docsy as a Hugo module and pull in the example site so we have a skeleton to work with. We’re using git, but we’ll keep it local for now.
git clone https://github.com/google/docsy-example.git .
hugo server
Browse to http://localhost:1313 and you should see the demo “Goldydocs” site.
Now you can proceed to configure Docsy!
The Docsy theme gets regular updates. To incorporate those you only have to run this command. Do this now, actually, to get any theme updates the example site hasn’t incoporated yet.
cd /path/to/my-existing-site
hugo mod get -u github.com/google/docsy
hugo
Error: Error building site: POSTCSS: failed to transform “scss/main.css” (text/css)>: Error: Loading PostCSS Plugin failed: Cannot find module ‘autoprefixer’
And then when you try to install the missing module
The following packages have unmet dependencies: nodejs : Conflicts: npm npm : Depends: node-cacache but it is not going to be installed
You may have already have installed Node.JS. Skip trying to install it from the OS’s repo and see if npm works. Then proceed with postcss install and such.
Let’s change the basics of the site in the config.toml file. I put some quick sed commands here, but you can edit by hand as well. Of note is the Github integration. We prepoulate it here for future use, as it allows quick edits in your browser down the road.
SITE=some.site.org
GITHUBID=someUserID
sed -i "s/Goldydocs/$SITE/" config.toml
sed -i "s/The Docsy Authors/$SITE/" config.toml
sed -i "s/example.com/$SITE/" config.toml
sed -i "s/example.org/$SITE/" config.toml
sed -i "s/google\/docsy-example/$GITHUBID\/$SITE/" config.toml
sed -i "s/USERNAME\/REPOSITORY/$GITHUBID\/$SITE/" config.toml
sed -i "s/https:\/\/policies.google.com//" config.toml
sed -i "s/https:\/\/github.com\/google\/docsy/https:\/\/github.com\/$GITHUBID/" config.toml
sed -i "s/github_branch/#github_branch/" config.toml
If you don’t plan to translate your site into different languages, you can dispense with some of the extra languages as well.
# Delete the 20 or so lines starting at "lLanguage] and stopping at the "[markup]" section,
# including the english section.
vi config.tml
# Delete the folders from 'content/' as well, leaving 'en'
rm -rf content/fa content/no
You should also set a default meta description or the engine will put in in the bootstrap default and google will summarize all your pages with that
vi config.toml
[params]
copyright = "some.site.org"
privacy_policy = "/privacy"
description = "My personal website to document what I know and how I did it"
Keep and eye on the site in your browser as you make changes. When you’re ready to start with the main part of adding content, take a look at the next section.
You can’t dispense with the en folder yet, as it breaks some github linking functionality you may want to take advantage of later
This is a quick excerpt from the Docsy Content and Customization docs. Definitely spend time with those after reading the overview here.
Content is, appropriately enough, in the content directory, and it’s subdirectories line up with the top-level navigation bar of the web site. About, Documentation, etc corresponds to content/about, content/docs and so on.
The directories and files you create will be the URL that you get with one important exception, filenames are converted to a ‘slug’, mimicking how index files work. For example, If you create the file docs/tech/mastadon.md the URL will be /docs/tech/mastadon/. This is for SEO (Search Engine Optimization).
The other thing you’ll see are _index.html files. In the example above, the URL /docs/tech/ has no content, as it’s a folder. But you can add a _index.md or .html to give it some. Avoid creating index.md or tech.md (a file that matches the name of a subdirectory). Either of those will block Hugo from generating content for any subdirectories.
The landing page itself is the content/_index.html file and the background is featured-background.jpg. The other top-nav pages are in the content folders with _index files. You may notice the special header variable “menu: main: weight: " and that is what flags that specific page as worth of being in the top menu. Removing that, or adding that (and a linkTitle) will change the top nav.
One of the most important features of the Docsy template is the well designed documentation section that features a Section menu, or left nav bar. This menu is built automatically from the files you put in the docs folder, as long as you give them a title. (See Front Matter, below). They are ordered by date but you can add a weight to change that.
It doesn’t collapse by default and if you have a lot of files, you’ll want to enable that.
# Search and set in your config.toml
sidebar_menu_compact = true
The example files have a section at the top like this. It’s not strictly required, but you must have at least the title or they won’t show up in the left nav tree.
---
title: "Examples"
---
In addition to normal markdown or html, you’ll see frequent use of ‘shortcodes’ that do things that normal markdown can’t. These are built in to Hugo and can be added by themes, and look like this;
{{% blocks/lead color="dark" %}}
Some Important Text
{{% /blocks/lead %}}
Docsy supports mermaid and a few other tools for creating illustrations from code, such as KaTeX, Mermaid, Diagrams.net, PlantUML, and MarkMap. Simply use a codeblock.
```mermaid
graph LR
one --> two
```
Once you’re satisfied with what you’ve got, tell hugo to generate the static files and it will populate the folder we configured earlier
hugo
Everything you need is in the public folder and all you need do is copy it to a web server. You can even use git, which I advise since we’re already using it to pull in and update the module.
If you have a large directory structure full of markdown files already, you can kick-start the process of adding frontmatter like this;
find . -type f | \
while read X
do
TITLE=$(basename ${X%.*})
FRONTMATTER=$(printf -- "---\ntitle = ${TITLE}\n---")
sed -i "1s/^/$FRONTMATTER\n/" "$X"
done
You may have noticed the links on the right like “Edit this page” that takes one to Github. Let’s set those up.
Go to github and create a new repository. Use the name of your side for the repo name, such as “some.site.org”. If you want to use something else, you can edit your config.toml file to adjust.
You man have noticed that Github suggested some next steps with a remote add using the name “origin”. Docsy is already using that, however, from when you cloned it. So we’ll have to pick a new name.
cd /path/to/my-existing-site
git remote add github https://github.com/yourID/some.site.org
Let’s change our default banch to “main” to match Github’s defaults.
git branch -m main
Now we can add, commit and push it up to Github
git add --all
git commit -m "first commit of new site"
git push github
You’ll notice something interesting when you go back to look at Github; all the contributers on the right. That’s because you’re dealing with a clone of Docsy and you can still pull in updates and changes from original project.
It may have been better to clone it via github
Grav is considered a CMS and is my first choice for anything that should alow direct page edits to update content. It also makes a pretty good wiki if combining that with flat files for your content is important.
Note: This content is circa 2018 so take it with a gain of salt for config details.
This has mostly to do with your registrar and hosting. Suffice to say, if you don’t have a domain name you won’t be able to complete the certificate steps and have meaningful SSL. Even if you don’t care about SSL, a DNS name is pretty useful.
We’re installing just the light version as we need only the minimal set of modules.
sudo apt install nginx-light
A minimal install of Debian 9 does not include PHP, nor would it be the right version if it did. You must install php 7.1 and the grav requirements1. The docs there mention php-openssl and php-session, but these seem to be compiled in and enabled by default in the Debian packages23. You also need to install php-fpm as there’s no php module like apache4.
Note: not sure about php-zip. It may have been included by one of the other packs because I went back to add and it was already installed
sudo apt install php-common php-curl php-ctype php-dom php-gd php-json php-mbstring php-simplexml php-xml php-fpm
# possibly sudo apt install php-zip
In this example we’re on Debian stable so we’re installing the stretch backports enabling us to install and use the current certbot, per their instruction5.
sudo echo "deb http://deb.debian.org/debian stretch-backports main" > /etc/apt/sources.list.d/stretch-backports.list
sudo apt update;sudo apt upgrade
sudo apt-get install certbot python-certbot-nginx -t stretch-backports
Let’s configure the firewall, nginx, certbot and file permissions.
We only need two ports; 80 and 443. On ubuntu you can use ufw. On debian you can issue iptables commands such as:
sudo iptables -A INPUT -p tcp -m tcp --dport 80 -m conntract --ctstate NEW,ESTABLISHED -j ACCEPT
sudo iptables -A INPUT -p tcp -m tcp --dport 443 -m conntract --ctstate NEW,ESTABLISHED -j ACCEPT
Nginx comes with a somewhat verbose default config. Let’s clean that up, arrange for a virtual host specific site, and block any random scans6. We can do that on the SSL site as well thanks to nginx’s support of SNI7. We also need to add a rewrite so that short-pathing works and you don’t get 404s for all sub pages8.
# Remove the default config files and content
sudo rm /etc/nginx/sites-available/*
sudo rm /etc/nginx/sites-enabled/*
sudo rm /var/www/html/*
# Make a directory for your grav deployment and the default site
sudo mkdir /var/www/html/default
sudo mkdir /var/www/html/cms.yourdomain.com
# Create a generic SSL key so we can inspect all requests[^7] and do some basic SSL testing
sudo mkdir /etc/nginx/ssl
sudo openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout /etc/nginx/ssl/nginx.key -out /etc/nginx/ssl/nginx.crt
# Define a default site simply drops requests (return code 444) if they didn't request a specific hostname
sudo vi /etc/nginx/sites-available/default
server {
listen 80 default_server;
server_name _;
return 444;
}
server {
listen 443 ssl default_server;
server_name _;
ssl_certificate /etc/nginx/ssl/nginx.crt;
ssl_certificate_key /etc/nginx/ssl/nginx.key;
return 444;
}
# Define the site that will host grav
sudo vi /etc/nginx/sites-available/cms.yourdomain.com
#
# Redirect requests for the site that are unencrypted
#
server {
listen 80;
server_name cms.yourdomain.com;
return 301 https://$host$request_uri;
}
#
# Site cms.yourdomain.com
#
server {
listen 443 ssl;
server_name cms.yourdomain.com;
ssl_certificate /etc/nginx/ssl/nginx.crt;
ssl_certificate_key /etc/nginx/ssl/nginx.key;
root /var/www/html/cms.yourdomain.com;
# Add index.php to the list if you are using PHP
index index.html index.htm index.nginx-debian.html index.php;
location / {
# if file doesn't exist (and it shouldn't unless you
# put a file there), pass to the index.php for routing
try_files $uri $uri/ /index.php?$query_string;
}
# Some additional security settings from the grav example
location ~* /(\.git|cache|bin|logs|backup|tests)/.*$ { return 403; }
location ~* /(system|vendor)/.*\.(txt|xml|md|html|yaml|yml|php|pl|py|cgi|twig|sh|bat)$ { return 403; }
location ~* /user/.*\.(txt|md|yaml|yml|php|pl|py|cgi|twig|sh|bat)$ { return 403; }
location ~ /(LICENSE\.txt|composer\.lock|composer\.json|nginx\.conf|web\.config|htaccess\.txt|\.htaccess) { return 403; }
location ~ \.php$ {
include snippets/fastcgi-php.conf;
fastcgi_pass unix:/var/run/php/php7.0-fpm.sock;
}
}
Now let’s do some basic testing to make sure it’s working
# Enable the sites, test for syntax errors then load the config in nginx
sudo ln -rs /etc/nginx/sites-available/default /etc/nginx/sites-enabled
sudo ln -rs /etc/nginx/sites-available/cms.yourdomain.com /etc/nginx/sites-enabled
sudo nginx -t
sudo systemctl reload nginx
# This tests that requests without the right server name are dropped
curl http://localhost
curl -k https://localhost
# This tests that the right server name does work and that non-ssl is redirected.
curl --header 'Host: cms.yourdomain.com' http://localhost
curl --header 'Host: cms.yourdomain.com' -k https://localhost
One enables php by including a ’location’ snippit in the nginx config file and allowing for index php files. We snuck that in above when we configured nginx. So let’s just create a php info file and make sure it’s working. As an added bonus the contents of that web page will tell you where your php.ini files is, if you do need to make any php config changes.
sudo vi /var/www/html/cms.yourdomain.com/info.php
<?php
phpinfo();
?>
curl --header 'Host: cms.yourdomain.com' -k https://localhost/info.php
sudo rm /var/www/html/cms.yourdomain.com/info.php
The Grav docs recommend some optional modules9 and state You should run a PHP opcache and usercache10….
Opcache has replaced apcu and should be installed and enabled by default11. Recreate the info.php from above and search for OPcache to verify
PECL-yaml is the simply the php native yaml. You can install it and test that’s it’s enabled by
php -i | grep yaml
sudo apt install php-yaml
php -i | grep yaml
In the advanced section they mention memcached and redis, but these are for distributed systems and are slower than OPcache
We’re using certbot to automate the installation and maintenance of free certificates from Let’s Encrypt. It will scan your config and ask which site you want to use. After it does it’s thing, it will ask you if you want to redirect. Choose no on that as you’ve already set that up.
When it’s done it will insert the appropriate stanza in your nginx config file and a cron job to renew the cert in your /etc/cron.d/ folder
sudo certbot --nginx
...
...
Which names would you like to activate HTTPS for?
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
1: cms.yourdomain.com
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Select the appropriate numbers separated by commas and/or spaces, or leave input
blank to select all options shown (Enter 'c' to cancel): 1
...
...
Please choose whether or not to redirect HTTP traffic to HTTPS, removing HTTP access.
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
1: No redirect - Make no further changes to the webserver configuration.
2: Redirect - Make all requests redirect to secure HTTPS access. Choose this for
new sites, or if you're confident your site works on HTTPS. You can undo this
change by editing your web server's configuration.
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Select the appropriate number [1-2] then [enter] (press 'c' to cancel): 1
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Congratulations! You have successfully enabled https://cms.yourdomain.com
You can verify this with curl by asking for the site without the insecure option
curl https://cms.yourdomain.com
Editing content means modifying files under the web-root. This is done by the web server itself (for on-line editing), syncing files via sftp and direct edits from the shell.
Managing this type of access is best done with group permissions and ACLs. Using simple file group ownership and umask breaks down when you have multiple ways in.
Luckily, there is already a group for this in debian:www-data. So lets add ourself to the group, own the directory to the group and set the existing and future permissions so the group can edit files12.
sudo usermod -aG www-data allen
sudo chown www-data:www-data /var/www/html/cms.yourdomain.com
sudo apt install acl
sudo setfacl --recursive --modify "g:www-data:rwx,d:g:www-data:rwx" /var/www/html/cms.yourdomain.com
Grav strongly recommends moving the extracted directory and not just copying the contents. I suspect this is so hidden .htaccess files and file permissions are kept, neither of which we need. But let’s go ahead and do as they ask, then change to be like we want.
sudo apt install unzip
cd ~
wget -O grav-admin.zip https://getgrav.org/download/core/grav-admin/latest
unzip grav-admin.zip
mv grav-admin /var/www/html/cms.yourdomain.com
sudo chown -R www-data:www-data /var/www/html/cms.yourdomain.com
sudo setfacl --recursive --modify "g:www-data:rwx,d:g:www-data:rwx" /var/www/html/cms.yourdomain.com
rm -rf grav-admin
You should be able to access the default page now at cms.yourdomain.com, and the admin panel at /admin. The admin panel will ask you set set a password and from there you’re free to move on to picking themes and adding content
https://askubuntu.com/questions/323005/php-openssl-extension-has-a-package ↩︎
https://support.qualityunit.com/021373-How-To-Enable-Session-Support-for-PHP ↩︎
https://www.digitalocean.com/community/tutorials/how-to-install-linux-nginx-mysql-php-lemp-stack-on-debian-9 ↩︎
https://certbot.eff.org/lets-encrypt/debianstretch-nginx.html ↩︎
https://serverfault.com/questions/559095/nginx-responding-to-unknown-host-names/841643#841643 ↩︎
http://nginx.org/en/docs/http/configuring_https_servers.html#sni ↩︎
https://github.com/getgrav/grav/blob/master/webserver-configs/nginx.conf ↩︎
https://learn.getgrav.org/15/basics/requirements#optional-modules ↩︎
https://learn.getgrav.org/15/advanced/performance-and-caching#performance ↩︎
https://support.cloud.engineyard.com/hc/en-us/articles/205411888-PHP-Performance-I-Everything-You-Need-to-Know-About-OpCode-Caches ↩︎
TODO Add a page about the ACLs ↩︎
Automating deployment as part of a general continuous integration strategy is best-practice these days. Web content should be similarly treated.
I.e. version controlled and deployed with git.
Let’s create a two-tiered system that goes from dev to prod using a post-commit trigger
graph LR Development --git / rsync---> Production
The Development system is your workstation and the Production system is a web server you can rsync to. Git commit will trigger a build and rsync.
I use Hugo in this example, but any system that has an output (or build) folder works similarly.
The first thing we need is a destination.
This server probably uses folders like /var/www/XXXXX for its web root. Use that or create a new folder and make yourself the owner.
sudo mkdir /var/www/some.site.org
sudo chown -R $USER /var/www/some.site.org
echo "Hello" > /var/www/some.site.org/index.html
Edit your web server’s config to make sure you can view that web page. Also check that rsync is available from the command line.
Hugo builds static html in a public directory. To generate the HTML, simply type hugo
cd /path/to/my-existing-site
hugo
ls public
We don’t actually want this folder in git and most themes (if you’re using Hugo) already have a .gitignore file. Take a look and create/add to it.
# Notice /public is at the top of the git ignore file
cat .gitignore
/public
package-lock.json
.hugo_build.lock
...
Assuming you have some content, let’s add and commit it.
git add --all
git commit -m "Initial Commit"
Note: All of these git commands work because pulling in a theme initialized the directory. If you’re doing something else you’ll need to git init.
The last step is to create a hook that will build and deploy after a commit.
cd /path/to/my-existing-site
touch .git/hooks/post-commit
chmod +x .git/hooks/post-commit
vi .git/hooks/post-commit
#!/bin/sh
hugo --cleanDestinationDir
rsync --recursive --delete public/ [email protected]:/var/www/some.site.org
This script ensures that the remote directory matches your local directory. When you’re ready to update the remote site:
git add --all
git commit --allow-empty -m "trigger update"
If you mess up the production files, you can just call the hook manually.
cd /path/to/my-existing-site
touch .git/hooks/post-commit
bash: line 1: rsync: command not found
Double check that the remote host has rsync.
When testing web servers you must occasionally be explicit as to what web site you’re asking for. This is done using Server Name Indication, or SNI. When testing What-Ifs, you must sometimes send the wrong name to see what the response is.
For curl, you use the --resolve flag for that.
# If you want to test the SNI function, use --resolve
curl -k -I --resolve cms.yourdomain.com:443:127.0.0.1 --header 'Host: cms.yourdomain.com' https://cms.yourdomain.com
Caddy is a web server that runs SSL by default by automatically grabing a cert from Let’s Encrypt. It comes as a stand-alone binary, written in Go, and makes a decent reverse proxy.
Caddy recommends “using our official package for your distro” and for debian flavors they include the basic instructions you’d expect. I recommend the testing build for the short-term there’s an bug with loging wild-cards sites (like *.you.org) that are handy to have available.
sudo apt install -y debian-keyring debian-archive-keyring apt-transport-https curl
curl -1sLf 'https://dl.cloudsmith.io/public/caddy/testing/gpg.key' | sudo gpg --dearmor -o /usr/share/keyrings/caddy-testing-archive-keyring.gpg
curl -1sLf 'https://dl.cloudsmith.io/public/caddy/testing/debian.deb.txt' | sudo tee /etc/apt/sources.list.d/caddy-testing.list
chmod o+r /usr/share/keyrings/caddy-testing-archive-keyring.gpg
chmod o+r /etc/apt/sources.list.d/caddy-testing.list
sudo apt update
sudo apt install caddy
If you ever add a module to caddy, chances are it will be lost when you next apt upgrade. This is because caddy phones home, requests a build for any non-standard module, and replaces itself. The devs don’t think this will be an issue. In any event, you should prepare for it, because you will be adding modules.
Because we’ve diverted, apt upgrade will not upgrade caddy. You must use caddy upgrade instead. Since this stops unattended-updates from any automation of vulnerability patching, maybe add a cron job if you like.
sudo dpkg-divert --divert /usr/bin/caddy.default --rename /usr/bin/caddy
sudo mv ./caddy /usr/bin/caddy.custom
sudo update-alternatives --install /usr/bin/caddy caddy /usr/bin/caddy.default 10
sudo update-alternatives --install /usr/bin/caddy caddy /usr/bin/caddy.custom 50
sudo systemctl restart caddy
The easiest way to configure Caddy is by editing the Caddyfile
sudo vi /etc/caddy/Caddyfile
You define websites with a block that includes a root and the file_server directive. Once you reload, and assuming you already have the DNS in place, Caddy will reach out to Let’s Encrypt, acquire a certificate, and automatically redirect from port 80 to 443
site.your.org {
root * /var/www/site.your.org
file_server
}
You can add basic auth to a site by creating a hash and adding a directive to the site.
caddy hash-password
site.your.org {
root * /var/www/site.your.org
file_server
basic_auth {
allen SomeBigLongStringFromTheCaddyHashPasswordCommand
}
}
Caddy also makes a decent reverse proxy.
site.your.org {
reverse_proxy * http://some.server.lan:8080
}
You can also take advantage of path-based reverse proxy. Note the rewrite to accommodate the trailing-slash potentially missing.
site.your.org {
rewrite /audiobooks /audiobooks/
handle_path /audiobooks/* {
uri strip_prefix /audiobooks/
reverse_proxy * http://some.server.lan:8080
}
}
You can define common elements at the top (snippets) or in files and import them multiple times to save duplication. This helps when you have many sites.
# At the top in the global section of your Caddyfile
(logging) {
log {
output file /var/log/caddy/access.log
}
}
site.your.org {
import logging
reverse_proxy * http://some.server.lan:8080
}
Caddy is a single binary so when adding a new module (aka feature) you are actually downloading a new binary that has it compiled in. You can find the list of packages at their download page.
Do this at the command line with caddy itself.
sudo caddy add-package github.com/mholt/caddy-webdav
systemctl restart caddy
Now you can add the webdav directive to a site. You can have the same both serve up a directory listing and handle webdav with a litte extra config. The new module uses different ‘verbs’ when working, allowing it to cooexist as long as file_server gets the first shot.
# Global Option Block
{
order webdav last
}
site.your.org {
root * /var/www/site.your.org
file_server browse
webdav *
}
You can test your config file and look at the logs like so
caddy validate --config /etc/caddy/Caddyfile
journalctl --no-pager -u caddy
Caddy has an individual cert for every virtual host you create. This is fine, but Let’s Encrypt publishes these as part of certificate transparency and the bad guys are watching. If you create a new site in caddy, you’ll see bots probing for weaknesses within 30 min - without you even having published the URL. There’s no security in anonymity, but need-to-know suggests we shouldn’t be informing the whole world about sites of limited scope.
One solution is a wildcard cert. It’s published as just ‘*.some.org’ so there’s no information disclosed. These are usally done with DNS integration. Caddy supports this and it requires a little extra work, but it does make it a lot easier to sort out the riff-raff later.
We’ll connect to CloudFlare’s DNS service. Check https://github.com/caddy-dns to see if your DNS provider is available. If you haven’t already configured a system-alternative for caddy, now is the time to do so.
sudo caddy add-package github.com/caddy-dns/cloudflare
sudo systemctl restart caddy.service
For Cloudflare, a decent example is below. Just use the ‘Getting the Cloudflare API Token’ part
https://roelofjanelsinga.com/articles/using-caddy-ssl-with-cloudflare/
Use the acme-dns global option and then create a single site (used to determine the cert) and match the actual vhosts with subsites.
{
acme_dns cloudflare alotcharactersandnumbershere
}
*.some.org, some.org {
@site1 host site1.some.org
handle @site1 {
reverse_proxy * http://localhost:3200
}
@site2 host site2.some.org
handle @site2 {
root * /srv/www/site2
}
}
In general, you should create a snippet and import into each block.
#
# Global Options Block
#
{
...
...
}
#
# Importable Snippets
#
(logging) {
log {
output file /var/log/caddy/access.log
}
}
#
# Host Blocks
#
site.your.org {
import logging
reverse_proxy * http://some.server.lan:8080
}
other.your.org {
import logging
reverse_proxy * http://other.server.lan:8080
}
If you want separate logs for each vhosts you can still use a snippit by adding a parameter for the site name.
#
# Global Options Block
#
{
...
...
}
#
# Importable Snippets
#
(logging) {
log {
output file /var/log/caddy/{args[0]}.log
}
#
# Host Blocks
#
site.your.org {
# Add the argument "site.your.org" at the end sets the log file name to "site.your.org.log"
import logging site.your.org
reverse_proxy * http://some.server.lan:8080
}
other.your.org {
import logging other.your.org
reverse_proxy * http://other.server.lan:8080
}
Wildcard sites only have one block but you must use the hostname matcher to separate logs. Note the slight difference in the handling of log names.
#
# Global Options Block
#
{
...
...
}
#
# Importable Snippets
#
(logging) {
log {
output file /var/log/caddy/{args[0]}
}
#
# Main Block
#
*.site.org, site.org {
# Everything goes to the file named here (access.log) unless it's filtered out by a more specific log block
import logging access.log
# This site will write to the access.log
@site host other.site.org
handle @site {
reverse_proxy * http://other.site
}
# This site will write to just this log file. Make sure to import logging BEFORE the '@www host' matcher or requests don't get logged here.
import logging www.site.org.log
@www host www.site.org
handle @www {
root * /var/www/www.site.org
file_server
}
If you want to track users that have authenticated, add a directive to the global headers so their usernames are included in the request log.
#
# Global Options Block
#
{
servers {
log_credentials
}
}
By default, only caddy can read the log files. This is a problem when you have a log analysis package. In recent versions of caddy however, you can set the mode.
log {
output file /var/log/caddy/access.log {
mode 644
}
}
If the log file doesn’t change modes, check the version of caddy. It must be newer than v2.8.4 for the change and you must restart.
You can have a case where the domain specific file never gets created. This usually happens when there us nothing to write to it. Check the hostname is correct.
Web servers are constantly under attack. So much so some people way back before Y2K termed this “internet background radiation”.
There are two broad categories of attacks; ones that are just probing IPs, and ones that know a little about your web presence.
If an adversary just connects to your IP without knowing the domain, you can:
The default configuration, and one that is consistent with the protocol, is to return an error. But the general consensus is to drop invalid requests. This prevents adversaries from profiling the system.
If they know your domains it more difficult. But they are usually asking for things that don’t exist, or trying to access a login page as fast as possible to guess passwords. In either case, you should apply a limit to how fast they can make requests.
So as a fist step, let’s drop or rate-limit every request.
A default config will take connections to port 80, announce that it’s a Caddy web server and redirect you to https. Instead, let’s look at the host headers and SNI (Server Name Indicator) first. Then we can immediately abort any requests for things we don’t even have.
#
# Global Options Block
#
{
...
...
}
#
# Importable Sections
#
...
...
#
# Site Blocks
#
# Accept secure requests specific to our domain
https://*.some.org, https://some.org {
# Serve any known host
@some host some.site.org
handle @some {
encode
reverse_proxy * http://some:8080
}
@other host other.site.org
handle @other {
import auth
encode
reverse_proxy * http://other:8080
}
# Abort any unknown host
handle {
abort
}
}
# Accept any insecure request to our domain
http://*.some.org, http://some.org {
# Redirect only known host headers to https
@secure {
host some.site.org
host other.site.org
}
handle @secure {
redir https://{host}{uri} permanent
}
# Abort anything else.
handle {
abort
}
}
# Handle requests without the right domain or SNI
# so we can abort them early
:80, :443 {
# Log it so we have a record of the probe
import logging
abort
}
Other attackers will connect to a valid domain and making requests for potential exploits. You’ll send them back 401 Unauthorized or 404 Not Found, but these are bots and they’ll just keep chugging along, ignoring the errors.
A basic protection is to rate-limit your responses. You’re already aborting requests for invalid sites, but if you’re hosting a web form or basic-auth protected site it can be useful. While a web app using a form may have it’s own rate limiting mechanism, there’s no inherent limit to the BA prompt. The only way to handle this is apply a rate limit to the specific site.
sudo caddy add-package github.com/mholt/caddy-ratelimit
...
...
https://*.some.org, https://some.org {
# A fairly generous over-all limit
rate_limit {
zone descriptive-name-1 {
key {http.request.remote_ip}
events 200
window 5s
}
}
@some host some.site.org
handle @some {
# A conservative one for this site
rate_limit {
zone descriptive-name-1 {
key {http.request.remote_ip}
events 20
window 5s
}
}
encode
reverse_proxy * http://other:8080
}
}
You’ll need to tune this as different sites may have radically different setups, some needing just one large request and another 50 small ones to fill a page load. It’s also mostly targeted at mitigating denial of service and credential stuffing. A distributed or patient bot makes it through just fine. Importantly, if you’re using a proxy service like Cloudflare, you’ll need to pull out the actual client IP.
For actual threat detection & response, consider adding Fail2Ban or CrowdSec
Caddy can also serve WebDAV requests with the appropriate module. This is important because for many clients, such as Kodi, WebDAV is significantly faster.
sudo caddy add-package github.com/mholt/caddy-webdav
sudo systemctl restart caddy
{ # Custom modules require order of precedence be defined
order webdav last
}
site.your.org {
root * /var/www/site.your.org
webdav *
}
You can combine WebDAV and Directly Listing - highly recommended - so you can browse the directory contents with a normal web browser as well. Since WebDAV doesn’t use the GET method, you can use the @get filter to route those to the file_server module so it can serve up indexes via the browse argument.
site.your.org {
@get method GET
root * /var/www/site.your.org
webdav *
file_server @get browse
}
https://github.com/mholt/caddy-webdav https://marko.euptera.com/posts/caddy-webdav.html
The package caddy-security offers a suite of auth functions. Among these is MFA and a portal for end-user management of tokens.
# Install a version of caddy with the security module
sudo caddy add-package github.com/greenpau/caddy-security
sudo systemctl restart caddy
/var/lib/caddy/.local/caddy/users.json
caddy hash-password
journalctl –no-pager -u caddy
Cloudflare offers and excellent free service that proxies your web server, removing you from the front lines of the internet and blocking many known bad-actors. That deserves it’s own section, but we should make changes to caddy’s logging to accommodate the proxy. Most specifically, to log the actual IP address of the end user.
Cloudflare will inject the header CF-Connecting-IP into the request and we can use that to identify the end-user. This is preferred over other headers, which can be easily spoofed. But in order to make sure it’s Cloudfare doing the injecting, and not some internet rando, we need to explicitly know all the Cloudflare exit node IPs to trust them. That would be hard to manage, but happily, there’s a handy [caddy-cloudflare-ip] module for that. Many thanks to WeidiDeng!
sudo caddy add-package github.com/WeidiDeng/caddy-cloudflare-ip
sudo vi /etc/caddy/Caddyfile
#
# Global Options Block
#
{
servers {
trusted_proxies cloudflare
client_ip_headers CF-Connecting-IP
}
}
After restarting Caddy, we can see the header change
sudo head /var/log/caddy/access.log | jq '.request'
sudo tail /var/log/caddy/access.log | jq '.request'
Before
"remote_ip": "172.68.15.223",
"client_ip": "172.68.15.223",
After
"remote_ip": "172.71.98.114",
"client_ip": "109.206.128.45",
You can now update any rate_limit from the remote_ip to the client_ip
rate_limit {
zone descriptive-name-1 {
key {http.request.client_ip} # Uses the real client IP
events 20
window 5s
}
}
HAProxy is the pre-eminent open source software load balancer, as evidenced by its nearly ubiquitous use by instagram, dropbox, twitter, etc. It incorporates many layer 4 (TCP) and layer 7 (HTTP) features that provide advanced load-balancing, protection and logging.
HAProxy can prompt for basic auth. This saves you from modifying back-end services.
Change a backend like this:
backend logs
mode http
server logs 10.40.0.8:5601 check
To:
backend logs
mode http
server logs 10.40.0.8:5601 check
acl authorized http_auth(users)
http-request auth unless authorized
http-request del-header Authorization
This requires a user list as well, so I included this at the bottom
userlist users
user admin password aBigLongHashHere
It seems one can add groups and such for more granular control. A remote password database would be a better solution.
https://blog.sleeplessbeastie.eu/2018/03/08/how-to-define-basic-authentication-on-haproxy/
Combine HAProxy and Certbot on the same container host allowing Certbot to keep certificates updated and HAProxy to proxy all traffic. HAProxy will redirect insecure requests to secure ports except for Certbot’s proof-of-ownership requests. Those are proxied to Certbot.
This example uses Docker.
Let’s create a private docker network for our containers to use. That will let them connect dynamically to each other by container name so we can avoid hard-coding IPs.
docker network create -d bridge proxy-nw
It’s also useful to have a local web server to test against. Throw up a darkhttpd web server (a very simple web server) on the same container host, redirecting a high-level port on the container host to it’s port 80.
mkdir /tmp/www
echo "Look ma, http" > /tmp/www/index.html
docker run --name darkhttpd --net=proxy-nw --detach --volume /tmp/www:/www --publish 8008:80 shelmangroup/darkhttpd
curl localhost:8008
If everything went as planned you should get the expected output and docker logs darkhttpd will show the web request.
The creators of HAProxy suggest avoiding local dependencies by adding config and certificate files to the official image to create a custom image. For our tests however, we’ll start with a local config as we need it to work with a local Certbot instance.
The first step is to create volumes to hold the local configs; one for HAProxy and one for Certbot. The docker image provided by HAProxy defaults to /usr/local/etc/haproxy/haproxy.cfg. Certbot defaults to /etc/letsencrypt. The cert itself is in a subdirectory under ’live’ named for the site, such as live/server.your.org, but you need all of the subdirectories to persist or renewals won’t work right. We’ll present to the containers in the run command below.
docker volume create haproxy
docker volume create certbot
# Check the location just in case
docker volume inspect haproxy
Next, create a config file for HAProxy to specifies the darkhttpd server as a back-end server. Notice that darkhttpd itself is listening on port 80. Port 8008 above is the translated-port we published on the container host for testing and not what you want HAProxy to connect to.
sudo vi /var/lib/docker/volumes/haproxy/_data/haproxy.cfg
global
maxconn 4000
tune.ssl.default-dh-param 2048
frontend http
bind *:80
mode http
default_backend http-server
timeout client 1h
backend http-server
mode http
server darkhttpd darkhttpd:80
timeout connect 1h
timeout server 1h
Now lets run the HAProxy image.
docker run \
--detach \
--mount type=volume,source=haproxy,target=/usr/local/etc/haproxy,readonly \
--mount type=volume,source=certbot,target=/etc/letsencrypt/,readonly \
--name haproxy \
--net=proxy-nw \
--publish 80:80 \
--publish 443:443 \
--restart always \
haproxy
curl localhost
If all went well, you can now access the host on port 80 and hit the darkhttpd server. Importantly, you must be able to hit this from the Internet, as Let’s Encrypt will be hitting it to prove you own the host in question. Issue a docker ps and docker logs haproxy if something failed. Once you’re done testing, stop the containers and remove the unneeded darkhttpd server.
docker stop haproxy
docker stop darkhttpd; docker rm darkhttpd; docker rmi shelmangroup/darkhttpd
The official image is designed to run-and-exit, requiring the container OS to have a cron job to manage renewals. Our goal is for Certbot manage the certificate without external dependencies, and to be protected from general internet traffic by HAProxy. So let’s create our own image based on theirs.
# Make a directory
mkdir ~/docker-certbot
cd ~/docker-certbot
# Create the Docker file and a script that will renew the cert
touch Dockerfile
touch cert_request
chmod +x cert_request
Create the following Docker file. It uses Certbot’s image as a base, places a cert request script and launches the cron daemon in the foreground (crond doesn’t run by default). With crond running this way you can examine the output easily with a docker log command.
vi Dockerfile
FROM certbot/certbot
COPY cert_request /etc/periodic/daily
ENTRYPOINT ["crond", "-f"]
Below is the content of the script. The COPY above puts it in a location where crond will run it once a day. Let’s Encrypt actually recommends twice a day but this is close enough. We also concatenate the public and private key into a single file as required by HAProxy.
The reason we’re going to all this trouble is that issued certs are valid for 90 days. You have to check often.
TODO We could probably do something slick here and not recreate that file if we spent some time thinking about it. We could also use –post-hook to tell haproxy something has changed or at least see if that’s needed.
#!/bin/sh
HOST=server.your.org
EMAIL=[email protected]
certbot certonly $1 --standalone -d $HOST --non-interactive --preferred-challenges http --agree-tos --email $EMAIL
cat /etc/letsencrypt/live/$HOST/fullchain.pem /etc/letsencrypt/live/$HOST/privkey.pem > /etc/letsencrypt/live/$HOST/$HOST.pem
Let’s build the image and publish port 80. We won’t want that published later on but it’s useful for a first test so we know Certbot itself doesn’t have any issues.
docker build -t allen/certbot .
docker run --name certbot --detach --publish 80:80 --mount type=volume,source=certbot,target=/etc/letsencrypt allen/certbot
Let’s run the first request manually against Let’s Encrypt’s test system to make sure it works as expected.
# Attach to the running container
docker exec -it certbot sh
# Execute the request script aginst let's encrypt's test system
/etc/periodic/daily/cert_request --staging
You should see output similar to:
- Congratulations! Your certificate and chain have been saved at: /etc/letsencrypt/live/server.your.org/fullchain.pem
If not, start digging into if your web server is reachable or the host name is wrong. Importantly, add –staging to the request while you are debugging. The production Let’s Encrypt service is rate-limited.
Now that you know Certbot works, recreate it without an external port (this is easier than a restart). We’ll use HAProxy to route requests to it. Note: now that it has a certificate, certbot will execute a renew operation which no longer requires the verification of accepting port 80 challenges. The private key it already has is used. But you still want to catch requests as any new certs you add later will require the verification.
docker stop certbot
docker rm certbot
docker run --name certbot --detach --net=proxy-nw --mount type=volume,source=certbot,target=/etc/letsencrypt --restart always allen/certbot
The HAProxy config file needs an update, both to handle reqeusts to Certbot and to take advantage of the certificate we got earlier.
sudo vi /var/lib/docker/volumes/haproxy/_data/haproxy.cfg
global
tune.ssl.default-dh-param 2048
defaults
timeout client 1h
timeout connect 1h
timeout server 1h
frontend http
bind *:80
mode http
acl certbot-acl path_beg /.well-known/acme-challenge/
http-request redirect scheme https if ! certbot-acl
use_backend certbot-backend if certbot-acl
frontend https
mode http
bind *:443 ssl crt /etc/letsencrypt/live/wiki.your.org/wiki.your.org.pem
backend certbot-backend
mode http
server certbot certbot:80
docker stop haproxy
docker start haproxy
Let’s do some basic tests
# Test that non-secure requests are redirected with a 302
curl -D - http://server.your.org/xyz
HTTP/1.1 302 Found
Cache-Control: no-cache
Content-length: 0
Location: https://wiki.your.org/xyz
# Test that non-secure requests to acme-challenge are not
curl http://server.your.org/.well-known/acme-challenge/
<html><body><h1>503 Service Unavailable</h1>
No server is available to handle this request.
</body></html>
# Take a look at the certificate we have
openssl s_client -showcerts -connect localhost:443
...
...
Verify return code: 20 (unable to get local issuer certificate)
These are as expected. Certbot isn’t isn’t in the middle of a request so nothing is listening. The cert displayed is from the testing server and so untrusted.
Now, let’s try requesting a cert from the production server and assuming it shows you success, restart HAProxy to use the new cert.
docker exec -it certbot sh
/etc/periodic/daily/cert_request --force-renew
exit
docker stop haproxy
docker start haproxy
openssl s_client -showcerts -connect localhost:443
...
...
Verify return code: 0 (ok)
Now that you have a working proxy, you’ll probably want to proxy some things other than certbot. Let’s add an example xwiki running on it’s own network.
docker network connect xwiki-nw haproxy
# To the haproxy conf file, add
...
...
frontend https
mode http
bind *:443 ssl crt /etc/letsencrypt/live/wiki.your.org/wiki.your.org.pem
acl is_wiki hdr(host) -i wiki.your.org
use_backend xwiki if is_wiki
backend xwiki
mode http
server xwiki xwiki:8080
...
...
docker stop haproxy
docker start haproxy
And enjoy!
Container exits immediately
Check the logs with docker logs haproxy to see what’s up.
‘server certbot’ : could not resolve address ‘certbot’…Failed to initialize server(s) addr.
HAProxy wont start if it can’t connect to the back-end server. Adding the init-addr option accomodates that.
cat: can’t open ‘/etc/letsencrypt/live/server.your.org/fullchain.pem**
Certbot can start saving certs under -0001 and such when you start requesting certs that it already knows about but can’t find. You can delete the /etc/letsencrypt/* dir
Moving a Custom Container
docker save allen/certbot >test.tar scp some.server:test.tar . docker load -i test.tar
To Remove old containers and images docker ps -a docker rm … docker images docker rmi … docker system prune –all docker image prune –all
Other Resources
https://www.lab-time.it/2018/09/20/running-haproxy-and-lets-encrypt-on-docker/ https://docs.docker.com/storage/volumes/ https://www.haproxy.com/blog/the-four-essential-sections-of-an-haproxy-configuration/ https://certbot.eff.org/docs/install.html https://wiki.alpinelinux.org/wiki/Alpine_Linux:FAQ#My_cron_jobs_don.27t_run.3F https://gist.github.com/andyshinn/3ae01fa13cb64c9d36e7
Login to the DNS server and create a CNAME to the proxy02 server
vi some file
some entry
sudo vi /var/lib/docker/volumes/haproxy/_data/haproxy.cfg
frontend https
...
...
acl SOMESERVER-acl hdr(host) SOMESERVER.marietta.edu
use_backend SOMESERVER-backend if SOMESERVER-acl
...
...
backend SOMESERVER-backend
mode http
server SOMESERVER 10.40.1.216:8080 check
(see below)
Login to your proxy server and attach to the running certbot
docker exec -it certbot sh
vi /etc/periodic/daily/cert_request
# add to the variable at top
/etc/periodic/daily/cert_request
vi /var/lib/docker/volumes/haproxy/_data/haproxy.cfg
# add 'cert path/certname.pem' along with the others
docker kill -s HUP haproxy
You can check the config by attaching and running the test parameter
docker exec -it haproxy bash
haproxy -c -V -f /usr/local/etc/haproxy/haproxy.cfg
The below does a graceful restart. This is preferred over the hard stop and start
docker kill -s HUP haproxy
Pronounced “lighty”, it describes itself as being two to three times faster than apache. More importantly, it consumes very few resources; important on a small system.
sudo apt-get install lighttpd
Configuration files are in /etc/lighttpd. Interestingly, one edits (or replaces) the files in the conf-available folder, which correspond to the available modules, and then turns them on with the lighty-enable-mod command.
Generate a self-signed certificate. (remember to use the public facing DNS name when it asks for Common Name)
sudo openssl req -new -x509 -keyout /etc/lighttpd/server.pem -out /etc/lighttpd/server.pem -days 3650 -nodes
sudo chmod 400 /etc/lighttpd/server.pem
Enable SSL and restart (see SSL notes below)
sudo lighty-enable-mod ssl
sudo service lighttpd restart
Let’s say we want to SSL wrap service, like transmission, that may not provide SSL on it’s own. In this example it’s running on the same host on port 9091
Edit the config file
cd /etc/lighttpd/conf-available
sudo cp -a 10-proxy.conf 10-proxy.conf.bak
sudo vim 10-proxy.conf
And make it look like so
server.modules += ( "mod_proxy" )
$HTTP["url"] =~ "^/transmission/" {
proxy.server = ( "" =>
(
( "host" => "127.0.0.1",
"port" => "9091"
)
)
)
}
And enable and restart
sudo lighty-enable-mod proxy
sudo service lighttpd force-reload
SSL -
At one time you edited the main conf file /etc/lighttpd/lighttpd.conf so it contained:
$SERVER["socket"] == ":443" {
ssl.engine = "enable"
ssl.pemfile = "/etc/lighttpd/server.pem"
}
But as of Ubuntu 13+, the above is included in a sub conf file referenced in the main file
include_shell "/usr/share/lighttpd/include-conf-enabled.pl"
So if you try to add it in the main, you get a red-herring error message of
(network.c.379) can’t bind to port: 443 Address already in use
The better way is a virtual host file that combines all the settings for one service, in one place.
One would do that by adding include “domain1.com.conf” at the bottom of lighttpd.conf file and start it with a $HTTP[“host”] == “www2.example.org” and then some directives. Haven’t tried it though.
http://redmine.lighttpd.net/projects/lighttpd/wiki/Docs:ModProxy#Example
Make the existing log files readable by apache and change the log rotation config so the new ones are too.
sudo chown root:www-data /var/log/apache2/*
sudo vi /etc/logrotate.d/apache2
...
...
create 640 root www-data
...
...
Make the apache logs viewable in the config
sudo vi /etc/apache2/sites-enabled/you.your.org.conf
<VirtualHost *:80>
ServerAdmin webmaster@localhost
DocumentRoot /var/www/test.marietta.edu
ErrorLog ${APACHE_LOG_DIR}/error.log
CustomLog ${APACHE_LOG_DIR}/access.log combined
...
...
...
Alias "/logs" "/var/log/apache2"
<Directory "/var/log/apache2">
Options Indexes
Require all granted
</Directory>
Note:
You may have to change the execute bit on the directory so the group has permission to list the contents.
https://serverfault.com/questions/961749/changing-default-apache-log-permissions
This replaced Apache’s Web Server as the go-to after 2010 and is still in the top spot.
When you’re proxying a back-end server that’s making let’s encrypt challenges for it’s own purposes, you need a webroot to handle those requests.
vi /etc/nginx/conf.d/some.server.org.conf
#
# Redirect requests for the site that are unencrypted, except for let's encrypt challenges
#
server {
listen 80;
server_name some.your.org;
location / {
return 301 https://$host$request_uri;
}
# Allow access to the ACME Challenge for Let's Encrypt
location ^~ /.well-known/acme-challenge {
allow all;
root /var/www/some.your.org/htdocs;
}
}
server {
listen 443 ssl;
...
...
With nginx, WebDAV is only available in edge. (or at least it was in 2018 or so)
sed -i -e 's/v[[:digit:]]\..*\//edge\//g' /etc/apk/repositories
apk update;apk upgrade
Then you can install nginx and the htpasswd too
apk add nginx-mod-http-dav-ext
apk add apache2-utils
htpasswd -c /etc/nginx/conf.d/passwords user1
sudo mkdir /var/lib/nginx/tmp/
server {
listen 80 default_server;
listen [::]:80 default_server;
# Try some media
location /video/ {
alias /srv/media/video/;
autoindex on;
dav_ext_methods PROPFIND OPTIONS;
auth_basic "Authentication Required";
auth_basic_user_file /etc/nginx/conf.d/passwords;
}
}
The best approach to maintaining content on a small-scale webserver is to use WebDAV. This lets you remain with to HTTP protocols and conventions on for the whole stack. No messing about with firewalls, ftp or windows sharing.
On the content creator’s side, it’s built into the Operating System so they can use the File Manager they use everyday to manage their web content just like a local file. This simplifies support greatly.
Apache is the only major web server that fully supports the native file managers on major OSs (as of 2022). Nginx almost get’s there in our testing, but write-level access doesn’t quite work with Mac OS Finder. IIS is similarly lacking.
We’re installing from repo and enabling the DAV modules using the debian tools rather than creating the symlinks ourselves.
sudo apt-get update
sudo apt-get install apache2
sudo a2enmod dav dav_fs dav_lock
The best method is to use a site-based approach where we put our content and config in their own locations. We’re using the debian conventions for file locations here.
# Create the content folder
sudo mkdir /var/www/www.your.org
sudo chown www-data:www-data /var/www/www.your.org
# Check ownership on the WebDAV file lock database folder
# and chown -R www-data:root as needed.
ls -la /var/lock/apache2
# Create the config file
sudo vi /etc/apache2/sites-enabled/www.your.org
Here’s our example site.
DavLockDB /var/lock/apache2/DavLock
<VirtualHost *:80>
ServerAdmin [email protected]
DocumentRoot /var/www/www.your.org
ServerName www.example.org
ErrorLog ${APACHE_LOG_DIR}/wwwerror.log
CustomLog ${APACHE_LOG_DIR}/access.log combined
<Directory /var/www/test.your.local/>
DAV On
AuthType Digest
AuthName "webdav"
AuthUserFile /etc/apache2/users.password
<LimitExcept GET POST OPTIONS HEAD>
Require user test
</LimitExcept>
</Directory>
</VirtualHost>
Useful notes:
If you leave out the DocumentRoot, apache will use a default value and present the default web site even when you have no sites enabled, driving you nuts until you finally find a hint about the default value being compiled differently for different distributions.
For authentication you should at least use HTTP Digest Authentication. Though it’s generally recognized that SSL should be deployed everywhere.
These two methods appear functionally equivalent. The first is the example from the Require Directive
<RequireAny>
Require method GET POST OPTIONS
Require valid-user
</RequireAny>
The second is the example from the WEBDAV module. NOTE - if you leave the ‘all granted’ you seem to mask
Require all granted
<LimitExcept GET POST OPTIONS>
Require valid-user
</LimitExcept>
Apache’s LimitExcept directive is part of the core module, and the webdav example uses it, so we’re going with it here.
This article suggest the three methods below. HEAD is a common one, so it makes sense.
<Directory />
<LimitExcept GET POST HEAD>
deny from all
</LimitExcept>
</Directory>
Or possibly
<LimitExcept GET POST OPTIONS PROPFIND>
The AuthBasicProvider is, in this case, optional, since file is the default value for this directive.
Now that you’re requiring auth, you have to compare it against a user database. LDAP is a possibility.
First, enable the LDAP module:
a2enmod authnz_ldap
Then config with an ldap source.
DavLockDB /usr/local/apache2/var/DavLock
<VirtualHost *:80>
ServerAdmin webmaster@localhost
DocumentRoot /var/www/test.your.local
ErrorLog ${APACHE_LOG_DIR}/error.log
CustomLog ${APACHE_LOG_DIR}/access.log combined
<Directory /var/www/test.your.local/>
DAV On
AuthName "YOUR"
AuthType Basic
AuthBasicProvider ldap
AuthLDAPBindDN [email protected]
# Creds are "svcapache/XXXXXX"
AuthLDAPBindPassword XXXXX
AuthLDAPURL "ldap://ldap.your.local:389/OU=People,DC=your,DC=local?uid?sub?(objectClass=*)"
Require all granted
<LimitExcept GET POST OPTIONS>
Require valid-user
</LimitExcept>
</Directory>
</VirtualHost>
You occasionally have to copy a web site. The best way is to copy the source with rsync. But if you’re in the position of somebody like The Internet Archive and don’t have source access, or the site has logic and databases you can’t recreate, you’ll need to suck it down with wget.
wget -nv -nH -E -r -l 1 -k -p >> http://www.some.site
-nv Non-verbose output Turn off verbose without being completely quiet (use -q for that), which means that error messages and basic information still get printed.
-nH no-host-directories. Disable generation of host-prefixed directories. By default, invoking wget with -r »http://fly.srk.fer.hr/ will create a structure of directories beginning with fly.srk.fer.hr/. This option disables such behavior. py the files.
-E html-extension. When you’re mirroring a remote site that uses .asp pages, but you want the mirrored pages to be viewable on your stock Apache server, this option will cause the suffix .html to be appended to the local filename.
-r recursive. Turn on recursive retrieving.
-l N-depth depth. Specify recursion maximum depth level depth.The default maximum depth is 5.
-k convert-links. After the download is complete, convert the links in the document to make them suitable for local viewing.
-p page-requisites. T his option causes Wget to download all the files that are necessary to properly display a given HTML page. This includes such things as inlined images, sounds, and referenced stylesheets.
wget -nv -m -np -nH -p >>http://www.site.org/files
-m tells wget(1) to turn on options suitable for mirroring (as in -r -N -l inf -nr).
-np (no-parent) option will prevent ascending to the parent directory.
-nH disables generation of host-prefixed directories, so you will not get a directory called www.site.org.
-p causes wget(1) to download all the files that are necessary to properly display a given HTML page.
adapted from »http://www.htdig.org/howto-mirror.html
You can also check for broken links via:
wget --recursive -nd -nv --delete-after >>http://www.some.site/ > wget.out 2>&1
grep -B 1 404 wget.out
wget --http-user=XXX --http-passwd=XXX https://some.site/some.file
When re-serving up a web site, you may need to edit the mime.types file, such as in /etc/apache2, by finding the text/html line and adding for example cfm (for coldfusion) as you must tell apache that is HTML too.
When renaming, that option doesn’t work as well as one would like as wget doesn’t handle image maps and flash links, so you have to just stick with the original name.
Monitoring is usually about metrics and alerts. You’re concerned about status and performance - is it up and how’s it doing? and when do we need to buy more?
Prometheus is a time series database, meaning it’s optimized to store and work with data organized in time order. It includes in it’s single binary:
This allows you to collect and manage data with fewer tools and less complexity than other solutions.
End-points normally expose metrics to Prometheus by making a web page available that it can poll. This is done by including a instrumentation library (provided by Prometheus) or simply adding a listener on a high-level port that spits out some text when asked.
For systems that don’t support Prometheus natively, there are a few add-on services to translate. These are called ’exporters’ and translate things such as SNMP into a web format Prometheus can ingest.
You can also alert on the data collected. This is through the Alert Manager, a second package that works closely with Prometheus.
You still need a dashboard tool like Grafana to handle visualizations, but you can get started quite quickly with just Prometheus.
Install from the Debian Testing repo. The normal repo can be up to a year behind.
# Add testing repo
echo 'deb http://deb.debian.org/debian testing main' | sudo tee -a /etc/apt/sources.list.d/testing.list
# Pin testing down to a low level so the rest of your packages don't get upgraded
sudo tee -a /etc/apt/preferences.d/not-testing << EOF
Package: *
Pin: release a=testing
Pin-Priority: 50
EOF
# Live Dangerously - this will pull in a lot of packages
sudo apt update
sudo apt install -t testing prometheus
# Observe the two new services
systemctl list-units --type service -q prometheus*
# If this is a VM, the pulled-in openipmi service isn't supported and can be removed
sudo apt remove openipmi
Let’s replace the default config with something slightly simpler, then remove anything old.
sudo mv /etc/prometheus/prometheus.yml /etc/prometheus/prometheus.yml.orig
sudo tee /etc/prometheus/prometheus.yml << EOF
scrape_configs:
- job_name: node
static_configs:
- targets: ["localhost:9100"]
EOF
sudo rm -rf /var/lib/prometheus/metrics2/*
sudo systemctl restart prometheus
The ‘scrapes’ are it’s jobs, and it has only one; check out the system at ’localhost:9100’ which happens to be the node exporter service running on itself.
For every target listed, the scraper makes a web request for /metrics/ and stores the results. It ingests all the data presented and stores them for 15 days. You can choose to ignore certain elements or retain differently by adding config, but by default it takes everything given.
You can access the Web UI to see the status of your targets at:
http://some.server:9090/classic/targets
The data can be viewed by selecting Graph at the top, and then choosing from the first dropdown labeled “insert metric at cursor”.
The data is often prefixed with a category. For instance, the node-exporter service is written in the GO language and it provides some stats about that, prefixed with ‘go_’. Information about the system will be prefixed with ’node_’.
A simple Graph tab is available as well.
You can also see the node-exporter service’s output by asking like Prometheus would. Hit it up directly in your browser.
http://some.server:9100/metrics
Data can be simple, like:
go_gc_duration_seconds_sum 3
Or it can be dimensional which is accomplished by adding labels. In the example below, the value of go_gc_duration_seconds has 5 labeled sub-sets.
go_gc_duration_seconds{quantile="0"} 4.5697e-05
go_gc_duration_seconds{quantile="0.25"} 7.814e-05
go_gc_duration_seconds{quantile="0.5"} 0.000103396
go_gc_duration_seconds{quantile="0.75"} 0.000143687
go_gc_duration_seconds{quantile="1"} 0.001030941
In this example, the value of net_conntrack_dialer_conn_failed_total has several.
net_conntrack_dialer_conn_failed_total{dialer_name="alertmanager",reason="refused"} 0
net_conntrack_dialer_conn_failed_total{dialer_name="alertmanager",reason="resolution"} 0
net_conntrack_dialer_conn_failed_total{dialer_name="alertmanager",reason="timeout"} 0
net_conntrack_dialer_conn_failed_total{dialer_name="alertmanager",reason="unknown"} 0
net_conntrack_dialer_conn_failed_total{dialer_name="default",reason="refused"} 0
net_conntrack_dialer_conn_failed_total{dialer_name="default",reason="resolution"} 0
net_conntrack_dialer_conn_failed_total{dialer_name="default",reason="timeout"} 0
net_conntrack_dialer_conn_failed_total{dialer_name="default",reason="unknown"} 0
net_conntrack_dialer_conn_failed_total{dialer_name="snmp",reason="refused"} 0
net_conntrack_dialer_conn_failed_total{dialer_name="snmp",reason="resolution"} 0
net_conntrack_dialer_conn_failed_total{dialer_name="snmp",reason="timeout"} 0
net_conntrack_dialer_conn_failed_total{dialer_name="snmp",reason="unknown"} 0
How is this useful? It allows you to do aggregations - such as looking at all the net_contrack failures, and also look at the failures that were specifically refused. All with the same data.
You may have a target you want to remove. Such as a typo hostname that is now causing a large red bar on a dashboard. You can remove that mistake by enabling the admin API and issuing a delete
sudo sed -i 's/^ARGS.*/ARGS="--web.enable-admin-api"/' /etc/default/prometheus
sudo systemctl reload prometheus
curl -s -X POST -g 'http://localhost:9090/api/v1/admin/tsdb/delete_series?match[]={instance="badhost.some.org:9100"}'
The default retention is 15 days. You may want less than that and you can configure --storage.tsdb.retention.time=1d similar to above. ALL data has the same retention, however. If you want historical data you must have a separate instance or use VictoriaMetrics.
Let’s get something interesting to see by adding some OS metrics
If you can’t start the prometheus server, it may be an issue with the init file. Test and Prod repos use different defaults. Add some values explicitly to get new versions running
sudo vi /etc/default/prometheus
ARGS="--config.file="/etc/prometheus/prometheus.yml --storage.tsdb.path="/var/lib/prometheus/metrics2/"
OpenIPMI
Pinning Repos
This is a service you install on your end-points to make CPU/Memory/Etc. metrics available to Prometheus.
On each device you want to monitor, install the node exporter with this command.
sudo apt install prometheus-node-exporter
Do a quick test to make sure it’s responding to scrapes.
curl localhost:9100/metrics
Back on your Prometheus server, add these new nodes as a job in the prometheus.yaml file. Feel free to drop the initial job where Prometheus was scraping itself.
global:
scrape_interval: 15s
scrape_configs:
- job_name: 'servers'
static_configs:
- targets:
- some.server:9100
- some.other.server:9100
- and.so.on:9100
sudo systemctl reload prometheus.service
You can check the status of your new targets at:
http://some.server:9090/classic/targets
A lot of data is provided by the node by default. This can have a noticeable impact on low-power systems. To reduce this, customize the node’s config to enable only specific collectors.
In the case below we reduce collection to just CPU, Memory, and Hardware metrics. When scraping a Pi 3B, this reduces the Scrape Duration from 500 to 50ms.
# Do this on the node you're monitoring
sudo sed -i 's/^ARGS.*/ARGS="--collector.disable-defaults --collector.hwmon --collector.cpu --collector.meminfo"/' /etc/default/prometheus-node-exporter
sudo systemctl restart prometheus-node-exporter
The available collectors are listed on the page:
SNMP is one of the most prevalent (and clunky) protocols still widely used on network-attached devices. But it’s a good general-purpose way to get data from lots of different makes of products in a similar way.
But Prometheus doesn’t understand SNMP. The solution is a translation service that acts a a middle-man and ’exports’ data from those devices in a way Prometheus can.
Assuming you’ve already installed Prometheus, add some SNMP tools and the exporter. You’ll need to enable an additional repository for the mib-downloader.
sudo apt install software-properties-common
sudo apt-add-repository non-free
sudo apt update
sudo apt install -t testing snmp snmp-mibs-downloader
sudo apt install -t testing prometheus-snmp-exporter
Change the SNMP tools config file to allow use of installed MIBs. It’s disabled by default.
# The line 'mibs:' in the file overrides the default path.
# Change it to include all MIBs
sudo sed -i 's/^mibs .*/mibs +ALL/' /etc/snmp/snmp.conf
We need a target, so assuming you have a switch somewhere and can enable SNMP on it, let’s query the switch for its name, AKA sysName. Here we’re using version “2c” of the protocol with the read-only password “public”. Pretty standard.
There are some well-known SNMP objects you can query, like System Name.
# Get the first value (starting at 0) of the sysName object
snmpget -Oqv -v 2c -c public some.switch.address sysName.0
Some-Switch
# If the .0 index isn't populated, you can use 'getnext' to find the first one that is
snmpgetnext -v 2c -c public some.switch.address sysName
Some vendors will release their own custom MIBs. These provide additional data for things that don’t have well-known objects. Add the MIBs to the system and ‘walk’ the tree to see what’s interesting.
# Unifi, for example
sudo cp UBNT-MIB.txt UBNT-UniFi-MIB.txt /usr/share/snmp/mibs
# snmpwalk doesn't look for enterprise sections by default
# You must look in the MIB and add the specific company's OID number.
grep enterprises UBNT-*
...
UBNT-MIB.txt: ubnt OBJECT IDENTIFIER ::= { enterprises 41112 }
...
snmpwalk -v2c -c public ubiquiti.switch.address enterprises.41112
UBNT-UniFi-MIB::unifiRadioIndex.1 = INTEGER: 0
...
...
Note: If you get back an error or something with a long number in the key (like SNMPv2-SMI::enterprises.41112.1.6.1.1.1.1.1 = INTEGER: 0) , double check the changes in the snmp.conf file.
You may see this error message when walking.
Bad operator (INTEGER): At line 73 in /usr/share/snmp/mibs/ietf/SNMPv2-PDU
If so, replace a file as described in this excellent post by telcoM.This indicates the SNMP
sudo wget --output-document=/usr/share/snmp/mibs/ietf/SNMPv2-PDU https://pastebin.com/raw/p3QyuXzZ
Failing to correct this will break the snmp-generator and you may want to use that later on.
Now that we can query data using command line tools, let’s configure Prometheus to do so.
Add a new job to the prometheus.yaml file. This job will include the targets as normal, but also the path (since it’s different than default) and an optional parameter called module that’s specific to the SNMP exporter.
This job also does uses something you may not have seen before, a relabel_config
This is because Prometheus isn’t actually taking to the switch, it’s talking to the local SNMP exporter service. So we put all the targets normally, and then at the bottom ‘oh, by the way, do a switcheroo’. This allows Prometheus to display all the data normally with no one the wiser.
...
...
scrape_configs:
- job_name: 'snmp'
static_configs:
- targets:
- some.switch.address
metrics_path: /snmp
params:
module: [if_mib]
relabel_configs:
- source_labels: [__address__]
target_label: __param_target
- source_labels: [__param_target]
target_label: instance
- target_label: __address__
replacement: 127.0.0.1:9116 # The SNMP exporter's real hostname:port.
No configuration on the exporter side is needed. Reload the config and check the target list. Then examine data in the graph section. Add additional targets as needed and the exporter will pull in the data.
http://some.server:9090/classic/targets
These metrics are considered well known and so will appear in the database named sysUpTime and upsBasicBatteryStatus and not be prefixed with snmp_ like you might expect.
If you have something non-standard, or you simply don’t want that huge amount of data in your system, look at the link below to customize the SNMP collection with the Generator.
SNMP Exporter Generator Customization
server returned HTTP status 400 Bad Request
When Prometheus queries the prometheus-snmp-exporter.service it may get that error back, indicating you have a SNMP error. This happened to me after an upgrade introduced a breaking change with the file produced from the snmp-generator.
You must re-generate any files it produced, and make sure to correct any MIB errors as with Fixing a Bad Operator above.
snmp-mibs-downloader isn’t found
The snmp-mibs-downloader is just a handy way to download a bunch of default MIBs so when you use the tools, all the cryptic numbers, like “1.3.6.1.2.1.17.4.3.1” are translated into pleasant names.
If you can’t find the mibs-downloader its probably because it’s in the non-free repo and that’s not enabled by default. We used a tool above, but you can do that manually as well if you search the internet for the how-to.
The generator is already installed. It came with the SNMP exporter. But if you have a device that supplies it’s own MIB (and many do), you should add that to the default location.
# Mibs are often named SOMETHING-MIB.txt
sudo cp -n *MIB.txt /usr/share/snmp/mibs/
You must identify the values you want to capture. Using snmpwalk is a good way to see what’s available, but it helps to have a little context.
The data is arranged like a folder structure that you drill-down though. The folder names are all numeric, with ‘.’ instead of slashes. So if you wanted to get a device’s sysName you’d click down through 1.3.6.1.2.1.1.5 and look in the file 0.
snmpwalk -v2c -c public 10.10.202.246 1.3.6.1.2.1.1.5
SNMPv2-MIB::sysName.0 = STRING: RickeySelby131
Why did sysName get put there? A bunch of folks got together (the ISO folks) and decided everything in advance. Since trying to remember all those numbers is a bit painful, they made some handy files (MIBs) and passed them out so you can jump right to what you want.
snmpwalk -v2c -c public 10.10.202.246 sysName.0
SNMPv2-MIB::sysName.0 = STRING: RickeySelby131
They allow vendors to create their own sections as well, for things that might not fit anywhere else.
A good place to start is looking at what the vendor made available. You see this by “walking” their section (descending down through all the folders from a starting point).
You’ll get descriptive names as long as you already coped their MIB and enabled all in the snmp.conf file. If not, you can put it in your working directory and include it at the command line.
# The SysobjectID identifies the vendor section
# Note use of the MIB name without the .txt
$ snmpwalk -m +SOMEVENDOR-MIB -v 2c -c public some.address SysobjectID
SNMPv2-MIB::sysObjectID.0 = OID: SOMEVENDOR-MIB::somevendoramerica
# Then walk the vendor section using the name from above
$ snmpwalk -m +SOMEVENDOR-MIB -v 2c -c some.address somevendoramerica
SOMEVENDOR-MIB::model.0 = STRING: SOME-MODEL
SOMEVENDOR-MIB::power.0 = INTEGER: 0
...
...
# Also check out the general System section
$ snmpwalk -m +SOMEVENDOR-MIB -v 2c -c public some.address system
# You can also walk the whole ISO tree. In some cases,
# there are thousands of entries and it's indecipherable
$ snmpwalk -m +SOMEVENDOR-MIB -v 2c -c public some.system iso
This can be a lot of information and you’ll need to do some homework to see what data you want to collect.
The exporter’s default configuration file is snmp.yml and contains about 57 Thousand lines of config. It’s designed to pull data from whatever you point it at. Basically, it doesn’t know what device it’s talking to, so it tries to cover all the bases.
This isn’t a file you should edit by hand. Instead, you create instructions for the generator and it look though the MIBs and create one for you. Here’s an example for a Samlex Invertor.
vim ~/generator.yml
modules:
samlex:
walk:
- sysLocation
- inverterMode
- power
- vin
- tempDD
- tempDA
Generate a new snmp.yml to replace the default.
prometheus-snmp-generator generate
sudo cp /etc/prometheus/snmp.yml /etc/prometheus/snmp.yml.orig
sudo cp ~/snmp.yml /etc/prometheus
sudo systemctl reload prometheus-snmp-exporter.service
Configuration in Prometheus remains the same - but since we picked a new module name we need to adjust that.
...
...
params:
module: [samlex]
...
...
sudo systemctl reload prometheus.service
by default, the names are all over the place. The SNMP Exporter Devs leave it this way because there are a lot of pre-built dashboards on downstream systems that expect the existing names.
If you are building your own downstream systems you can prefix (as is best-practice) as you like with a post generation step. This example cases them all to be prefixed with samlex_.
prometheus-snmp-generator generate
sed -i 's/name: /name: samlex_/' snmp.yml
sudo cp ~/snmp.yml /etc/prometheus
sudo systemctl reload prometheus-snmp-exporter.service
You can combine multiple systems in the generator file to create one snmp.yml file, and refer to them by the module name in the Prometheus file.
modules:
samlex:
walk:
- sysLocation
- inverterMode
- power
- vin
- tempDD
- tempDA
ubiquiti:
walk:
- something
- somethingElse
As before, you can get a preview directly from the exporter (using a link like below). This data should show up in the Web UI too.
http://some.server:9116/snmp?module=samlex&target=some.device
server returned HTTP status 400 Bad Request
If you see this error message in your http://some.server:9090/classic/targets, and:
Unknown auth ‘public_v2’
When you look directly at http://some.server:9116/snmp?module=samlex&target=a.ups.your.local
Then you may need to add an auth section to your file as in this posting.
sudo vi /etc/prometheus/snmp.yml
# Add this right at the top
auths:
public_v1:
version: 1
public_v2:
community: public
security_level: noAuthNoPriv
auth_protocol: MD5
priv_protocol: DES
version: 2
sudo systemctl restart prometheus-snmp-exporter.service
Possible version missmatch between generator and snmp_exporter. Make sure generator and snmp_exporter are the same version."
If you upgrade and the snmp exporter doesn’t start, you may see this in your log. It’s possible your snmp.yml files has old syntax. You can run the generate and cp commands to generate a current one.
level=ERROR source=main.go:137 msg=“Failing on reported parse error(s)”
You might have that operator syntax error as described in the exporter config. Double check that you can snmpwalk without error.
If you’re running prometheus-snmp-generator generate and get the error:
NetSNMP reported parse error(s)
you must update a mib file. This is described in some detail but the short answer is to run this command;
sudo wget --output-document /usr/share/snmp/mibs/ietf/SNMPv2-PDU https://pastebin.com/raw/p3QyuXzZ
https://github.com/prometheus/snmp_exporter/tree/main/generator
The blackbox exporter allows you to check the status of endpoints over HTTP, HTTPS, DNS, TCP, ICMP and gRPC.
This is mostly used as a ‘ping’ test to see if the device is on the network, or a web scrape to see if the web server is responding.
This is most useful when you can’t or don’t want to install the node exporter on the target.
sudo apt install -t testing prometheus-blackbox-exporter
- job_name: 'phones'
metrics_path: /probe
params:
module: [icmp]
static_configs:
- targets:
- some.end.point
- some.other.point
relabel_configs:
- source_labels: [__address__]
target_label: __param_target
- source_labels: [__param_target]
target_label: instance
- target_label: __address__
replacement: 127.0.0.1:9115
You can also monitor things that don’t have an exporter by writing it yourself. Here’s an example for an environmental DHT sensors.
DHT stands for Digital Humidity and Temperature. At less than $5 they are cheap and can be hooked to a Raspberry Pi easily. Add a Prometheus exporter to do at scale.
These usually come as a breakout-board with three leads you can connect to the Raspberry PI GPIO pins as follows;
(From https://github.com/rnieva/Playing-with-Sensors---Raspberry-Pi)
Use the Raspberry Pi Imager to Provision the Pi with Raspberry PI OS Lite 64 bit. Next, install the “adafruit_blinka” library as adapted from their instructions and test.
# General updates
sudo apt update
sudo apt -y upgrade
sudo apt -y autoremove
sudo reboot
# These python components may already be installed, but making sure
sudo apt -y install python3-pip
sudo apt -y install --upgrade python3-setuptools
sudo apt -y install python3-venv
# Make a virtual environment for the python process
sudo mkdir /usr/local/bin/sensor-dht
sudo python3 -m venv /usr/local/bin/sensor-dht --system-site-packages
cd /usr/local/bin/sensor-dht
sudo chown -R ${USER}:${USER} .
source bin/activate
# Build and install the library
pip3 install --upgrade adafruit-python-shell
wget https://raw.githubusercontent.com/adafruit/Raspberry-Pi-Installer-Scripts/master/raspi-blinka.py
sudo -E env PATH=$PATH python3 raspi-blinka.py
After logging back in, test the blinka lib.
cd /usr/local/bin/sensor-dht
source bin/activate
wget https://learn.adafruit.com/elements/2993427/download -O blinkatest.py
python3 blinkatest.py
Then install the DHT library from CircuitPython and create a script to test the sensor.
cd /usr/local/bin/sensor-dht
source bin/activate
pip3 install adafruit-circuitpython-dht
vi sensortest.py
import board
import adafruit_dht
dhtDevice = adafruit_dht.DHT11(board.D4)
temp = dhtDevice.temperature
humidity = dhtDevice.humidity
print(
"Temp: {:.1f} C Humidity: {}% ".format(temp, humidity)
)
dhtDevice.exit()
You can get occasional errors like RuntimeError: Checksum did not validate. Try again. that are safe to ignore. These DHTs are not 100% solid.
Add the Prometheus pips.
cd /usr/local/bin/sensor-dht
source bin/activate
pip3 install prometheus_client
And create a script like this.
nano sensor.py
import board
import adafruit_dht
import time
from prometheus_client import start_http_server, Gauge
dhtDevice = adafruit_dht.DHT11(board.D4)
temperature_gauge= Gauge('dht_temperature', 'Local temperature')
humidity_gauge = Gauge('dht_humidity', 'Local humidity')
start_http_server(8000)
while True:
try:
temperature = dhtDevice.temperature
temperature_gauge.set(temperature)
humidity = dhtDevice.humidity
humidity_gauge.set(humidity)
except:
# Errors happen fairly often as DHT's are hard to read. Just continue on.
continue
finally:
time.sleep(60)
Create a service
sudo nano /lib/systemd/system/sensor.service
[Unit]
Description=Temperature and Humidity Sensing Service
After=network.target
[Service]
Type=idle
Restart=on-failure
User=root
ExecStart=/bin/bash -c 'cd /usr/local/bin/sensor-dht && source bin/activate && python sensor.py'
[Install]
WantedBy=multi-user.target
Enable and start it
sudo systemctl enable --now sensor.service
curl http://localhost:8000/metrics
This service is running as root. You should consider creating a sensor account.
sudo useradd --home-dir /usr/local/bin/sensor-dht --system --shell /usr/sbin/nologin --comment "Sensor Service" sensor
sudo usermod -aG gpio sensor
sudo systemctl stop sensor.service
sudo chown -R sensor:sensor /usr/local/bin/sensor-dht
sudo sed -i 's/User=root/User=sensor/' /lib/systemd/system/sensor.service
sudo systemctl daemon-reload
sudo systemctl start sensor.service
Adding it requires logging into your Prometheus server and adding a job like below.
sudo vi /etc/prometheus/prometheus.yml
...
...
- job_name: 'dht'
static_configs:
- targets:
- 192.168.1.45:8000
You will be able to find the node in your server at http://YOUR-SERVER:9090/targets?search=#pool-dht and data will show up with a leading dht_....
https://randomnerdtutorials.com/raspberry-pi-dht11-dht22-python/
You may want to raise errors to the log as in the above source.
It’s useful to have an internal speed test. Browser based is best for end-users.
The two main choices are LibreSpeed or OpenSpeedTest. The former is generally preferred for it’s simplicity.
Install a minimal Debian instance by using the netinst image, de-selecting the desktop system at the top, while adding ‘common system tools’ near the bottom.
You can use an Apache server with PHP, or a binary service. If you have a simple use case, the latter has fewer moving parts
You’ll need the need the web/assets folder and the pre-compiled binary. Let’s use the rust version with the theory being it’s safer and faster. (though in casual testing, this is debatable)
Look for the latest release at:
https://github.com/librespeed/speedtest-rust/releases
# Download the source and extract the assets folder
wget https://github.com/librespeed/speedtest-rust/archive/refs/tags/v1.3.8.tar.gz
tar --strip-components=1 -xzf v1.3.8.tar.gz speedtest-rust-1.3.8/assets/
# Download the binary, extract and make executable
wget https://github.com/librespeed/speedtest-rust/releases/download/v1.3.8/librespeed-rs-x86_64-unknown-linux-gnu.tar.xz
tar xf librespeed-rs-x86_64-unknown-linux-gnu.tar.xz
chmod +x librespeed-rs
# Disable the stored results database
sed -i 's/database_type="sqlite"/database_type="none"/' configs.toml
# Change the labels in the page
sed -i 's/Your server name here/speedtest.your.org/' assets/servers_list.js
sed -i 's/LibreSpeed Example/speedtest.your.org/' assets/index.html
# Launch the server and test
./librespeed-rs
firefox http://localhost:8080
Assuming that works as intended, you can create a service file and enable it.
You might need to do more, such as run some code before letting people hit your speed test. Use the PHP version for this.
# Add the prerequisites. PHP is enabled for apache by default
apt install apache2 php openssl
# Look for the latest release at:
firefox https://github.com/librespeed/speedtest/releases
# Download, extract and copy a few things to your web folder
wget https://github.com/librespeed/speedtest/archive/refs/tags/5.4.1.tar.gz
tar xzf 5.4.1.tar.gz
cd speedtest-5.4.1
sudo cp -r index.html speedtest.js speedtest_worker.js favicon.ico backend /var/www/html/
Sometimes you want to compare LAN/WAN speeds across multiple networks but want to keep just a single URL. Do this by using multiple (or one server with multiple interfaces) and use a .php to redirect users to the right place based on their IP.
Create a new index page.
sudo vi /var/www/html/index.php
<!DOCTYPE html>
<html>
<head>
<?php
# Extract the client's IP address
$chunks = explode('.', $_SERVER['REMOTE_ADDR']);
# Assume we have an interface on the clients LAN at .55 and
# redirect them to it.
$network = "$chunks[0].$chunks[1].$chunks[2].55";
# Use the .replace method so the next request client has a
# referring page, used later on.
echo "<script>";
echo "window.location.replace(\"http://$network/speed.html\")";
echo "</script>";
?>
</head>
</html>
Adjust apache so index.php takes precedence.
sudo vi /etc/apache2/mods-available/dir.conf
# Move index.php to be right at the front.
DirectoryIndex index.php index.html index.cgi index.pl index.php index.xhtml index.htm
# Reload so changes take affect
sudo systemctl reload apache2.service
Some users will bookmark the speed test page. If they change subnets and use the old bookmark, this will no longer be a LAN test.
Let’s copy the delivered index.html to a new page that includes referrer check. This makes sure they hit the main URL first and get redirected every time. (note the script above refers to the page “speed.html” so let’s use that)
sudo cp /var/www/html/index.html /var/www/html/speed.html
sudo vi /var/www/html/speed.html
<script>
// Redirects to the main URL in case somebody books the wrong subnet page
correct_referrer="http://speedtest.marietta.edu/";
actual_referrer=document.referrer;
if (actual_referrer != correct_referrer) window.location.replace(correct_referrer);
</script>
If you added interfaces to a server to test LAN connections, your clients will connect to the WAN IP address. But the server has an interface on the same network as the client, so it replies using that. The client will drop these unexpected replies.
You need to a routing policy so the server responds back on the same interface the client sent it’s request to.
Note: We are justing using table ID 1, and not bothering to create a rt_tables file to give it a custom name.
# On the WAN interface, 192.168.1.5 used below.
allow-hotplug eth0
iface eth0 inet static
address ...
gateway ...
dns-nameservers ...
up ip rule add from 192.168.1.5 table 1 prio 1
up ip route add default via 192.168.1.5 dev eth0 table 1
I’ve been using Smokeping for at least 20 years. Every so often I look at the competitors, but it’s still the best for a self-contained latency monitoring system.
On Debian Stretch:
# Install smokeping - apache gets installed automatically and the config enabled
sudo apt-get install smokeping
# Install and enable SpeedyCGI if you can find it, otherwise, fastCGI
sudo apt install libapache2-mod-fcgid
Edit the General Config file
sudo vim /etc/smokeping/config.d/General
owner = Some Org
contact = [email protected]
mailhost = localhost
cgiurl = http://some.server.you.just.deployed.on/cgi-bin/smokeping.cgi
# specify this to get syslog logging
syslogfacility = local0
# each probe is now run in its own process
# disable this to revert to the old behaviour
# concurrentprobes = no
@include /etc/smokeping/config.d/pathnames
Edit the pathnames file. You must put in a value for sendmail (If you don’t have it) so that smoke ping will run.
sudo vim /etc/smokeping/config.d/pathnames
sendmail = /bin/false
imgcache = /var/cache/smokeping/images
imgurl = ../smokeping/images
...
...
Edit the Alerts
sudo vim /etc/smokeping/config.d/Alerts
to = [email protected]
from = [email protected]
Edit the Targets
# Add your own targets that you will measure by appending them to the bottom of the targets file.
# These will show up in a menu on the left of the generated web page. You add an entry starting with a + to create a top level entry, and subsequent lines with ++ that will show up as sub entries like so:
# + My Company
# ++ My Company's Web Server 1
# ++ My Company's Web Server 2
# Actual config requires a few extra lines, as below;
sudo vim /etc/smokeping/config.d/Targets
+ My_Company
menu = My Company
title = My Company
++ Web_Server_1
menu = Web Server 1
title = Web Server 1
host = web.server.org
# Restart smokeping and apache
sudo service smokeping restart
sudo service apache2 reload
Access smokeping at:
http://some.server.name/smokeping
The default resolution - i.e. polling frequency is 20 requests over 5 min - or 1 request every 15 seconds
http://collaboration.cmc.ec.gc.ca/science/rpn/biblio/ddj/Website/articles/SA/v12/i07/a5.htm
ArchWiki suggests a workaround for sendmail
SNMP is a protocol that’s almost always used to monitor things. Usually metrics. Though it can be used as a management interface as well.
Every time I use it I imagine a conclave of exports, meeting together and sequestering themselves for a year, then coming out of a back room and proclaiming “Here it is, the perfect protocol for doing everything”.
And it’s completely different than anything else. The one true faith protocol.
Eaton offers network management cards for many of their UPS lines that provide a web management console and SNMP server. The “Eaton Gigabit Network Card (NETWORK-M2)” is the offering at time of writing.
For SNMP, this card implements the RFC 1628 UPS standard as well as an Eaton specific set of objects1 starting at 1.3.6.1.4.1.534.
snmpwalk -v1 -c public your.ups.address
snmpwalk -v1 -c public your.ups.address 1.3.6.1.4.1.534
To make sense of these you need the Eaton MIBs2. If you download and extract to your working folder you can query like this.
snmpwalk -v1 -c public -m +./EATON-OIDS.txt:./EATON-EMP-MIB.txt:./XUPS-MIB.txt your.ups.address 1.3.6.1.4.1.534
Some useful MIBs are:
| sysLocation | Physical location for the device being monitored |
| xupsInputVoltage | The measured input voltage from the UPS meters in volts |
| xupsOutputSource | The present source of output power |
| xupsBatVoltage | Battery voltage as reported by the UPS meters |
| xupsInputFrequency | The utility line frequency in tenths of Hz |
| xupsOutputWatts | The measured real output power in watts |
| xupsEnvAmbientTemp | The reading of the ambient temperature in the vicinity |
You’d query these like so:
snmpget -v1 -c public -m +./EATON-OIDS.txt:./EATON-EMP-MIB.txt:./XUPS-MIB.txt your.ups.address \
sysLocation.0 xupsInputVoltage.1 xupsOutputSource.0 xupsBatVoltage.0 xupsInputFrequency.0 xupsOutputWatts.1 xupsEnvAmbientTemp.0
Some objects need a .0 here and others a .1 as shown above.
The object xupsOutputSource is generally used to determine if the UPS is ‘on battery’3. There is a trap specifically for OnBattery, but you can’t query the traps.
xupsOutputSource OBJECT-TYPE
SYNTAX INTEGER {
other(1),
none(2),
normal(3),
bypass(4),
battery(5),
booster(6),
reducer(7),
parallelCapacity(8),
parallelRedundant(9),
highEfficiencyMode(10),
maintenanceBypass(11),
essMode(12)
}
There is no such variable name in this MIB
You might need to “walk” the object to get the instance ID as .0 is only for scalar objects, and not tabular.
snmpget -v1 -c public -m +./EATON-OIDS.txt:./EATON-EMP-MIB.txt:./XUPS-MIB.txt your.ups.address xupsInputVoltage.0
Error in packet
Reason: (noSuchName) There is no such variable name in this MIB.
Failed object: XUPS-MIB::xupsInputVoltage.0
snmpwalk -v1 -c public -m +./EATON-OIDS.txt:./EATON-EMP-MIB.txt:./XUPS-MIB.txt your.ups.address xupsInputVoltage
XUPS-MIB::xupsInputVoltage.1 = INTEGER: 116 RMS Volts
snmpget -v1 -c public -m +./EATON-OIDS.txt:./EATON-EMP-MIB.txt:./XUPS-MIB.txt your.ups.address xupsInputVoltage.1
XUPS-MIB::xupsInputVoltage.1 = INTEGER: 116 RMS Volts
https://www.eaton.com/content/dam/eaton/products/backup-power-ups-surge-it-power-distribution/power-management-software-connectivity/connectups-e-web-snmp-device/Eaton-connect-UPS-web-snmp-user-guide-164201819.pdf.pdf ↩︎
https://www.eaton.com/us/en-us/site-search.html.searchTerm$mib.tabs$all.html ↩︎
https://thwack.solarwinds.com/product-forums/network-performance-monitor-npm/f/forum/68430/monitor-when-ups-is-on-battery-power ↩︎
Samlex makes a line of rack-mount inverters with a web management console and respond to SNMP requests. Their PSR-1200-24 and -48 models. They supply the MIB on an included USB thumb drive. If you lose it you may never find it again on the web. Here’s a copy just in case.
If you probe them without the MIB or knowing their enterprise OID, you don’t get much back.
snmpwalk -v2c -c public your.samlex.ip
However, you can probe the enterprise section and you’ll see they have thier own section with some 64 values and traps.
snmpwalk -v2c -c public your.samlex.ip 1.3.6.1.4.1
To make sense of these, you’ll need that MIB file from above.
snmpwalk -v2c -c public -m ./SAMLEXAMERICA-MIB.txt your.samlex.ip 1.3.6.1.4.1
The most useful values are probably the ones that tell you:
You can pull those in a loop like so
while read X;do
snmpget -v2c -m +./SAMLEXAMERICA-MIB.TXT -c public $X \
sysLocation.0 inverterMode.0 vin.0 gridvout.0 power.0 tempDD.0 tempDA.0
echo
done < file_of_device_IPs
You can also pull this into Prometheus
https://samlexamerica.com/wp-content/uploads/2021/08/11001-PSR-1200-24-48-0821_Lrez.pdf
Grafana is an easy to use metrics dashboard. You create and arrange panels that display charts, graphs, and tables. These query data from multiple sources. But mostly influxdb and prometheus.
# Install the prerequisite packages
sudo apt-get install -y apt-transport-https software-properties-common wget
# Import the GPG key
sudo mkdir -p /etc/apt/keyrings/
wget -q -O - https://apt.grafana.com/gpg.key | gpg --dearmor | sudo tee /etc/apt/keyrings/grafana.gpg > /dev/null
echo "deb [signed-by=/etc/apt/keyrings/grafana.gpg] https://apt.grafana.com stable main" | sudo tee -a /etc/apt/sources.list.d/grafana.list
# Updates the list of available packages
sudo apt-get update
# Install the latest Enterprise release
sudo apt-get install grafana-enterprise
Note: Grafana Enterprise is the recommended and default edition1. It’s free, but there are other versions available if desired.
The service isn’t enabled or running by default.
sudo systemctl enable --now grafana-server.service
If you want to allow anonymous view access, enable it in the .conf.
sudo vi /etc/grafana/grafana.ini
[auth.anonymous]
# enable anonymous access
enabled = false
# specify role for unauthenticated users
org_role = Viewer
The Web UI is available on port 3000. The default credentials are “admin/admin”. It will prompt you to change the admin password.
On the left menu:
Connections -> Data Sources -> Add data source
Choose ‘Prometheus’ and use the defaults, the URL being http://localhost:9090 by default. Hit the save and test button at the bottom, and you should see success
Design and use is a large topic I won’t cover on this page, but here’s some handy queries to refer to.
Surprisingly, this important stat is one of the hardest to produce.
# Display CPU usage
(100 - (avg by(instance) (irate(node_cpu_seconds_total{mode="idle", job="someJob"}[5m])) * 100))
# Use the label_replace function shorten the chart labels.
# Turns 'host.subdomain.your.org' to just 'host'
label_replace((100 - (avg by(instance) (irate(node_cpu_seconds_total{mode="idle", job="screens"}[5m])) * 100)), "instance", "$1", "instance", "(.*).subdomain.*")
All instances
label_replace(node_memory_MemAvailable_bytes / 1000000, "instance", "$1", "instance", "(.*).sign.*")
Detail on a single instance
Total
node_memory_MemTotal_bytes{instance="some.your.org:9100",job="screens"}
Used
node_memory_MemTotal_bytes{instance="some.your.org:9100",job="screens"} - node_memory_MemFree_bytes{instance="some.your.org:9100",job="screens"}
Cache
node_memory_Cached_bytes{instance="some.your.org:9100",job="screens"} + node_memory_Buffers_bytes{instance="some.your.org:9100",job="screens"} + node_memory_Active_bytes{instance="some.your.org:9100",job="screens"}
label_replace(up{job="screens"},"instance", "$1", "instance", "(.*).sign.*")
# set the legend to {{instance}}
Under-voltage
label_replace(node_hwmon_in_lcrit_alarm_volts{job="screens"},"instance", "$1", "instance", "(.*).sign.*")
A Samlex Invertor Dashboard
{
"__inputs": [
{
"name": "DS_PROMETHEUS",
"label": "prometheus",
"description": "",
"type": "datasource",
"pluginId": "prometheus",
"pluginName": "Prometheus"
}
],
"__elements": {},
"__requires": [
{
"type": "grafana",
"id": "grafana",
"name": "Grafana",
"version": "12.1.1"
},
{
"type": "datasource",
"id": "prometheus",
"name": "Prometheus",
"version": "1.0.0"
},
{
"type": "panel",
"id": "table",
"name": "Table",
"version": ""
},
{
"type": "panel",
"id": "timeseries",
"name": "Time series",
"version": ""
}
],
"annotations": {
"list": [
{
"builtIn": 1,
"datasource": {
"type": "grafana",
"uid": "-- Grafana --"
},
"enable": true,
"hide": true,
"iconColor": "rgba(0, 211, 255, 1)",
"name": "Annotations & Alerts",
"target": {
"limit": 100,
"matchAny": false,
"tags": [],
"type": "dashboard"
},
"type": "dashboard"
}
]
},
"editable": true,
"fiscalYearStartMonth": 0,
"graphTooltip": 0,
"id": null,
"links": [],
"panels": [
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"fieldConfig": {
"defaults": {
"color": {
"mode": "thresholds"
},
"custom": {
"align": "left",
"cellOptions": {
"type": "auto"
},
"inspect": false
},
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": 0
},
{
"color": "red",
"value": 26
}
]
},
"unit": "volt"
},
"overrides": [
{
"matcher": {
"id": "byName",
"options": "instance"
},
"properties": [
{
"id": "custom.width",
"value": 262
}
]
}
]
},
"gridPos": {
"h": 8,
"w": 12,
"x": 0,
"y": 0
},
"id": 10,
"options": {
"cellHeight": "sm",
"footer": {
"countRows": false,
"fields": "",
"reducer": [
"sum"
],
"show": false
},
"showHeader": true,
"sortBy": [
{
"desc": true,
"displayName": "Value"
}
]
},
"pluginVersion": "12.1.1",
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"disableTextWrap": false,
"editorMode": "builder",
"exemplar": false,
"expr": "label_replace(samlex_vin{job=\"samlex\"}, \"instance\", \"$1\", \"instance\", \"(.*).ups.*\") / 10",
"format": "table",
"fullMetaSearch": false,
"includeNullMetadata": true,
"instant": true,
"legendFormat": "{{instance}}",
"range": false,
"refId": "A",
"useBackend": false
}
],
"title": "Battery Voltage",
"transformations": [
{
"id": "organize",
"options": {
"excludeByName": {
"Time": true,
"Value": false,
"__name__": true,
"instance": false,
"job": true
},
"indexByName": {},
"renameByName": {
"Time": ""
}
}
}
],
"type": "table"
},
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"fieldConfig": {
"defaults": {
"color": {
"mode": "thresholds"
},
"custom": {
"align": "left",
"cellOptions": {
"mode": "gradient",
"type": "gauge"
},
"filterable": false,
"inspect": false
},
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": 0
},
{
"color": "#EAB839",
"value": 45
},
{
"color": "red",
"value": 50
}
]
},
"unit": "celsius"
},
"overrides": [
{
"matcher": {
"id": "byName",
"options": "instance"
},
"properties": [
{
"id": "custom.cellOptions",
"value": {
"type": "auto"
}
},
{
"id": "custom.hidden",
"value": false
}
]
}
]
},
"gridPos": {
"h": 8,
"w": 12,
"x": 12,
"y": 0
},
"id": 12,
"options": {
"cellHeight": "sm",
"footer": {
"countRows": false,
"enablePagination": false,
"fields": "",
"reducer": [
"sum"
],
"show": false
},
"showHeader": true,
"sortBy": [
{
"desc": true,
"displayName": "DC to AC Temp"
}
]
},
"pluginVersion": "12.1.1",
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"disableTextWrap": false,
"editorMode": "builder",
"exemplar": false,
"expr": "label_replace(samlex_tempDA{job=\"samlex\"}, \"instance\", \"$1\", \"instance\", \"(.*).ups.*\")",
"format": "table",
"fullMetaSearch": false,
"includeNullMetadata": true,
"instant": true,
"legendFormat": "{{instance}}",
"range": false,
"refId": "A",
"useBackend": false
},
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"editorMode": "builder",
"exemplar": false,
"expr": "label_replace(samlex_tempDD{job=\"snmp\"}, \"instance\", \"$1\", \"instance\", \"(.*).ups.*\")",
"format": "table",
"hide": false,
"instant": true,
"legendFormat": "{{instance}}",
"range": false,
"refId": "B"
}
],
"title": "Temp",
"transformations": [
{
"id": "joinByField",
"options": {
"byField": "instance",
"mode": "outer"
}
},
{
"id": "organize",
"options": {
"excludeByName": {
"Time 1": true,
"Time 2": true,
"__name__ 1": true,
"__name__ 2": true,
"job 1": true,
"job 2": true
},
"indexByName": {},
"renameByName": {
"Value #A": "DC to AC Temp",
"Value #B": "DC to DC Temp"
}
}
}
],
"type": "table"
},
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"fieldConfig": {
"defaults": {
"color": {
"mode": "palette-classic"
},
"custom": {
"axisBorderShow": false,
"axisCenteredZero": false,
"axisColorMode": "text",
"axisLabel": "",
"axisPlacement": "auto",
"axisSoftMax": 30,
"axisSoftMin": 22,
"barAlignment": 0,
"barWidthFactor": 0.6,
"drawStyle": "line",
"fillOpacity": 0,
"gradientMode": "none",
"hideFrom": {
"legend": false,
"tooltip": false,
"viz": false
},
"insertNulls": false,
"lineInterpolation": "linear",
"lineWidth": 1,
"pointSize": 5,
"scaleDistribution": {
"type": "linear"
},
"showPoints": "auto",
"spanNulls": false,
"stacking": {
"group": "A",
"mode": "none"
},
"thresholdsStyle": {
"mode": "off"
}
},
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": 0
},
{
"color": "red",
"value": 80
}
]
},
"unit": "volt"
},
"overrides": []
},
"gridPos": {
"h": 8,
"w": 12,
"x": 0,
"y": 8
},
"id": 4,
"options": {
"legend": {
"calcs": [],
"displayMode": "list",
"placement": "bottom",
"showLegend": true
},
"tooltip": {
"hideZeros": false,
"mode": "single",
"sort": "none"
}
},
"pluginVersion": "12.1.1",
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"editorMode": "builder",
"expr": "label_replace(samlex_vin, \"instance\", \"$1\", \"instance\", \"(.*).ups.*\") / 10",
"legendFormat": "{{instance}}",
"range": true,
"refId": "A"
}
],
"title": "Battery Voltage",
"type": "timeseries"
},
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"fieldConfig": {
"defaults": {
"color": {
"mode": "palette-classic"
},
"custom": {
"axisBorderShow": false,
"axisCenteredZero": false,
"axisColorMode": "text",
"axisLabel": "",
"axisPlacement": "auto",
"axisSoftMin": 20,
"barAlignment": 0,
"barWidthFactor": 0.6,
"drawStyle": "line",
"fillOpacity": 0,
"gradientMode": "none",
"hideFrom": {
"legend": false,
"tooltip": false,
"viz": false
},
"insertNulls": false,
"lineInterpolation": "linear",
"lineWidth": 1,
"pointSize": 5,
"scaleDistribution": {
"type": "linear"
},
"showPoints": "auto",
"spanNulls": false,
"stacking": {
"group": "A",
"mode": "none"
},
"thresholdsStyle": {
"mode": "off"
}
},
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": 0
},
{
"color": "red",
"value": 80
}
]
},
"unit": "celsius"
},
"overrides": []
},
"gridPos": {
"h": 8,
"w": 12,
"x": 12,
"y": 8
},
"id": 8,
"options": {
"legend": {
"calcs": [],
"displayMode": "list",
"placement": "bottom",
"showLegend": true
},
"tooltip": {
"hideZeros": false,
"mode": "single",
"sort": "none"
}
},
"pluginVersion": "12.1.1",
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"disableTextWrap": false,
"editorMode": "builder",
"expr": "label_replace(samlex_tempDD{job=\"samlex\"}, \"instance\", \"$1 DC\", \"instance\", \"(.*).ups.*\")",
"fullMetaSearch": false,
"includeNullMetadata": true,
"legendFormat": "{{instance}}",
"range": true,
"refId": "A",
"useBackend": false
},
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"editorMode": "builder",
"exemplar": false,
"expr": "label_replace(samlex_tempDA{job=\"snmp\"}, \"instance\", \"$1 AC\", \"instance\", \"(.*).ups.*\")",
"format": "time_series",
"hide": false,
"instant": false,
"legendFormat": "{{instance}}",
"range": true,
"refId": "B"
}
],
"title": "Inverter Temp",
"type": "timeseries"
},
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"fieldConfig": {
"defaults": {
"color": {
"mode": "palette-classic"
},
"custom": {
"axisBorderShow": false,
"axisCenteredZero": false,
"axisColorMode": "text",
"axisLabel": "",
"axisPlacement": "auto",
"axisSoftMin": 0,
"barAlignment": 0,
"barWidthFactor": 0.6,
"drawStyle": "line",
"fillOpacity": 0,
"gradientMode": "none",
"hideFrom": {
"legend": false,
"tooltip": false,
"viz": false
},
"insertNulls": false,
"lineInterpolation": "linear",
"lineWidth": 1,
"pointSize": 5,
"scaleDistribution": {
"type": "linear"
},
"showPoints": "auto",
"spanNulls": false,
"stacking": {
"group": "A",
"mode": "none"
},
"thresholdsStyle": {
"mode": "off"
}
},
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": 0
},
{
"color": "red",
"value": 80
}
]
},
"unit": "volt"
},
"overrides": []
},
"gridPos": {
"h": 9,
"w": 12,
"x": 0,
"y": 16
},
"id": 2,
"options": {
"legend": {
"calcs": [],
"displayMode": "list",
"placement": "bottom",
"showLegend": true
},
"tooltip": {
"hideZeros": false,
"mode": "single",
"sort": "none"
}
},
"pluginVersion": "12.1.1",
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"editorMode": "builder",
"expr": "label_replace(samlex_gridvout, \"instance\", \"$1\", \"instance\", \"(.*).ups.*\") / 10",
"legendFormat": "{{instance}}",
"range": true,
"refId": "A"
}
],
"title": "Grid Voltage",
"type": "timeseries"
},
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"fieldConfig": {
"defaults": {
"color": {
"mode": "palette-classic"
},
"custom": {
"axisBorderShow": false,
"axisCenteredZero": false,
"axisColorMode": "text",
"axisLabel": "",
"axisPlacement": "auto",
"barAlignment": 0,
"barWidthFactor": 0.6,
"drawStyle": "line",
"fillOpacity": 0,
"gradientMode": "none",
"hideFrom": {
"legend": false,
"tooltip": false,
"viz": false
},
"insertNulls": false,
"lineInterpolation": "linear",
"lineWidth": 1,
"pointSize": 5,
"scaleDistribution": {
"type": "linear"
},
"showPoints": "auto",
"spanNulls": false,
"stacking": {
"group": "A",
"mode": "none"
},
"thresholdsStyle": {
"mode": "off"
}
},
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": 0
},
{
"color": "red",
"value": 80
}
]
},
"unit": "watt"
},
"overrides": []
},
"gridPos": {
"h": 8,
"w": 12,
"x": 12,
"y": 16
},
"id": 6,
"options": {
"legend": {
"calcs": [],
"displayMode": "list",
"placement": "bottom",
"showLegend": true
},
"tooltip": {
"hideZeros": false,
"mode": "single",
"sort": "none"
}
},
"pluginVersion": "12.1.1",
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"editorMode": "builder",
"expr": "label_replace(samlex_power, \"instance\", \"$1\", \"instance\", \"(.*).ups.*\") / 10",
"legendFormat": "{{instance}}",
"range": true,
"refId": "A"
}
],
"title": "Inverter Power Output",
"type": "timeseries"
}
],
"refresh": "",
"schemaVersion": 41,
"tags": [],
"templating": {
"list": []
},
"time": {
"from": "now-30m",
"to": "now"
},
"timepicker": {},
"timezone": "",
"title": "Samlex",
"uid": "UuR5GSAVz",
"version": 4,
"weekStart": ""
}
A Raspberry Pi Status Dasboard
{
"__inputs": [
{
"name": "DS_PROMETHEUS",
"label": "prometheus",
"description": "",
"type": "datasource",
"pluginId": "prometheus",
"pluginName": "Prometheus"
}
],
"__elements": {},
"__requires": [
{
"type": "grafana",
"id": "grafana",
"name": "Grafana",
"version": "12.1.1"
},
{
"type": "datasource",
"id": "prometheus",
"name": "Prometheus",
"version": "1.0.0"
},
{
"type": "panel",
"id": "state-timeline",
"name": "State timeline",
"version": ""
},
{
"type": "panel",
"id": "timeseries",
"name": "Time series",
"version": ""
}
],
"annotations": {
"list": [
{
"builtIn": 1,
"datasource": {
"type": "grafana",
"uid": "-- Grafana --"
},
"enable": true,
"hide": true,
"iconColor": "rgba(0, 211, 255, 1)",
"name": "Annotations & Alerts",
"target": {
"limit": 100,
"matchAny": false,
"tags": [],
"type": "dashboard"
},
"type": "dashboard"
}
]
},
"editable": true,
"fiscalYearStartMonth": 0,
"graphTooltip": 0,
"id": null,
"links": [],
"panels": [
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"fieldConfig": {
"defaults": {
"color": {
"mode": "thresholds"
},
"custom": {
"axisPlacement": "auto",
"fillOpacity": 70,
"hideFrom": {
"legend": false,
"tooltip": false,
"viz": false
},
"insertNulls": false,
"lineWidth": 0,
"spanNulls": false
},
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "red",
"value": 0
},
{
"color": "green",
"value": 1
}
]
}
},
"overrides": []
},
"gridPos": {
"h": 16,
"w": 12,
"x": 0,
"y": 0
},
"id": 10,
"options": {
"alignValue": "left",
"legend": {
"displayMode": "list",
"placement": "bottom",
"showLegend": false
},
"mergeValues": true,
"rowHeight": 0.9,
"showValue": "never",
"tooltip": {
"hideZeros": false,
"mode": "single",
"sort": "none"
}
},
"pluginVersion": "12.1.1",
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"editorMode": "code",
"expr": "label_replace(up{job=\"screens\"},\"instance\", \"$1\", \"instance\", \"(.*).sign.*\")",
"legendFormat": "{{instance}}",
"range": true,
"refId": "A"
}
],
"title": "State",
"transformations": [
{
"id": "concatenate",
"options": {}
}
],
"type": "state-timeline"
},
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"fieldConfig": {
"defaults": {
"color": {
"mode": "palette-classic"
},
"custom": {
"axisBorderShow": false,
"axisCenteredZero": false,
"axisColorMode": "text",
"axisLabel": "",
"axisPlacement": "auto",
"barAlignment": 0,
"barWidthFactor": 0.6,
"drawStyle": "line",
"fillOpacity": 0,
"gradientMode": "none",
"hideFrom": {
"legend": false,
"tooltip": false,
"viz": false
},
"insertNulls": false,
"lineInterpolation": "linear",
"lineWidth": 1,
"pointSize": 5,
"scaleDistribution": {
"type": "linear"
},
"showPoints": "auto",
"spanNulls": false,
"stacking": {
"group": "A",
"mode": "none"
},
"thresholdsStyle": {
"mode": "off"
}
},
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": 0
},
{
"color": "red",
"value": 80
}
]
}
},
"overrides": []
},
"gridPos": {
"h": 8,
"w": 12,
"x": 12,
"y": 0
},
"id": 4,
"options": {
"legend": {
"calcs": [],
"displayMode": "list",
"placement": "bottom",
"showLegend": true
},
"tooltip": {
"hideZeros": false,
"mode": "single",
"sort": "none"
}
},
"pluginVersion": "12.1.1",
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"editorMode": "code",
"expr": "label_replace((100 - (avg by(instance) (irate(node_cpu_seconds_total{mode=\"idle\", job=\"screens\"}[5m])) * 100)), \"instance\", \"$1\", \"instance\", \"(.*).sign.*\")",
"legendFormat": "__auto",
"range": true,
"refId": "A"
}
],
"title": "CPU %",
"type": "timeseries"
},
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"fieldConfig": {
"defaults": {
"color": {
"mode": "palette-classic"
},
"custom": {
"axisBorderShow": false,
"axisCenteredZero": false,
"axisColorMode": "text",
"axisLabel": "",
"axisPlacement": "auto",
"barAlignment": 0,
"barWidthFactor": 0.6,
"drawStyle": "line",
"fillOpacity": 0,
"gradientMode": "none",
"hideFrom": {
"legend": false,
"tooltip": false,
"viz": false
},
"insertNulls": false,
"lineInterpolation": "linear",
"lineWidth": 1,
"pointSize": 5,
"scaleDistribution": {
"type": "linear"
},
"showPoints": "auto",
"spanNulls": false,
"stacking": {
"group": "A",
"mode": "none"
},
"thresholdsStyle": {
"mode": "line"
}
},
"mappings": [],
"max": 1000,
"min": 0,
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": 0
}
]
},
"unit": "decmbytes"
},
"overrides": []
},
"gridPos": {
"h": 8,
"w": 12,
"x": 12,
"y": 8
},
"id": 18,
"options": {
"legend": {
"calcs": [],
"displayMode": "list",
"placement": "bottom",
"showLegend": true
},
"tooltip": {
"hideZeros": false,
"mode": "single",
"sort": "none"
}
},
"pluginVersion": "12.1.1",
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"editorMode": "code",
"expr": "label_replace(node_memory_MemAvailable_bytes / 1000000, \"instance\", \"$1\", \"instance\", \"(.*).sign.*\")",
"legendFormat": "{{instance}}",
"range": true,
"refId": "A"
}
],
"title": "Memory Available",
"type": "timeseries"
},
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"fieldConfig": {
"defaults": {
"color": {
"mode": "palette-classic"
},
"custom": {
"axisBorderShow": false,
"axisCenteredZero": false,
"axisColorMode": "text",
"axisLabel": "",
"axisPlacement": "auto",
"barAlignment": 0,
"barWidthFactor": 0.6,
"drawStyle": "line",
"fillOpacity": 0,
"gradientMode": "none",
"hideFrom": {
"legend": false,
"tooltip": false,
"viz": false
},
"insertNulls": false,
"lineInterpolation": "linear",
"lineWidth": 1,
"pointSize": 5,
"scaleDistribution": {
"type": "linear"
},
"showPoints": "auto",
"spanNulls": false,
"stacking": {
"group": "A",
"mode": "none"
},
"thresholdsStyle": {
"mode": "off"
}
},
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": 0
},
{
"color": "red",
"value": 80
}
]
}
},
"overrides": []
},
"gridPos": {
"h": 8,
"w": 12,
"x": 0,
"y": 16
},
"id": 6,
"options": {
"legend": {
"calcs": [],
"displayMode": "list",
"placement": "bottom",
"showLegend": true
},
"tooltip": {
"hideZeros": false,
"mode": "single",
"sort": "none"
}
},
"pluginVersion": "12.1.1",
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"editorMode": "code",
"exemplar": false,
"expr": "label_replace(node_hwmon_in_lcrit_alarm_volts{job=\"screens\"},\"instance\", \"$1\", \"instance\", \"(.*).sign.*\")",
"instant": false,
"legendFormat": "{{instance}}",
"range": true,
"refId": "A"
}
],
"title": "Undervoltage",
"type": "timeseries"
},
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"fieldConfig": {
"defaults": {
"color": {
"mode": "palette-classic"
},
"custom": {
"axisBorderShow": false,
"axisCenteredZero": false,
"axisColorMode": "text",
"axisLabel": "",
"axisPlacement": "left",
"barAlignment": 0,
"barWidthFactor": 0.6,
"drawStyle": "line",
"fillOpacity": 0,
"gradientMode": "none",
"hideFrom": {
"legend": false,
"tooltip": false,
"viz": false
},
"insertNulls": false,
"lineInterpolation": "linear",
"lineWidth": 1,
"pointSize": 5,
"scaleDistribution": {
"type": "linear"
},
"showPoints": "auto",
"spanNulls": false,
"stacking": {
"group": "A",
"mode": "none"
},
"thresholdsStyle": {
"mode": "off"
}
},
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": 0
},
{
"color": "red",
"value": 80
}
]
},
"unit": "percentunit"
},
"overrides": []
},
"gridPos": {
"h": 8,
"w": 12,
"x": 12,
"y": 16
},
"id": 14,
"options": {
"legend": {
"calcs": [],
"displayMode": "list",
"placement": "bottom",
"showLegend": true
},
"tooltip": {
"hideZeros": false,
"mode": "single",
"sort": "none"
}
},
"pluginVersion": "12.1.1",
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"disableTextWrap": false,
"editorMode": "builder",
"expr": "avg without(cpu) (irate(node_cpu_seconds_total{job=\"screens\", instance=\"admissions.sign.some.org:9100\", mode!=\"idle\"}[5m]))",
"fullMetaSearch": false,
"includeNullMetadata": true,
"legendFormat": "{{mode}}",
"range": true,
"refId": "A",
"useBackend": false
}
],
"title": "CPU Admissions",
"type": "timeseries"
},
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"fieldConfig": {
"defaults": {
"color": {
"mode": "palette-classic"
},
"custom": {
"axisBorderShow": false,
"axisCenteredZero": false,
"axisColorMode": "text",
"axisLabel": "",
"axisPlacement": "auto",
"barAlignment": 0,
"barWidthFactor": 0.6,
"drawStyle": "line",
"fillOpacity": 0,
"gradientMode": "none",
"hideFrom": {
"legend": false,
"tooltip": false,
"viz": false
},
"insertNulls": false,
"lineInterpolation": "linear",
"lineWidth": 1,
"pointSize": 5,
"scaleDistribution": {
"type": "linear"
},
"showPoints": "auto",
"spanNulls": false,
"stacking": {
"group": "A",
"mode": "none"
},
"thresholdsStyle": {
"mode": "off"
}
},
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": 0
},
{
"color": "red",
"value": 80
}
]
},
"unit": "bytes"
},
"overrides": []
},
"gridPos": {
"h": 8,
"w": 12,
"x": 12,
"y": 24
},
"id": 16,
"options": {
"legend": {
"calcs": [],
"displayMode": "list",
"placement": "bottom",
"showLegend": true
},
"tooltip": {
"hideZeros": false,
"mode": "single",
"sort": "none"
}
},
"pluginVersion": "12.1.1",
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"editorMode": "code",
"expr": "node_memory_MemTotal_bytes{instance=\"admissions.sign.marietta.edu:9100\",job=\"screens\"}",
"hide": false,
"legendFormat": "Total",
"range": true,
"refId": "A"
},
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"editorMode": "code",
"expr": "node_memory_MemTotal_bytes{instance=\"admissions.sign.marietta.edu:9100\",job=\"screens\"} - node_memory_MemFree_bytes{instance=\"admissions.sign.marietta.edu:9100\",job=\"screens\"}",
"hide": false,
"legendFormat": "Used",
"range": true,
"refId": "B"
},
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"editorMode": "code",
"expr": "node_memory_Cached_bytes{instance=\"admissions.sign.marietta.edu:9100\",job=\"screens\"} + node_memory_Buffers_bytes{instance=\"admissions.sign.marietta.edu:9100\",job=\"screens\"} + node_memory_Active_bytes{instance=\"admissions.sign.marietta.edu:9100\",job=\"screens\"}",
"hide": false,
"legendFormat": "Cache",
"range": true,
"refId": "C"
}
],
"title": "Memory Admissions",
"type": "timeseries"
}
],
"refresh": "",
"schemaVersion": 41,
"tags": [],
"templating": {
"list": []
},
"time": {
"from": "now-6h",
"to": "now"
},
"timepicker": {},
"timezone": "",
"title": "Signage",
"uid": "Dz5puQT4z",
"version": 3,
"weekStart": ""
}
Supermicro often bundles the firmware for the motherboard and BMC together.
Upgrading the BIOS remotely requires a license key. If your system is out of support, you can generate that getting the IMPI MAC address from the System page and using this command.
echo -n 'THEMACADDRESS' | xxd -r -p | openssl dgst -sha1 -mac HMAC -macopt hexkey:8544E3B47ECA58F9583043F8 | awk '{print $2}' | cut -c 1-24 | sed 's/..../& /g'
You can then download the BIOS from the support page, uncompress it, enter the BIOS update settings and then apply it.
(Accept the defaults)
The Baseboard Management Controller, or BMC, is a small embedded computer inside the server that allows you lights-out management. I.e. you can control the system even when its powered off. One of it’s key features is to provide the IMPI (Intelligent Platform Management Interface), a way to mange the server over the network. Its a good idea to enable it and keep it up to date.
Find the model of your server (as from the PO or BIOS screen) and search for it’s support page, for example:
Supermicro 5048R-E1CR36L
<https://www.supermicro.com/en/products/system/4U/5048/SSG-5048R-E1CR36L.cfm?
View the server’s Quick Start document to find which network socket on the server the BMC uses. Enter the BIOS (if needed) to configure the IPMI network settings and to determine what IP address (if using DHCP) is in use. It will also display this during boot. You can then access it over the network, change the password and go to maintenance.
Go to Maintenance at the top, then Firmware Update at the left. Enter Update Mode and supply the new firmware, downloadable from the server support page above.
(Leave the defaults checked)
For some/all models, the BIOS and BMC firmware are bundled together and you can skip the next step
The recommended[^2] firmware (by TruNAS) is version 16.00.12.00. This is newer than the version available at Supermicro’s site.[^3].
Download the firmware package from Supermicro and the firmware binary from TrueNAS. You will need the former for the flashing utility.
You may be able to create an ISO from it and mount it virtually. See the [Virtual CD-ROM.md] notes. If it works, you can look at mapped devices with the command and change to the device
If that fails, you can extract it to a GPT FAT32 USB drive, travel to the site and insert it. Connect to the network KVM and reboot. During POST, record the SAS address by hitting Ctrl-C during it’s screen and taking a screen shot
Then you can reboot and invoke the boot menu with F11 and select built-in EUFI shell. (Enable the built in shell in BIOS Boot order if needed)
You should flash it to IT mode as you’ll not be booting from it
# Look for removable HardDisks as likely suspects
map -b
fsrB:
cd 3008*\IT\UEFI
SMC3008T.NSH
You may get stuck at the end if your netkvm keeps interrupting you before you can get the whole string in and you’ll have to look at the NSH and see what the last command is. It will be something like this:
sas3flash.efi -o -sasaddhi 50030
In that case, if your address was 50030480:1CDC4001 you can execute
sas3flash.efi -o -sasadd 500304801CDC4001
TrueNAS discovered issues[^2] with the 3008 and sata drives and recommends a special firmware they have worked with Broadcom to create. I have seen those issues first hand with the firmware available from Supermicro and recommend updating to firmware 16.00.12.00 from TrueNAS
https://forums.servethehome.com/index.php?threads/lsi-9300-8i-flashing.28140/
Instructions to use the java based KVM that’s part of the IPMI on older Aupermicro servers.
Supermicro’s IPMI will only mount NTLMv1 shares for virtual CD-ROM drives. Do that like so:
Enable NT1 protocol in your samba server
vi /etc/samba/smb.conf
[global]
map to guest = bad user
hosts allow = 192.168.1.
log file = /var/log/samba/%m.log
min protocol = NT1
[srv]
path = /srv
read only = no
guest ok = yes
guest only = yes
In the Virtual Media CD-ROM add the host and path as below and hit Save before you attempt to mount.
Share Host: 192.168.1.X Path to Image: \srv\VMware-VMvisor-Installer-7.0U3c-19193900.x86_64.iso
Waveshare makes many SBC accessories
This is perhaps the most portable NAS you can put together. Paired with 5V/3A DC power supply like the ones [geekworm] makes (that don’t reset when input power is applied) it travels in the car well.
One great use is to add some media, enable it as a HotSpot and share things up with plain-old SMB. Most tablets will easily connect and play with VLC.
If you have an EMMC based CM4, flip the switch on the back to put it in target mode and power it on while plugged into your PC. You’ll see it appear as a block device and you can image it.
After installation, run these commands on it to set it to US locale and settings.
sudo raspi-config nonint do_change_locale en_US-UTF-8
sudo raspi-config nonint do_configure_keyboard us
sudo raspi-config nonint do_wifi_country US
sudo timedatectl set-timezone America/New_York
These make the buttons and LCD work as intended. Refer to their wiki for installation - though this may be out of date and require some work as Raspberry Pi updates their OS.
After you’ve configured the storage, with mdadm, btrfs or mergerfs, you’ll want to share it up simply.
Setup SMB
The easiest way to create a hotspot is with nmcli. It’s already installed and will do the configuration for you. Here’s the command to add a 5GHz connection on channel 161.
sudo nmcli device wifi hotspot ssid SOMENAME band a channel 161 password SOMEPASSWORD ifname wlan0
sudo nmcli connection modify Hotspot connection.autoconnect yes
This will give you about 72MBits of bandwidth. If you’re getting less, look to see if anything else is using 161 and change as needed.
# What channels can I use?
iw phy phy0 info | grep "MHz \[" | grep -v "no IR" | grep -v "disabled"
# What other things are around me?
sudo nmcli device wifi
# Try on of the ones that might be free
sudo nmcli connection modify Hotspot wifi.channel 149
Not every channel is available in every country. In fact, if you tried to use the default when creating the AP you’d get these errors1.
Error: Connection activation failed: 802.1X supplicant took too long to authenticate brcmfmac: brcmf_set_channel: set chanspec 0xd022 fail, reason -52
If you want to use it as a router as well, say because you’ve plugged it into a LAN cable, you can enable masquerade.
You can double-down and use this as a media player, in addition to serving up content over the network. A great way to do this is with LibreELEC
Though as Jeff Geerling found you may be doing something dangerous or harmful, so be careful!
This requires either flashing the SD card, or putting the CM4 into USB target mode and using rpiboot to mount the eMMC as a drive so you can use the Raspberry Pi Imager to flash the LibreELEC image.
The LibreELEC folks recently added AHCI support to their CM4 image so you don’t have to re-spin things.
You may want to increase the WiFi performance by installing hostapd, but some assembly is required.
You can also install services like Jellyfin. For large libraries, you may want to switch it’s metadata directory by editing:
/storage/.kodi/userdata/addon_data/service.jellyfin
For smaller CPUs, it can help to use the rsync daemon on-demand to remove the overhead of SSH. Establish SSH keys and write a simple script like below.
Setup Rsync Daemon
The USB port isn’t working
Check that the config.txt file contains
dtoverlay=dwc2,dr_mode=host
Jellyfin transcoding is slow
Yes, this is just a Pi4. But you can find versions of ffmpeg optimized for the pi4 hardware and rest replacing the shipped version. Though you’d think that one was already optimized.
https://unix.stackexchange.com/questions/612498/hotspot-network-creation-took-too-long-and-failed-activation [geekworm]:https://geekworm.com/products/x-ups1 ↩︎
https://www.waveshare.com/ups-hat-e.htm
The pogo pins didn’t supply voltage until I used a pencil eraser to clean the bottom of the Pi’s contacts, and them too for good measure.
Once working, I downloaded the python same as described in
https://www.waveshare.com/wiki/UPS_HAT_(E)
You must sudo the first example, but the second injects into the GUI seemingly OK (though I didn’t test to see if it was accurate).
Then I used ChatGPT to make a UPS monitoring script. It did a really good job. I’d have hacked something up in bash, but python is better for reading registers (probably)
I had the gui version of RaspOS installed so there might have been other libraries than described in the wiki that were used without being explicitly called out.
sudo vi /usr/local/bin/ups-monitor.py
sudo chmod +x /usr/local/bin/ups-monitor.py
#!/usr/bin/env python3
import time
import subprocess
from smbus2 import SMBus
I2C_ADDR = 0x2D # UPS HAT (E) I2C address
RUNTIME_LOW = 0x28 # low 8-bit of remaining time (minutes)
RUNTIME_HIGH = 0x29 # high 8-bit
SHUTDOWN_LIMIT = 20 # minutes
def get_remaining_minutes():
try:
with SMBus(1) as bus:
lo = bus.read_byte_data(I2C_ADDR, RUNTIME_LOW)
hi = bus.read_byte_data(I2C_ADDR, RUNTIME_HIGH)
remaining = (hi << 8) | lo
return remaining
except Exception as e:
print(f"UPS read error: {e}")
return None
def shutdown():
print("Remaining runtime below threshold — initiating shutdown …")
subprocess.run(["sudo", "shutdown", "-h", "now"])
def main():
while True:
remaining = get_remaining_minutes()
if remaining is not None:
print(f"[UPS HAT E] Remaining runtime: {remaining} minutes")
if remaining < SHUTDOWN_LIMIT:
shutdown()
break
else:
print("Could not read remaining runtime from UPS — retrying")
time.sleep(30) # check every 30 seconds
if __name__ == "__main__":
main()
sudo nano /etc/systemd/system/ups-monitor.service
[Unit]
Description=Waveshare UPS Runtime Monitor
After=multi-user.target
StartLimitIntervalSec=0
[Service]
Type=simple
ExecStart=/usr/local/bin/ups-monitor.py
Restart=always
RestartSec=5
User=root
[Install]
WantedBy=multi-user.target
sudo systemctl daemon-reload
sudo systemctl enable ups-monitor.service
sudo systemctl start ups-monitor.service
Why do we put something in front of tomcat? Do we need it? I’ll quote the apache docs.
When using a single server, the performance when using a native webserver in front of the Tomcat instance is most of the time significantly worse than a standalone Tomcat with its default HTTP connector, even if a large part of the web application is made of static files.
Of course, this changes when we need a load balancer….or do we?
Tomcat 5 provides load balancing in three different ways: using the JK native connector, using Apache 2 with mod_proxy and mod_rewrite, or using the balancer web app.
The balancer web app is for tomcat native load balancing and provides useful features such as cluster wide sessions that the apache solutions (and hardware based ones) do not.
Below are some links on using tomcat native clustering.
http://www.onjava.com/pub/a/onjava/2004/03/31/clustering.html
http://www.onjava.com/pub/a/onjava/2004/04/14/clustering.html
And of course:
http://tomcat.apache.org/tomcat-5.5-doc/cluster-howto.html
Note: I imported this from a pot I wrote 20 years ago! Does anyone use Java anymore? I haven’t since 2016. But I like the thought.
Note: Some very old info on tomcat container auth
The conventional approach when connecting Java Servlets to Active Directory is to use the Java Naming and Directory Interface (JNDI) built into the Web or Servlet Container. This is done by configuring a security realm in the Container, making it available to all servlets inside.
Servlets such as the Tomcat Manager Application, which use the security-restraint mechanism, can immediately take advantage of this interface. Other servlets can be converted to use container based relatively easily. Configuration of both the Manager App and an example of a servlet with form-based login are detailed below. Directory Prep
In order for an application to talk to Active Directory, the service must have an account. This is known as a service account. Contact your AD Org Unit Administrator and ask to have a development account and groups created for your system. Alternatively, use your favorite tool and follow the procedure Creating Tomcat Resources in AD, For Development
Configuring tomcat means editing the server.xml file. This is usually found in [tomcat home]/conf/server.xml, tomcat home being where you installed tomcat at.
Tomcat ships with the JNDI resource “UserDatabase” in the server.xml. This is a connection to a flat text file. By substituting our own JNDI resource, we can take advantage of Active Directory.
Note: We use the service account “OIT-tomcat-boss4” in the Org Unit Folder “Ohio/OIT/OIT-SysOps/Service Accounts”. Yours will be what your OUAdmin gives you, or what you create.
Comment out:
And then right below it Add
<Realm className="org.apache.catalina.realm.JNDIRealm"
connectionName="CN=jjjjj,OU=Service Accounts,OU=jjjjj"
connectionPassword="SomePassWord"
connectionURL="ldap://jjjjj:389"
userRoleName="member"
userBase="CN=jjjjj"
userPattern="CN={0},OU=jjjjjj"
roleBase="ou=Groups,ou=jjjjj"
roleName="cn"
roleSearch="(member={0})"
roleSubtree="false"
userSubtree="false"
/>
Note: This is a ‘un-encrypted’ connection to AD via the LDAP protocol. The problem isn’t passwords, they are one-way hashed during the bind process. What is at risk is the data. Who is in what group is passing in plain text. This is in most cases marginally OK. However, if group membership is deemed to be a security risk you can implement LDAPS. (How To Needed)
It is also possible to use MD5 passwords and SASL mechanisms. (How To Needed)
[tomcat home]/webapps/manager/WEB-INF/web.xml
Web App Side Config
The webapp requires a two-part change;
The Form
To use this with an existing webapp, that has a form (that you don’t want to change) simply change the details of your form so it uses the same method, action and names as the following.
<form method="POST" action="j_security_check">
<input type="text" name="j_username">
<input type="password" name="j_password">
</form>
Tomcat will take these values and pass them to the JNDI module, which will check against Active Directory.
The Web.xml
You add the controls you want by creating rolls and limiting what Resources and Methods that role can access.
If a user is not in the “AD Dev Portal” they will not be able to edit, rename, upload, etc. (see the pages and access listed in the security constraints).
Note that the data constraints are commented out because we have this on the inside network and aren’t using SSL. See Section above on SSL implementation.
<!-- This application has two basic areas; -->
<!-- the webroot of the application accessed by all users(/MyWebApp) -->
<!-- and the admin pages (/MyWebApp/Admin) -->
<!-- We assign the role-name 'Users' to first, and 'Admin' to the second -->
<security-constraint>
<display-name>All Users</display-name>
<web-resource-collection>
<web-resource-name>All Users</web-resource-name>
<url-pattern>/MyWebApp</url-pattern>
</web-resource-collection>
<auth-constraint>
<role-name>Users</role-name>
</auth-constraint>
</security-constraint>
<security-constraint>
<display-name>Admin Users</display-name>
<web-resource-collection>
<web-resource-name>Admin Users</web-resource-name>
<url-pattern>/MyWebApp/Admin</url-pattern>
</web-resource-collection>
<auth-constraint>
<role-name>Admins</role-name>
</auth-constraint>
</security-constraint>
<!-- And here we map the User and Admin roles to the corresponding Active Directory groups -->
<security-role>
<description>CBW Webapp Admins</description>
<role-name>Admin</role-name>
<group-name>CBW-WebappAdmin</group-name>
</security-role>
<security-role>
<description>CBW Webapp Users</description>
<role-name>User</role-name>
<group-name>CBW-WebappUser</group-name>
</security-role>
<!-- We can also specify how to handle log in -->
<login-config>
<auth-method>FORM</auth-method>
<form-login-config>/MyWebApp/login.jsp</form-login-config>
<form-error-page>/MyWebApp/error_login.jsp</form-error-page>
</login-config>
More web.xml
The previous example did not seem to work, but here is another possible configuration. In the web.xml for the application (Tomcat manager, lambda probe, etc), add a security-role for your Active Directory group. Then add references to that role along side any references to other security-roles that came with the application. In this way you extend the original configuration rather than alter it.
For example, within the Tomcat Manager web.xml file:
<!-- Define a Security Constraint on this Application -->
<security-constraint>
<web-resource-collection>
<web-resource-name>HTMLManger and Manager command</web-resource-name>
<url-pattern>/jmxproxy/*</url-pattern>
<url-pattern>/html/*</url-pattern>
<url-pattern>/list</url-pattern>
<url-pattern>/expire</url-pattern>
<url-pattern>/sessions</url-pattern>
<url-pattern>/start</url-pattern>
<url-pattern>/stop</url-pattern>
<url-pattern>/install</url-pattern>
<url-pattern>/remove</url-pattern>
<url-pattern>/deploy</url-pattern>
<url-pattern>/undeploy</url-pattern>
<url-pattern>/reload</url-pattern>
<url-pattern>/save</url-pattern>
<url-pattern>/serverinfo</url-pattern>
<url-pattern>/status/*</url-pattern>
<url-pattern>/roles</url-pattern>
<url-pattern>/resources</url-pattern>
</web-resource-collection>
<auth-constraint>
<!-- NOTE: This role is not present in the default users file -->
<!-- Ohio University: Added the OIT-Tomcat-Admin AD group with same privs as a manager. -->
<role-name>manager</role-name>
<role-name>OIT-Tomcat-Admin</role-name>
</auth-constraint>
</security-constraint>
<!-- Define the Login Configuration for this Application -->
<login-config>
<auth-method>BASIC</auth-method>
<realm-name>Tomcat Manager Application</realm-name>
</login-config>
<!-- Security roles referenced by this web application -->
<security-role>
<description>The role that is required to log in to the Manager Application</description>
<role-name>manager</role-name>
</security-role>
<!-- Ohio University: Added the OIT-Tomcat-Admin Active Directory Group. -->
<security-role>
<description>Declares an OIT AD Group to the Tomcat Manager Application</description>
<role-name>OIT-Tomcat-Admin</role-name>
</security-role>
Adding the AD CA to the keystore that tomcat is configured to use, similar to how we add the Geotrust CA, should do the trick
In the webapp, you can ensure transport security by adding a constraint as below - if - you have setup LDAPS on the server
<security-constraint>
<display-name>All Users</display-name>
<web-resource-collection>
<web-resource-name>All Users</web-resource-name>
<url-pattern>/MyWebApp</url-pattern>
</web-resource-collection>
<auth-constraint>
<role-name>Users</role-name>
</auth-constraint>
<user-data-constraint>
<transport-guarantee>CONFIDENTIAL</transport-guarantee>
</user-data-constraint>
</security-constraint>
Example of ldap to role mapping: https://jazz.net/learn/LearnItem.jsp?href=content/tech-notes/jazz-team-server-1_0-user-management-in-tomcat/index.html
Note: Circa 2008
The best way to have tomcat run on port 80, but not be root, is to use iptables.
One can change the tomcat connector, but since non root processes don’t have access to ports under 1024, nothing will happen. Since iptables is part of the kernel and already running on most servers, it’s a relatively short jump to turn on forward from port 80 to 8080, tomcat’s native port.
The iptables command is:
iptables -t nat -A PREROUTING -p tcp --dport 80 -i eth0 -j REDIRECT --to-port 8080
To make this happen with SuSE, you need to tag into the SuSEfirewall2 system ( this is a series of scripts that do all the heavy lifting of iptables in SuSE so you don’t have to)
You could edit the /etc/sysconfig/SuSEfirewall2 config file, but since we already have the iptables command we can simply insert it into the /etc/sysconfig/scripts/SuSEfirewall2-custom config file.
Don’t forget to activate the custom-config file in your main config file. (/etc/sysconfig/SuSEfirewall2 section 25)
My thanks to Ramon Casha whose article the iptables command and idea is pulled from http://linux.org.mt/article/tomcat-ports
Most of the instructions on how to do this are at http://jakarta.apache.org/tomcat/tomcat-5.0-doc/ssl-howto.html
However, there are some peculiarities to using the IBM jre as opposed to the sun.
In the tomcat connector, one has to use algorithm=“IbmX509” and sslProtocol=“SSL”
If you don’t use the right algorithm, you’ll see it in your tomcat log. The second item IExplorer doesn’t work with.
As with the port 80 instructions, you’ll need to add a hook in the custom firewall rules (it’s detailed in the other inst)
Random guy on the Internet:
Fun fact: Debian’s pronunciation in Chinese means poo
Response:
Wrong, at least in Mandarin. Poop is ‘dà biàn‘’. It’s two words with a ah sound instead of an eh sound.
That back and forth pretty much sums up the rest of the internet.
Building Debian packages is the first-half of distributing software in their ecosystem. In private use, it’s how you can build it in one place and and install it somewhere else. Keeps you from installing a bunch of build-tools (which can get big) everywhere.
Here’s an example with libTorrent and rTorrent.
I’ll use the example of libtorrent here.
# Install the deb build basics, plus a couple things the source needs
sudo apt install git debhelper devscripts dh-make libcppunit-dev pkg-config
# We want the latest release so check GitHub as needed
git clone https://github.com/rakshasa/libtorrent --branch v0.16.6 --depth 1 libtorrent-0.16.6
# If you cloned head for some reason, you can set the package version explicitly
# dh_make --createorig -s -y -p libtorrent_0.16.6
dh_make --createorig -s -y
# The readme says it needs (in addition to make and autoconf) curl-libcurlpp-dev
sudo apt install libcurlpp-dev
# Create the Debian template files
cd libtorrent-*
dh_make --createorig -s -y
Edit the control file with the build details. This git project will actually build three packages;
| libtorrent36 | The main package you’re building |
| libtorrent | The source package. The other packages will use this when they build |
| libtorrent-dev | The development files for anyone else building with libTorrent |
Notice that we put 36 at the end of the package. The the Debian Policy Manual says to add the SONAME (shared object name) to the end to keep things separate.
You can get that from the LibTool number in the autoconf file: grep LIBTORRENT_CURRENT= libtorrent*/configure. You can consult the Debian Library Packaging Guide for more info.
vi debian/control
Source: libtorrent
Section: libs
Priority: optional
Maintainer: you <[email protected]>
Rules-Requires-Root: no
Build-Depends: debhelper-compat (= 13),
pkg-config,
libtool,
automake,
autoconf,
libcppunit-dev,
libcurlpp-dev,
libcurl4-openssl-dev,
libssl-dev,
zlib1g-dev
Standards-Version: 4.7.2
Homepage: https://github.com/rakshasa/libtorrent
Package: libtorrent36
Architecture: any
Multi-Arch: same
Depends: ${shlibs:Depends}, ${misc:Depends}
Description: BitTorrent library with focus on high performance (Rakshasa version)
libtorrent is a BitTorrent library written in C++ with a focus on
high performance and good code. This package contains the shared
library (SONAME 36).
Package: libtorrent-dev
Section: libdevel
Architecture: any
Multi-Arch: same
Depends: libtorrent36 (= ${binary:Version}),
libsigc++-2.0-dev,
libssl-dev,
${misc:Depends}
Description: development files for libtorrent (Rakshasa version)
This package contains the header files and static libraries for
developing applications that use Rakshasa's libtorrent.
Add a rule to handle the lack of a Makefile and make sure it looks in tmp for files it will need between the packages
vi debian/rules
...
...
%:
dh $@
# This override runs BEFORE the automatic configuration step
override_dh_autoreconf:
autoreconf -ivf
dh_autoreconf
# dh_make generated override targets.
# This is an example for Cmake (see <https://bugs.debian.org/641051>).
# Force installation to debian/tmp so multiple packages can pick from it
override_dh_auto_install:
dh_auto_install --destdir=debian/tmp
Create some install files so the packager knows what output files go with what package
vi debian/libtorrent36.install
usr/lib/*/libtorrent.so.36*
vi debian/libtorrent-dev.install
usr/include/torrent/*
usr/lib/*/libtorrent.so
usr/lib/*/pkgconfig/libtorrent.pc
The project builds a ’libtool archive’ file that the packager frowns upon. Add a rule so it knows explicitly that we are going to package it.
echo "usr/lib/*/libtorrent.la" > debian/not-installed
Run the build
debuild -us -uc
Install with
sudo dpkg -i libtorrent36_0.16.6-1_amd64.deb libtorrent-dev_0.16.6-1_amd64.deb
You may need to clean up, add an ignore file and re-build
# Create a ignore file so the builder is OK with the directory have a previous build product in it
vi debian/source/options
extend-diff-ignore = "(^|/)(Makefile\.in|config\.h\.in|scripts/.*\.m4|configure|aclocal\.m4|compile|depcomp|install-sh|missing|ltmain\.sh|config\.sub|config\.guess)$"
debian/rules clean
debuild -us -uc
You’ll need libTorrent, but hopefully you’ve just built it. If you haven’t installed it, do that now.
Note: You probably want XPC enabled as per https://github.com/rakshasa/rtorrent-doc/blob/master/RPC-Setup-XMLRPC.md. We’ll add a configure flag for that.
# Check the current version before you clone
git clone https://github.com/rakshasa/rtorrent --branch v0.16.6 --depth 1 rtorrent-0.16.6
sudo apt install libncurses-dev
# Create the Debian template files
cd rtorrent-*
dh_make --createorig -s -y
Change the debian/control file to be
Source: rtorrent
Section: net
Priority: optional
Maintainer: you <[email protected]>
Rules-Requires-Root: no
Build-Depends:
debhelper-compat (= 13),
pkg-config,
libtool,
automake,
autoconf,
libtorrent-dev,
libncursesw5-dev,
libcurl4-openssl-dev,
libcppunit-dev,
libxmlrpc-c++9-dev
Standards-Version: 4.7.2
Homepage: https://github.com/rakshasa/rtorrent
#Vcs-Browser: https://salsa.debian.org/debian/rtorrent
#Vcs-Git: https://salsa.debian.org/debian/rtorrent.git
Package: rtorrent
Architecture: any
Depends:
${shlibs:Depends},
${misc:Depends},
adduser,
tmux
Description: ncurses BitTorrent client based on libtorrent
rtorrent is a BitTorrent client that uses ncurses and aims to be a
lean, yet powerful terminal application.
Make the debian/rules like this:
#!/usr/bin/make -f
# See debhelper(7) (uncomment to enable).
# Output every command that modifies files on the build system.
#export DH_VERBOSE = 1
# See FEATURE AREAS in dpkg-buildflags(1).
#export DEB_BUILD_MAINT_OPTIONS = hardening=+all
# See ENVIRONMENT in dpkg-buildflags(1).
# Package maintainers to append CFLAGS.
#export DEB_CFLAGS_MAINT_APPEND = -Wall -pedantic
# Package maintainers to append LDFLAGS.
#export DEB_LDFLAGS_MAINT_APPEND = -Wl,--as-needed
%:
dh $@
# Generate the configure script before building
override_dh_autoreconf:
autoreconf -ivf
dh_autoreconf
# dh_make generated override targets.
# This is an example for Cmake (see <https://bugs.debian.org/641051>).
#override_dh_auto_configure:
# dh_auto_configure -- \
# -DCMAKE_LIBRARY_PATH=$(DEB_HOST_MULTIARCH)
override_dh_auto_configure:
dh_auto_configure -- --with-xmlrpc-c
Include a service unit file
vi debian/rtorrent.service
[Unit]
Description=rTorrent System Service
After=network.target
[Service]
Type=forking
User=rtorrent
Group=rtorrent
WorkingDirectory=/var/lib/rtorrent
# Capture the stdin and out to send to the journal
ExecStart=/usr/bin/tmux new-session -d -s rtorrent 'rtorrent 2>&1'
StandardOutput=journal
StandardError=journal
SyslogIdentifier=rtorrent
ExecStop=/usr/bin/tmux send-keys -t rtorrent C-q
TimeoutStopSec=30
Restart=on-failure
[Install]
WantedBy=multi-user.target
Add a template file.
vi debian/rtorrent.rc
# rTorrent System Configuration
directory.default.set = /var/lib/rtorrent/downloads
session.path.set = /var/lib/rtorrent/.session
# Set port statically for VPN incomming
network.port_range.set = 50000-50000
network.port_random.set = no
protocol.encryption.set = allow_incoming,try_outgoing,enable_retry
dht.mode.set = auto
# Adjust as needed for a remote web server if used. But secure thoroughly as
# this interface allows for remote code execution (I've read)
network.scgi.open_port = 0.0.0.0:8080
# Use a unix domain socket if possible.
# Make sure the webserver and rtorrent share a group and set permissions for that
#network.scgi.open_local = /home/user/rtorrent/rpc.socket
#schedule2 = scgi_permission,0,0,"execute.nothrow=chmod,\"g+w,o=\",/home/user/rtorrent/rpc.socket"
# Redirect all 'info', 'error', and 'critical' logs to the tmux screen (stdout) if desired.
log.add_output = "info", "stdout"
log.add_output = "error", "stdout"
log.add_output = "critical", "stdout"
Create debian/install so the template file is included
vi debian/install
debian/rtorrent.rc etc/
Create a post install to create a user and dirs
vi debian/postinst
#!/bin/sh
set -e
if [ "$1" = "configure" ]; then
# Create system user if missing
if ! getent passwd rtorrent >/dev/null; then
adduser --system --group --home /var/lib/rtorrent \
--shell /bin/false --quiet rtorrent
fi
# Create required directories
mkdir -p /var/lib/rtorrent/downloads
mkdir -p /var/lib/rtorrent/.session
# Link the global config to the user's home
if [ ! -f /var/lib/rtorrent/.rtorrent.rc ]; then
ln -sf /etc/rtorrent.rc /var/lib/rtorrent/.rtorrent.rc
fi
# Fix permissions
chown -R rtorrent:rtorrent /var/lib/rtorrent
fi
#DEBHELPER#
exit 0
Make it executable
chmod +x debian/postinst
Build as before
debuild -us -uc
And install
sudo dpkg -i rtorrent_0.16.6-1_amd64.deb
Start and attach with
sudo -u rtorrent tmux -L rtorrent attach -t rtorrent
Note:
To install on other servers
sudo apt install ./libtorrent* ./rtorrent_0.16.6-1_amd64.deb
Most computers come with something called firmware. This is just software written semi-permanently on a chip in the PC. This is the first thing loaded at power-on. Modern versions include a GUI interface to change settings and test hardware. But it’s most important job is to load the main operating system from storage.
The main OS is normally loaded from a local disk, but it can also be loaded over the network. There are usually two goals;
This saves you from carrying around a boot disk. But most importantly, it allows unattended deployment. You can remotely re-image workstations without a tech on-site. As great as this sounds, it’s not easy and requires solid user management to make it useful. So there hasn’t been wide adoption outside of line-of-business situations. Though it does make updating install sources easier.
This seems like a good idea, and it works at scale. But the cost of a thin client is usually more than an equivalent PC so it’s mostly limited to line-of-business again. And the cost of storage is so cheap that a technical investment in netboot rarely pays off. But it can work when repurposing old PCs with worn out drives for kiosk use.
Either way, you’ll need a boot server
NetBoot systems rely on the network DHCP (Dynamic Host Configuration Protocol) to request info on how to load an OS. So your first step is a DHCP server. Since you probably already have one, and would prefer not to break it while testing out netboot, let’s use a feature that allows the netboot server to work alongside your existing DHCP server; PXE Proxy
This server adds additional DHCP services and HTTP/TFTP related to PXE without interfering with your main IP allocation services.
dnsmasq supports both DHCP, for telling clients about itself, and TFTP, for getting the files to them. We’ll also add lighttpd for HTTP support. That protocol is much faster for systems that can use it.
sudo apt install dnsmasq lighttpd
Use a static IP and a hostname for the server that resolves correctly. We use the server name netboot.lan.
No configuration is needed. It serves up content from /var/www/html folder by default.
When configured in proxy dhcp mode: “…dnsmasq simply provides the information given in –pxe-prompt and –pxe-service to allow netbooting”. So only certain settings are available. This is a bit vague, but testing reveals that you must set the boot file name with the dhcp-boot directive, rather than setting it with the more general DHCP option ID 67, for example.
# Add a file in the drop folder
sudo vi /etc/dnsmasq.d/netboot.conf
# Disable DNS
port=0
# Set for DHCP PXE Proxy mode. It will only answer request from this range.
dhcp-range=192.168.1.0,proxy
# Respond to clients that use 'HTTPClient' or 'PXEClient' to identify themselves.
dhcp-pxe-vendor=PXEClient,HTTPClient
# Send the BOOTP information for the clients using HTTP
dhcp-boot="http://netboot.lan/debian.iso"
# Specify a boot menu option for PXE clients. If there is only one, it's booted immediately.
pxe-service=x86-64_EFI,"Network Boot"
pxe-service=x86-64_EFI,"Network Boot (UEFI)",boot/bootmgfw.efi
pxe-service=x86-64_EFI,"iPXE (UEFI)", "ipxe.efi"
# Enable TFTP for the PXE clients.
enable-tftp
tftp-root=/var/www/html
# Restart DNSMasq to enable
sudo systemctl restart dnsmasq.service
The simplest thing possible is to just drop an ISO on the web server. Take a look at the current debian ISO (the numbering changes) at https://www.debian.org/CD/netinst and download.
sudo wget https://cdimage.debian.org/debian-cd/current/amd64/iso-cd/debian-13.1.0-amd64-netinst.iso -P /var/www/html -O debian.iso
Clients may need to have the “Enable UEFI Network Stack” enabled in thier firmware setup. The debian boot loader is signed and works with secure boot.
You didn’t get any choices when booting. A menu with some options is a good thing to add next.
A good way to see what’s going on is to enable dnsmasq logging.
# Add these to the dnsmasq config file
log-queries
log-dhcp
# Restart and follow to see what's happening
sudo systemctl restart dnsmasq.service
sudo journalctl -u dnsmasq -f
If you’ve enabled logging in dnsmasq and it’s not seeing any requests, you may need to look at your networking. Some virtual environments suppress DHCP broadcasts when they are managing the IP range. You can also have an issue with the client resolving DNS. Test with IPs if needed.
PXEClient:Arch:00007:UNDI:003016
If you see this the client isn’t attempting HTTPBoot. Dells specifically lack this feature even circa 2019
You can also see what’s being requested from the web server if you enable access logs.
cd /etc/lighttpd/conf-enabled
sudo ln -s ../conf-available/10-accesslog.conf
sudo systemctl restart lighttpd.service
sudo cat /var/log/lighttpd/access.log
In addition to ISOs, you can also specify .efi binaries like grubx64.efi. This allows you to extract the files and add a pre-seed. Some distributions support this, though Debian itself may have issues.
For a simple menu use the pxe-service built into dnsmasq.
Configure dnsmasq to serve up the ipxe.efi binary for both types of clients.
# add to the settings from the last example
pxe-service=X86-64_EFI,"Network Boot UEFI x86_64",grub/grubx64.efi
You can add netboot.xyz to your iPXE menu to run Live CDs, OS installers and utilities they provide. This can save a lot of time and their list is always improving.
You’re going to connect to the web for this, so there’s nothing to install. You can download their efi bootloader manually if you’d like to keep things HTTPS, but they update it regularly so you may fall behind.
Add a menu item to your autoexec.ipxe. When you select it, iPXE will chainload (in their parlance) the netboot.xyz bootloader.
#!ipxe
echo ${cls}
:MAIN
menu Local Netboot Menu
item --gap Local Network Installation
item WINDOWS ${space} Windows 11 LTSC Installation
item DEBIAN ${space} Debian Installation
item --gap Connect to Internet Sources
item NETBOOT ${space} Netboot.xyz
choose selection && goto ${selection} || goto ERROR
:WINDOWS
echo Some windows things here
sleep 3
goto MAIN
:DEBIAN
dhcp
imgfree
set base http://netboot/debian-installer/amd64
kernel ${base}/linux
initrd ${base}/initrd.gz
boot || goto ERROR
:NETBOOT
dhcp
chain --autofree http://boot.netboot.xyz || goto ERROR
:ERROR
echo There was a problem with the selection. Exiting...
sleep 3
exit
Netboot.xyz detects that it’s working with a Proxy PXE server and behaves a little differently. For example, you can’t insert your own local menu.ipxe. One helpful addition is a local settings file to speed up boot.
sudo vi /var/www/html/local-vars.ipxe
#!ipxe
set use_proxydhcp_settings true
You can choose the new menu item and load netboot.xyz. It will take you out the web for more selections. Not everything will load on every client, of course. But it gives you a lot of options.
We glossed over how to install Windows. That’s a useful item.
tftp://192.168.0.1/local-vars.ipxe....Connection timed out
Local vars file not found... attempting TFTP boot...
DHCP proxy detected, press p to boot from 192.168.0.2...
If your boot client is attempting to connect to the main DHCP server, that server is probably sending value next server: 192.168.0.1 in it’s packets. This isn’t a DNS option per say, but it affects netboot. Dnsmasq does this though Kea doesn’t.
sudo systemctl -u dnsmasq -f
...
...
next server: 192.168.0.1
...
...
The boot still works, it’s just annoying. You can usually ignore the message and don’t have to hit ‘p’.
Could not boot: Exec format error (https://ipxe.org/2e008081)
You may see this flash by. Check your menus and local variables file to make sure you’ve in included the #!pxe shebang.
You can also host your own local instance.
It would be useful to have some choices when you netboot. You can use the pxe-service built into dnsmasq but a more flexible option is the menu system provided by the iPXE project.
Set up a http/pxe net-boot server if you haven’t already.
Configure dnsmasq to serve up the ipxe.efi binary for both types of clients.
# Disable DNS
port=0
# Use in DHCP PXE Proxy mode
dhcp-range=192.168.0.0,proxy
# Tell dnsmasq to provide proxy PXE service to both PXE and HTTP clients
dhcp-pxe-vendor=PXEClient,HTTPClient
# Send the BOOTP information for the clients using HTTP
dhcp-boot="http://netboot/ipxe.efi"
# Specify a boot menu option for PXE clients. If there is only one, it's booted immediately.
pxe-service=x86-64_EFI,"iPXE (UEFI)", "ipxe.efi"
# We also need to enable TFTP for the PXE clients
enable-tftp
tftp-root=/var/www/html
Change the autoexec.ipxe to display a menu.
sudo vi /var/www/html/autoexec.ipxe
#!ipxe
echo ${cls}
:MAIN
menu Local Netboot Menu
item --gap Local Network Installation
item WINDOWS ${space} Windows 11 LTSC Installation
item DEBIAN ${space} Debian Installation
choose selection && goto ${selection} || goto ERROR
:WINDOWS
echo Some windows things here
sleep 3
goto MAIN
:DEBIAN
dhcp
imgfree
set base http://netboot/debian-installer/amd64
kernel ${base}/linux
initrd ${base}/initrd.gz
boot || goto ERROR
:ERROR
echo There was a problem with the selection. Exiting...
sleep 3
exit
You’ll doubtless find additional options to add. You may want to add the netboot.xyz project to your local menu too.
To install windows, have iPXE load wimboot then WinPE. From there you can connect to a samba share and start the Windows installer. Just like back in the gold-ole administrative installation point days.
Getting a copy of WinPE the official way is a bit of a hurdle, but definitely less work than setting up a full Windows imaging solution.
On the netboot server, install wimboot and Samba.
sudo wget https://github.com/ipxe/wimboot/releases/latest/download/wimboot -P /var/www/html
sudo apt install samba
On a Windows workstation, download the ADK and PE Add-on and install as per Microsoft’s ADK Install Doc.
Prepare the netboot server to receive the Windows files.
sudo vi /etc/samba/smb.conf
[global]
map to guest = bad user
log file = /var/log/samba/%m.log
[install]
path = /var/www/html
browseable = yes
read only = no
guest ok = yes
guest only = yes
sudo mkdir /var/www/html/winpe
sudo mkdir /var/www/html/win11
sudo chmod o+w /var/www/html/win*
sudo systemctl restart smbd.service
On the Windows workstation, start the deployment environment as an admin and create the working files as below. More info is in Microsoft’s Create Working Files document.
copype amd64 c:\winpe\amd64
Add the required additions for Windows 11 with the commands below. These are the optional components WinPE-WMI and WinPE-SecureStartup and more info is in Microsoft’s Customization Section.
mkdir c:\winpe\offline
dism /mount-Image /Imagefile:c:\winpe\amd64\media\sources\boot.wim /index:1 /mountdir:c:\winpe\offline
dism /image:c:\winpe\offline /add-package /packagepath:"..\Windows Preinstallation Environment\amd64\WinPE_OCs\WinPE-WMI.cab" /packagepath:"..\Windows Preinstallation Environment\amd64\WinPE_OCs\WinPE-SecureStartup.cab"
dism /unmount-image /mountdir:c:\winpe\offline /commit
Make the ISO in case you want to HTTP Boot from it later and keep the shell open for later.
MakeWinPEMedia /ISO C:\winpe\amd64 C:\winpe\winpe_amd64.iso
Now that you’ve got a copy of WinPE, copy it to the netboot server.
net use q: \\netboot\install
xcopy /s c:\winpe\* q:\winpe
Also create some auto-start files for setup. The first is part to the WinPE system and tells it (generically) what to do after it starts up.
notepad q:\winpe\amd64\winpeshl.ini
[LaunchApps]
"install.bat"
This the second is more specific and associated with the thing you are installing. We’ll mix and match these in the PXE menu later so we can install different things.
notepad q:\win11\install.bat
wpeinit
net use \\netboot
\\netboot\install\win11\setup.exe
pause
You also need to obtain the latest ISO and extract the contents.
Bck on the netboot server, customize the WINDOWS section of your autoexex.ipxe like this.
:WINDOWS
dhcp
imgfree
set winpe http://netboot/winpe/amd64
set source http://netboot/win11
kernel wimboot
initrd ${winpe}/media/sources/boot.wim boot.wim
initrd ${winpe}/media/Boot/BCD BCD
initrd ${winpe}/media/Boot/boot.sdi boot.sdi
initrd ${winpe}/winpeshl.ini winpeshl.ini
initrd ${source}/install.bat install.bat
boot || goto MAIN
You can add other installs by copying this block and changing the :WINDOWS header and source variable.
Add some more installation sources and take a look at the Windows zero touch install.
A given client may be unable to connect to the SMB share, or it may fail once, but then connect on a retry a moment later. I suspect it’s because the client doesn’t have an IP yet, though I’ve not looked at it closely. You can usually just retry.
You can also comment out the winpeshl.ini line and you’ll boot to a command prompt that will let you troubleshoot. Sometimes you just don’t have an IP yet from the DHCP server and you can edit the install.bat file to add a sleep or other things. See then [zero touch deployment] page for some more ideas.
This may be related to the executable bit. If you’ve copied from the ISO they should be set. But if after that you’ve changed anything you could have lost the x bit from setup.exe. It’s hard to know what’s supposed to be set once it’s gone, so you may want to recopy the files.
Many older systems can’t HTTP Boot so let’s add PXE support with some dnsmasq options.
Install as in the httpboot page.
Older clients don’t handle ISOs well, so grab and extract the Debian netboot files.
sudo wget http://ftp.debian.org/debian/dists/bookworm/main/installer-amd64/current/images/netboot/netboot.tar.gz -O - | sudo tar -xzvf - -C /var/www/html
Grub is famous for ignoring proxy dhcp settings, so let’s start off the boot with something else; iPXE. It can do a lot, but isn’t signed so you must disable secure boot on your clients.
sudo wget https://boot.ipxe.org/ipxe.efi -P /var/www/html
Debian is ready to go, but you’ll want to create an auto-execute file for iPXE so you don’t have to type in the commands manually.
sudo vi /var/www/html/autoexec.ipxe
#!ipxe
set base http://netboot/debian-installer/amd64
dhcp
kernel ${base}/linux
initrd ${base}/initrd.gz
boot
HTTP and PXE clients need different information to boot. We handle this by adding a filename to the PXE service option. This will override the dhcp-boot directive for PXE clients.
sudo vi /etc/dnsmasq.d/netboot.conf
# Disable DNS
port=0
# Use in DHCP PXE Proxy mode
dhcp-range=192.168.0.0,proxy
# Respond to both PXE and HTTP clients
dhcp-pxe-vendor=PXEClient,HTTPClient
# Send the BOOTP information for the clients using HTTP
dhcp-boot="http://netboot/debian.iso"
# Specify a boot menu option for PXE clients. If there is only one, it's booted immediately.
pxe-service=x86-64_EFI,"iPXE (UEFI)", "ipxe.efi"
# We also need to enable TFTP for the PXE clients
enable-tftp
tftp-root=/var/www/html
Both types of client should now work. The debian installer will pull the rest of what it needs from the web.
You can create a boot-menu by adding multiple pxe-service entries in dnsmasq, or by customizing the iPXE autoexec.ipxe files. Take a look at that in the menu page.
This is most often a symptom of secure boot still being enabled.
These configs are aimed at UEFI clients. If you have old BIOS clients, you can try the pxe-service tag for those.
pxe-service=x86-64_EFI,"iPXE (UEFI)", "ipxe.efi"
pxe-service=x86PC,"iPXE (UEFI)", "ipxe.kpxe"
This may not work and there’s a few client flavors so enable the dnsmasq logs to see how they identify themselves. You can also try booting pxelinux as in the Debian docs.
Dnsmasq also has a whole tag system that you can set and use similar to this:
dhcp-match=set:PXE-BOOT,option:client-arch,7
dhcp-option=tag:PXE-BOOT,option:bootfile-name,"netboot.xyz.efi"
However, dnsmasq in proxy mode limits what you can send to the clients, so we’ve avoided DHCP options and focused on PXE service directives.
*ERROR* CPU pipe B FIFO underrun
You probably need to use the non-free firmware
No Boot option
Try entering the computers bios setup and adding a UEFI boot option for the OS you just installed. You may need to browse for the file \EFI\debian\grubx64.efi
https://documentation.suse.com/sles/15-SP2/html/SLES-all/cha-deployment-prep-uefi-httpboot.html https://github.com/ipxe/ipxe/discussions/569 https://linuxhint.com/pxe_boot_ubuntu_server/#8
It’s possible to use secure boot if you’re willing to implement a chain of trust. Here’s an example used by FOG to boot devices.
https://forums.fogproject.org/topic/13832/secureboot-issues/3
If you’re deploying Windows servers, Server Core is best practice1. Install from USB and it will offer that as a choice - it’s fairly painless. But these instances are designed to be remote-managed so you’ll need to perform a few post-install tasks to help with that.
The IP is DHCP by default and that’s fine if you create a reservation at the DHCP server or just use DNS. If you require a manual address, however:
# Access the PowerShell interface (you can use the server console if desired)
# Identify the desired interface's index number. You'll see multiple per adapter for IP4 and 6 but the interface index will repeat.
Get-NetIPInterface
# Set a manual address, netmask and gateway using that index (12 in this example)
New-NetIPaddress -InterfaceIndex 12 -IPAddress 192.168.0.2 -PrefixLength 24 -DefaultGateway 192.168.0.1
# Set DNS
Set-DNSClientServerAddress –InterfaceIndex 12 -ServerAddresses 192.168.0.1
This is normally a useful feature, though it depends on your security needs.
Set-NetFirewallRule -Name FPS-ICMP4-ERQ-In -Enabled True
Server core allows ‘Remote Management’ by default2. That is specifically the Server Manager application that ships with Windows Server versions and is included with the Remote Server Admin Tools on Windows 10 professional3 or better. For more detailed work you’ll need to use the Computer Management feature as well. If you’re all part of AD, this is reported to Just Work(TM). If not, you’ll need to allow several ports for SMB and RPC.
# Port 445
Set-NetFirewallRule -Name FPS-SMB-In-TCP -Enabled True
# Port 135
Set-NetFirewallRule -Name WMI-RPCSS-In-TCP -Enabled True
maybe
FPS-NB_Name-In-UDP
NETDIS-LLMNR-In-UDP
If you’re using windows 10/11, install it on a workstation by going to System -> Optional features -> View features and enter Server Manager in the search box to select and install.
When you’re all in the same Domain then everything just works (TM). Or so I’ve read.
If you’re not using Active Directory, you’ll have to do a few extra steps before using the app.
Tell your workstation you trust the remote server you are about to manage4 (yes, seems backwards). Use either the hostname or IP address depending on how your planning to connect - i.e. if you didn’t set up DNS use IPs. Start an admin powershell and enter:
Set-Item wsman:\localhost\Client\TrustedHosts 192.168.5.1 -Concatenate -Force
Start up Server Manager and select Manage -> Add Servers -> DNS and search for the IP or DNS name. Pay attention the server’s name that it detects. If DNS happens to reslove the IP address you put in, as server-1.local for example, you’ll need to repeat the above TrustedHosts command with that specific name.
You may notice that after adding the server, the app tries to connect and fails. You’ll need to right-click it and select Manage As… and enter credentials in the form of server-1\Administrator and select Remember me to have this persist. Here you’ll need to use the actual server name and not the IP. If unsure, you can get this on the server with the hostname command.
The server you added should now say that it’s performance counters are not started. Right-click to and you can select to start them. The server should now show up as Online and you can perform some basic tasks.
server-1.local\Administrator
Server Manager is the default management tool and newer servers allow remote management by default. The client needs a few things, however.
On the system where you start the the Server Manager app - usually where you are sitting - ensure you can resolve the remote host via DNS. You may want to edit your hosts file if not.
notepad c:\Windows\System32\drivers\etc\hosts
You can now add the remote server.
Manage -> Add Servers -> DNS -> Search Box (enter the other servers hostname) -> Magnifying Glass -> Select the server -> Right Arrow Icon -> OK
(You man need to select Manage As on it)
You can right-click on a remote server and select Computer Management after doing this
Set-NetFirewallProfile -Profile Domain, Public, Private -Enabled False
https://learn.microsoft.com/en-us/windows-server/get-started/install-options-server-core-desktop-experience ↩︎
https://learn.microsoft.com/en-us/windows-server/administration/server-core/server-core-sconfig#configure-remote-management ↩︎
https://www.microsoft.com/en-us/download/details.aspx?id=45520 ↩︎
https://learn.microsoft.com/en-us/windows-server/administration/server-manager/configure-remote-management-in-server-manager#to-enable-server-manager-remote-management-by-using-the-windows-interface ↩︎
The simplest way to zero-touch install Windows is with a web-generated answer file. Go to a site like schneegans and just create it. This removes the need for the complexity of MDS WDS SCCM etc. systems for normal deployments.
Visit schneegans. Start with some basic settings, leaving most at the default, and increase complexity with successive iterations. A problematic setting will just dump you out of the installer and it can be hard to determine what went wrong.
Download the file and use it one of the following ways;
After creating the USB installer, copy the file (autounattend.xml) to the root of the USB drive (or one of these locations) and setup will automatically detect it.
For a netboot install, copy the file to the sources folder of the Windows files.
scp autounattend.xml netboot:/var/www/html/win11/sources
Additionally, some scripting elements of the install don’t support UNC paths so we must map a drive. Back in the Windows netboot page, we created an install.bat to start the installation. Let’s modify that like so
vi /var/www/html/win11/install.bat
wpeinit
SET SERVER=netboot
:NET
net use q: \\%SERVER%\install
REM If there was a problem with the net use command,
REM ping, pause and loop back to try again
IF %ERRORLEVEL% NEQ 0 (
ping %SERVER%
pause
GOTO NET
) ELSE (
q:
cd win11
setup.exe
)
The installer can also add 3rd party software packages by adding commands in the Run custom scripts section to run at initial log-in. We’ll use HTTP to get the files as some versions of windows block anonymous SMB.
On the netboot server, create an apps folder for your files and download packages there.
mkdir /var/www/html/apps; cd /var/www/html/apps
wget https://get.videolan.org/vlc/3.0.9.2/win64/vlc-3.0.9.2-win64.msi
wget https://statics.teams.cdn.office.net/production-windows-x64/enterprise/webview2/lkg/MSTeams-x64.msix
It’s easiest to add this in the web form rather than try and edit the XML file. Go to this section and add a line like this one to the third block of custom scripts. It must run at initial user login as the network isn’t available before that.
Navigate to the block that says:
Scripts to run when the first user logs on after Windows has been installed
These and handled as .cmd files as in field 1.
msiexec /package http://netboot/apps/GoogleChromeStandaloneEnterprise64.msi /quiet
msiexec /package http://netboot/apps/vlc-3.0.9.2-win64.msi /quiet
These are handled as .ps1 files as in field 2.
Add-AppPackage -path http://netboot/apps/MSTeams-x64.msix
These are are also handled in the .ps1 files in field 2. They require more work however, as you must download, run, then remove them.
(New-Object System.Net.WebClient).DownloadFile("http://netboot/apps/WindowsSensor.MaverickGyr.exe","$env:temp\crowd.exe")
Start-Process $env:temp\crowd.exe -ArgumentList "/install /quiet CID=239023847023984098098" -wait
Remove-Item "$env:temp\crowd.exe"
Specifying the KMS product key won’t always allow you to skip the “Select Image” screen. This may be due to an ISO being pre-licensed or have something to do with Windows releases. To fix this, add an InstallFrom stanza to the OSImage block of your unattended.xml file.
<ImageInstall>
<OSImage>
<InstallTo>
<DiskID>0</DiskID>
<PartitionID>3</PartitionID>
</InstallTo>
<InstallFrom>
<MetaData wcm:action="add">
<Key>/Image/Description</Key>
<Value>Windows 11 Enterprise</Value>
</MetaData>
</InstallFrom>
</OSImage>
</ImageInstall>
https://www.tenforums.com/installation-upgrade/180022-autounattend-no-product-key.html
Windows Product Keys https://gist.github.com/rvrsh3ll/0810c6ed60e44cf7932e4fbae25880df
MeshCentral allows remote management of computers over the network and internet. It’s a web-based open source system that’s full featured and allows delegated permissions.
It looks a bit dated at this point but there are no other open competitors that match it.
Create a new linux server to host it, and provision like this:
sudo useradd -r -d /opt/meshcentral -s /sbin/nologin meshcentral
sudo mkdir /opt/meshcentral
cd /opt/meshcentral
sudo apt install nodejs -y
sudo apt install npm -y
sudo npm install meshcentral
sudo -u meshcentral node ./node_modules/meshcentral
sudo chown -R meshcentral:meshcentral /opt/meshcentral
sudo chmod 755 –R /opt/meshcentral/meshcentral-*
sudo nano /etc/systemd/system/meshcentral.service
[Unit]
Description=MeshCentral Server
[Service]
Type=simple
LimitNOFILE=1000000
ExecStart=/usr/bin/node /home/default/node_modules/meshcentral
WorkingDirectory=/home/default
Environment=NODE_ENV=production
User=default
Group=default
Restart=always
RestartSec=10
# Set port permissions capability
AmbientCapabilities=cap_net_bind_service
[Install]
WantedBy=multi-user.target
sudo systemctl enable --now meshcentral.service
The clients look for a ’local’ server by default. Look in
/usr/local/mesh_services/meshagent/meshagent_osx64.msh
And change the matching line to something like:
MeshServer=wss://some.server.org:8443/agent.ashx
It doesn’t come up with a reboot so you have to change the launch deamon as per https://github.com/Ylianst/MeshCentral/issues/4822#issuecomment-1542789876
/Library/LaunchDaemons/meshagent_osx64_LaunchDaemon.plist
cat /Library/LaunchDaemons/meshagent_osx64_LaunchDaemon.plist
Then you have to allow the binary to screen record and remote control when prompted
To get remote logins when the user isn’t logged in, or when he login screen has reappeared from inactivity, you must enable VNC
In the beginning, users time-shared CPUs and virtualization was without form and void. And IBM said “Let there be System/370”. This was in the 70’s and involved men with crew-cuts, horn-rimmed glasses and pocket protectors. And ties. Lots of ties.
At the turn of the millennium, VMWare brought this to intel server that used on average 5% of their capacity up to that point. It was expensive, but worth it.
Today, virtualization and it’s successor, containerization, have been driven to commodities and are now simply part of Windows and Linux kernels.
Specifically with containers, there is just one kernel and it keeps groups processes separate from each other. This is possible because Linux implemented kernel namespaces around 2008 - mostly work by IBM, suitably enough. The program used to work with this is named LXC and you’d use commands like sudo lxc-create --template download --name u1 --dist ubuntu --release jammy --arch amd64. Tool like LXD and Docker (originally) just run on top to provide more management.
You can script a backup without too much trouble. If you’re backing up to something foreign to Windows (Like TruNAS) you can use use the WSL to get access to rsync
On Windows, Get a List of Running VMs
Get-VM |Where-Object { $_.State -eq ‘Running’ } | Select-Object -ExpandProperty Name
Create those on TruNAS as separate datasets (so you manipulate them more easily down the road)
while read X;do zfs create pool01/replication/$X;done < list
Adjust Permissions
chmod go+w create pool01/replication/*
Install Debian
wsl –install –distribution Debian
Install Tools
sudo apt install rsync openssh-client
Create and Copy a Key
ssh-keygen ssh-copy-id [email protected]
Create a PowerShell script
notepad test.ps1
$vmNames = Get-VM |
Where-Object { $_.State -eq 'Running' } |
Select-Object -ExpandProperty Name
foreach ($name in $vmNames) {
$DestPath = "[email protected]:/mnt/pool01/replication/$name/"
$SourcePath = "E:\$name\Virtual Hard Disks"
Write-Host "Creating checkpoint for $name..."
Checkpoint-VM -Name $name -SnapshotName "TempRsync"
# Note the single quotes to pass the file globs through to linux
Write-Host "Launching WSL for rsync"
wsl --cd "$SourcePath" rsync -avz --inplace --progress '*.vh*' "$DestPath"
Write-Host "Merging checkpoint..."
Get-VMSnapshot -VMName $name -Name "TempRsync" | Remove-VMSnapshot
}
Inucs is a light-weight manager for LXC containers and KVM virtual machines. It provides a web and command line interface and handles all the normal lifecycle tasks, plus clustering and storage.
It was forked from Canonical’s widely admired LXD project. Canonical’s decision to move LXD in-house led the lead developer and most of the community to fork, thus giving us Incus.
Incus is currently experiencing very high, active development.
A Linux server is required and Debian is fine, though you may prefer Ubuntu which is what the developers use. This is a light-weight solution and 4 cores and 4G RAM is all you need to get started.
You usually run out of RAM first. If you want to cluster and run Windows VMs, start with 16 Gigs.
The sweet spot for a small cluster is 3 nodes with 16G RAM. Though after a few years you’ll have a lot of sprawl and wish you had 32G. Avoid ceph for small scale. It’s complex and RAM intensive.
It’s a good idea to use the hosts file so cluster members can easily find each other. If you have more than one interface and want to use a dedicated network for cluster communication, this is a good place to start.
sudo sh -c 'cat <<EOF >> /etc/hosts
10.0.0.1 debian1
10.0.0.2 debian2
10.0.0.3 debian3
10.0.0.4 debian4
EOF'
Incus recommends an empty zfs or btrfs1 partition so that you can snapshot your VMs and containers. It’s not required, but useful. If you don’t have a spare disk, create an extra partition to your boot disk during the OS install.
I use /dev/sdb in this example. Wipe it and install the BTRFS programs, but let Incus handle the rest.
sudo wipefs -a /dev/sdb
sudo apt install btrfs-progs
Running many containers can use up the default number of file handles. You should increase that on the host.
sudo vi /etc/security/limits.d/10-nofile.conf
# <domain> <type> <item> <value>
* soft nofile 65535
* hard nofile 65535
sudo vi /etc/systemd/system.conf
DefaultLimitNOFILE=65535
# also increase the system wide file limits
sudo vi /etc/sysctl.d/99-limits.conf
fs.file-max = 200000
sudo reboot
Your stock kernel from Debian is fine, but the current kernel is about 10% faster for some things (especially for ZFS at time or writing). It also happens that Stéphane Graber, the lead developer of Incus, as well as Linus Torvalds and others run the current kernel.
Though I’ve had several showstopping bugs on older hardware and it’s caused DKMS issues, so I recommend skipping this at this point.
But if you must, use a build forked directly from the torvalds/linux repo2.
Note
sudo apt install -y curl gpg
sudo mkdir -p /etc/apt/keyrings/
sudo curl -fsSL https://pkgs.zabbly.com/key.asc -o /etc/apt/keyrings/zabbly.asc
sudo sh -c 'cat <<EOF > /etc/apt/sources.list.d/zabbly-kernel-stable.sources
Enabled: yes
Types: deb
URIs: https://pkgs.zabbly.com/kernel/stable
Suites: $(. /etc/os-release && echo ${VERSION_CODENAME})
Components: main
Architectures: $(dpkg --print-architecture)
Signed-By: /etc/apt/keyrings/zabbly.asc
EOF'
# This will take the kernel from 6.12 to 6.18 on trixie at the time of writting
sudo apt update
sudo apt -y install linux-zabbly
# Reboot to apply the new kernel
sudo reboot
You can install Incus from your distro’s repo, but Zabbly’s (from the lead developer’s company) is a bit newer.
As per https://github.com/zabbly/incus
# Add their key
# sudo apt install -y curl gpg
sudo mkdir -p /etc/apt/keyrings/
sudo wget -O /etc/apt/keyrings/zabbly.asc https://pkgs.zabbly.com/key.asc
# Add the LTS repo, needed for the web user interface
sudo sh -c 'cat <<EOF > /etc/apt/sources.list.d/zabbly-incus-lts-6.0.sources
Enabled: yes
Types: deb
URIs: https://pkgs.zabbly.com/incus/lts-6.0
Suites: $(. /etc/os-release && echo ${VERSION_CODENAME})
Components: main
Architectures: $(dpkg --print-architecture)
Signed-By: /etc/apt/keyrings/zabbly.asc
EOF'
# Then add the stable repo, to access the latest stable version of incus
sudo sh -c 'cat <<EOF > /etc/apt/sources.list.d/zabbly-incus-stable.sources
Enabled: yes
Types: deb
URIs: https://pkgs.zabbly.com/incus/stable
Suites: $(. /etc/os-release && echo ${VERSION_CODENAME})
Components: main
Architectures: $(dpkg --print-architecture)
Signed-By: /etc/apt/keyrings/zabbly.asc
EOF'
Now you have a choice;
A Full Install - includes the Web UI and support for both containers and VMs
# A full install
sudo apt update
sudo apt -y install incus incus-ui-canonical
A Minimal Install - command line and only supports containers.
# A minimal install
sudo apt update
sudo apt -y install incus-base
The full install can add up to a gig of dependencies. If keeping it trim is important, start with a full install on just one node, and add the rest as needed. Both versions fully support clustering and all other features.
Install Incus on your other nodes, then start the configuration.
“The recommended backends are zfs and btrfs” at https://github.com/zabbly/linux ↩︎
https://linuxcontainers.org/incus/docs/main/howto/initialize/ ↩︎
Start the initialization on the node with the full install. Node 1 for us.
ssh debain1
sudo incus admin init
You’re safe to take the defaults for all but a couple questions.
Would you like to use clustering? (yes/no) [default=no]: yes
Would you like to use an existing empty block device (e.g. a disk or partition)? (yes/no) [default=no]: yes
Path to the existing block device: /dev/sdb
The initializer uses the hostname by default. In some cases it resolves that as 127.0.1.1 based on the first line in the hosts file. That will cause problems for the other hosts and prevent more esoteric things like split-brain DNS. You can check and possibly correct this with the command incus admin cluster.
If you have multiple NICs you must tell Incus to listen on them (or at least the one you want).
# Be careful about changing the port later on, as it's also used for cluster communication
sudo incus config set core.https_address 0.0.0.0:8443
Before you leave the 1st host, create a cluster join token.
# Assuming the next server you add will be NODENAME2
sudo incus cluster add debian2
Repeat the config on successive hosts, adding the disk similar to before.
sudo incus admin init
Would you like to use clustering? (yes/no) [default=no]: yes
Are you joining an existing cluster? (yes/no) [default=no]: yes
You can now connect to the Web user interface. It will prompt you to generate a key pair. I suggest not using a password as it complicates the import. You can then copy the key from the server to your workstation and import it into your browser.
Incus creates a virtual network for your guests and provides DHCP and DNS. It does this by creating a linux bridge device incusbr0, wrapping an instance of dnsmasq, and creating a nftable to handle masquerade.
If you want to isolate guests, you create a new network and a profile that uses that, Then assign guests to that profile.
If you have a cluster you probably want instances to talk to each other. The simplest way to do that is to put them on the LAN. Do that by installing bridge-utils and creating a bridge.
On Debian 13 use systemd-networkd for this.
# Install the bridge utilities
sudo apt install bridge-utils
# Create a network device file to define the bridge device
vi /etc/systemd/network/10-br0.netdev
[NetDev]
Name=br0
Kind=bridge
# Define the network details in a network file
vi /etc/systemd/network/10-br0.network
[Match]
Name=br0
[Network]
Address=192.168.250.100/24
Gateway=192.168.250.1
DNS=192.168.250.1
# And then reload
sudo systemctl restart systemd-networkd.service
# Tell Incus about it
incus network create LAN-BRIDGE --type=physical parent=br0
# Change the default profile so instances use the bridge by default
incus profile device remove default eth0
incus profile device add default eth0 nic nictype=bridged parent=br0 name=eth0
Note: Incus may need a restart before it accepts br0
An interesting edge case is testing netboot where you need to turn off Incus’s DHCP. For that you must create a network without those services at the command line.
incus network create test ipv4.address=none ipv6.address=none
incus profile copy default isolated
Incus uses the concept of a storage pool. This is a location to hold disk images, ISOs, profiles and such. You can see the default with:
sudo incus storage show default
This is a shared space but instances are kept separate. Each gets their own volume inside the pool. This is thin-provisioned so it only takes as much space as used, but there is a max size of 10G. In some cases this isn’t enough so you can change the default.
sudo incus storage set default size 100GiB
The best way is to mount it on your host and pass it to your containers with a bind-mount. This keeps the kernel modules and caching out of the containers where you’d otherwise need escalated privileges and have less efficient caching.
User IDs are normally mis-matched because containers have their own name-space. But you can line them up easily with the shift option.
incus config device add syncthing srv disk shift=true source=/srv/media path=/srv/media
If you’re transcoding media it helps to have access to a video card, or at least the integrated video card hardware in modern CPUs.
Here’s a good discussion on it.
https://discuss.linuxcontainers.org/t/sharing-a-gpu-with-multiple-containers/21913/2
If you don’t have an existing media or render group, you can use the GUI of the process owner. For a server like Jellyfin, you can use the GID of the jellyfin user.
You can sudo Incus commands, but you can also add users to the admin group.
sudo adduser YOUR-USERNAME incus-admin
incus admin init
You can access the Web UI at https://debian1:8443/ui/. Authentication is handled by client certificate and the GUI will step you through it. The main thing to know is that after creating a certificate pair, you must copy the private key from the server to your workstation and import it in your browser. You may want to generate the cert without a password to facilitate this.
This is the most efficient use of resources and the OCI is built in. Simply pick one from the list. This only allows for linux as it’s a kernel-sharing technology.
Some containers may run out of file handles if they are handling lots of streams. You can increase the limit per-container with:
sudo incus config set SOMECONTAINER limits.kernel.nofile 65535
Download the Windows installation ISO and attach it. Also, since Windows does’nt include VirtIO drivers, you’ll need that too.
# Check the on the web for the current URLs before you wget them
wget https://fedorapeople.org/groups/virt/virtio-win/direct-downloads/latest-virtio/virtio-win.iso
wget https://drive.massgrave.dev/en-us_windows_11_consumer_editions_version_24h2_updated_sep_2025_x64_dvd_6d1ad20d.iso -O Win11_24H2_English_x64.iso
# Create a new VM and allow access to the TPM
incus init win11vm --empty --vm
incus config device add win11vm vtpm tpm path=/dev/tpm0
# Increase the hardware over the defaults
incus config device override win11vm root size=64GiB io.bus=nvme
incus config set win11vm limits.cpu=4 limits.memory=8GiB
# Attach the install and driver ISOs
incus config device add win11vm install disk source=/home/allen/Downloads/Win11_24H2_English_x64.iso io.bus=usb boot.priority=10
incus config device add win11vm virtio disk source=/home/allen/Downloads/virtio-win.iso io.bus=usb boot.priority=5
# Start the VM and attach
incus start win11vm --console=vga
# When done, shutdown and remove the ISOs
# (after installing the rest of the VirtIO Drivers)
incus config device remove win11vm install
incus config device remove win11vm virtio
Windows will automatically find the storage drivers, but not the network. You can manually search when prompted, or you can use the shift + F10 + OOBE\BYPASSNRO trick. This will restart the config and let you bypass that part. Then you can install the full set of drivers from the root of the driver disk, post-install.
The process is similar for Server 2025, but you may need a newer version of the VirtIO drivers. The above is usually the latest, but check https://github.com/qemus/virtiso/releases if you run into problems.
incus init win25 --empty --vm
incus config device add win25 vtpm tpm path=/dev/tpm0
incus config device override win25 root size=64GiB io.bus=nvme
incus config set win25 limits.cpu=4 limits.memory=8GiB
incus config device add win25 install disk source=/home/allen/Downloads/26100.1742.240906-0331.ge_release_svc_refresh_SERVER_EVAL_x64FRE_en-us.iso io.bus=usb boot.priority=10
incus config device add win25 virtio disk source=/home/allen/Downloads/virtio-win-0.1.285.iso io.bus=usb boot.priority=5
incus start win25 --console=vga
# As the system restarts, use:
incus console win25 --type=vga
incus config device remove win25 install
incus config device remove win25 virtio
If you do a lot of Windows installs, you may want to create your own ISO that includes some installation logic as well as the VirtIO drivers.
Simos Xenitellis has some great info on that on his blog as does Scott Thompson who’s info I use a lot.
You normally want just containers. But if you must run a full VM, you should install the incus agent.
sudo apt install incus-agent
You may need to manually mount an install source if the above command fails.
There’s no built-in facility for this. But it’s fairly trivial to create script like this;
#!/bin/bash
BACKUP_DIR="/mnt/gluster/media/backup"
INSTANCES=$(incus list --format csv -c n)
for instance in $INSTANCES; do
incus export "$instance" "$BACKUP_DIR"/"$instance"_backup_$(date +%Y%m%d%H%M) --instance-only --optimized-storage
# Optional: Add logic here to delete old backups in BACKUP_DIR
done
You may need to dedicate temp space for this. When exporting, the server first creates the backup artifact (a compressed tarball) at /var/lib/incus/backups/. If there’s not enough space there, the export will fail.
If you need a dedicated location, use these commands on all your nodes.
# Assuming you have a large pool named 'local' to create a volume in
sudo incus storage volume create local backup-tmpfiles
sudo incus config set storage.backups_volume local/backup-tmpfiles
Scott at scottibyte is the main resource for the windows install.
Proxmox PVE is a distro from the company Proxmox that makes it easy to manage manage containers and virtual machines. It’s built on top of Debian and allows a lot of customization. This can be good and bad compared to VMWare or XCP-NG that keep you on the straight and narrow. But it puts the choice in your hands.
Download the ISO and make a USB installer. It’s a hybrid image so you can write it directly to a USB drive.
sudo dd if=Downloads/proxmox*.iso of=/dev/sdX bs=1M conv=fdatasync
EUFI boot works fine. It will set a static IP address durning the install so be prepared for that.
If your system has an integrated NIC, check out the troubleshooting section after installing for potential issues with RealTek hardware.
After installation has finished and the system rebooted, update it. If you skip this step you may have problems with containers.
# Remove the pop-up warning if desired
sed -Ezi.bak "s/(function\(orig_cmd\) \{)/\1\n\torig_cmd\(\);\n\treturn;/g" /usr/share/javascript/proxmox-widget-toolkit/proxmoxlib.js && systemctl restart pveproxy.service
# Remove the enterprise subscription repos
rm /etc/apt/sources.list.d/pve-enterprise.list
rm /etc/apt/sources.list.d/ceph.list
# Add the non-subscription PVE repo
. /etc/os-release
echo "deb http://download.proxmox.com/debian/pve $VERSION_CODENAME pve-no-subscription" > /etc/apt/sources.list.d/pve-no-subscription.list
# Add the non-subscription Ceph repo - this will change so consult
# Check in your browser --> https://pve.proxmox.com/wiki/Package_Repositories
echo "deb http://download.proxmox.com/debian/ceph-reef bookworm no-subscription" > /etc/apt/sources.list.d/ceph-no-subscription.list
# Alternately, here's a terrible way to get the latest ceph release
LATEST=$(curl https://enterprise.proxmox.com/debian/ | grep ceph | sed 's/.*>\(ceph-.*\)\/<.*\(..-...-....\) .*/\1,\2/' | sort -t- -k3,3n -k2,2M -k1,1n | tail -1 | cut -f 1 -d ",")
echo "deb http://download.proxmox.com/debian/$LATEST $VERSION_CODENAME no-subscription" > /etc/apt/sources.list.d/ceph-no-subscription.list
# Update, upgrade and reboot
apt update
apt upgrade -y
reboot
The template list is updated on a schedule, but you can get a jump on it while you’re logged in. More information at:
https://pve.proxmox.com/wiki/Linux_Container#pct_container_images
pveam update
pveam available
pveam download local (something from the list)
The default config is fine for most use-cases and you can skip the Network section. If you’re in a larger environment, you may want to employ an overlap network, or use VLANs.
PVE creates bridge interface named vmbr0 and assigns a management IP there. As containers and VMs come up, their virtual interfaces will be connected to this bridge so they can have their own MAC addresses.
You can see this in the GUI or in the traditional Debian interfaces file.
cat /etc/network/interfaces
auto lo
iface lo inet loopback
iface enp1s0 inet manual
auto vmbr0
iface vmbr0 inet static
address 192.168.1.11/24
gateway 192.168.100.1
bridge-ports enp1s0
bridge-stp off
bridge-fd 0
You may want to add some additional LANs for your guests, or to separate your management from the rest of the network. You can do this by simply adding some additional LAN addresses.
After changing IPs, take a look further down at how to restrict access.
To add additional DHCP IPs, say because you get a DHCP address from the wall but don’t want PVE management on that, use the up directive. In this example the 192 address is LAN only and the gateway comes from DHCP.
auto lo
iface lo inet loopback
iface enp1s0 inet manual
auto vmbr0
iface vmbr0 inet static
address 192.168.1.11/24
up dhclient vmbr0
bridge-ports enp1s0
bridge-stp off
bridge-fd 0
You should1 use the modern debian method.
auto lo
iface lo inet loopback
iface enp1s0 inet manual
auto vmbr0
iface vmbr0 inet static
address 192.168.1.11/24
bridge-ports enp1s0
bridge-stp off
bridge-fd 0
iface vmbr0 inet static
address 192.168.64.11/24
gateway 192.168.64.1
To can add VLANS in the /etc/network/interfaces file as well.
auto lo
iface lo inet loopback
iface enp1s0 inet manual
auto vmbr0
iface vmbr0 inet manual
bridge-ports enp1s0
bridge-stp off
bridge-fd 0
bridge-vlan-aware yes
bridge-vids 2-4094
auto vmbr0.1337
iface vmbr0.1337 inet static
address 10.133.7.251/24
gateway 10.133.7.1
auto vmbr0.1020
iface vmbr0.1020 inet static
address 10.20.146.14/16
You can use the pvefirewall or the pveproxy settings. There’s an anti-lockout rule on the firewall however, that requires an explicit block, so you may prefer to set controls on the proxy.
The management web interface listens on all addresses by default. You can change that here. Other services, such as ssh, remain the same.
vi /etc/default/pveproxy
LISTEN_IP="192.168.32.11"
You can also combine or substitute a control list. The port will still accept connections, but the application will reset them.
ALLOW_FROM="192.168.32.0/24"
DENY_FROM="all"
POLICY="allow"
pveproxy restart
VMs and Containers work best when they’re light-weight and that means saving data somewhere else, like a NAS. Containers are the lightest of all but using NFS in a container causes a security issue.
Instead, mount on the host and bind-mount to the container with mp.
vi /etc/pve/lxc/100.conf
# Add this line.
# mount point ID: existing location on the server, location to mount inside the guest
mp0: /mnt/media,mp=/mnt/media,shared=1
#mp1: and so on as you need more.
This next thing you’ll notice is that users inside the containers don’t match users outside. That’s because they’re shifted for security. To get them to line up you need a map.
# In the host, edit these files to allow root, starting at 1000, to map the next 11 UIDs and GIDs (in addition to what's there already)
# cat /etc/subuid
root:1000:11
root:100000:65536
# cat /etc/subgid
root:1000:11
root:100000:65536
# Also on the host, edit the container's config
vi /etc/pve/lxc/100.conf
# At the bottom add these
# By default, the container users are shifted up by 100,000. Keep that in place for the first 1000 with this section
## Starting with uid 0 in the container, map it to 100000 in the host and continue mapping for 1000 entries. (users 0-999)
lxc.idmap = u 0 100000 1000
lxc.idmap = g 0 100000 1000
# Map the next 10 values down low so they match the host (10 is just an arbitrary number. Map as many or as few as you need)
## Starting in the container at uid 1000, jump to 1000 in the host and map 10 values. (users 1000-1009)
lxc.idmap = u 1000 1000 10
lxc.idmap = g 1000 1000 10
# Then go back to mapping the rest up high
## Starting in the container at uid 1010, map it 101010 and continue for the next 64525 entries (65535 - 1010)
lxc.idmap = u 1010 101010 64525
lxc.idmap = g 1010 101010 64525
If you want to add mapping to an existing container, user IDs are probably already in place and you’ll have to adjust them. Attempts to do so in the container will result in a permission denied, even as root. Mount them in the PVE host and change them there.
pct mount 119
# For a user ID number of 1000
find /var/lib/lxc/119/rootfs -user 101000 -exec chown -h 1000 {} \;
find /var/lib/lxc/119/rootfs -group 101000 -exec chgrp -h 1000 {} \;
pct unmount 119
Sometimes, you have to change a service to match between different containers. Log into your container and do the following.
# find the service's user account, make note of it and stop the service
ps -ef
service someService stop
# get the exiting uid and gid, change them and change the files
id someService
> uid=112(someService) gid=117(someService) groups=117(someService),117(someService)
usermod -u 1001 someService
groupmod -g 1001 someService
# Change file ownership -xdev so it won't traverse remote volumes
find / -xdev -group 1001 -exec chgrp -h someService {} \;
find / -xdev -user 1001 -exec chown -h someService {} \;
Edit the /etc/hosts file to ensure that the IP address reflects any changes you’ve made (such as the addition on a specific management address). Ideally, add the hostname and IP of all of the impending cluster members and ensure they can all ping each other by that name.
The simple way to create and add members is done at the command line.
# On the first cluster member
pvecm create CLUSTERNAME
# On the other members
pvecm add FIRST-NODE-HOSTNAME
You can also refer to the notes at:
https://pve.proxmox.com/wiki/Cluster_Manager
You can access the Web GUI at:
https://forum.proxmox.com/threads/cannot-log-into-containers.39064/
pct enter 100
You may want to start it in foreground mode to see the error up close
lxc-start -n ID -F -l DEBUG -o /tmp/lxc-ID.log
Containers use LVM by default. If it fails to start and you suspect a disk error, you can fsck it. You can access the content as well. There are also direct ways2 if these fail.
pct list
pct fsck 108
pct mount 108
If your PVE server periodically drops the network with an error message about the realtek firmware, consider updating the driver.
# Add the non free and firmware to the apt source main line
sed -i '/bookworm main contrib/s/$/ non-free non-free-firmware/' /etc/apt/sources.list
apt update
# Install the kernel headers and the dkms driver.
apt -y install linux-headers-$(uname -r)
apt install r8168-dkms
You can’t do this the normal Debian way it seems. In testing, the bridge doesn’t accept mixing types directly. You must use the ip command.
PVE local node address: cannot use IP not found on local node! 500 Can’t connect to XXXX:8006 (hostname verification failed)
Make sure the hosts files on all the nodes match and they can ping each other by hostname. Use hostnames to add cluster members, not IPs.
https://forum.proxmox.com/threads/proxmox-host-is-getting-unavailable.125416/ https://www.reddit.com/r/Proxmox/comments/10o58uq/how_to_install_r8168dkms_package_on_proxmox_ve_73/ https://wiki.archlinux.org/title/Dynamic_Kernel_Module_Support https://pve.proxmox.com/wiki/Unprivileged_LXC_containers https://pve.proxmox.com/wiki/Unprivileged_LXC_containers#Using_local_directory_bind_mount_points https://www.reddit.com/r/homelab/comments/6p3xdw/proxmoxlxc_mount_host_folder_in_an_unprivileged/
Cloud file storage, like dropbox, onedrive, etc. , is a great alternative to a full cloud application suite when you just need to work with files. You don’t get all the features, but it’s dramatically less work. Of these, OpenCloud is the best mix of simplicity and function. Importantly, your files are just saved to disk in the folder structure you created. You can directly access it making everything easy and portable.
Even better, if all you want is the web manager and text/markdown editor you can deploy it as a simply binary in just a few moments. No databases or docker needed.
Check https://github.com/opencloud-eu/opencloud/releases/latest/ for the URL of the latest binary to download and adjust the wget below.
# Download the binary and make it executible
wget https://github.com/opencloud-eu/opencloud/releases/download/v5.0.2/opencloud-5.0.2-linux-amd64
chmod +x ./open*
# Move it to local bin directoy and create a symlink so we can launch it as just 'opencloud'
sudo mv open* /usr/local/bin
sudo ln -s /usr/local/bin/open* /usr/local/bin/opencloud
# Inistialize it right in your home folder. Maybe create a service account if you feel production-ish about it
cd
opencloud init --insecure true --admin-password admin
# Move the directory to /opt and create a symlink to it
sudo mv .opencloud /opt/opencloud
ln -s /opt/opencloud .opencloud
Important - you need to tell the binary about how it will be accessed via environment variables. Otherwise you’ll only be able to access it via localhost. Assuming you’re going to use a secure proxy like nginx or caddy, tell it about those too. You can ONLY access it this way. Anything else will give you a cryptic error message.
sudo vi /etc/systemd/system/opencloud.service
[Unit]
Description=OpenCloud
After=network.target
[Service]
Environment="OC_INSECURE=true"
Environment="OC_URL=https://docs.your.org"
Environment="PROXY_TLS=false"
ExecStart=/usr/local/bin/opencloud server
Restart=on-failure
User=allen
[Install]
WantedBy=multi-user.target
If you were only going to access it inside your LAN, you’d only need the single OC_URL setting.
Environment="OC_URL=https://docs.lan:9200"
You’ve setup OpenCloud in /opt/opencloud and it has a subfolder named storage where it saves user files. If you have any meaningful amout of data you should put that on dedicated storage. Especially if you’re running in a container. You don’t want all that user data gumming things up. If your back-end is local you can just use a bind mount. But if it’s network storage, you’ll want to be a but more granular to keep the metadata and lock files off the back-end.
# Disable OpenCloud while you work on it
sudo systemctl disable --now opencloud
# If you have local storage, you can move the whole storage folder and create a new one for the bind mount
sudo mv /opt/opencloud/storage /opt/opencloud/storage.orig
sudo mkdir /opt/opencloud/storage
# OR #
# For network storage move just the user folder.
sudo mv /opt/opencloud/storage/users /opt/opencloud/storage/users.orig
sudo mkdir /opt/opencloud/storage/users
# Now is the time to bind mount from your host or such to the new folder
# and bouce the instance so it can access the new location
# for gluster and incus you may need to adjust the user IDs with a
# sudo incus config set files raw.idmap "both 1000 1000"
# Copy the content over, preserving the XATTRs that OpenCloud uses
rsync -aHAXv /opt/opencloud/storage/users.orig/ /opt/opencloud/storage/users
After profiling thigs, the only interesting folders on the remote filesystem are
users/uploads users//.oc-nodes users//.oc-nodes/locks users/*/.oc-tmp i
It might be worth symlinking those to directores on a fast SSD but unless the OpenCloud server is busy it’s not worth the complexity out of the gate.
I’ve used many, from SnipSnap to Atlassian’s Confluence. You could argue that SharePoint is in there too. It’s a better alternative than a folder full of word files by far.
And in that time I’ve had to haul my content around as I’ve changed solutions and it’s always a pain.
You know what’s less of a pain, but never needs to change? a directory of text files (or markdown since 2018). And it turns out you can use a JAMstack generator like Hugo to publish that if you really want to get it on the web.
Though sometimes it would be really nice to do a quick edit when you’re looking at something without going through a process.
For group collaboration, I still think a wiki is better. But definitely put some structure around it.
Grav makes a pretty good Wiki, too.
I’d call this the enterprise-like choice. Best if you’re going to build processes and flows around it.
These notes are circa 2020 so use with some caution.
We’ve chosen the docker option detailed in the installation options on their web site and gone with PostgreSQL simply because Oracle now owns MySQL. Minor changes to the docker start scripts are detailed below.
This gets the wiki running on port 8080. To add a layer of security, pair this with secure reverse proxy such as [HAProxy].
This is based on the well documented docker option with a restart option added so they come up after a container reboot.
The first step is to create the user-defined network so the containers to talk to each other privately and connect using their container names as hostnames1.
docker network create -d bridge xwiki-nw
Run the database. We deviate from the instructions and install the current version of postgres. Set the passwords and take note of one you use for YYYYYY and reuse below
docker run \
--net=xwiki-nw \
--name postgres-xwiki \
-v /my/own/postgres:/var/lib/postgresql/data \
-e POSTGRES_ROOT_PASSWORD=XXXXXXXX \
-e POSTGRES_USER=xwiki \
-e POSTGRES_PASSWORD=YYYYYYYY \
-e POSTGRES_DB=xwiki \
-e POSTGRES_INITDB_ARGS="--encoding=UTF8" \
--detach \
--restart=always \
postgres:9.5
Run the xwiki tomcat container with the corresponding password.
docker run \
--net=xwiki-nw \
--name xwiki \
-p 8080:8080 \
-v /my/own/xwiki:/usr/local/xwiki \
-e DB_USER=xwiki \
-e DB_PASSWORD=YYYYYYYY \
-e DB_DATABASE=xwiki \
-e DB_HOST=postgres-xwiki \
--detach \
--restart=always \
xwiki:stable-postgres-tomcat
The install will prompt you to pick a flavor. Definitely pick the Standard Flavor. Picking none leaves you without all the default pages that are handy.
The installation takes a very, very long time.
If all went as planned, you can now connect to your docker host on port 8080 and see the wiki. If not, connect with a bash shell and investigate the logs folder
docker exec -it xwiki bash
We’ve followed the upgrade several times, checked the release note each time and not had a successful upgrade.
You can export the pages, install a fresh wiki and import.
Email is a commodity service, but critical for many things - so you can get it anywhere, but you better not mess it up.
Your options, in increasing order of complexity, are:
Email sent to [email protected] is simply forwarded to someplace like gmail. It’s free and easy, and you don’t need any infrastructure. Most registrars like GoDaddy, NameCheap, CloudFlare, etc, will handle it.
You can even reply from [email protected] by integrating with SendGrid or a similar provider.
If you want more, Google and Microsoft have full productivity suites. Just edit your DNS records, import your users, and pay them $5 a head per month. You still have to ‘do email’ but it’s a little less work than if you ran the whole stack. In most cases, companies that specialize in email do it better than you can.
If you are considering local email, let me paraphrase Kenji López-Alt. The first step is, don’t. The big guys can do it cheaper and better. But if it’s a philosophical, control, or you just don’t have the funding, press on.
Most of the cost is user support. Hosting means someone else gets purchase and patch a server farm, but you still have to talk to users. My (anecdotal) observation is that fully hosting saves 10% in overall costs and it soothes out expenses. The more users you have, the more that 10% starts to matter.
This is the best solution for a small number of users. You configure it at your registrar and rely on google (or someone similar) to do all the work for free.
If you want your out-bound emails to come from your domain name (and you do), add an out-bound relay. This is also free for minimal use.
This is different per registrar, but normally involves creating an address and it’s destination
Cloudflare
Once validated, email will begin arriving at the destination.
The registrars is only forwarding email, not sending it. To get your sent mail to from from your domain, you must integrate with a mail service such as SendGrid
SendGrid
Gmail
After validating the code Gmail sends you, there will be a drop down in the From field of new emails.
This is more in the software-as-a-service category. You get an admin dashboard and are responsible for managing users and mail flow. The hosting service provide will help you with basic things, but you’re doing most of the work yourself.
Having manged 100K+ user mail systems and migrated from on-prem sendmail to exchange and then O365 and Google, I can confidently say the infrastructure and even platform amounts to less than 10% of the cost of providing the service.
The main advantage to hosting is that you’re not managing the platform, installing patches and replacing hardware. The main disadvantage is is that you have little control and sometimes things are broken and you can’t do anything about it.
Medium sized organizations benefit most from hosting. You probably need a productivity suite anyways, and email is usually wrapped up in that. It saves you from having to specialize someone in email and the infrastructure associated with it.
But if controlling access to your data is paramount, then be aware that you have lost that and treat email as a public conversation.
When you self-host, you develop expertise in email itself, arguably a commodity service where such expertise has small return. But, you have full control and your data is your own.
The generally accepted best practice is install Postfix and Dovecot. This is the simplest path and what I cover here. But there are some pretty decent all-in-one packages such as Mailu, Modoboa, etc. These usually wrap Postfix and Dovecot to spare you the details and improve your quality of life, at the cost of not really knowing how they really work.
You’ll also need to configure a relay. Many ISPs block basic mail protocol and many recipient servers are rightly suspicious of random emails from unknown IPs in cable modem land.
This is the first step - a server that handles and stores email. You’ll be able to check messages locally at the console. (Remote client access such as with Thunderbird comes later.)
You need:
We use Debian Bookworm (12) in this example but any derivative will be similar. At large scale you’d setup virtual users, but we’ll stick with the default setup and use your system account. Budget about 10M per 100 emails stored.
Mail protocol uses port 25. Simply forward that to your internal mail server and you’re done.
You need an normal ‘A’ record for your server and a special ‘MX’ record for your domain root. That way, mail sent to [email protected] will get routed to the server.
| Name | Type | Value |
|---|---|---|
| the-server | A | 20.236.44.162 |
| @ | MX | the-server |
Mail servers see [email protected] and look for records of type ‘MX’ for ‘your.org’. Seeing that ’the-server’ is listed, they lookup it’s ‘A’ record and connect. A message to [email protected] is handled the same way, though when there is no ‘MX’ record it just delivers it to the ‘A’ record for ’the-server.your.org’. If you have both, the ‘MX’ takes precedence.
Some configuration is done at install time by the package so you must make sure your hostname is correct. We use the hostname ‘mail’ in this example.
# Correct internal hostnames as needed. 'mail' and 'mail.home.lan' are good suggestions.
cat /etc/hostname /etc/hosts
# Set the external host name and run the package installer. If postfix is already installed, apt remove it first
EXTERNAL="mail.your.org"
sudo debconf-set-selections <<< "postfix postfix/mailname string $EXTERNAL"
sudo debconf-set-selections <<< "postfix postfix/main_mailer_type string 'Internet Site'"
sudo apt install --assume-yes postfix
# Add the main domain to the destinations as well
DOMAIN="your.org"
sudo sed -i "s/^mydestination = \(.*\)/mydestination = $DOMAIN, \1/" /etc/postfix/main.cf
sudo systemctl reload postfix.service
Test with telnet - use your unix system ID for the rcpt address below.
telnet localhost 25
ehlo localhost
# wait for server response
mail from: <[email protected]>
rcpt to: <[email protected]>
data
# wait for server response
Subject: Wish List
Red Ryder BB Gun
.
quit
Assuming that ‘you’ matches your shell account, Postfix will have accepted the message and used it’s Local Delivery Agent to store it in the local message store. That’s in /var/mail.
cat /var/mail/YOU
Postfix uses mbox storage format by default. This is one big file with all your mail in it and doesn’t scale well. Switch to the newer maildir format where your messages are stored as individual files.
# Change where Postfix delivers mail.
sudo postconf -e "home_mailbox = Maildir/"
# Reload the service and delete the old mailstore
sudo systemctl reload postfix.service
sudo rm /var/mail/you
Postfix will use the untrusted “snakeoil” that comes with debian to opportunistically encrypt communication between it and other mail servers. Surprisingly, most other servers will accept this cert (or fall back to non-encrypted), so lets proceed for now. We’ll generate a trusted one later.
The default config is secured so that it won’t relay messages, but it will accept message from Santa, and is subject to backscatter and a few other things. Let’s tighten it up.
sudo tee -a /etc/postfix/main.cf << EOF
# Tighten up formatting
smtpd_helo_required = yes
disable_vrfy_command = yes
strict_rfc821_envelopes = yes
# Error codes instead of bounces
invalid_hostname_reject_code = 554
multi_recipient_bounce_reject_code = 554
non_fqdn_reject_code = 554
relay_domains_reject_code = 554
unknown_address_reject_code = 554
unknown_client_reject_code = 554
unknown_hostname_reject_code = 554
unknown_local_recipient_reject_code = 554
unknown_relay_recipient_reject_code = 554
unknown_virtual_alias_reject_code = 554
unknown_virtual_mailbox_reject_code = 554
unverified_recipient_reject_code = 554
unverified_sender_reject_code = 554
EOF
sudo systemctl reload postfix.service
PostFix has some recommendations as well. Some of these, like smtpd_relay_restrictions may already be in your distributions package so look for messages about that at reload to delete any superfluous settings.
sudo tee -a /etc/postfix/main.cf << EOF
# PostFix Suggestions
smtpd_helo_restrictions = reject_unknown_helo_hostname
smtpd_sender_restrictions = reject_unknown_sender_domain
smtpd_recipient_restrictions =
permit_mynetworks,
permit_sasl_authenticated,
reject_unauth_destination
smtpd_relay_restrictions =
permit_mynetworks,
permit_sasl_authenticated,
reject_unauth_destination
smtpd_data_restrictions = reject_unauth_pipelining
EOF
sudo systemctl reload postfix.service
If you test a message from Santa now, Postfix will do some checks and realize it’s bogus.
550 5.7.27 [email protected]: Sender address rejected: Domain northpole.org does not accept mail (nullMX)
Postfix will attach a Received: header to outgoing emails that has details of your internal network and mail client. That’s information you don’t need to broadcast. You can remove that with a “cleanup” step as the message is sent.
# Insert a header check after the 'cleanup' line in the smtp section of the master file and create a header_checks file
sudo sed -i '/^cleanup.*/a\\t-o header_checks=regexp:/etc/postfix/header_checks' /etc/postfix/master.cf
echo "/^Received:/ IGNORE" | sudo tee -a /etc/postfix/header_checks
Note - there is some debate on if this triggers a higher spam score. You may want to replace instead.
You can now receive mail to [email protected] and [email protected]. Try this to make sure you’re getting messages. Feel free to install mutt if you’d like a better client at the console.
You usually can’t send mail and there are several reasons why.
Many ISPs block outgoing port 25 to keep a lid on spam bots. This prevents you from sending any messages. You can test that by trying to connect to gmail on port 25 from your server.
nc -zv gmail-smtp-in.l.google.com 25
Also, many mail servers will reverse-lookup your IP to see who it belongs to. That request will go to your ISP (who owns the IPs) and show their DNS name instead of yours. You’re often blocked at this step, though some providers will work with you if you contact them.
Even if you’re not blocked and your ISP has given you a static IP with a matching reverse-lookup, you will suffer from a lower reputation score as you’re not a well-known email provider. This can cause your sent messages to be delayed while being considered for spam.
To solve these issues, relay your email though a email provider. This will improve your reputation score (used to judge spam), ease the additional security layers such as SPF, DKIM, DMARC, and is usually free at small volume.
Postfix even calls this using a ‘Smarthost’
Now that you can get email, let’s make it so you can also send it.
When adding Postfix’s anti-spam suggestions, we left off the smtpd_client_restrictions and smtpd_end_of_data_restrictions as they created problems during testing.
You may get a warning from Postfix that one of the settings you’ve added is overriding one of the earlier settings. Simply delete the first instance. These are usually default settings that we’re overriding.
Use ‘@’ to view the logs from all the related services.
sudo journalctl -u [email protected]
If you change your server’s DNS entry, make sure to update mydestination in your /etc/postfix/main.cf and sudo systemctl reload postfix*.
Postfix only accepts messages for users in the “local recipient table” which is built from the unix password file and the aliases file. You can add aliases for other addresses that will deliver to your shell account, but only shell users can receive mail right now. See virtual mailboxes to add users without shell accounts.
In the alias file, you’ll see “Postmaster” (and possibly others) are aliased to root. Add root as an alias to you at the bottom so that mail gets to your mailbox.
echo "root: $USER" | sudo tee -a /etc/aliases
sudo newaliases
They’ve been around for many years and are included in the recommendations from postfix.
smtpd_recipient_restrictions =
reject_rbl_client zen.spamhaus.org,
reject_rhsbl_reverse_client dbl.spamhaus.org,
reject_rhsbl_helo dbl.spamhaus.org,
reject_rhsbl_sender dbl.spamhaus.org
However, if you’re using one of the large public DNS resolvers, like Google, Cloudflare or Quad 9, this won’t work. The service uses DNS to transmit data and Spamhaus has recently started rejecting 1 the large providers. You’ll see errors like this in your journal;
sudo journalctl -u [email protected] | grep spam
May 05 14:01:23 mail postfix/smtpd[527]: NOQUEUE: reject: RCPT from i-ii.cloudflare-email.net[104.30.8.88]: 554 5.7.1 Service unavailable; Client host [104.30.8.88] blocked using zen.spamhaus.org; Error: open resolver; https://check.spamhaus.org/returnc/pub/172.70.41.5/;
And the sender wil get return notices like this;
The response from the remote server was:
521 5.3.0 Upstream error
and
Rejected reason:
upstream (mail.some.org) error: failed to initialize: Unknown error: permanent error (554): 5.7.1 Service unavailable; Unverified Client host [i-bgb.cloudflare-email.net] blocked using dbl.spamhaus.org; Error: open resolver; https://check.spamhaus.org/returnc/pub/172.71.189.5/
The recommended solution is to create an account with Spamhaus. This works with any DNS resolver. Alternatively, you should be able to use your providers DNS server if you don’t mind them data-mining your traffic. Or deploy your own recursive DNS server, like Unbound. Though neither of these worked reliably for me and I didn’t pursue the reasons.
So sign up for a free account and configure postfix’s domain query service daemon with their instructions.
A relay is simply another mail server you send everything to, rather than try to deliver it yourself.
There are many companies that specialize in this. Sign up for a free account and they give you the block of text to add to your postfix config. Some popular ones are:
They allow anywhere between 50 and 300 a day for free. Amazon charges a miniscule fee after a year.
Note: I originally used SendGrid, but they removed their free tier. I migrated to SMTP2GO but didn’t take the best of notes so be suspicious of this doc.
This is the preferred service when you are only interested in a mail relay.
https://www.smtp2go.com/setupguide/postfix/ https://ivansalloum.com/how-to-configure-postfix-for-external-smtp-relay/
Restart Postfix and use mutt to send an email. It works! the only thing you’ll notice is that your message has a “On Behalf Of” notice in the message letting you know it came from SendGrid. Follow the section below to change that.
To integrate your domain fully, add DNS records using these instructions.
https://support.smtp2go.com/hc/en-gb/articles/115004408567-Verified-Senders
If you’re familiar with email domain-based security, you’ll see that two of the records are links to DKIM keys so they can sign emails as you. The other record (emXXXX) is the host they will use to send email. The SPF record for that host will include a SPF record that includes multiple pools of IPs so that SPF checks will pass. They use CNAMEs on your side so they can rotate keys and pool addresses without changing DNS entries.
If none of this makes sense to you, then that’s really the point. You don’t have to know any of it - they take care of it for you.
Your server can now send email too. All shell users on your sever rejoice!
To actually use your mail server, you’ll want to add some remote client access.
Dovecot is an IMAP (Internet Message Access Protocol) server that allows remote clients to access their mail. There are other protocols and servers, but Dovecot has about 75% of the internet and is a good choice.
Use the same server you installed Postfix on. Just forward a couple additional ports.
Since we’re on Trixie we’ll get a fairly current dovecot 2.4. It’s config is slightly different than what came with pervious versions. You can also use Dovecot’s repo but there’s not a significant advantage.
sudo apt install dovecot-imapd dovecot-submissiond
Debian adds some defaults that prevent Dovecot from auto-detecting things. Let’s correct that.
sudo vi /etc/dovecot/conf.d/10-mail.conf
# Search for these under 'Debian defaults'
# Comment these out
#mail_driver = mbox
#mail_home = /home/%{user | username}
#mail_path = %{home}/mail
#mail_inbox_path = /var/mail/%{user}
# Add these right below
mail_driver = maildir
mail_path = %{owner_home}/Maildir
#TODO - should these be in namespace inbox?
Dovecot comes with it’s own default cert. This isn’t trusted, but Thunderbird will prompt you and you can choose to accept it. This will be fine for now. We’ll generate a valid cert later.
Dovecot checks passwords against the local unix system by default. This is normally of the form you. If you’d like to allow [email protected] as well, make this edit.
sudo vi /etc/dovecot/conf.d/10-auth.conf
# Uncomment this line to allow optional your.org after username
auth_username_format = %{user|username|lower}
RFC 8314 strongly recommends IMAPS (Implicit TLS) on port 993 over explicit on port 143. Dovecot accepts both by default and it’s generally recommended to leave them as long as ssl=required remains (set by default) so that clear-text passwords are not accepted.
We’re not forwarding port 143 (Explicit) here, but you can if you want to support older clients.
One potential surprise is that IMAP is only for viewing mail. To send it, you use the SMTP protocol to talk directly to Postfix.
That doesn’t support authentication however, and since we don’t want just anyone relaying messages, it’s disabled for non-local IPs. Instead, you use a separate process called the submission service.
We’ve installed Dovecot’s submission service as it’s newer and easier to set up. Postfix even suggests considering it over theirs. It really just a simply proxy for Postfix that requires you to login.
The RFC recommends clients connect by implicit TLS to port 465. Both types (implicit and explicit) are enabled by default and it’s generally recommended to leave it, as long as plain text passwords remain prohibited as they are by default. Port forward as needed.
The only configuration needed it to set the localhost as the relay.
# Set the relay as localhost where postfix runs
sudo sed -i 's/#submission_relay_host =/submission_relay_host = localhost/' /etc/dovecot/conf.d/20-submission.conf
sudo systemctl reload dovecot.service
If you do want to disable either of these services, set the port = 0. Commenting them out just leaves them on by default.
nc -zv mail.your.org 993
nc -zv mail.your.org 465
If it’s working from outside your network, but not inside, you may need to enable [reflection] aka hairpin NAT. This will be different per firewall vendor, but in OPNSense it’s:
Firewall -> Settings -> Advanced
# Enable these settings
Reflection for port forwards
Reflection for 1:1
Automatic outbound NAT for Reflection
Thunderbird and others will successfully discover the config via ‘heuristics’ some of the time. To make sure, check out the autodiscover page as a last step.
Now that you’ve got the basics working, let’s secure things a little more
We should use valid certificates. The best way to do that is with the certbot utility.
Certbot automates the process of getting and renewing certs, and only requires a brief connection to port 80 as proof it’s you. There’s also a DNS based approach, but we use the port method for simplicity. It only runs once every 60 days so there is little risk of exploit.
You probably already have a web server and can’t just change where port 80 goes. To integrate certbot, add a name-based virtual host proxy to that web server.
# Here is a caddy example. Add this block to your Caddyfile
http://mail.your.org {
reverse_proxy * mail.internal.lan
}
# You can also use a well-known URL if you're already using that vhost
http://mail.your.org {
handle /.well-known/acme-challenge/ {
reverse_proxy mail.internal.lan
}
}
Once the port forwarding is in place, you can install certbot and use it to request a certificate. Note the --deploy-hook argument. This reloads services after a cert is obtained or renewed. Else, they’ll keep using an expired one.
DOMAIN=your.org
sudo apt install certbot
sudo certbot certonly --standalone --domains mail.$DOMAIN --non-interactive --agree-tos -m postmaster@$DOMAIN --deploy-hook "service postfix reload; service dovecot reload"
Once you have a cert, Certbot will keep keep it up-to-date by launching periodically from a cronjob in /etc/cron.d and scanning for any needed renewals.
Tell Postfix about the cert by using the postconf utility. This will warn you about any potential configuration errors.
sudo postconf -e "smtpd_tls_cert_file = /etc/letsencrypt/live/mail.$DOMAIN/fullchain.pem"
sudo postconf -e "smtpd_tls_key_file = /etc/letsencrypt/live/mail.$DOMAIN/privkey.pem"
sudo postfix reload
Change the Dovecot to use the cert as well.
sudo sed -i "s/^ssl_server_cert_file = .*/ssl_server_cert_file = \/etc\/letsencrypt\/live\/mail.$DOMAIN\/fullchain.pem/" /etc/dovecot/conf.d/10-ssl.conf
sudo sed -i "s/^ssl_server_key_file = .*/ssl_server_key_file = \/etc\/letsencrypt\/live\/mail.$DOMAIN\/privkey.pem/" /etc/dovecot/conf.d/10-ssl.conf
sudo dovecot reload
You can view the certificates with the commands:
openssl s_client -connect mail.$DOMAIN:143 -starttls imap -servername mail.$DOMAIN
openssl s_client -starttls smtp -showcerts -connect mail.$DOMAIN:587 -servername mail.$DOMAIN
You can take advantage of Cloudflare (or other) services to accept and inspect your email before forwarding it on to you. As far as the Internet is concerned, Cloudflare is your email server. The rest is private.
Take a look at the Forwarding section, and simply forward your mail to your own server instead of Google’s. That will even allow you to remove your mail server from DNS and drop connections other than CloudFlare if desired.
In my testing it takes less than an hour before someone discovers and attempts to break into your mail server. You may wish to GeoIP block or otherwise limit connections. You can also use crowdsec.
Crowdsec is an open-source IPS that monitors your log files and blocks suspicious behavior.
Install as per their instructions.
curl -s https://packagecloud.io/install/repositories/crowdsec/crowdsec/script.deb.sh | sudo bash
sudo apt install -y crowdsec
sudo apt install crowdsec-firewall-bouncer-nftables
sudo cscli collections install crowdsecurity/postfix
Most services now log to the system journal rather than a file. You can view them with the journalctl command
# What is the exact service unit name?
sudo systemctl status | grep postfix
# Anything having to do with that service unit
sudo journalctl --unit [email protected]
# Zooming into just the identifiers smtp and smtpd
sudo journalctl --unit [email protected] -t postfix/smtp -t postfix/smtpd
Crowdsec accesses the system journal by adding a block to it’s log acquisition directives.
sudo tee -a /etc/crowdsec/acquis.yaml << EOF
source: journalctl
journalctl_filter:
- "[email protected]"
labels:
type: syslog
---
EOF
sudo systemctl reload crowdsec
Install the dovecot collection as well.
sudo cscli collections install crowdsecurity/dovecot
sudo tee -a /etc/crowdsec/acquis.yaml << EOF
source: journalctl
journalctl_filter:
- "_SYSTEMD_UNIT=dovecot.service"
labels:
type: syslog
---
EOF
sudo systemctl reload crowdsec
Is it working? You won’t see anything at first unless you’re actively under attack. But after 24 hours you may see some examples of attempts to relay spam.
allen@mail:~$ sudo cscli alerts list
╭────┬────────────────────┬────────────────────────────┬─────────┬──────────────────────────────────────────────┬───────────┬─────────────────────────────────────────╮
│ ID │ value │ reason │ country │ as │ decisions │ created_at │
├────┼────────────────────┼────────────────────────────┼─────────┼──────────────────────────────────────────────┼───────────┼─────────────────────────────────────────┤
│ 60 │ Ip:187.188.233.58 │ crowdsecurity/postfix-spam │ MX │ 17072 TOTAL PLAY TELECOMUNICACIONES SA DE CV │ ban:1 │ 2023-05-24 06:33:10.568681233 +0000 UTC │
│ 54 │ Ip:177.229.147.166 │ crowdsecurity/postfix-spam │ MX │ 13999 Mega Cable, S.A. de C.V. │ ban:1 │ 2023-05-23 20:17:49.912754687 +0000 UTC │
│ 53 │ Ip:177.229.154.70 │ crowdsecurity/postfix-spam │ MX │ 13999 Mega Cable, S.A. de C.V. │ ban:1 │ 2023-05-23 20:15:27.964240044 +0000 UTC │
│ 42 │ Ip:43.156.25.237 │ crowdsecurity/postfix-spam │ SG │ 132203 Tencent Building, Kejizhongyi Avenue │ ban:1 │ 2023-05-23 01:15:43.87577867 +0000 UTC │
│ 12 │ Ip:167.248.133.186 │ crowdsecurity/postfix-spam │ US │ 398722 CENSYS-ARIN-03 │ ban:1 │ 2023-05-20 16:03:15.418409847 +0000 UTC │
╰────┴────────────────────┴────────────────────────────┴─────────┴──────────────────────────────────────────────┴───────────┴─────────────────────────────────────────╯
If you’d like to get into the details, take a look at the Crowdsec page .
Now that your server is secure, let’s take a look at how email is authenticated and how to ensure yours is.
Email authentication prevents forgery. People can still send unsolicited email, but they can’t fake who it’s from. If you set up a Relay for Postfix, the relayer is doing it for you. But otherwise, proceed onward to prevent your outgoing mail being flagged as spam.
You need three things
SPF, or Sender Policy Framework, is the oldest component. It’s a DNS TXT record that lists the servers authorized to send email for a domain.
A receiving server looks at a messages’s return path (aka RFC5321.MailFrom header) to see what domain the email purports to be from. It then looks up that domain’s SPF record and if the server that sent the email isn’t included, the email is considered forged.
Note - this doesn’t check the From: header the user sees. Messages can appear (to the user) to be from anywhere. So it’s is mostly a low-level check to prevent spambots.
The DNS record for your Postfix server should look like:
Type: "TXT"
NAME: "@"
Value: "v=spf1 a:mail.your.org -all"
The value above shows the list of authorized servers (a:) contains mail.your.org. Mail from all other servers is considered forged (-all).
To have your Postfix server check SPF for incoming messages add the SPF policy agent.
sudo apt install postfix-policyd-spf-python
sudo tee -a /etc/postfix/master.cf << EOF
policyd-spf unix - n n - 0 spawn
user=policyd-spf argv=/usr/bin/policyd-spf
EOF
sudo tee -a /etc/postfix/main.cf << EOF
policyd-spf_time_limit = 3600
smtpd_recipient_restrictions =
permit_mynetworks,
permit_sasl_authenticated,
reject_unauth_destination,
check_policy_service unix:private/policyd-spf
EOF
sudo systemctl restart postfix
DKIM, or DomainKeys Identified Mail, signs the emails as they are sent ensuring that the email body and From: header (the one you see in your client) hasn’t been changed in transit and is vouched for by the signer.
Receiving servers see the DKIM header that includes who signed it, then use DNS to check it. Unsigned mail simply isn’t checked. (There is no could-but-didn’t in the standard).
Note - There is no connection between the domain that signs the message and what the user sees in the From: header. Messages can have a valid DKIM signature and still appear to be from anywhere. DKIM is mostly to prevent man-in-the-middle attacks from altering the message.
For Postfix, this requires installation of OpenDKIM and a connection as detailed here. Make sure to sign with the domain root.
https://tecadmin.net/setup-dkim-with-postfix-on-ubuntu-debian/
Once you’ve done that, create the following DNS entry.
Type: "TXT"
NAME: "default._domainkey"
Value: "v=DKIM1; h=sha256; k=rsa; p=MIIBIjANBgkq..."
Having a DMARC record is the final piece that instructs servers to check the From: header the user sees against the domain return path from the SPF and DKIM checks, and what to do on a fail.
This means mail “From: [email protected]” sent though mail.your.org mail servers will be flagged as spam.
The DNS record should look like:
Type: "TXT"
NAME: "_dmarc"
Value: "v=DMARC1; p=reject; adkim=s; aspf=r;"
Reject (p=reject) indicates that email servers should “reject” emails that fail DKIM or SPF tests, and skip quarantine.
Strict DKIM alignment (=s) means that the SPF Return-Path domain or the DKIM signing domain must be an exact match with the domain in the From: address. A DKIM signature from your.org would exactly match [email protected].
Relaxed SPF alignment (=r) means subdomains of the From: address are acceptable. I.e. the server mail.your.org from the SPF test aligns with an email from: [email protected].
You can also choose quarantine mode (p=quarantine) or report-only mode (p=none) where the email will be accepted and handled as such by the receiving server, and a report sent to you like below.
v=DMARC1; p=none; rua=mailto:[email protected]
DMARC is an or test. In the first example, if either the SPF or DKIM domains pass, then DMARC passes. You can choose to test one, both or none at all (meaning nothing can pass DMARC) as the the second DMARC example.
To implement DMARC checking in Postfix, you can install OpenDMARC and configure a mail filter as described below.
https://www.linuxbabe.com/mail-server/opendmarc-postfix-ubuntu
Now that you are hadnling email securely and authentically, let’s help ease client connections
You’ll want this if you have a lot of clients. There is an RFC but most fail to implement it. The conventional solution is to serve up the config from a web site. It’s best to create a dedicated vhost as each client has a slightly different approach.
In your DNS or provider, create two entries; one for Thunderbird and one for Outlook. These should be CNAME records pointing to your web server.
On your web server, create a website to handle these. If you’re using a wildcard and handle setup (and you should be), a config block would look like
@mail-discovery host autoconfig.your.org autodiscover.your.org
handle @mail-discovery {
encode
root * /var/www/autoconfig.your.org
file_server
}
Create the directory and file using this template. This is set up with the current standard of implicit SSL.
mkdir -p /var/www/autoconfig.your.org/mail
vi /var/www/autoconfig.your.org/mail/config-v1.1.xml
<?xml version="1.0" encoding="UTF-8"?>
<clientConfig version="1.1">
<emailProvider id="your.org">
<domain>your.org</domain>
<displayName>Your Mail</displayName>
<displayShortName>Your</displayShortName>
<!-- Incoming Mail Server (IMAP) -->
<incomingServer type="imap">
<hostname>mail.your.org</hostname>
<port>993</port>
<socketType>SSL</socketType>
<authentication>password-cleartext</authentication>
<username>%EMAILADDRESS%</username>
</incomingServer>
<!-- Outgoing Mail Server (SMTP) -->
<outgoingServer type="smtp">
<hostname>mail.your.org</hostname>
<port>465</port>
<socketType>SSL</socketType>
<authentication>password-cleartext</authentication>
<username>%EMAILADDRESS%</username>
</outgoingServer>
</emailProvider>
</clientConfig>
mkdir -p /var/www/autoconfig.your.org/autodiscover
vi /var/www/autoconfig.your.org/autodiscover/autodiscover.xml
<?xml version="1.0" encoding="utf-8"?>
<Autodiscover xmlns="http://microsoft.com">
<Response xmlns="http://microsoft.com">
<Account>
<AccountType>email</AccountType>
<Action>settings</Action>
<Protocol>
<Type>IMAP</Type>
<Server>mail.your.org</Server>
<Port>993</Port>
<SSL>on</SSL>
<AuthRequired>on</AuthRequired>
<SPA>off</SPA>
<LoginName>%EmailAddress%</LoginName>
</Protocol>
<Protocol>
<Type>SMTP</Type>
<Server>mail.your.org</Server>
<Port>465</Port>
<SSL>on</SSL>
<AuthRequired>on</AuthRequired>
<SPA>off</SPA>
<LoginName>%EmailAddress%</LoginName>
</Protocol>
</Account>
</Response>
</Autodiscover>
The mail app won’t configure itself directly, but you can visit the URL in Safari and it should prompt you. I’ve not tested this so take with a grain of salt. This would be http://autodiscover.your.org/apple/gattis.mobileconfig.
Update your Caddy config block with the content type. (not tested to see if actually needed)
@mail-discovery host autoconfig.your.org autodiscover.your.org
handle @mail-discovery {
header *.xml Content-Type "text/xml; charset=utf-8"
header *.mobileconfig Content-Type "application/x-apple-aspen-config; charset=utf-8"
encode
root * /var/www/autoconfig.your.org
file_server
}
```bash
mkdir -p /var/www/autoconfig.your.org/apple
vi /var/www/autoconfig.your.org/apple/your.mobileconfig
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://apple.com">
<plist version="1.0">
<dict>
<key>PayloadContent</key>
<array>
<dict>
<key>PayloadDescription</key>
<string>Configures mail settings for your.org</string>
<key>PayloadDisplayName</key>
<string>Your Mail Account</string>
<key>PayloadIdentifier</key>
<string>org.your.mail</string>
<key>PayloadType</key>
<string>com.apple.mail.managed</string>
<key>PayloadUUID</key>
<string>6A2B84D2-E123-4567-89AB-CDEF01234567</string>
<key>PayloadVersion</key>
<integer>1</integer>
<key>EmailAccountType</key>
<string>EmailTypeIMAP</string>
<key>EmailAccountDescription</key>
<string>Your Mail</string>
<key>IncomingMailServerHostName</key>
<string>mail.your.org</string>
<key>IncomingMailServerPortNumber</key>
<integer>993</integer>
<key>IncomingMailServerUseSSL</key>
<true/>
<key>IncomingMailServerUsername</key>
<string></string> <!-- Left blank so user inputs their specific prefix -->
<key>OutgoingMailServerHostName</key>
<string>mail.your.org</string>
<key>OutgoingMailServerPortNumber</key>
<integer>465</integer>
<key>OutgoingMailServerUseSSL</key>
<true/>
<key>OutgoingMailServerUsername</key>
<string></string>
</dict>
</array>
<key>PayloadDisplayName</key>
<string>Your Email Configuration</string>
<key>PayloadIdentifier</key>
<string>org.your.profile</string>
<key>PayloadRemovalDisallowed</key>
<false/>
<key>PayloadType</key>
<string>Configuration</string>
<key>PayloadUUID</key>
<string>1F4E92A1-ABCD-EF01-2345-67890ABCDEF1</string>
<key>PayloadVersion</key>
<integer>1</integer>
</dict>
</plist>
And of course, the best thing about standards are that there are so many to pick from. So I suppose we can create entries as per the RFC.
# Secure IMAP (Implicit TLS - Port 993)
_imaps._tcp.gattis.org. SRV 10 10 993 mail.your.org.
# Secure SMTP Submission (STARTTLS - Port 587)
_submission._tcp.gattis.org. SRV 10 10 587 mail.your.org.
It’s traditional to match server names to protocols and we would have used “imap.your.org” and “smtp.your.org”. But using ‘mail’ is popular now and it simplifies setup at several levels.
Thunderbird will try to guess at your server names, attempting to connect to smtp.your.org for example. But many Postfix configurations have spam prevention that interfere.
https://cweiske.de/tagebuch/claws-mail-autoconfig.htm
https://www.hardill.me.uk/wordpress/2021/01/24/email-autoconfiguration/
I’ve taken a look at a few of the clients you’d run on a server. The notable ones are:
RTorrent is a contender for the best, light-weight, torrent client. It’s terminal based, though there are web interfaces for it. In my testing it was second-best in terms of resource use.
The developer has recently picked it up again and is pulling in community contributions. My recent (Jan 2026) builds of it had segfault issues, but those may be cleaned up by now.
You can build and install on the same box, but if you’re running a seed box you may prefer to package it for deployment, so I’ve included both so you can:
or
RTorrent depends on LibTorrent, also from this developer.
# on Debian 13 (Trixie)
sudo apt install git
git clone https://github.com/rakshasa/libtorrent
cd libtorrent
sudo apt install build-essential autoconf libcurlpp-dev libtool
autoreconf -ivf
./configure
make
sudo make install
# Assuming you've already installed and built libtorrent on this box. Adjust branch after checking
git clone https://github.com/rakshasa/rtorrent --branch v0.16.6 --depth 1 rtorrent-0.16.6
sudo apt install libncurses-dev
autoreconf -ivf
./configure
make
sudo make install
Packaging things into a .deb is a bit of an adventure. But give it a try. Once you get it going you can probably use some sort of system to keep it up do date. Maybe do a repo and go full-debian on.
Assuming you’ve built and copied the packages:
sudo apt install ./libtorrent36_0.16.6-1_amd64.deb ./rtorrent_0.16.6-1_amd64.deb
# Connect to the console with
sudo -u rtorrent tmux a
# Disconnect from tmux to leave it running with
Ctrl-b d
You probably want to link it to other tools, like sonarr. They don’t talk SCGI, so you’ll need a proxy. Lightty is an easy way to do that.
sudo apt install lighttpd
sudo vi /etc/lighttpd/conf-available/10-rtorrent.conf
server.modules += ( "mod_scgi" )
scgi.server = (
"/RPC2" =>
( "127.0.0.1" =>
(
"socket" => "/home/user/rtorrent/rpc.socket",
"check-local" => "disable",
"disable-time" => 0, # don't disable SCGI if connection fails
)
# YOU MUST ADD SOME KIND OF AUTH
)
)
or possibly
"host" => "127.0.0.1",
"port" => 8080,
sudo lighty-enable-mod rtorrent
sudo systemctl restart lighttpd
You can test with a curl or xmlrpc (though you may need to add the latter)
curl -H "Content-Type: text/xml" --data "<?xml version='1.0'?><methodCall><methodName>system.listMethods</methodName></methodCall>" http://127.0.0.1/RPC2
xmlrpc tpub/RPC2 system.listMethods
Transmission is a very clean and easy to use client. It runs efficiently and includes a web interface. The only downside is under heavy load it suffers performance issues. Say over a hundred torrents and a thousand connections.
But for casual use, it’s my preferred client.
sudo apt install transmission-daemon
One note about installing transmission; make sure to stop the service before editing the config file. It will replace your settings with it’s running settings when it exits, otherwise.
sudo apt install transmission-daemon
sudo service transmission-daemon stop
sudo vim /etc/transmission-daemon/settings.json
"dht-enabled": false,
"rpc-whitelist": "127.0.0.1,192.168.*.*",
"peer-limit-global": 960,
"peer-limit-per-torrent": 288,
"preallocation": 2,
"rename-partial-files": false,
"watch-dir": "xxxxx",
"watch-dir-enabled": true
# Legacy setting not needed anymore
"max-peers-global": 960,
Most systems will require adjustment of the iptables firewall, should you be using such, for both the web admin port and the torrent traffic port. If you’re using something other than the standard port, adjust to suit. If you’re using random ports, you may want to open a range.
# The web admin port is tcp 9091 and peer traffic is 51413 tcp/udp by default
sudo iptables -A INPUT -p tcp -s 192.168.1.1/24 --dport 9091 -m conntrack --ctstate NEW,ESTABLISHED -j ACCEPT
sudo iptables -A INPUT -p tcp --dport 51413 -m conntrack --ctstate NEW,ESTABLISHED -j ACCEPT
sudo iptables -A INPUT -p udp --dport 51413 -m conntrack --ctstate NEW,ESTABLISHED -j ACCEPT
Should you have a router and NAT tranmission will use upnp to forward the appropriate ports. You can also check out upnpc for manipulating that device manually or via script.
If you’re using a VPN you may want to restrict traffic to that connection. You can configure transmission for that. Disable the transmission server so that you can start it as part of the VPN service.
sudo systemctl disable --now transmission-daemon
Here’s an example using PIA. Note: This may be somewhat dated at this point.
# These are not your normal credentials. Even though you may be using wireguard, set them up under the openvpn service.
# Get these and the client ID from their website support section
USER=xxxx
PASS=xxxxx
CLIENT_ID=xxxxx
IP=$(ip -o -4 addr list tun0 | awk '{print $4}')
PORT=$(curl -d "user=$USER&pass=$PASS&client_id=$CLIENT_ID&local_ip=$IP" https://www.privateinternetaccess.com/vpninfo/port_forward_assignment | grep -o '[0-9]\+')
echo $IP $PORT
# Just in case it's already running
sudo service transmission-daemon stop
sudo sed -i "s/bind-address-ipv4.*/bind-address-ipv4\": \"$IP\",/" /etc/transmission-daemon/settings.json
sudo sed -i "s/peer-port\".*/peer-port\": $PORT, /" /etc/transmission-daemon/settings.json
sudo iptables -A INPUT -i tun0 -p tcp -m tcp --dport $PORT -m conntrack --ctstate NEW,ESTABLISHED -j ACCEPT
sudo iptables -A INPUT -i tun0 -p udp -m udp --dport $PORT -m conntrack --ctstate NEW,ESTABLISHED -j ACCEPT
sudo service transmission-daemon start
Bitmagnet is a tool that crawls the Distributed Hash Table (DHT) and attempts in index all known torrents. It’s normally distributed via docker, but you can also deploy without it by installing:
You may want to use a VPN so you don’t get flagged. Even though you’re not downloading files with this tool, it’s a bittorrent tool and that could get you swept up in something.
# Insall postgres
sudo apt install -y postgresql postgresql-contrib
# Create the database
createdb bitmagnet
# Run the psql command as the system user 'postgres' who has superuser rights on the DB
# and set the postgres user password
sudo -i -u postgres psql
ALTER USER postgres WITH PASSWORD 'SOMETHINGRANDOM';
exit
Note - you may need to add a line to the postgres client auth config file. If you can’t connect with psql --host localhost --user postgres --password you may need to add a line like local all all scram-sha-256 to the /etc/postgresql/17/main/pg_hba.conf file.
You can install with go as the setup page suggests but they also make a .deb you can more easily install.
wget https://github.com/bitmagnet-io/bitmagnet/releases/download/v0.10.0/bitmagnet_0.10.0_linux_amd64.deb
sudo apt install ./bitmagnet_0.10.0_linux_amd64.deb
Then you can create a service file like this.
sudo vi /etc/systemd/system/bitmagnet-web.service
[Unit]
Description=bitmagnet Web GUI
After=network-online.target
After=pia-vpn.service
Requires=pia-vpn.service
[Service]
Type=simple
User=root
WorkingDirectory=/opt/bitmagnet
ExecStart=/usr/bin/bitmagnet worker run --all
Environment=POSTGRES_HOST=localhost
Environment=POSTGRES_PASSWORD=postgres
Environment=TMDB_API_KEY=theTmdbKey
Restart=on-failure
[Install]
WantedBy=multi-user.target
sudo systemctl daemon-reload
sudo systemctl enable --now bitmagnet-web.service
To ensure you bind to the VPN address, you may consider a complicated ExexStart as below.
[Unit]
Description=bitmagnet Web GUI
After=network-online.target
# Use whatever service your VPN runs as, though this is a bit unreliable
After=pia-vpn.service
Requires=pia-vpn.service
[Service]
Type=simple
User=root
Environment=POSTGRES_HOST=localhost
Environment=POSTGRES_PASSWORD=postgres
Environment=TMDB_API_KEY=theTmdbKey
# Instead of the normal start, wait 30 sec and bail if no VPN
# We're looking for "dev pia" so adjust as needed
#ExecStart=/usr/bin/bitmagnet worker run --all
ExecStart=/bin/sh -c '\
SECONDS=0; \
while [ $SECONDS -lt 30 ]; do \
VPN_IP=$(ip -4 addr show dev pia | awk '\''/inet / { split($2,a,"/"); print a[1] }'\''); \
[ -n "$VPN_IP" ] && break; \
sleep 1; \
done; \
if [ -z "$VPN_IP" ]; then \
echo "Could not get VPN IP on pia interface" >&2; exit 1; \
fi; \
echo "Starting client bound to IP $VPN_IP"; \
export BITMAGNET_DHT_BINDADDRESS=${VPN_IP}:6881 ; \
exec /usr/bin/bitmagnet worker run --all \
'
Restart=on-failure
[Install]
WantedBy=multi-user.target
Audioserve is a web audiobook server with a minimalist design. It focuses on delivering audio based on the directory structure you’ve created, eschewing tags embedded in the audio files. Anyone who has worked with internet-sourced audio content will know wisdom of this approach.
Create an audioserve user and obtain the latest static binary and web client
sudo adduser --system --home /opt/audioserve --disabled-password --disabled-login audioserve
# Download the latest binary to a version folder and create a link for it.
cd /opt/audioserve/
wget -q -O - https://github.com/izderadicka/audioserve/releases/latest/download/audioserve_static.tar.gz | sudo tar xvz
sudo ln -s audioserve_static*/* audioserve
# Download the web-client.
# You'll have to check on the web at https://github.com/izderadicka/audioserve-web/releases/latest/ to see what the current relase is.
# and updat the VER variable accordingly (and check the sketchy nature of using this naming convention in a script)
VER=0.3.3
sudo mkdir /opt/audioserve/audioserve-web-release_v${VER}
cd /opt/audioserve/audioserve-web-release_v${VER}
wget -q -O - https://github.com/izderadicka/audioserve-web/releases/download/v${VER}/audioserve-web-release_v${VER}.tgz | sudo tar xvz
sudo ln -s /opt/audioserve/audioserve-web-release_v${VER} /opt/audioserve/web
Get a ffmpeg binary. Audioserve needs ffmpeg for transcoding media that clients cant directly play. The apt package is quite large, but you can download the static binary at a fraction of the size.
wget -q -O - https://johnvansickle.com/ffmpeg/builds/ffmpeg-git-amd64-static.tar.xz | \
sudo tar -C /usr/local/bin -xvJ --strip-components 1 --wildcards "*ffmpeg" "*ffprobe"
Create a service script. This has the parameter no-authentication as we’ll be relying on a web server proxy to do that for us.
sudo vi /etc/systemd/system/audioserve.service
[Unit]
Description=Audioserve
[Service]
ExecStart=/opt/audioserve/audioserve --config /etc/audioserve.conf --no-authentication
User=audioserve
[Install]
WantedBy=multi-user.target
sudo systemctl daemon-reload
sudo systemctl enable audioserve.service
sudo vi /etc/audioserve.conf
behind_proxy: true
client_dir: /opt/audioserve/web
base_dirs:
- /some/place/where/your/audio/is
- /some/place/where/other/audio/is
Start the service and check it with:
sudo systemctl start audioserve.service
sudo systemctl status audioserve.service
sudo journalctl -u audioserve.service
You can access the web client on port 3000. It may display a banner about the lack of SSL but that is safe to ignore as you test it out. Having more than one base_dir in the config will populate a drop-down at the top. This is convenient if you have separated your library into genres, like Fantasy, Fiction, etc.
To display metadata, simply put image and text file in the folders alongside the audio files. In each folder, the first available image (jpeg or png) is taken as a cover/author picture and first text file (html, txt, md) is taken as description.
A good choice is Caddy, and a config file with just these lines in it will work just fine - as long as you actually own the some.web.server domain. This will pop up a login box ala web basic auth and serve it up at the root of that DNS name.
caddy hash-password
sudo vi /etc/caddy/Caddyfile
(auth) {
basicauth {
you ABIGLONGHASHFromAbove
}
}
some.web.server {
import auth
reverse_proxy * http://localhost:3000
}
sudo caddy reload
After you have things working the way you want, you can disable direct access to audioserve by adding this to the config
listen: "127.0.0.1:3000"
Note: Securing the web client this way is more secure, but it prevents the android client from working. Workarounds with wireguard and such can be done but take additional setup.
# Launch it in the foreground to see the errors up close
/opt/audioserve/audioserve --no-authentication --config /etc/audioserve.conf
# Print out all the options
/opt/audioserve/audioserve --help
# Figure out what should be in a YAML file by specifiying all the options on the command line and then printing the config. Can be tricky
/opt/audioserve/audioserve --config /etc/audioserve.conf --no-authentication --print-config
If you decide not to proxy with a web server, you’ll want to:
--no-authentication parameter from the service unit fileshared_secret: somePassword and remove behind_proxy from the conf fileThere are a handful of solutions for web eBook delivery but Calibre-Web looks and works the best. It also integrates directly with the Calibre eBook management app.
This is a python app and that pip for installation. Pip is a bit heavy at 275M but even if you wanted to install Calibre-Web manually you still need pip, so let’s go for it. The successive packages in the brackets are optional, but may be of interest to you.
NOTE: for debian 12 and higher you must set up python venv as pip is unhappy mixing it’s repo with debian. You can’t just delete the /usr/lib/python3.11/EXTERNALLY-MANAGED as some of the required packages want write access.
sudo apt -y install python3-pip
sudo apt install python3-venv
# Optional override of IDs to match existing permissions
sudo groupadd --gid 1001 calibre-web
sudo adduser --uid 1001 --gid 1001 --disabled-password --home /opt/calibre-web calibre-web
sudo -i -u calibre-web
python3 -m venv .venv
source .venv/bin/activate
pip install calibreweb
pip install calibreweb[metadata]
pip install calibreweb[comics]
pip install calibreweb[comics]
pip install calibreweb[goodreads]
# You can change the shell to /usr/sbin/nologin now that we're done with pip
# sudo usermod -s /usr/sbin/nologin calibre-web
exit
sudo vi /etc/systemd/system/calibre-web.service
[Unit]
Description=Calibre-Web
[Service]
ExecStart=/opt/calibre-web/.venv/bin/cps
User=calibre-web
[Install]
WantedBy=multi-user.target
sudo systemctl daemon-reload
sudo systemctl enable --now calibre-web.service
All the configuration settings are in the app.db file. You can install new and then copy that file.
scp /opt/calibre-web/.calibre-web/app.db some-other-server:
# On the other server
sudo mv app.db /opt/calibre-web/.calibre-web/app.db
sudo chown calibre-web:calibre-web /opt/calibre-web/.calibre-web/app.db
You can login with the default creds at
http://192.168.1.18:8083/ admin admin123
It will prompt you for the location of your Calibre database. If it’s writeable, you’ll be able to upload books and make edits from web app, but it’s not strictly required.
Change the admin password by:
Admin User Icon (top right) –> Password –> Save
If you proxy Calibre-Web and authenticate before you allow access, you may whish to allow anonymous browsing and reserve logins for admin access. You can enable that by:
Admin Dial Icon (top right) –> Edit Basic Configuration –> Feature Configuration –> Enable Anonymous Browsing
You can then return the the Admin screen and you’ll see Guest listed as a user. Select that user to turn on the various features. I recommend turn on all except Allow Edit. Selecting the the ‘show..’ selections on the left is best to show all the features.
Jellyfin is the best full-featured personal media server that doesn’t rely on the cloud. It allows for GPU acceleration without a paid subscription. (Looking at you, Plex.)
In the Incus GUI, create a new Trixie instance and attach the GPU in the devices section. It will list the hardware available for selection.
Jellyfin can update and save metadata and artwork alongside the media files instead of in server itself. To get the permissions right, set user numeric IDs in advance by pre-creating the system account.
sudo apt install curl
sudo addgroup --gid 1001 jellyfin
sudo adduser --uid 1001 --gid 1001 --home /var/lib/jellyfin --comment "Jellyfin default user" --disabled-login --disabled-password jellyfin
curl https://repo.jellyfin.org/install-debuntu.sh | sudo bash
The full instructions are at:
https://jellyfin.org/docs/general/installation/linux
On linux, you can use VA-API or QSV, the latter being preferred1 for better performance. Since debian 13 only works with QSV, it’s an easy choice.
You need the intel-opencl-icd package and since it’s not in the Trixie repos right now, you’ll need to install it from Intel.
To quote the docs:
This package may not be available for newer distro since it currently relies on LLVM 14, which may not be in releases like Debian Trixie. If this is the case a release from the Intel compute-runtime repository may be used instead.
https://github.com/intel/compute-runtime/releases
The jellyfin installer will add the service account to the video group for you, so it should just work by selecting ‘Intel Quicksync (QSV)’ in the Playback -> Transcoding settings and ‘/dev/dri/renderD128’ under that.
Leave everything else at the defaults
On the host, install the GPU tools sudo apt install intel-gpu-tools and observe with sudo intel_gpu_top during a transcode.
Note: The Jellyfin docs may will lead you test VA-API too, without being clear you don’t need it anymore.
One of the best systems for a handling media is LibreELEC. It’s both a Kodi box and a server appliance that’s resistant to abuse. With the right hardware (like a ROCKPro64 or Waveshare) it also makes an excellent portable server for traveling.
Download an image from https://libreelec.tv/downloads/ and flash as directed. Enable SSH during the initial setup.
RAID is a useful feature but only BTRFS works directly. This is fine, but with a little extra work you can add MergerFS, a popular option for combining disks.
Create the RAID set on another PC. If your disks are of different sizes you can use the ‘single’ profile, but leave the metadata mirrored.
sudo mkfs.btrfs -f -L pool -d single -m raid1 /dev/sda /dev/sdb /dev/etc...
After attaching to LibreELEC, the array will be automatically mounted at /media/pool based on label pool you specified above.
This is a good option if you just want to combine disks and unlike most other RAID technologies, if you loose a disk the rest will keep going. Many people combine this with SnapRAID for off-line parity.
But it’s a bit more work.
You may want to manage the fan. The RockPro64 has a PWM fan header and LibreELEC loads the pwm_fan module.
The kodi process can use a significant amount of CPU even at rest. If you’re using this primarily as a file server you can disable kodi from starting automatically.
cp /usr/lib/systemd/system/kodi.service /storage/.config/system.d/kodi-alt.service
systemctl mask kodi
To start kodi, you can enter systemctl start kodi-alt
Plug in a cheap Fm4 style remote and it ‘just works’ with kodi. But if you want to customize some remote buttons, say to start kodi manually, you still can.
To share your media, simply copy the sample file, remove all the preconfigured shares (unless you want them), and add one for your storage pool. Then just enable Samba and reboot (so the file is picked up)
cp /storage/.config/samba.conf.sample /storage/.config/samba.conf
vi /storage/.config/samba.conf
[media]
path = /storage/pool
available = yes
browseable = yes
public = yes
writeable = yes
Config --> LibreELEC --> Services --> Enable Samba
Config --> LibreELEC --> Network --> Wireless Networks
Enable Active and Wireless Access Point and it just works!
This is a good way handle things like Jellyfin or Plex if you must. In the GUI, go to add-ons, search for the items below and install.
Then you must make sure the docker starts starts after the storage is up or the containers will see an empty folder instead of a mounted one.
vi /storage/.config/system.d/service.system.docker.service
[Unit]
...
...
After=network.target storage-pool.mount
If that fails, you can also tell docker to wait a bit
ExecStartPre=/usr/bin/sleep 120
You may be called upon to look at something remotely. Sadly, there’s no remote access to the GUI but you can use things like autossh to create a persistent remote tunnel, or wireguard to create a VPN connection. Wireguard is usually better.
You can also use this platform as a server. This seems counter-intuitive at first; to use a media player OS as a server. But in practice it is rock-solid. I have a mixed fleet of 10 or so devices and LibreELEC has better uptime stats than TrueNAS.
The device playing content on your TV is also the media server for the rest of the house. I wouldn’t advertise this as an enterprise solution, but I can’t dispute the results.
Common tools like rsync, as well as server software like Jellyfin are available. You can browse as descriped below, or use the search tool if you’re looking for something specific.
If you’re on ARM or want more frequent updates, you may want to add Docker and the LinuxServer.io repository.
Drill down and add Jellyfin, for example.
This allows you to setup and monitor a remote tunnel as the easiest wat to manage remote clients is to let them come to you. To accomplish this, we’ll set up a server, create client keys, test a reverse tunnel, and setup autossh.
This is simply a server somewhere that everyone can reach via SSH. Create a normal user account with a password and home directory, such as with adduser remote. We will be connecting from our clients for initial setup with this.
Use SSH to connect to the LibreELEC client, generate a ssh key pair and copy it to the remote server
ssh [email protected]
ssh-keygen -f ~/.ssh/id_rsa -q -P ""
# ssh-copy-id isn't available so you must use the rather harder command below
cat ~/.ssh/id_rsa.pub | ssh -t [email protected] "cat - >> ~/.ssh/authorized_keys"
ssh [email protected]
If all went well you can back out and then test logging in with no password. Make sure to do this and accept the key so th
SSH normally connects your terminal to a remote server. Think of this as a encrypted tunnel where your keystrokes are sent to the server and it’s responses are sent back to you. You can send more than your keystrokes, however. You can take any port on your system and send it as well In our case, we’ll take port 22 (where ssh just happens to be listening) and send it to the rendezvous server on port 2222. SSH will continue to accept local connections while also taking connections from the remote port we are tunneling in.
# On the client, issue this command to connect the (-R)remote port 2222 to localhost:22, i.e. the ssh server on the client
ssh -N -R 2222:localhost:22 -o ServerAliveInterval=240 -o ServerAliveCountMax=2 [email protected]
# Leave that running while you login to the rendezvois server and test if you can now ssh to the client by connecting to the forwarded port.
ssh [email protected]
ssh root@localhost -p 2222
# Now exit both and set up Autossh below
Autossh is a daemon that monitors ssh sessions to make sure they’re up and operational, restarting them as needed, and this is exactly what we need to make sure the ssh session from the client stays up. To run this as a service, a systemd service file is needed. For LibreELEC, these are in /storage/.config.
vi /storage/.config/system.d/autossh.service
[Unit]
Description=autossh
Wants=network-online.target
After=network-online.target
[Service]
Type=simple
User=root
EnvironmentFile=/storage/.config/autossh
ExecStart=/storage/.kodi/addons/virtual.system-tools/bin/autossh $SSH_OPTIONS
Restart=always
RestartSec=60
[Install]
WantedBy=multi-user.target
vi /storage/.config/autossh
AUTOSSH_POLL=60
AUTOSSH_FIRST_POLL=30
AUTOSSH_GATETIME=0
AUTOSSH_PORT=22034
SSH_OPTIONS="-N -R 2222:localhost:22 [email protected] -i /storage/.ssh/id_rsa"
systemctl enable autossh.service
systemctl start autossh.service
systemctl status autossh.service
At this point, the client has a SSH connection to your server on port 22, opened port 2222 the ssh server and forwarded that back to it’s own ssh server. You can now connect by:
ssh [email protected]
ssh root@localhost -p 2222
If not, check the logs for errors and try again.
journalctl -b 0 --no-pager | less
Now that you have the client connected, you can use your Rendezvous Server as a Jump Host to access things on the remote client such as it’s web interface and even the console via VNC. Your connection will generally take the form of:
ssh localport:libreelec:libreelec_port -J rendezvoisServer redevoisServer -p autosshPort
The actual command is hard to read as are going through the rendezvois server twice and connecting to localhost on the destination.
ssh -L 8080:localhost:32400 -J [email protected] root@localhost -p 2222
This works best in an Ubuntu container.
LibreELECT Notes
Installed but no sata hdd. Found this
RPi4 has zero support for PCIe devices so why is it “embarrasing” for LE to omit support for PCIe SATA things in our RPi4 image?
Feel free to send a pull-request to GitHub enabling the kernel config that’s needed.
https://forum.libreelec.tv/thread/27849-sata-controller-error/
Went though thier resouces beginners guid to git https://wiki.libreelec.tv/development/git-tutorial#forking-and-cloning building basics https://wiki.libreelec.tv/development/build-basics specific build commands https://wiki.libreelec.tv/development/build-commands/build-commands-le-12.0.x
and then failed because jammy wasn’t compatibile enough
Created a jammy container and restarted
https://ubuntu.com/server/docs/lxc-containers
sudo lxc-create –template download –name u1 ubuntu jammy amd64 sudo lxc-start –name u1 –daemon sudo lxc-attach u1
Used some of the notes from
https://www.artembutusov.com/libreelec-raid-support/
Did as fork, clone and a
git fetch –all
but couldnt get all the downloads as alsa.org site was down
On a side note, these are needed in the config.txt so that USB works
otg_mode=1,dtoverlay=dwc2,dr_mode=host
I tried a menuconfig and selected ..sata? and got
CONFIG_ATA=m < CONFIG_ATA_VERBOSE_ERROR=y < CONFIG_ATA_FORCE=y CONFIG_ATA_SFF=y CONFIG_ATA_BMDMA=y
Better compare the .config file again
Edited and commited a config.txt but it didn’t show up in the image. Possibly the wrong file or theres another way to realize that chagne
Enabled the SPI interface
https://raspberrypi.stackexchange.com/questions/48228/how-to-enable-spi-on-raspberry-pi-3 https://wiki.libreelec.tv/configuration/config_txt
sudo apt install lxc
# This didn't work for some reason
sudo lxc-create --template download --name u1 --dist ubuntu --release jammy --arch amd64
sudo lxc-create --template download --name u1
sudo lxc-start --name u1 --daemon
sudo lxc-attach u1
# Now inside, build
apt update
apt upgrade
apt-get install gcc make git wget
apt-get install bc patchutils bzip2 gawk gperf zip unzip lzop g++ default-jre u-boot-tools texinfo xfonts-utils xsltproc libncurses5-dev xz-utils
# login and fork so you can clone more easily. Some problem with the creds
cd
git clone https://github.com/agattis/LibreELEC.tv
cd LibreELEC.tv/
git fetch --all
git tag
git remote add upstream https://github.com/LibreELEC/LibreELEC.tv.git
git fetch --all
git checkout libreelec-12.0
git checkout -b CM4-AHCI-Add
PROJECT=RPi ARCH=aarch64 DEVICE=RPi4 tools/download-tool
ls
cat /etc/passwd
pwd
ls /home/
ls /home/ubuntu/
ls
cd ..
mv LibreELEC.tv/ /home/ubuntu/
cd /home/ubuntu/
ls -lah
chown -R ubuntu:ubuntu LibreELEC.tv/
ls -lah
cd LibreELEC.tv/
ls
ls -lah
cd
sudo -i -u ubuntu
ip a
cat /etc/resolv.conf
ip route
sudo -i -u ubuntu
apt install tmux
sudo -i -u ubuntu tmux a
# And back home you can write
ls -lah ls/u1/rootfs/home/ubuntu/LibreELEC.tv/target/
Add this to the /storage/bin and create a service unit.
vi /storage/.config/system.d/fancontrol.service
systemctl enable fancontrol
#!/bin/sh
# Summary
#
# Adjust fan speed by percentage when CPU/GPU is between user set
# Min and Max temperatures.
#
# Notes
#
# Temp can be gleaned from the sysfs termal_zone files and are in
# units millidegrees meaning a reading of 30000 is equal to 30.000 C
#
# Fan speed is read and controlled by he pwm_fan module and can be
# read and set from a sysfs file as well. The value can be set from 0 (off)
# to 255 (max). It defaults to 255 at start
## Set Points
# CPU Temp set points
MIN_TEMP=40 # Min desired CPU temp
MAX_TEMP=60 # Max desired CPU temp
# Fan Speeds set points
FAN_OFF=0 # Fan is off
FAN_MIN=38 # Some fans need a minimum of 15% to start from a dead stop.
FAN_MAX=255 # Max cycle for fan
# Frequency
CYCLE_FREQ=6 # How often should we check, in seconds
SHORT_CYCLE_PERCENT=20 # If we are shutting on or of more than this percent of the
# time, just run at min rather than shutting off
## Sensor and Control files
# CPU and GPU sysfs locations
CPU=/sys/class/thermal/thermal_zone0/temp
GPU=/sys/class/thermal/thermal_zone1/temp
# Fan Control files
FAN2=/sys/devices/platform/pwm-fan/hwmon/hwmon2/pwm1
FAN3=/sys/devices/platform/pwm-fan/hwmon/hwmon3/pwm1
## Logic
# The fan control file isn't available until the module loads and
# is unpredictable in path. Wait until it comes up
FAN=""
while [[ -z $FAN ]];do
[[ -f $FAN2 ]] && FAN=$FAN2
[[ -f $FAN3 ]] && FAN=$FAN3
[[ -z $FAN ]] && sleep 1
done
# The sensors are in millidegrees so adjust the user
# set points to the same units
MIN_TEMP=$(( $MIN_TEMP * 1000 ))
MAX_TEMP=$(( $MAX_TEMP * 1000 ))
# Short cycle detection requires us to track the number
# of on-off flips to cycles
CYCLES=0
FLIPS=0
while true; do
# Set TEMP to the highest GPU/CPU Temp
TEMP=""
read TEMP_CPU < $CPU
read TEMP_GPU < $GPU
[[ $TEMP_CPU -gt $TEMP_GPU ]] && TEMP=$TEMP_CPU || TEMP=$TEMP_GPU
# How many degress above or below our min threshold are we?
DEGREES=$(( $TEMP-$MIN_TEMP ))
# What percent of the range between min and max is that?
RANGE=$(( $MAX_TEMP-$MIN_TEMP ))
PERCENT=$(( (100*$DEGREES/$RANGE) ))
# What number between 0 and 255 is that percent?
FAN_SPEED=$(( (255*$PERCENT)/100 ))
# Override the calculated speed for some special cases
if [[ $FAN_SPEED -le $FAN_OFF ]]; then # Set anything 0 or less to 0
FAN_SPEED=$FAN_OFF
elif [[ $FAN_SPEED -lt $FAN_MIN ]]; then # Set anything below the min to min
FAN_SPEED=$FAN_MIN
elif [[ $FAN_SPEED -ge $FAN_MAX ]]; then # Set anything above the max to max
FAN_SPEED=$FAN_MAX
fi
# Did we just flip on or off?
read -r OLD_FAN_SPEED < $FAN
if ( ( [[ $OLD_FAN_SPEED -eq 0 ]] && [[ $FAN_SPEED -ne 0 ]] ) || \
( [[ $OLD_FAN_SPEED -ne 0 ]] && [[ $FAN_SPEED -eq 0 ]] ) ); then
FLIPS=$((FLIPS+1))
fi
# Every 10 cycles, check to see if we are short-cycling
CYCLES=$((CYCLES+1))
if [[ $CYCLES -ge 10 ]] && [[ ! $SHORT_CYCLING ]]; then
FLIP_PERCENT=$(( 100*$FLIPS/$CYCLES ))
if [[ $FLIP_PERCENT -gt $SHORT_CYCLE_PERCENT ]]; then
SHORT_CYCLING=1
echo "Short-cycling detected. Fan will run at min speed rather than shutting off."
else
CYCLES=0;FLIPS=0
fi
fi
# If we are short-cycling and would turn the fan off, just set to min
if [[ $SHORT_CYCLING ]] && [[ $FAN_SPEED -le $FAN_MIN ]]; then
FAN_SPEED=$FAN_MIN
fi
# Every so often, exit short cycle mode to see if conditions have changed
if [[ $SHORT_CYCLING ]] && [[ $CYCLES -gt 10000 ]]; then # Roughly half a day
echo "Exiting short-cycling"
SHORT_CYCLING=""
fi
# Write that to the fan speed control file
echo $FAN_SPEED > $FAN
# Log the stats everyone once in a while
# if [[ $LOG_CYCLES ]] && [[ $LOG_CYCLES -ge 10 ]]; then
# echo "Temp was $TEMP fan set to $FAN_SPEED"
# LOG_CYCLES=""
# else
# LOG_CYCLES=$(($LOG_CYCLES+1))
# fi
sleep $CYCLE_FREQ
done
# Also look at drive temps. The sysfs filesystem isn't useful for
# all drives on RockPro64 so use smartctl instead
#ls -1 /dev/sd? | xargs -n1 smartctl -A | egrep ^194 | awk '{print $4}'
The Hotspot capability built-in to LibreELEC works, but is rudimentary. It’s primary use case is to allow an operator to authenticate through hotel network portals and such upstream of the device. It’s designed to be as widely compatible as possible, with the trade off of limited performance, especially with multiple clients.
You can instead deploy hostapd. This is the most widely used linux authentication and wireless access point software.
What follows are my rough notes from cobbling things together. I may come back and do it again and clean things up.
Download hostapd and libnl to the folder /storage/.kodi/userdata/scripts. This location was recommended so it would get backed up.
wget http://mirror.archlinuxarm.org/aarch64/extra/hostapd-2.10-4-aarch64.pkg.tar.xz
wget http://mirror.archlinuxarm.org/aarch64/core/libnl-3.9.0-1-aarch64.pkg.tar.xz
# Extract and copy to the scripts folder
- hostapd
- libnl-3.so.200
- libnl-genl-3.so.200
export LD_LIBRARY_PATH=/storage/bin:$LD_LIBRARY_PATH
export PATH=/storage/bin:$PATH
# You'll also need dnsmasq to hand out IPs.
wget http://mirror.archlinuxarm.org/aarch64/extra/dnsmasq-2.90-1-aarch64.pkg.tar.xz
wget http://mirror.archlinuxarm.org/aarch64/core/libnetfilter_conntrack-1.0.9-2-aarch64.pkg.tar.xz
wget http://mirror.archlinuxarm.org/aarch64/extra/nftables-1:1.0.9-3-aarch64.pkg.tar.xz
cp /storage/test/usr/lib/libnftables.so.1 .
wget http://mirror.archlinuxarm.org/aarch64/core/libnfnetlink-1.0.2-2-aarch64.pkg.tar.xz
cp /storage/test/usr/lib/libnfnetlink.so.0 .
wget http://mirror.archlinuxarm.org/aarch64/core/libnftnl-1.2.6-1-aarch64.pkg.tar.xz
cp /storage/test/usr/lib/libnftnl.so.11 /storage/bin/
wget http://mirror.archlinuxarm.org/aarch64/core/jansson-2.14-4-aarch64.pkg.tar.xz
cp /storage/test/usr/lib/libjansson.so.4 /storage/bin/
wget http://mirror.archlinuxarm.org/aarch64/core/readline-8.2.010-1-aarch64.pkg.tar.xz
cp /storage/test/usr/lib/libreadline.so.8 /storage/bin/
wget http://mirror.archlinuxarm.org/aarch64/core/ncurses-6.5-3-aarch64.pkg.tar.xz
You’ll need a service for all this.
vi /storage/.config/system.d/router.service
[Unit]
Description = Router
After=sys-subsystem-net-devices-wlan0.device
Wants=sys-subsystem-net-devices-wlan0.device
[Service]
Environment=LD_LIBRARY_PATH=/storage/bin
Type=forking
WorkingDirectory=/storage/bin
ExecStart=/storage/bin/ap-up
ExecStop=killall hostapd
ExecStop=killall dnsmasq
[Install]
WantedBy=default.target
with a the script in bin of
#!/bin/bash
ip link set wlan0 down
ip addr flush dev wlan0
ip link set wlan0 up
ip addr add 10.0.0.1/24 dev wlan0
sleep 2
cd /storage/bin
pgrep dnsmasq || /storage/bin/dnsmasq --conf-file=/storage/bin/dnsmasq.conf --dhcp-leasefile=/storage/bin/dnsmasq.leases
iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE
iptables -A FORWARD -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
iptables -A FORWARD -i wlan0 -o eth0 -j ACCEPT
sysctl -w net.ipv4.ip_forward=1
/storage/bin/hostapd -B /storage/bin/hostapd.conf
You’ll need to change up a but from the normal way of deploying. The process outlined here prevents disks from auto-mounting. Another strategy might be to create to ensure the mergerfs unit delays.
Prepare and exempt the file systems from auto-mounting1 so you can supply your own mount options and make sure they are up before you start MergerFS.
# Format and label each disk the same
mkfs.ext4 /dev/sda
e2label /dev/sda pool-member
# Copy the udev rule for editing
cp /usr/lib/udev/rules.d/95-udevil-mount.rules /storage/.config/udev.rules.d
vi /storage/.config/udev.rules.d/95-udevil-mount.rules
Edit this section by adding the pool-member label from above
# check for special partitions we don't want to auto-mount
IMPORT{builtin}="blkid"
ENV{ID_FS_LABEL}=="EFI|BOOT|Recovery|RECOVERY|SETTINGS|boot|root0|share0|pool-member", GOTO="exit"
Test this by rebooting and making sure the drives are not mounted.
Each filesystem requires a mount unit like below. Create one for each drive named disk1, disk2, etc. Note: The name of the file is import and to mount /storage/disk1 the name of the file must be storage-disk1.mount
vi /storage/.config/system.d/storage-disk1.mount
[Unit]
Description=Mount sda
Requires=dev-sda.device
After=dev-sda.device
[Mount]
What=/dev/sda
Where=/storage/disk1
Type=ext4
Options=rw,noatime,nofail
[Install]
WantedBy=multi-user.target
systemctl enable --now storage-disk1.mount
MergerFS isn’t available as an add-on, but you can get it directly from the developer. LibreELEC (or CoreELEC) on ARM have a 32 bit[^2] user space so you’ll need the armhf version.
wget https://github.com/trapexit/mergerfs/releases/latest/download/mergerfs-static-linux_armhf.tar.gz
tar --extract --file=./mergerfs-static-linux_armhf.tar.gz --strip-components=3 usr/local/bin/mergerfs
mkdir bin
mv mergerfs bin/
Mount the drives and run a test like below. Notice the escaped *. That’s needed at the command line to prevent shell globbing.
mkdir /storage/pool
/storage/bin/mergerfs /storage/disk\* /storage/pool/
vi /storage/.config/system.d/mergerfs.service
[Unit]
Description = MergerFS Service
After=storage-disk1.mount storage-disk2.mount storage-disk3.mount storage-disk4.mount
Requires=storage-disk1.mount storage-disk2.mount storage-disk3.mount storage-disk4.mount
[Service]
Type=forking
ExecStart=/storage/bin/mergerfs -o category.create=mfs,noatime /storage/disk* /storage/pool/
ExecStop=umount /storage/pool
[Install]
WantedBy=default.target
systemctl enable --now mergerfs.service
Your content should now be available in /storage/pool after boot.
Most remotes just work. Newer ones emulate a keyboard and send well-known multimedia keys like ‘play’ and ‘volume up’. If you want to change what a button does, you can tell Kodi what to do pretty easily. In addition, LibreELEC also supports older remotes using eventlircd and popular ones are already configured. You can add unusual ones as well as get normal remotes to perform arbitrary actions when kodi isn’t running (like telling the computer to start kodi or shutdown cleanly).
If you plug in a remote receiver and the kernel makes reference to a keyboard you have a modern remote and Kodi will talk to it directly.
dmesg
input: BESCO KSL81P304 Keyboard as ...
hid-generic 0003:2571:4101.0001: input,hidraw0: USB HID v1.11 Keyboard ...
If you want to change a button action, put kodi into log mode, tail the logfile, and press the button in question to see what event is detected.
# Turn on debug
kodi-send -a toggledebug
# Tail the logfile
tail -f /storage/.kodi/temp/kodi.log
debug <general>: Keyboard: scancode: 0xac, sym: 0xac, unicode: 0x00, modifier: 0x0
debug <general>: HandleKey: browser_home (0xf0b6) pressed, window 10000, action is ActivateWindow(Home)
In this example, we pressed the ‘home’ button on the remote. That was detected as a keyboard press of the browser_home key. This is just one of many defined keys like ’email’ and ‘calculator’ that can be present on a keyboard. Kodi has a default action of that and you can see what it is in the system keymap
# View the system keyboard map to see what's happening by default
cat /usr/share/kodi/system/keymaps/keyboard.xml
To change what happens, create a user keymap. Any entries in it will override the default.
# Create a user keymap that takes you to 'Videos' instead of 'Home'
vi /storage/.kodi/userdata/keymaps/keyboard.xml
<keymap>
<global>
<keyboard>
<browser_home>ActivateWindow(Videos)</browser_home>
</keyboard>
</global>
</keymap>
kodi-send -a reloadkeymaps
Some receivers don’t send well-known keys. For these, there’s eventlircd. LibreELEC has a list of popular remotes that fall into this category and will dynamically use it as needed. For instance, pair an Amazon Fire TV remote and udev will fire, match a rule in /usr/lib/udev/rules.d/98-eventlircd.rules, and launch eventlircd with the buttons mapped in /etc/eventlircd.d/aftvsremote.evmap.
These will interface with Kodi using it’s “LIRC” (Linux Infrared Remote Contoll) interface. And just like with keyboards, there’s a set of well-known remote keys Kodi will accept. Some remotes don’t know about these so eventlircd does some pre-translation before relaying to Kodi. If you look in the aftvsremote.evmap file for example, you’ll see that KEY_HOMEPAGE = KEY_HOME.
To find out if your remote falls into this category, enable logging, tail the log, and if your remote has been picked up for handling by eventlircd you’ll see some entries like this.
debug <general>: LIRC: - NEW 66 0 KEY_HOME devinput (KEY_HOME)
debug <general>: HandleKey: percent (0x25) pressed, window 10000, action is PreviousMenu
In the first line, Kodi notes that it’s LIRC interface received a KEY_HOME button press. (Eventlircd actually translated it, but that happened before kodi saw anything.) In the second line, Kodi says it received the key ‘percent’, and preformed the action ‘Back’. The part where Kodi says ‘percent (0x25)’ was pressed seems resistent to documentation, but the action of PreviousMenu is the end result. The main question is why?
Turns out that Kodi has a pre-mapping file for events relayed to it from LIRC systems. There’s a mapping for ‘KEY_HOME’ that kodi translates to the well-known key ‘start’. Then Kodi checks the normal keymap file and ‘start’ translates to the Kodi action ‘Back’
Take a look at the system LIRC mapping file to see for yourself.
# The Lircmap file has the Kodi well-known button (start) surrounding the original remote command (KEY_HOME)
grep KEY_HOME /usr/share/kodi/system/Lircmap.xml
<start>KEY_HOME</start>
Then take a look at the normal mapping file to see how start get’s handled
# The keymap file has the well-known Kodi button surrounding the Kodi action,
grep start /usr/share/kodi/system/keymaps/remote.xml
<start>PreviousMenu</start>
You’ll actually see quite a few things are mapped to ‘start’ as it does different things depending on what part of Kodi you are accessing at the time.
You have a few options an they are listed here in increasing complexity. Specifically, you can
To change what the KEY_HOME button does you can create a user keymap like before and override it. It just needs a changed from keyboard to remote for entering through the LIRC interface. In this example we’ve set it to actually take you home via the kodi function ActivateWindow(Home).
vi /storage/.kodi/userdata/keymaps/remote.xml
<keymap>
<global>
<remote>
<start>ActivateWindow(Home)</start>
</remote>
</global>
</keymap>
This can occasionally cause problems though - such as when you have another button that already gets translated to start and you want it to keep working the same. In this case, you make an edit at the Lircmap level to translate KEY_HOME to some other button first, then map that button to the action you want. (You can’t put the Kodi function above in the Lircmap file so you have to do a double hop.)
First, let’s determine what the device name should be with the irw command.
irw
# Hit a button and the device name will be at the end
66 0 KEY_HOME devinput
Now let’s pick a key. My remote doesn’t have a ‘red’ key, so lets hijack that one. Note the device name devinput from the above.
vi /storage/.kodi/userdata/Lircmap.xml
<lircmap>
<remote device="devinput">
<red>KEY_HOME</red>
</remote>
</lircmap>
Then map the key restart kodi (the keymap reload command doesn’t handle Lircmap)
vi /storage/.kodi/userdata/keymaps/remote.xml
<keymap>
<global>
<remote>
<red>ActivateWindow(Home)</red>
</remote>
</global>
</keymap>
systemctl restart kodi
You can also change what evenlircd does. If LibreELEC wasn’t a read-only filesystem you’d have done this first. But you can do it with a but more work than the above if you prefer.
# Copy the evmap files
cp -r /etc/eventlircd.d /storage/.config/
# Override where the daemon looks for it's configs
systemctl edit --full eventlircd
# change the ExecStart line to refer to the new location - add vvv to the end for more log info
ExecStart=/usr/sbin/eventlircd -f --evmap=/storage/.config/eventlircd.d --socket=/run/lirc/lircd -vvv
# Restart, replug the device and grep the logs to see what evmap is in use
systemctl restart eventlircd
journalctl | grep evmap
# Edit that map to change how home is mapped (yours may not use the default map)
vi /storage/.config/eventlircd.d/default.evmap
KEY_HOMEPAGE = KEY_HOME
Sometimes, you’ll have a button that does nothing at all.
debug <general>: LIRC: - NEW ac 0 KEY_HOMEPAGE devinput (KEY_HOMEPAGE)
debug <general>: HandleKey: 0 (0x0, obc255) pressed, window 10016, action is
In this example Kodi received the KEY_HOMEPAGE button, consulted it’s Lircmap.xml and didn’t find anything. This is because eventlircd didn’t recognize the remote and translate it to KEY_HOME like before. That’s OK, we can just add a user LIRC mapping. If you look through the system file you’ll see things like ‘KEY_HOME’ are tto the ‘start’ button. So let’s do the same.
vi /storage/.kodi/userdata/Lircmap.xml
<lircmap>
<remote device="devinput">
<start>KEY_HOMEPAGE</start>
</remote>
</lircmap>
systemctl restart kodi
Check the log and you’ll see that you now get
debug <general>: LIRC: - NEW ac 0 KEY_HOMEPAGE devinput (KEY_HOMEPAGE)
debug <general>: HandleKey: 251 (0xfb, obc4) pressed, window 10025, action is ActivateWindow(Home)
You may want a remote to work outside of kodi too - say because you want to start kodi with a remote button. If you have a modern remote that eventlircd didn’t capture, you must first add your remote to the list of udev rules.
First you must identify the remote with lsusb. It’s probably the only non-hub device listed.
lsusb
...
...
Bus 006 Device 002: ID 2571:4101 BESCO KSL81P304
^ ^
Vendor ID -------------/ \--------- Model ID
...
Then, copy the udev rule file and add a custom rule for your remote.
cp /usr/lib/udev/rules.d/98-eventlircd.rules /storage/.config/udev.rules.d/
vi /storage/.config/udev.rules.d/98-eventlircd.rules
...
...
...
ENV{ID_USB_INTERFACES}=="", IMPORT{builtin}="usb_id"
# Add the rule under the above line so the USB IDs are available.
# change the numbers to match the ID from lsusb
ENV{ID_VENDOR_ID}=="2571", ENV{ID_MODEL_ID}=="4101", \
ENV{eventlircd_enable}="true", \
ENV{eventlircd_evmap}="default.evmap"
...
Now, reboot, turn on logging and see what the buttons show up as. You can also install the system tools add-on in kodi, and at the command line, stop kodi and the eventlircd service, then run evtest and press some buttons. You should see something like
Testing ... (interrupt to exit)
Event: time 1710468265.112925, type 4 (EV_MSC), code 4 (MSC_SCAN), value c0223
Event: time 1710468265.112925, type 1 (EV_KEY), code 172 (KEY_HOMEPAGE), value 1
Event: time 1710468265.112925, -------------- SYN_REPORT ------------
Event: time 1710468265.200987, type 4 (EV_MSC), code 4 (MSC_SCAN), value c0223
Event: time 1710468265.200987, type 1 (EV_KEY), code 172 (KEY_HOMEPAGE), value 0
Event: time 1710468265.200987, -------------- SYN_REPORT ------------
Now that you have seen the event, you must have the irexec process watching for it to take action. Luckily, LibreELEC already includes it.
vi /storage/.config/system.d/irexec.service
[Unit]
Description=IR Remote irexec config
After=eventlircd.service
Wants=eventlircd.service
[Service]
ExecStart=/usr/bin/irexec --daemon /storage/.lircrc
Type=forking
[Install]
WantedBy=multi-user.target
We’ll create a the config file next. The config is the command or script to run. systemctl start kodi in our case.
vi /storage/.lircrc
begin
prog = irexec
button = KEY_HOMEPAGE
config = systemctl start kodi
end
Let’s enable and start it up
systemctl enable --now irexec
Go ahead and stop kodi, then press the KEY_HOMEPAGE button on your remote. Try config entries like echo start kodi > /storage/test-results if you have issues and wonder if it’s running.
You may notice that eventlircd is always running, even if it has no remotes. That’s of a unit file is in /usr/lib/systemd/system/multi-user.target.wants/. I’m not sure of why this is the case when there is no remote in play.
Plex is the preeminent personal media server. While they’ve locked some features behind a subscription and are branching out into cloud providers, it remains a solid choice. They have clients for a lot of different Tablets and TVs, and importantly make connecting your family to your home videos easy.
Plex has jumped on the pipe-to-shell installation method and you can now run this to install.
curl -LsSf https://repo.plex.tv/scripts/setupRepo.sh | sudo bash
This sets up the repo and installation you’d have done manually in the past.
To allow unattended updates of plex, install the unattended upgrade package and a site to the Origins-Pattern section.
Note: The Plex installer may be doing this already at this point, but I’ve left this hear just in case.
sudo apt -y install unattended-upgrades
sudo vi /etc/apt/apt.conf.d/50unattended-upgrades
Unattended-Upgrade::Origins-Pattern {
...
...
"site=downloads.plex.tv";
};
Most private ranges are automatically granted access, but you may need to adjust.
sudo vim "/var/lib/plexmediaserver/Library/Application Support/Plex Media Server/Preferences.xml"
... allowedNetworks="192.168.1.0/24" ...
The metadata library can get very large. Many tens of gigabytes in some cases. So you may wish to relocate it for space or even security reasons.
sudo service stop plexmediaserver
sudo mv /var/lib/plexmediaserver /mnt/crypt
sudo ln -s /mnt/crypt/plexmediaserver /var/lib/plexmediaserver
sudo chown plex:plex /var/lib/plexmediaserver
If you’re using Ubuntu and have UFW running, adjust as below.
sudo vim /etc/ufw/applications.d/plexmediaserver
[plexmediaserver]
title=Plex Media Server
description=Plex Media Server is the back-end media server component of Plex
ports=32400/tcp
# Add custom rules as needed
sudo ufw allow from xxx.xxx.xx.0/23 to any app plexmediaserver
sudo ufw deny from xxx.xxx.0.0/16 to any app plexmediaserver
sudo ufw allow plexmediaserver
You can kick off scans via the command line, but you have to invoke the commands as the plex user.
#!/bin/bash
sudo su -c "export LD_LIBRARY_PATH=/usr/lib/plexmediaserver;/usr/lib/plexmediaserver/Plex\ Media\ Scanner --scan" plex
You can also just become the plex user, of course, if it’s not too awkward
sudo -i -u plex
/usr/lib/plexmediaserver/Plex\ Media\ Scanner --list --section 1
When you have a large collection, it becomes almost impossible to find where some items land at. You can however, search the database
sudo apt install sqllite3
sqlite3 "/var/lib/plexmediaserver/Library/Application Support/Plex Media Server/Plug-in Support/Databases/com.plexapp.plugins.library.db"
select media_item_id,file from media_parts where file like "%Kid Party E07E01.mkv%";
select metadata_item_id,hints from media_items where id=1011881;
select title from metadata_items where id=173491;
Or for a complex join
SELECT
metadata_items.title,media_parts.file
FROM
metadata_items
JOIN media_parts JOIN media_items
WHERE
metadata_items.library_section_id=1
AND
metadata_items.id=media_items.metadata_item_id
AND
media_items.id=media_parts.media_item_id
AND
media_parts.file LIKE "%Kid Party E07E01.mkv%";
tail -f /var/lib/plexmediaserver/Library/Application\ Support/Plex\ Media\ Server/Logs/Plex\ Media\ Server.log
Database info: https://forums.plex.tv/t/howto-query-data-from-plex-on-a-mac/11636
Anthias (AKA Screenly) is a simple, open-source digital signage system that runs well on a raspberry pi or Intel PC. When plugged into a monitor, it displays images, video or web sites in slideshow fashion. It’s managed directly though a web interface on the device and there are fleet and support options.
Use the Raspberry Pi Imager to create a 64 bit Raspberry Pi OS Lite image. Select the gear icon at the bottom right to enable SSH, create a user, configure networking, and set the locale. Use SSH continue configuration.
setterm --cursor on
sudo raspi-config nonint do_change_locale en_US-UTF-8
sudo raspi-config nonint do_configure_keyboard us
sudo raspi-config nonint do_wifi_country US
sudo timedatectl set-timezone America/New_York
sudo raspi-config nonint do_hostname SOMENAME
sudo apt update;sudo apt upgrade -y
sudo reboot
Boot from the Debian installer ISO and make the following choices.
After installing, add sudo and allow the user you created during install to use it without prompt. (as per Athnias docs)
# Become root
apt install sudo
adduser SOMEBODY sudo
echo "SOMEBODY ALL=(ALL) NOPASSWD:ALL" > /etc/sudoers.d/SOMEBODY
chmod 0440 /etc/sudoers.d/SOMEBODY
Use the newer iwd service for better handling of outages. It expects resolved for DNS.
sudo apt install iwd systemd-resolved
sudo systemctl enable --now iwd systemd-resolved
Set iwd to handle DHCP for WiFi. This is more reliable for outage recovery.
sed -i 's/^#\s*EnableNet.*/EnableNetworkConfiguration=true/' /etc/iwd/main.conf
Tell iwd which WiFi network to join. It will connect as soon as you save the config.
# Generate a preshared key from the SSID and Passphrase
python3 -c 'import hashlib; print(hashlib.pbkdf2_hmac("sha1", b"Your_Passphrase", b"Your_SSID", 4096, 32).hex())'
sudo vi /var/lib/iwd/Your_SSID.psk
# Add this section if it's a hidden SSID
[Settings]
Hidden=true
[Security]
PreSharedKey=paste_your_python_output_hash_here
Let’s make the PC more resilient to power loss and hangs. Using BTRFS helps a lot, but we can also configure the BIOS, update the kernel settings and use a watchdog timer.
Enter the BIOS and under power settings, configure to power-on on power recovery.
Configure the system to reboot instead of hang on a kernel panic.
sudo vi /etc/sysctl.d/99-panic-reboot.conf
kernel.panic = 10
kernel.panic_on_oops = 1
Change GRUB to handle failed boots better.
echo "GRUB_RECORDFAIL_TIMEOUT=2" | sudo tee -a /etc/default/grub
sudo update-grub
Your chipset may provide a watchdog timer that once engaged, will trigger a power-cycle if the operating system stops updating it.
ls -la /dev/watchdog*
If you have any devices listed, you can install and configure the Watchdog service.
sudo apt install watchdog
sudo systemctl stop watchdog.service
# Load the module early in the boot
echo "iTCO_wdt" | sudo tee /etc/modules-load.d/watchdog.conf
sudo vi /etc/watchdog.conf
Add these to the bottom
# Tell the daemon to talk to your Dell's hardware chip
watchdog-device = /dev/watchdog
# Send a heartbeat to the hardware chip every 10 seconds
interval = 10
# Force a hardware reset if the system remains frozen for 60 seconds
watchdog-timeout = 60
# Test that you can allocate memory
allocatable-memory = 1
sudo systemctl start watchdog.service
You normally want to apply security automatically, and control updates to the rest. If you don’t have a good structure you can of course apply all, as this config exemplifies.
sudo apt -y install unattended-upgrades
sudo tee /etc/apt/apt.conf.d/52unattended-upgrades-local > /dev/null <<'EOF'
Unattended-Upgrade::Origins-Pattern {
"origin=Debian";
};
Unattended-Upgrade::Remove-Unused-Kernel-Packages "true";
Unattended-Upgrade::Remove-Unused-Dependencies "true";
Unattended-Upgrade::Remove-New-Unused-Dependencies "true";
Unattended-Upgrade::Automatic-Reboot "true";
Unattended-Upgrade::Automatic-Reboot-Time "02:00";
Unattended-Upgrade::Allow-APT-Mark-Fallback "true";
EOF
sudo systemctl restart unattended-upgrades.service
sudo apt install curl
# When prompted, install the latest but don't allow Anthias to manage the network
bash <(curl -sL https://www.screenly.io/install-ose.sh)
You can add a regular update by dropping a cronjob for the Anthias update script.
sudo tee /etc/cron.d/anthias-update > /dev/null <<'EOF'
# Anthias weekly update
0 3 * * 0 root /home/SOMEBODY/screenly/bin/run_upgrade.sh
EOF
Navigate to the Web UI at the IP address of the device. You may wish to enter the settings and add authentication and change the device name.
You may add common graphic types, mp4, web and youtube links. It will let you know if it fails to download the youtube video. Some heavy web pages fail to render correctly, but most do.
Images must be sized to for the screen. In most cases this is 1080. Larger images are scaled down, but smaller images are not scaled up. For example, PowerPoint is often used to create slides, but it exports at 720. On a 1080 screen creates black boarders. You can change the resolution on the Pi with rasp-config or add a registry key to Windows to change PowerPoint’s output size.
Windows Registry Editor Version 5.00
[HKEY_CURRENT_USER\Software\Microsoft\Office\16.0\PowerPoint\Options]
"ExportBitmapResolution"=dword:00000096
You may want to turn off the display during non-operation hours. The vcgencmd command can turn off video output and some displays will choose to enter power-savings mode. Some displays misbehave or ignore the command, so testing is warranted.
Note: This is only for the Raspberry Pi. Most PCs lack the CEC interface needed without a dedicated add-on.
sudo tee /etc/cron.d/screenpower << EOF
# m h dom mon dow usercommand
# Turn monitor on
30 7 * * 1-5 root /usr/bin/vcgencmd display_power 1
# Turn monitor off
30 19 * * 1-5 root /usr/bin/vcgencmd display_power 0
# Weekly Reboot just in case
0 7 * * 1 root /sbin/shutdown -r +10 "Monday reboot in 10 minutes"
EOF
You may find you must download the video manually and then upload to Anthias. Use the utility yt-dlp to list and then download the mp4 version of a video
yt-dlp --list-formats https://www.youtube.com/watch?v=YE7VzlLtp-4
yt-dlp --format 22 https://www.youtube.com/watch?v=YE7VzlLtp-4
WiFi can go up and down, and some variants of the OS do not automatically reconnect. You way want to add the following script to keep connected. This is mostly for the Pi as iwd is fault tolerant.
sudo touch /usr/local/bin/checkwifi
sudo chmod +x /usr/local/bin/checkwifi
sudo vim.tiny /usr/local/bin/checkwifi
#!/bin/bash
# Exit if WiFi isn't configured
grep -q ssid /etc/wpa_supplicant/wpa_supplicant.conf || exit
# In the case of multiple gateways (when connected to wired and wireless)
# the `grep -m 1` will exit on the first match, presumably the lowest metric
GATEWAY=$(ip route list | grep -m 1 default | awk '{print $3}')
ping -c4 $GATEWAY > /dev/null
if [ $? != 0 ]
then
logger checkwifi fail `date`
service wpa_supplicant restart
service dhcpcd restart
else
logger checkwifi success `date`
fi
sudo tee /etc/cron.d/checkwifi << EOF
# Check WiFi connection
*/5 * * * * /usr/bin/sudo -H /usr/local/bin/checkwifi >> /dev/null 2>&1"
EOF
If you didn’t set up WiFi during imaging, you can use raspi-config after boot, but you must add a line if it’s a hidden network, and reboot.
sudo sed -i '/psk/a\ scan_ssid=1' /etc/wpa_supplicant/wpa_supplicant.conf
You may want to turn this off or adjust it.
# You can turn off the splash screen in the GUI or in the .conf
sed -i 's/show_splash =.*/show_splash = off/' /home/pi/.screenly/screenly.conf
# Or you can correct it in the docker file
vi ./screenly/docker-compose.yml
Anthias works best when the graphics are the correct size. It will attempt to display images that are too large, but this flashes a white screen and eventually hangs the box (at least in the current version). Not all users get the hang of sizing things correctly, so if you have issues, try this script.
#!/bin/bash
# If this device isn't running signage, exit
[ -d /home/pi/screenly_assets ] || { echo "No screenly image asset directory, exiting"; exit 1; }
# Check that mediainfo and imagemagick convert are available
command -v mediainfo || { echo "mediainfo command not available, exiting"; exit 1; }
command -v convert || { echo "imagemagick convert not available, exiting"; exit 1; }
cd /home/pi/screenly_assets
for FILE in *.png *.jpe *.gif
do
# if the file doesn't exist, skip this iteration
[ -f $FILE ] || continue
# Use mediainfo to get the dimensions at it's much faster than imagemagick
read -r NAME WIDTH HEIGHT <<<$(echo -n "$FILE ";mediainfo --Inform="Image;%Width% %Height%" $FILE)
# if it's too big, use imagemagick's convert. (the mogify command doesn't resize reliably)
if [ "$WIDTH" -gt "1920" ] || [ "$HEIGHT" -gt "1080" ]
then
echo $FILE $WIDTH x $HEIGHT
convert $FILE -resize 1920x1080 -gravity center $FILE
fi
done
If the display is off when you boot the pi, it may decide there is no monitor. When someone does turn on the display, there is no output. Enable hdmi_force_hotplug in the `/boot/config.txt`` to avoid this problem, and specify the group and mode to 1080 and 30hz.
sed -i 's/.*hdmi_force_hotplug.*/hdmi_force_hotplug=1/' /boot/config.txt
sed -i 's/.*hdmi_group=.*/hdmi_group=2/' /boot/config.txt
sed -i 's/.*hdmi_mode=.*/hdmi_mode=81/' /boot/config.txt
If you do regular deployments you can create an image. A reasonable approach is to:
Or you can use PiShrink to script all that.
wget https://raw.githubusercontent.com/Drewsif/PiShrink/master/pishrink.sh
chmod +x pishrink.sh
sudo mv pishrink.sh /usr/local/bin
# Capture and shrink the image
sudo dd if=/dev/mmcblk0 of=anthias-raw.img bs=1M
sudo pishrink.sh anthias-raw.img anthias.img
# Copy to a new card
sudo dd if=anthias.img of=/dev/mmcblk0 bs=1M
If you need to modify the image after creating it you can mount it via loop-back.
sudo losetup --find --partscan anthias.img
sudo mount /dev/loop0p2 /mnt/
# After you've made changes
sudo umount /mnt
sudo losetup --detach-all
If you have access to a graphical desktop environment, use GParted. It will resize the filesystem and partitions for you quite easily.
# Mount the image via loopback and open it with GParted
sudo losetup --find --partscan anthias-raw.img
# Grab the right side of the last partition with your mouse and
# drag it as far to the left as you can, apply and exit
sudo gparted /dev/loop0
Now you need to find the last sector and truncate the file after that location. Since the truncate utility operates on bytes, you convert sectors to bytes with multiplication.
# Find the End of the last partition. In the below example, it's Sector *9812664*
$ sudo fdisk -lu /dev/loop0
Units: sectors of 1 * 512 = 512 bytes
Device Boot Start End Sectors Size Id Type
/dev/loop0p1 8192 532479 524288 256M c W95 FAT32 (LBA)
/dev/loop0p2 532480 9812664 9280185 4.4G 83 Linux
sudo losetup --detach-all
sudo truncate --size=$[(9812664+1)*512] anthias-raw.img
If you don’t have a GUI, you can do it with a combination of commands.
# Mount the image via loopback
sudo losetup --find --partscan anthias-raw.img
# Check and resize the file system
sudo e2fsck -f /dev/loop0p2
sudo resize2fs -M /dev/loop0p2
... The filesystem on /dev/loop0p2 is now 1149741 (4k) blocks long
# Now you can find the end of the resized filesystem by:
# Finding the number of sectors.
# Bytes = Num of blocks * block size
# Number of sectors = Bytes / sector size
echo $[(1149741*4096)/512]
# Finding the start sector (532480 in the example below)
sudo fdisk -lu /dev/loop0
Device Boot Start End Sectors Size Id Type
/dev/loop0p1 8192 532479 524288 256M c W95 FAT32 (LBA)
/dev/loop0p2 532480 31116287 30583808 14.6G 83 Linux
# Adding the number of sectors to the start sector. Add 1 because you want to end AFTER the end sector
echo $[532480 + 9197928 + 1]
# And resize the part to that end sector (ignore the warnings)
sudo parted resizepart 2 9730409
Great! Now you can follow the remainder of the GParted steps to find the new last sector and truncate the file.
It’s handy to compress the image. xz is pretty good for this
xz anthias-raw.img
xzcat anthias-raw.img | sudo dd of=/dev/mmcblk0
In these procedures, we make a copy of the SD card before we do anything. Another strategy is to resize the SD card directly, and then use dd and read in X number of sectors rather than read it all in and then truncate it. A bit faster, if a but less recoverable from in the event of a mistake.
The API docs on the web refer to screenly. Anthias uses an older API. However, you can access the API docs for the version your working with at
http://sign.your.domain/api/docs/
You’ll have to correct the swagger form with correct URL, but after that you can see what you’re working with.
What started several years ago as a better chat-bot, is now able to perform simple chain-of-thought reasoning. In neural complexity we’are at a squirrel’s level, I’ve read.
When asked how big this is going to be, I usually say it is “the next cloud computing”.
My personal viewpoint has changed. It’s not so much that I’m so impressed at LLMs, but more that I’m less impressed by our own consciousness - we may not be special as we like to think.
Note: This field is changing so fast that the trials I did a year are invalid. You can probably disregard anything you find under this section
An “Ollama RAG app” is a web service that uses Ollama and a Vector DB to provide “Retrieval-Augmented Generation”. Here we set up a local LLM instance with ollama and chroma db for result augmentation.
Rough Notes from the initial test, mostly based from hackernoon.
# Check that we have Video Card Support
lspci | grep VGA
00:02.0 VGA compatible controller: Intel Corporation HD Graphics 530 (rev 06)
01:00.0 VGA compatible controller: NVIDIA Corporation GM107GLM [Quadro M1000M] (rev a2)
# Verify "Quadro" supports compute 5 at https://developer.nvidia.com/cuda-gpus
# install ollama as per https://github.com/ollama/ollama/blob/main/README.md#quickstart
curl -fsSL https://ollama.com/install.sh | sh
ollama run llama3.2
>>> what is your knowledge cutoff?
My knowledge cutoff is currently December 2023. This means that I have information up to that date, but I may not be aware of events, updates, or developments that have occurred after that time.
Install ChromaDB and connect it to ollama.
# install python deps
pip install --q chromadb
pip install --q unstructured langchain langchain-text-splitters
pip install --q "unstructured[all-docs]"
pip install --q flask
# Install the text embedding model
ollama pull nomic-embed-text
# Is ollama running? the CURL install add it as a service, I suspect
curl localhost:11434
Ollama is running
# Add a Markdown Document about the holiday schedule
curl --request POST \
--url http://localhost:8080/embed \
--header 'Content-Type: multipart/form-data' \
--form file=@/fall_schedule.md
{
"message": "File embedded successfully"
}
# Ask it a question about an event
curl --request POST \
--url http://localhost:8080/query \
--header 'Content-Type: application/json' \
--data '{ "query": "When is fall break?" }'
{
"message": "Fall break occurs from October 9-12."
}