This is the multi-page printable view of this section. Click here to print.

Return to the regular view of this page.

Metrics

1 - Prometheus

Overview

Prometheus is a time series database, meaning it’s optimized to store and work with data organized in time order. It includes in it’s single binary:

  • Database engine
  • Collector
  • Simple web-based user interface

This allows you to collect and manage data with fewer tools and less complexity than other solutions.

Data Collection

End-points normally expose metrics to Prometheus by making a web page available that it can poll. This is done by including a instrumentation library (provided by Prometheus) or simply adding a listener on a high-level port that spits out some text when asked.

For systems that don’t support Prometheus natively, there are a few add-on services to translate. These are called ’exporters’ and translate things such as SNMP into a web format Prometheus can ingest.

Alerting

You can also alert on the data collected. This is through the Alert Manager, a second package that works closely with Prometheus.

Visualization

You still need a dashboard tool like Grafana to handle visualizations, but you can get started quite quickly with just Prometheus.

1.1 - Installation

Install from the Debian Testing repo. The normal repo can be up to a year behind.

# Add testing repo
echo 'deb http://deb.debian.org/debian testing main' | sudo tee -a /etc/apt/sources.list.d/testing.list

# Pin testing down to a low level so the rest of your packages don't get upgraded
sudo tee -a /etc/apt/preferences.d/not-testing << EOF
Package: *
Pin: release a=testing
Pin-Priority: 50
EOF

# Live Dangerously - this will pull in a lot of packages
sudo apt update
sudo apt install -t testing prometheus

# Observe the two new services
systemctl list-units --type service -q prometheus*

# If this is a VM, the pulled-in openipmi service isn't supported and can be removed
sudo apt remove openipmi

Configuration

Let’s replace the default config with something slightly simpler, then remove anything old.

sudo mv /etc/prometheus/prometheus.yml /etc/prometheus/prometheus.yml.orig

sudo tee /etc/prometheus/prometheus.yml << EOF
scrape_configs:
  - job_name: node
    static_configs:
      - targets: ["localhost:9100"]
EOF

sudo rm -rf /var/lib/prometheus/metrics2/*
sudo systemctl restart prometheus

The ‘scrapes’ are it’s jobs, and it has only one; check out the system at ’localhost:9100’ which happens to be the node exporter service running on itself.

For every target listed, the scraper makes a web request for /metrics/ and stores the results. It ingests all the data presented and stores them for 15 days. You can choose to ignore certain elements or retain differently by adding config, but by default it takes everything given.

Operation

User Interface

You can access the Web UI to see the status of your targets at:

http://some.server:9090/classic/targets

The data can be viewed by selecting Graph at the top, and then choosing from the first dropdown labeled “insert metric at cursor”.

The data is often prefixed with a category. For instance, the node-exporter service is written in the GO language and it provides some stats about that, prefixed with ‘go_’. Information about the system will be prefixed with ’node_’.

A simple Graph tab is available as well.

You can also see the node-exporter service’s output by asking like Prometheus would. Hit it up directly in your browser.

http://some.server:9100/metrics

Data Composition

Data can be simple, like:

go_gc_duration_seconds_sum 3

Or it can be dimensional which is accomplished by adding labels. In the example below, the value of go_gc_duration_seconds has 5 labeled sub-sets.

go_gc_duration_seconds{quantile="0"} 4.5697e-05
go_gc_duration_seconds{quantile="0.25"} 7.814e-05
go_gc_duration_seconds{quantile="0.5"} 0.000103396
go_gc_duration_seconds{quantile="0.75"} 0.000143687
go_gc_duration_seconds{quantile="1"} 0.001030941

In this example, the value of net_conntrack_dialer_conn_failed_total has several.

net_conntrack_dialer_conn_failed_total{dialer_name="alertmanager",reason="refused"} 0
net_conntrack_dialer_conn_failed_total{dialer_name="alertmanager",reason="resolution"} 0
net_conntrack_dialer_conn_failed_total{dialer_name="alertmanager",reason="timeout"} 0
net_conntrack_dialer_conn_failed_total{dialer_name="alertmanager",reason="unknown"} 0
net_conntrack_dialer_conn_failed_total{dialer_name="default",reason="refused"} 0
net_conntrack_dialer_conn_failed_total{dialer_name="default",reason="resolution"} 0
net_conntrack_dialer_conn_failed_total{dialer_name="default",reason="timeout"} 0
net_conntrack_dialer_conn_failed_total{dialer_name="default",reason="unknown"} 0
net_conntrack_dialer_conn_failed_total{dialer_name="snmp",reason="refused"} 0
net_conntrack_dialer_conn_failed_total{dialer_name="snmp",reason="resolution"} 0
net_conntrack_dialer_conn_failed_total{dialer_name="snmp",reason="timeout"} 0
net_conntrack_dialer_conn_failed_total{dialer_name="snmp",reason="unknown"} 0

How is this useful? It allows you to do aggregations - such as looking at all the net_contrack failures, and also look at the failures that were specifically refused. All with the same data.

Removing Data

You may have a target you want to remove. Such as a typo hostname that is now causing a large red bar on a dashboard. You can remove that mistake by enabling the admin API and issuing a delete

sudo sed -i 's/^ARGS.*/ARGS="--web.enable-admin-api"/' /etc/default/prometheus

sudo systemctl reload prometheus

curl -s -X POST -g 'http://localhost:9090/api/v1/admin/tsdb/delete_series?match[]={instance="badhost.some.org:9100"}'

The default retention is 15 days. You may want less than that and you can configure --storage.tsdb.retention.time=1d similar to above. ALL data has the same retention, however. If you want historical data you must have a separate instance or use VictoriaMetrics.

Next Steps

Let’s get something interesting to see by adding some OS metrics

Troubleshooting

If you can’t start the prometheus server, it may be an issue with the init file. Test and Prod repos use different defaults. Add some values explicitly to get new versions running

sudo vi /etc/default/prometheus

ARGS="--config.file="/etc/prometheus/prometheus.yml  --storage.tsdb.path="/var/lib/prometheus/metrics2/"

Sources

OpenIPMI

https://medium.com/@MahdiAlimohammadi/why-openipmi-service-failed-on-my-debian-12-server-and-how-i-fixed-it-9c3fef441ad0

Pinning Repos

https://unix.stackexchange.com/questions/647204/aptdefault-release-stable-isnt-sufficient-to-stop-packages-being-automatica

1.2 - Node Exporter

This is a service you install on your end-points to make CPU/Memory/Etc. metrics available to Prometheus.

Installation

On each device you want to monitor, install the node exporter with this command.

sudo apt install prometheus-node-exporter

Do a quick test to make sure it’s responding to scrapes.

curl localhost:9100/metrics

Configuration

Back on your Prometheus server, add these new nodes as a job in the prometheus.yaml file. Feel free to drop the initial job where Prometheus was scraping itself.

global:
  scrape_interval: 15s
scrape_configs:
  - job_name: 'servers'
    static_configs:
    - targets:
      - some.server:9100
      - some.other.server:9100
      - and.so.on:9100

sudo systemctl reload prometheus.service

Operation

You can check the status of your new targets at:

http://some.server:9090/classic/targets

A lot of data is provided by the node by default. This can have a noticeable impact on low-power systems. To reduce this, customize the node’s config to enable only specific collectors.

In the case below we reduce collection to just CPU, Memory, and Hardware metrics. When scraping a Pi 3B, this reduces the Scrape Duration from 500 to 50ms.

# Do this on the node you're monitoring
sudo sed -i 's/^ARGS.*/ARGS="--collector.disable-defaults --collector.hwmon --collector.cpu --collector.meminfo"/' /etc/default/prometheus-node-exporter
sudo systemctl restart prometheus-node-exporter

The available collectors are listed on the page:

https://github.com/prometheus/node_exporter

1.3 - SNMP Exporter

SNMP is one of the most prevalent (and clunky) protocols still widely used on network-attached devices. But it’s a good general-purpose way to get data from lots of different makes of products in a similar way.

But Prometheus doesn’t understand SNMP. The solution is a translation service that acts a a middle-man and ’exports’ data from those devices in a way Prometheus can.

Installation

Assuming you’ve already installed Prometheus, add some SNMP tools and the exporter. You’ll need to enable an additional repository for the mib-downloader.

sudo apt install software-properties-common
sudo apt-add-repository non-free
sudo apt update

sudo apt install -t testing snmp snmp-mibs-downloader
sudo apt install -t testing prometheus-snmp-exporter

Change the SNMP tools config file to allow use of installed MIBs. It’s disabled by default.

# The line 'mibs:' in the file overrides the default path. 
# Change it to include all MIBs
sudo sed -i 's/^mibs .*/mibs +ALL/' /etc/snmp/snmp.conf

Preparation

We need a target, so assuming you have a switch somewhere and can enable SNMP on it, let’s query the switch for its name, AKA sysName. Here we’re using version “2c” of the protocol with the read-only password “public”. Pretty standard.

Industry Standard Query

There are some well-known SNMP objects you can query, like System Name.

# Get the first value (starting at 0) of the sysName object
snmpget -Oqv -v 2c -c public some.switch.address sysName.0

Some-Switch

# If the .0 index isn't populated, you can use 'getnext' to find the first one that is
snmpgetnext -v 2c -c public some.switch.address sysName

Vendor Specific Query

Some vendors will release their own custom MIBs. These provide additional data for things that don’t have well-known objects. Add the MIBs to the system and ‘walk’ the tree to see what’s interesting.

# Unifi, for example
sudo cp UBNT-MIB.txt UBNT-UniFi-MIB.txt  /usr/share/snmp/mibs

# snmpwalk doesn't look for enterprise sections by default
# You must look in the MIB and add the specific company's OID number.
grep enterprises UBNT-*
...
UBNT-MIB.txt:    ubnt OBJECT IDENTIFIER ::= { enterprises 41112 }
...

snmpwalk -v2c -c public ubiquiti.switch.address enterprises.41112 

UBNT-UniFi-MIB::unifiRadioIndex.1 = INTEGER: 0
...
...

Note: If you get back an error or something with a long number in the key (like SNMPv2-SMI::enterprises.41112.1.6.1.1.1.1.1 = INTEGER: 0) , double check the changes in the snmp.conf file.

Fixing a Bad Operator

You may see this error message when walking.

Bad operator (INTEGER): At line 73 in /usr/share/snmp/mibs/ietf/SNMPv2-PDU

If so, replace a file as described in this excellent post by telcoM.This indicates the SNMP

sudo wget --output-document=/usr/share/snmp/mibs/ietf/SNMPv2-PDU https://pastebin.com/raw/p3QyuXzZ

Failing to correct this will break the snmp-generator and you may want to use that later on.

Configuration

Now that we can query data using command line tools, let’s configure Prometheus to do so.

Add a new job to the prometheus.yaml file. This job will include the targets as normal, but also the path (since it’s different than default) and an optional parameter called module that’s specific to the SNMP exporter.

This job also does uses something you may not have seen before, a relabel_config

This is because Prometheus isn’t actually taking to the switch, it’s talking to the local SNMP exporter service. So we put all the targets normally, and then at the bottom ‘oh, by the way, do a switcheroo’. This allows Prometheus to display all the data normally with no one the wiser.

...
...
scrape_configs:
  - job_name: 'snmp'
    static_configs:
      - targets:
        - some.switch.address    
    metrics_path: /snmp
    params:
      module: [if_mib]
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: 127.0.0.1:9116  # The SNMP exporter's real hostname:port.

Operation

No configuration on the exporter side is needed. Reload the config and check the target list. Then examine data in the graph section. Add additional targets as needed and the exporter will pull in the data.

http://some.server:9090/classic/targets

These metrics are considered well known and so will appear in the database named sysUpTime and upsBasicBatteryStatus and not be prefixed with snmp_ like you might expect.

Next Steps

If you have something non-standard, or you simply don’t want that huge amount of data in your system, look at the link below to customize the SNMP collection with the Generator.

SNMP Exporter Generator Customization

Troubleshooting

server returned HTTP status 400 Bad Request

When Prometheus queries the prometheus-snmp-exporter.service it may get that error back, indicating you have a SNMP error. This happened to me after an upgrade introduced a breaking change with the file produced from the snmp-generator.

You must re-generate any files it produced, and make sure to correct any MIB errors as with Fixing a Bad Operator above.

snmp-mibs-downloader isn’t found

The snmp-mibs-downloader is just a handy way to download a bunch of default MIBs so when you use the tools, all the cryptic numbers, like “1.3.6.1.2.1.17.4.3.1” are translated into pleasant names.

If you can’t find the mibs-downloader its probably because it’s in the non-free repo and that’s not enabled by default. We used a tool above, but you can do that manually as well if you search the internet for the how-to.

1.4 - SNMP Generator

Installation

The generator is already installed. It came with the SNMP exporter. But if you have a device that supplies it’s own MIB (and many do), you should add that to the default location.

# Mibs are often named SOMETHING-MIB.txt
sudo cp -n *MIB.txt /usr/share/snmp/mibs/

Preparation

You must identify the values you want to capture. Using snmpwalk is a good way to see what’s available, but it helps to have a little context.

The data is arranged like a folder structure that you drill-down though. The folder names are all numeric, with ‘.’ instead of slashes. So if you wanted to get a device’s sysName you’d click down through 1.3.6.1.2.1.1.5 and look in the file 0.

snmpwalk -v2c -c public 10.10.202.246 1.3.6.1.2.1.1.5
SNMPv2-MIB::sysName.0 = STRING: RickeySelby131

Why did sysName get put there? A bunch of folks got together (the ISO folks) and decided everything in advance. Since trying to remember all those numbers is a bit painful, they made some handy files (MIBs) and passed them out so you can jump right to what you want.

snmpwalk -v2c -c public 10.10.202.246 sysName.0
SNMPv2-MIB::sysName.0 = STRING: RickeySelby131

They allow vendors to create their own sections as well, for things that might not fit anywhere else.

A good place to start is looking at what the vendor made available. You see this by “walking” their section (descending down through all the folders from a starting point).

You’ll get descriptive names as long as you already coped their MIB and enabled all in the snmp.conf file. If not, you can put it in your working directory and include it at the command line.

# The SysobjectID identifies the vendor section
# Note use of the MIB name without the .txt
$ snmpwalk -m +SOMEVENDOR-MIB -v 2c -c public some.address SysobjectID

SNMPv2-MIB::sysObjectID.0 = OID: SOMEVENDOR-MIB::somevendoramerica

# Then walk the vendor section using the name from above
$ snmpwalk -m +SOMEVENDOR-MIB -v 2c -c some.address somevendoramerica

SOMEVENDOR-MIB::model.0 = STRING: SOME-MODEL
SOMEVENDOR-MIB::power.0 = INTEGER: 0
...
...

# Also check out the general System section
$ snmpwalk -m +SOMEVENDOR-MIB -v 2c -c public some.address system

# You can also walk the whole ISO tree. In some cases,
# there are thousands of entries and it's indecipherable
$ snmpwalk -m +SOMEVENDOR-MIB -v 2c -c public some.system iso

This can be a lot of information and you’ll need to do some homework to see what data you want to collect.

Configuration

The exporter’s default configuration file is snmp.yml and contains about 57 Thousand lines of config. It’s designed to pull data from whatever you point it at. Basically, it doesn’t know what device it’s talking to, so it tries to cover all the bases.

This isn’t a file you should edit by hand. Instead, you create instructions for the generator and it look though the MIBs and create one for you. Here’s an example for a Samlex Invertor.

vim ~/generator.yml

modules:
  samlex:
    walk:
      - sysLocation
      - inverterMode
      - power
      - vin
      - tempDD
      - tempDA

Generate a new snmp.yml to replace the default.

prometheus-snmp-generator generate
sudo cp /etc/prometheus/snmp.yml /etc/prometheus/snmp.yml.orig
sudo cp ~/snmp.yml /etc/prometheus
sudo systemctl reload prometheus-snmp-exporter.service

Configuration in Prometheus remains the same - but since we picked a new module name we need to adjust that.

    ...
    ...
    params:
      module: [samlex]
    ...
    ...

sudo systemctl reload prometheus.service

Adding Data Prefixes

by default, the names are all over the place. The SNMP Exporter Devs leave it this way because there are a lot of pre-built dashboards on downstream systems that expect the existing names.

If you are building your own downstream systems you can prefix (as is best-practice) as you like with a post generation step. This example cases them all to be prefixed with samlex_.

prometheus-snmp-generator generate
sed -i 's/name: /name: samlex_/' snmp.yml
sudo cp ~/snmp.yml /etc/prometheus
sudo systemctl reload prometheus-snmp-exporter.service

Combining MIBs

You can combine multiple systems in the generator file to create one snmp.yml file, and refer to them by the module name in the Prometheus file.

modules:
  samlex:
    walk:
      - sysLocation
      - inverterMode
      - power
      - vin
      - tempDD
      - tempDA
  ubiquiti:
    walk:
      - something
      - somethingElse

Operation

As before, you can get a preview directly from the exporter (using a link like below). This data should show up in the Web UI too.

http://some.server:9116/snmp?module=samlex&target=some.device

Troubleshooting

SNMP Exporter

server returned HTTP status 400 Bad Request

If you see this error message in your http://some.server:9090/classic/targets, and:

Unknown auth ‘public_v2’

When you look directly at http://some.server:9116/snmp?module=samlex&target=a.ups.your.local

Then you may need to add an auth section to your file as in this posting.

sudo vi /etc/prometheus/snmp.yml

# Add this right at the top
auths:
  public_v1:
    version: 1
  public_v2:
    community: public
    security_level: noAuthNoPriv
    auth_protocol: MD5
    priv_protocol: DES
    version: 2

sudo systemctl restart prometheus-snmp-exporter.service

Possible version missmatch between generator and snmp_exporter. Make sure generator and snmp_exporter are the same version."

If you upgrade and the snmp exporter doesn’t start, you may see this in your log. It’s possible your snmp.yml files has old syntax. You can run the generate and cp commands to generate a current one.

level=ERROR source=main.go:137 msg=“Failing on reported parse error(s)”

You might have that operator syntax error as described in the exporter config. Double check that you can snmpwalk without error.

NetSNMP

If you’re running prometheus-snmp-generator generate and get the error:

NetSNMP reported parse error(s)

you must update a mib file. This is described in some detail but the short answer is to run this command;

sudo wget --output-document /usr/share/snmp/mibs/ietf/SNMPv2-PDU https://pastebin.com/raw/p3QyuXzZ

Sources

https://github.com/prometheus/snmp_exporter/tree/main/generator

1.5 - Blackbox

The blackbox exporter allows you to check the status of endpoints over HTTP, HTTPS, DNS, TCP, ICMP and gRPC.

This is mostly used as a ‘ping’ test to see if the device is on the network, or a web scrape to see if the web server is responding.

This is most useful when you can’t or don’t want to install the node exporter on the target.

sudo apt install  -t testing prometheus-blackbox-exporter

 - job_name: 'phones'
    metrics_path: /probe
    params:
      module: [icmp]
    static_configs:
      - targets:
        - some.end.point
        - some.other.point
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: 127.0.0.1:9115

1.6 - Custom Sensor

You can also monitor things that don’t have an exporter by writing it yourself. Here’s an example for an environmental DHT sensors.

DHT stands for Digital Humidity and Temperature. At less than $5 they are cheap and can be hooked to a Raspberry Pi easily. Add a Prometheus exporter to do at scale.

  • Connect the Senor
  • Provision and Install the Python Libraries
  • Test the Libraries and the Sensor
  • Install the Prometheus Exporter as a Service
  • Create a Service Account
  • Add to Prometheus

Connect The Sensor

These usually come as a breakout-board with three leads you can connect to the Raspberry PI GPIO pins as follows;

  • Positive lead to pin 1 (power)
  • Negative lead to pin 6 (ground)
  • Middle or ‘out’ lead to pin 7 (that’s GPIO 4)

Image of DHT Connection

(From https://github.com/rnieva/Playing-with-Sensors---Raspberry-Pi)

Provision and Install Libraries

Use the Raspberry Pi Imager to Provision the Pi with Raspberry PI OS Lite 64 bit. Next, install the “adafruit_blinka” library as adapted from their instructions and test.

# General updates
sudo apt update
sudo apt -y upgrade
sudo apt -y autoremove
sudo reboot

# These python components may already be installed, but making sure
sudo apt -y install python3-pip
sudo apt -y install --upgrade python3-setuptools
sudo apt -y install python3-venv

# Make a virtual environment for the python process
sudo mkdir /usr/local/bin/sensor-dht
sudo python3 -m venv /usr/local/bin/sensor-dht --system-site-packages
cd /usr/local/bin/sensor-dht
sudo chown -R ${USER}:${USER} .
source bin/activate

# Build and install the library
pip3 install --upgrade adafruit-python-shell
wget https://raw.githubusercontent.com/adafruit/Raspberry-Pi-Installer-Scripts/master/raspi-blinka.py
sudo -E env PATH=$PATH python3 raspi-blinka.py

Test the Libraries and the Sensor

After logging back in, test the blinka lib.

cd /usr/local/bin/sensor-dht
source bin/activate
wget https://learn.adafruit.com/elements/2993427/download -O blinkatest.py
python3 blinkatest.py

Then install the DHT library from CircuitPython and create a script to test the sensor.

cd /usr/local/bin/sensor-dht
source bin/activate
pip3 install adafruit-circuitpython-dht

vi sensortest.py

import board
import adafruit_dht

dhtDevice = adafruit_dht.DHT11(board.D4)
temp = dhtDevice.temperature
humidity = dhtDevice.humidity

print(
  "Temp: {:.1f} C    Humidity: {}% ".format(temp, humidity)
)

dhtDevice.exit()

You can get occasional errors like RuntimeError: Checksum did not validate. Try again. that are safe to ignore. These DHTs are not 100% solid.

Create an Export Script

Add the Prometheus pips.

cd /usr/local/bin/sensor-dht
source bin/activate
pip3 install prometheus_client

And create a script like this.

nano sensor.py

import board
import adafruit_dht
import time
from prometheus_client import start_http_server, Gauge

dhtDevice = adafruit_dht.DHT11(board.D4)

temperature_gauge= Gauge('dht_temperature', 'Local temperature')
humidity_gauge = Gauge('dht_humidity', 'Local humidity')

start_http_server(8000)
    
while True:

 try:
  temperature = dhtDevice.temperature
  temperature_gauge.set(temperature)

  humidity = dhtDevice.humidity
  humidity_gauge.set(humidity)
 
 except:
  # Errors happen fairly often as DHT's are hard to read. Just continue on.
  continue

 finally:
  time.sleep(60)

Create a service

sudo nano /lib/systemd/system/sensor.service

[Unit]
Description=Temperature and Humidity Sensing Service
After=network.target

[Service]
Type=idle
Restart=on-failure
User=root
ExecStart=/bin/bash -c 'cd /usr/local/bin/sensor-dht && source bin/activate && python sensor.py'

[Install]
WantedBy=multi-user.target

Enable and start it

sudo systemctl enable --now sensor.service

curl http://localhost:8000/metrics

Create a Service Account

This service is running as root. You should consider creating a sensor account.

sudo useradd --home-dir /usr/local/bin/sensor-dht --system --shell /usr/sbin/nologin --comment "Sensor Service" sensor
sudo usermod -aG gpio sensor

sudo systemctl stop  sensor.service
sudo chown -R sensor:sensor /usr/local/bin/sensor-dht
sudo sed -i 's/User=root/User=sensor/' /lib/systemd/system/sensor.service

sudo systemctl daemon-reload 
sudo systemctl start sensor.service

Add to Prometheus

Adding it requires logging into your Prometheus server and adding a job like below.

sudo vi /etc/prometheus/prometheus.yml

...
...
  - job_name: 'dht' 
    static_configs: 
      - targets: 
        - 192.168.1.45:8000

You will be able to find the node in your server at http://YOUR-SERVER:9090/targets?search=#pool-dht and data will show up with a leading dht_....

Sources

https://randomnerdtutorials.com/raspberry-pi-dht11-dht22-python/

You may want to raise errors to the log as in the above source.

2 - LibreSpeed

It’s useful to have an internal speed test. Browser based is best for end-users.

The two main choices are LibreSpeed or OpenSpeedTest. The former is generally preferred for it’s simplicity.

Preparation

Install a minimal Debian instance by using the netinst image, de-selecting the desktop system at the top, while adding ‘common system tools’ near the bottom.

Installation

You can use an Apache server with PHP, or a binary service. If you have a simple use case, the latter has fewer moving parts

Binary

You’ll need the need the web/assets folder and the pre-compiled binary. Let’s use the rust version with the theory being it’s safer and faster. (though in casual testing, this is debatable)

Look for the latest release at:

https://github.com/librespeed/speedtest-rust/releases

# Download the source and extract the assets folder
wget https://github.com/librespeed/speedtest-rust/archive/refs/tags/v1.3.8.tar.gz
tar --strip-components=1 -xzf v1.3.8.tar.gz speedtest-rust-1.3.8/assets/

# Download the binary, extract and make executable
wget https://github.com/librespeed/speedtest-rust/releases/download/v1.3.8/librespeed-rs-x86_64-unknown-linux-gnu.tar.xz
tar xf librespeed-rs-x86_64-unknown-linux-gnu.tar.xz
chmod +x librespeed-rs

# Disable the stored results database
sed -i 's/database_type="sqlite"/database_type="none"/' configs.toml

# Change the labels in the page
sed -i 's/Your server name here/speedtest.your.org/' assets/servers_list.js
sed -i 's/LibreSpeed Example/speedtest.your.org/' assets/index.html

# Launch the server and test
./librespeed-rs

firefox http://localhost:8080

Assuming that works as intended, you can create a service file and enable it.

Web Server

You might need to do more, such as run some code before letting people hit your speed test. Use the PHP version for this.

# Add the prerequisites. PHP is enabled for apache by default
apt install apache2 php openssl

# Look for the latest release at:
firefox https://github.com/librespeed/speedtest/releases

# Download, extract and copy a few things to your web folder
wget https://github.com/librespeed/speedtest/archive/refs/tags/5.4.1.tar.gz
tar xzf 5.4.1.tar.gz
cd speedtest-5.4.1
sudo cp -r index.html speedtest.js speedtest_worker.js favicon.ico backend /var/www/html/

LAN Speeds

Sometimes you want to compare LAN/WAN speeds across multiple networks but want to keep just a single URL. Do this by using multiple (or one server with multiple interfaces) and use a .php to redirect users to the right place based on their IP.

Add A Redirecting Page

Create a new index page.

sudo vi /var/www/html/index.php

<!DOCTYPE html>
<html>
<head>
<?php

# Extract the client's IP address
$chunks = explode('.', $_SERVER['REMOTE_ADDR']);

# Assume we have an interface on the clients LAN at .55 and 
# redirect them to it.
$network = "$chunks[0].$chunks[1].$chunks[2].55";

# Use the .replace method so the next request client has a 
# referring page, used later on.
echo "<script>";
echo "window.location.replace(\"http://$network/speed.html\")";
echo "</script>";

?>
</head>
</html>

Adjust apache so index.php takes precedence.

sudo vi /etc/apache2/mods-available/dir.conf

# Move index.php to be right at the front.
DirectoryIndex index.php index.html index.cgi index.pl index.php index.xhtml index.htm

# Reload so changes take affect
sudo systemctl reload apache2.service

Modify The Main Page

Some users will bookmark the speed test page. If they change subnets and use the old bookmark, this will no longer be a LAN test.

Let’s copy the delivered index.html to a new page that includes referrer check. This makes sure they hit the main URL first and get redirected every time. (note the script above refers to the page “speed.html” so let’s use that)

sudo cp /var/www/html/index.html  /var/www/html/speed.html
sudo vi /var/www/html/speed.html

<script>
        // Redirects to the main URL in case somebody books the wrong subnet page
        correct_referrer="http://speedtest.marietta.edu/";
        actual_referrer=document.referrer;
        if (actual_referrer != correct_referrer) window.location.replace(correct_referrer);
</script>

Special Asymmetric Routing Bonus

If you added interfaces to a server to test LAN connections, your clients will connect to the WAN IP address. But the server has an interface on the same network as the client, so it replies using that. The client will drop these unexpected replies.

You need to a routing policy so the server responds back on the same interface the client sent it’s request to.

Note: We are justing using table ID 1, and not bothering to create a rt_tables file to give it a custom name.

# On the WAN interface, 192.168.1.5 used below.
allow-hotplug eth0
iface eth0 inet static
        address ...
        gateway ...
        dns-nameservers ...
        up ip rule add from 192.168.1.5 table 1 prio 1
        up ip route add default via 192.168.1.5 dev eth0 table 1

3 - Smokeping

I’ve been using Smokeping for at least 20 years. Every so often I look at the competitors, but it’s still the best for a self-contained latency monitoring system.

Installation

On Debian Stretch:

# Install smokeping - apache gets installed automatically and the config enabled
sudo apt-get install smokeping

# Install and enable SpeedyCGI if you can find it, otherwise, fastCGI
sudo apt install libapache2-mod-fcgid

Configure

Edit the General Config file

sudo vim /etc/smokeping/config.d/General

owner    = Some Org
contact  = [email protected]
mailhost = localhost

cgiurl   = http://some.server.you.just.deployed.on/cgi-bin/smokeping.cgi

# specify this to get syslog logging
syslogfacility = local0

# each probe is now run in its own process
# disable this to revert to the old behaviour
# concurrentprobes = no

@include /etc/smokeping/config.d/pathnames

Edit the pathnames file. You must put in a value for sendmail (If you don’t have it) so that smoke ping will run.

sudo vim /etc/smokeping/config.d/pathnames


sendmail = /bin/false
imgcache = /var/cache/smokeping/images
imgurl   = ../smokeping/images
...
...

Edit the Alerts

sudo vim /etc/smokeping/config.d/Alerts


to = [email protected]
from = [email protected]

Edit the Targets

# Add your own targets that you will measure by appending them to the bottom of the targets file. 
# These will show up in a menu on the left of the generated web page. You add an entry starting with a + to create a top level entry, and subsequent lines with ++ that will show up as sub entries like so:

#        + My Company

#        ++ My Company's Web Server 1

#        ++ My Company's Web Server 2


#    Actual config requires a few extra lines, as below;


sudo vim /etc/smokeping/config.d/Targets


    + My_Company

    menu =  My Company
    title = My Company


    ++ Web_Server_1
    menu = Web Server 1
    title = Web Server 1
    host = web.server.org


# Restart smokeping and apache

sudo service smokeping restart
sudo service apache2 reload

Access smokeping at:

http://some.server.name/smokeping

Notes

The default resolution - i.e. polling frequency is 20 requests over 5 min - or 1 request every 15 seconds

http://collaboration.cmc.ec.gc.ca/science/rpn/biblio/ddj/Website/articles/SA/v12/i07/a5.htm

ArchWiki suggests a workaround for sendmail

https://wiki.archlinux.org/index.php/smokeping

4 - SNMP

SNMP is a protocol that’s almost always used to monitor things. Usually metrics. Though it can be used as a management interface as well.

Every time I use it I imagine a conclave of exports, meeting together and sequestering themselves for a year, then coming out of a back room and proclaiming “Here it is, the perfect protocol for doing everything”.

And it’s completely different than anything else. The one true faith protocol.

4.1 - Eaton UPS

Eaton offers network management cards for many of their UPS lines that provide a web management console and SNMP server. The “Eaton Gigabit Network Card (NETWORK-M2)” is the offering at time of writing.

For SNMP, this card implements the RFC 1628 UPS standard as well as an Eaton specific set of objects1 starting at 1.3.6.1.4.1.534.

snmpwalk -v1 -c public your.ups.address

snmpwalk -v1 -c public your.ups.address 1.3.6.1.4.1.534

To make sense of these you need the Eaton MIBs2. If you download and extract to your working folder you can query like this.

snmpwalk -v1 -c public -m +./EATON-OIDS.txt:./EATON-EMP-MIB.txt:./XUPS-MIB.txt your.ups.address 1.3.6.1.4.1.534

Some useful MIBs are:
sysLocation Physical location for the device being monitored
xupsInputVoltage The measured input voltage from the UPS meters in volts
xupsOutputSource The present source of output power
xupsBatVoltage Battery voltage as reported by the UPS meters
xupsInputFrequency The utility line frequency in tenths of Hz
xupsOutputWatts The measured real output power in watts
xupsEnvAmbientTemp The reading of the ambient temperature in the vicinity

You’d query these like so:

snmpget  -v1 -c public -m +./EATON-OIDS.txt:./EATON-EMP-MIB.txt:./XUPS-MIB.txt your.ups.address \
sysLocation.0 xupsInputVoltage.1 xupsOutputSource.0 xupsBatVoltage.0 xupsInputFrequency.0 xupsOutputWatts.1 xupsEnvAmbientTemp.0

Some objects need a .0 here and others a .1 as shown above.

Notes

The object xupsOutputSource is generally used to determine if the UPS is ‘on battery’3. There is a trap specifically for OnBattery, but you can’t query the traps.

xupsOutputSource OBJECT-TYPE
    SYNTAX  INTEGER {
        other(1),
        none(2),
        normal(3),              
        bypass(4),
        battery(5),
        booster(6),             
        reducer(7),             
        parallelCapacity(8),    
        parallelRedundant(9),   
        highEfficiencyMode(10), 
        maintenanceBypass(11),   
        essMode(12)   
        }

Troubleshooting

There is no such variable name in this MIB

You might need to “walk” the object to get the instance ID as .0 is only for scalar objects, and not tabular.

snmpget  -v1 -c public -m +./EATON-OIDS.txt:./EATON-EMP-MIB.txt:./XUPS-MIB.txt your.ups.address xupsInputVoltage.0
Error in packet
Reason: (noSuchName) There is no such variable name in this MIB.
Failed object: XUPS-MIB::xupsInputVoltage.0

snmpwalk  -v1 -c public -m +./EATON-OIDS.txt:./EATON-EMP-MIB.txt:./XUPS-MIB.txt your.ups.address xupsInputVoltage
XUPS-MIB::xupsInputVoltage.1 = INTEGER: 116 RMS Volts

snmpget  -v1 -c public -m +./EATON-OIDS.txt:./EATON-EMP-MIB.txt:./XUPS-MIB.txt your.ups.address xupsInputVoltage.1
XUPS-MIB::xupsInputVoltage.1 = INTEGER: 116 RMS Volts

Resources

  1. https://www.eaton.com/content/dam/eaton/products/backup-power-ups-surge-it-power-distribution/power-management-software-connectivity/connectups-e-web-snmp-device/Eaton-connect-UPS-web-snmp-user-guide-164201819.pdf.pdf ↩︎

  2. https://www.eaton.com/us/en-us/site-search.html.searchTerm$mib.tabs$all.html ↩︎

  3. https://thwack.solarwinds.com/product-forums/network-performance-monitor-npm/f/forum/68430/monitor-when-ups-is-on-battery-power ↩︎

4.2 - Samlex UPS

Samlex makes a line of rack-mount inverters with a web management console and respond to SNMP requests. Their PSR-1200-24 and -48 models. They supply the MIB on an included USB thumb drive. If you lose it you may never find it again on the web. Here’s a copy just in case.

Download

If you probe them without the MIB or knowing their enterprise OID, you don’t get much back.

snmpwalk -v2c -c public your.samlex.ip

However, you can probe the enterprise section and you’ll see they have thier own section with some 64 values and traps.

snmpwalk -v2c -c public your.samlex.ip 1.3.6.1.4.1

To make sense of these, you’ll need that MIB file from above.

snmpwalk -v2c -c public -m ./SAMLEXAMERICA-MIB.txt your.samlex.ip 1.3.6.1.4.1

The most useful values are probably the ones that tell you:

  • which system is this (sysLocation)
  • are you on battery or grid right now (inverterMode)
  • what is your battery voltage (vin)
  • what is your grid voltage (gridvout)
  • how much power you’re putting out (power)
  • what your temps are (tempDD tempDA)

You can pull those in a loop like so

while read X;do 
        snmpget -v2c -m +./SAMLEXAMERICA-MIB.TXT -c public $X \
                sysLocation.0 inverterMode.0 vin.0 gridvout.0 power.0 tempDD.0 tempDA.0
         echo
done < file_of_device_IPs

You can also pull this into Prometheus

https://samlexamerica.com/wp-content/uploads/2021/08/11001-PSR-1200-24-48-0821_Lrez.pdf