SNMP Exporter

SNMP is one of the most prevalent (and clunky) protocols still widely used on network-attached devices. But it’s a good general-purpose way to get data from lots of different makes of products in a similar way.

But Prometheus doesn’t understand SNMP. The solution is a translation service that acts a a middle-man and ’exports’ data from those devices in a way Prometheus can.

Installation

Assuming you’ve already installed Prometheus, add some SNMP tools and the exporter. You’ll need to enable an additional repository for the mib-downloader.

sudo apt install software-properties-common
sudo apt-add-repository non-free
sudo apt update

sudo apt install -t testing snmp snmp-mibs-downloader
sudo apt install -t testing prometheus-snmp-exporter

Change the SNMP tools config file to allow use of installed MIBs. It’s disabled by default.

# The line 'mibs:' in the file overrides the default path. 
# Change it to include all MIBs
sudo sed -i 's/^mibs .*/mibs +ALL/' /etc/snmp/snmp.conf

Preparation

We need a target, so assuming you have a switch somewhere and can enable SNMP on it, let’s query the switch for its name, AKA sysName. Here we’re using version “2c” of the protocol with the read-only password “public”. Pretty standard.

Industry Standard Query

There are some well-known SNMP objects you can query, like System Name.

# Get the first value (starting at 0) of the sysName object
snmpget -Oqv -v 2c -c public some.switch.address sysName.0

Some-Switch

# If the .0 index isn't populated, you can use 'getnext' to find the first one that is
snmpgetnext -v 2c -c public some.switch.address sysName

Vendor Specific Query

Some vendors will release their own custom MIBs. These provide additional data for things that don’t have well-known objects. Add the MIBs to the system and ‘walk’ the tree to see what’s interesting.

# Unifi, for example
sudo cp UBNT-MIB.txt UBNT-UniFi-MIB.txt  /usr/share/snmp/mibs

# snmpwalk doesn't look for enterprise sections by default
# You must look in the MIB and add the specific company's OID number.
grep enterprises UBNT-*
...
UBNT-MIB.txt:    ubnt OBJECT IDENTIFIER ::= { enterprises 41112 }
...

snmpwalk -v2c -c public ubiquiti.switch.address enterprises.41112 

UBNT-UniFi-MIB::unifiRadioIndex.1 = INTEGER: 0
...
...

Note: If you get back an error or something with a long number in the key (like SNMPv2-SMI::enterprises.41112.1.6.1.1.1.1.1 = INTEGER: 0) , double check the changes in the snmp.conf file.

Fixing a Bad Operator

You may see this error message when walking.

Bad operator (INTEGER): At line 73 in /usr/share/snmp/mibs/ietf/SNMPv2-PDU

If so, replace a file as described in this excellent post by telcoM.This indicates the SNMP

sudo wget --output-document=/usr/share/snmp/mibs/ietf/SNMPv2-PDU https://pastebin.com/raw/p3QyuXzZ

Failing to correct this will break the snmp-generator and you may want to use that later on.

Configuration

Now that we can query data using command line tools, let’s configure Prometheus to do so.

Add a new job to the prometheus.yaml file. This job will include the targets as normal, but also the path (since it’s different than default) and an optional parameter called module that’s specific to the SNMP exporter.

This job also does uses something you may not have seen before, a relabel_config

This is because Prometheus isn’t actually taking to the switch, it’s talking to the local SNMP exporter service. So we put all the targets normally, and then at the bottom ‘oh, by the way, do a switcheroo’. This allows Prometheus to display all the data normally with no one the wiser.

...
...
scrape_configs:
  - job_name: 'snmp'
    static_configs:
      - targets:
        - some.switch.address    
    metrics_path: /snmp
    params:
      module: [if_mib]
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: 127.0.0.1:9116  # The SNMP exporter's real hostname:port.

Operation

No configuration on the exporter side is needed. Reload the config and check the target list. Then examine data in the graph section. Add additional targets as needed and the exporter will pull in the data.

http://some.server:9090/classic/targets

These metrics are considered well known and so will appear in the database named sysUpTime and upsBasicBatteryStatus and not be prefixed with snmp_ like you might expect.

Next Steps

If you have something non-standard, or you simply don’t want that huge amount of data in your system, look at the link below to customize the SNMP collection with the Generator.

SNMP Exporter Generator Customization

Troubleshooting

server returned HTTP status 400 Bad Request

When Prometheus queries the prometheus-snmp-exporter.service it may get that error back, indicating you have a SNMP error. This happened to me after an upgrade introduced a breaking change with the file produced from the snmp-generator.

You must re-generate any files it produced, and make sure to correct any MIB errors as with Fixing a Bad Operator above.

snmp-mibs-downloader isn’t found

The snmp-mibs-downloader is just a handy way to download a bunch of default MIBs so when you use the tools, all the cryptic numbers, like “1.3.6.1.2.1.17.4.3.1” are translated into pleasant names.

If you can’t find the mibs-downloader its probably because it’s in the non-free repo and that’s not enabled by default. We used a tool above, but you can do that manually as well if you search the internet for the how-to.