Loki

Loki is a system for handling logs (unstructured data) but is lighter-weight than Elasticsearch. It also has fewer add-ons. But if you’re already using Prometheus and Grafana and you want to do it yourself, it can be a better solution.

Installation

Install Loki and Promtail together. These are available in the debian stable repos at current version. No need to go to backports or testing

sudo apt install loki promtail
curl localhost:3100/metrics

Configuration

Default config files are create in /etc/loki and /etc/promtail. Promtail is tailing /var/log/*log file, pushing them to localhost loki on the default port (3100) and loki is saving data in the /tmp directory. This is fine for testing.

Promtail runs as the promtail user (not root) and can’t read anything useful, so add them to the adm group.

sudo usermod -a -G adm promtail
sudo systemctl restart promtail

Grafana Integration

In grafana, add a datasource.

Configuration –> Add new data source –> Loki

Set the URL to http://localhost:3100

Then view the logs

Explore –> Select label (filename) –> Select value (daemon)

Troubleshooting

error notifying frontend about finished query

Edit the timeout setting in your loki datasource. The default may be too short so set it to 30s or some such

Failed to load log volume for this query

If you added a logfmt parser like the gui suggested, you may find not all your entries can be parsed, leading to this error.:w