Access Logging

• 1: GoAccess

1 - GoAccess

GoAccess is a lightweight web stats visualizer than can display in a terminal window or in a browser. It supports Caddy’s native JSON log format and can also be run as a real-time service with a little work.

Installation

If you have Debian/Ubuntu, you can add the repo as the [official docs] show.

# Note: The GoAccess docs use the `lsb_release -cs` utility that some Debains don't have, so I've substituted the $VERSION_CODENAME variable from the os-release file
wget -O - https://deb.goaccess.io/gnugpg.key | gpg --dearmor | sudo tee /usr/share/keyrings/goaccess.gpg >/dev/null 
source /etc/os-release 
echo "deb [signed-by=/usr/share/keyrings/goaccess.gpg arch=$(dpkg --print-architecture)] https://deb.goaccess.io/ $VERSION_CODENAME main" | sudo tee /etc/apt/sources.list.d/goaccess.list
sudo apt update
sudo apt install goaccess

Basic Operation

No configuration is needed if your webserver is logging in a supported format¹. Though you may need to adjust file permissions so the log file can be read by the user running GoAccess.

To use in the terminal, all you need to is invoke it with a couple parameters.

sudo goaccess --log-format CADDY --log-file /var/log/caddy/access.log

To produce a HTML report, just add an output file somewhere your web server can find it.

sudo touch  /var/www/some.server.org/report.html
sudo chown $USER /var/www/some.server.org/report.html
goaccess --log-format=CADDY --output /var/www/some.server.org/report.html --log-file /var/log/caddy/access.log

Retaining History

History is useful and GoAccess lets you persist your data and incorporate it on the next run. This updates your report rather than replace it.

# Create a database location
sudo mkdir -p /var/lib/goaccess-db
sudo chown $USER /var/lib/goaccess-db

goaccess \
    --persist --restore --db-path /var/lib/goaccess-db \
    --log-format CADDY --output /var/www/some.server.org/report.html --log-file /var/log/caddy/access.log

GeoIP

To display country and city you need GeoIP databases, preferably the GeoIP2 format. An easy way to get started is with DB-IP’s Lite GeoIP2 databases that have a permissive license.

# These are for Jan 2025. Check https://db-ip.com/db/lite.php for the updated links
wget --directory-prefix /var/lib/goaccess-db \
    https://download.db-ip.com/free/dbip-asn-lite-2025-01.mmdb.gz \
    https://download.db-ip.com/free/dbip-city-lite-2025-01.mmdb.gz \
    https://download.db-ip.com/free/dbip-country-lite-2025-01.mmdb.gz  
    
gunzip /var/lib/goaccess-db/*.gz

goaccess \
    --persist --restore --db-path /var/lib/goaccess-db \
    --geoip-database /var/lib/goaccess-db/dbip-asn-lite-2025-01.mmdb  \
    --geoip-database /var/lib/goaccess-db/dbip-city-lite-2025-01.mmdb  \
    --geoip-database /var/lib/goaccess-db/dbip-country-lite-2025-01.mmdb \
    --log-format CADDY --output /var/www/some.server.org/report.html --log-file /var/log/caddy/access.log

This will add a Country and ASN panel, and populate the city column on the “Visitor Hostnames and IPs” panel.

This one-time download is “good enough” for most purposes. But if you want to automate updates of the GeoIP data, you can create an account with a provider and get an API key. Maxmind offers free accounts. Sign up here and you can sudo apt install geoipupdate to get regular updates.

Automation

A reasonable way to automate is with a logrotate hook. Most systems already have this in place to handle their logs so it’s an easy add. If you’re using Apache or nginx you probably already have one that you can just add a prerotate hook to. For Caddy, something like this should be added.

sudo vi  /etc/logrotate.d/caddy

/var/log/caddy/access.log {
    daily
    rotate 7
    compress
    missingok
    prerotate
        goaccess \
        --persist --restore --db-path /var/lib/goaccess-db \
        --geoip-database /var/lib/goaccess-db/dbip-asn-lite-2025-01.mmdb  \
        --geoip-database /var/lib/goaccess-db/dbip-city-lite-2025-01.mmdb  \
        --geoip-database /var/lib/goaccess-db/dbip-country-lite-2025-01.mmdb \
        --log-format CADDY --output /var/www/some.server.org/report.html --log-file /var/log/caddy/access.log
    endscript
    postrotate
        systemctl restart caddy
    endscript
}

You can test this works with force option. If it works, you’ll be left with updated stats, an empty access.log, and a newly minted access.log.1.gz

sudo logrotate --force /etc/logrotate.d/caddy

For Caddy, you’ll also need to disable it’s built-in log rotation

sudo vi /etc/caddy/Caddyfile


log  { 
    output file /path/to/your/log.log {
        roll_disabled
    }
}

sudo systemctl restart caddy

Of course, this runs as root. You can design that out, but you can also configure systemd to trigger a timed execution and run as non-root. Arnaud Rebillout has some good info on that.

Real-Time Service

You can also run GoAccess as a service for real-time log analysis. It works in conjunction with your web server by providing a Web Socket to push the latest data to the browser.

In this example, Caddy is logging vhosts to individual files as described in the Caddy logging page. This is convenient as it allows you view vhosts separately which is often desired. Adjust as needed.

Prepare The System

# Create a system user with no home
sudo adduser --system --group --no-create-home goaccess

# Add the user to whatever group the logfiles are set to (ls -lah /var/log/caddy/ or whatever)
sudo adduser goaccess caddy

# Put the site in a variable to make edits easier
SITE=www.site.org

# Create a database location
sudo mkdir -p /var/lib/goaccess-db/$SITE
sudo chown goaccess:goaccess /var/lib/goaccess-db/$SITE

# Possibly create the report location
sudo touch /var/www/$SITE/report-$SITE.html
sudo chown goaccess /var/www/$SITE/report-$SITE.html

# Test that the goaccess user can create the report
sudo -u goaccess goaccess \
    --log-format CADDY \
    --output /var/www/$SITE/report-$SITE.html \
    --persist \
    --restore \
    --db-path /var/lib/goaccess-db/$SITE \
    --log-file /var/log/caddy/$SITE.log    

Create The Service

sudo vi /etc/systemd/system/goaccess.service

[Unit]
Description=GoAccess Web log report
After=network.target

# Note: Variable braces are required for sysmtemd variable expansion
[Service]
Environment="SITE=www.site.org"
Type=simple
User=goaccess
Group=goaccess
Restart=always
ExecStart=/usr/bin/goaccess \
    --log-file /var/log/caddy/${SITE}.log \
    --log-format CADDY \
    --persist \
    --restore \
    --db-path /var/lib/goaccess-db/${SITE} \
    --output /var/www/${SITE}/report-${SITE}.html \
    --real-time-html
StandardOutput=null
StandardError=null

[Install]
WantedBy=multi-user.target

sudo systemctl enable --now goaccess.service
sudo systemctl status goaccess.service
sudo journalctl -u  goaccess.service

If everything went well, it should be listening on the default port of 7890

nc -zv localhost 7890
localhost [127.0.0.1] 7890 (?) open

BUT you can’t access that port unless you’re on the same LAN. You can start forwarding that port and even setup SSL in the WS config, but in most cases it’s better to handle it with a proxy.

Configure Access via Proxy

To avoid adding additional port forwarding you can convert the websocket connection from a high-level port to a proxy path. This works with cloudflare as well.

Edit your GoAccess service unit to indicate the proxied URL.

ExecStart
    ...
    # Note the added escape slash the the formerly last line
    --real-time-html \  
    --ws-url wss://www.site.org:443/ws/goaccess 
    # If you don't add the port explicitly, GoAccess
    # will 'helpfully' add the internal port (which isn't helpful), silently.

Add a proxy line to your web server. If using Caddy, add a path handler and proxy like this, and restart.

some.server.org {
        ...
        ...
        handle_path /ws/goaccess* {
                reverse_proxy * http://localhost:7890
        }
}

Take your browser to the URL, and you should see the gear icon top left now has a green dot under it.

If the dot isn’t green you’re not connected so take a look at the troubleshooting section.

Create Multiple Services vs Multiple Reports

When you have lots of vhosts its useful to separate them at the log level and report separately. To do that you can use a systemd template so as to create multiple instances. Arnaud Rebillout has some details on that.

But scaling that becomes onerous. My preference is to automate report generation more frequently and skip the realtime.

Troubleshooting

No Data

Check the permissions on the files. If you accidentally typed caddy start as root it will be running as root and later runs may not be able to write log entries.

GoAccess Isn’t Live

Your best bet is to open the developer tools in your browser, check the network tab and refresh. If proxying is wrong it will give you some hints.

What About The .conf

The file /etc/goaccess/goaccess.conf can be used, just make sure to remove the arguments from the unit file so there isn’t a conflict.

Web Socket Intermittent

Some services, such as Cloudflare, support WS but can cause intermittent disconnections. Try separating your stats from your main traffic site.