Caddy

• 1: Install
• 2: Configure
• 3: Wildcard Sites
• 4: Log
• 5: Secure
• 6: WebDAV
• 7: MFA
• 8: Cloudflare

1 - Install

Installation

Caddy recommends “using our official package for your distro” and for debian flavors they include the basic instructions you’d expect. I recommend the testing build for the short-term there’s an bug with loging wild-cards sites (like *.you.org) that are handy to have available.

sudo apt install -y debian-keyring debian-archive-keyring apt-transport-https curl
curl -1sLf 'https://dl.cloudsmith.io/public/caddy/testing/gpg.key' | sudo gpg --dearmor -o /usr/share/keyrings/caddy-testing-archive-keyring.gpg
curl -1sLf 'https://dl.cloudsmith.io/public/caddy/testing/debian.deb.txt' | sudo tee /etc/apt/sources.list.d/caddy-testing.list
chmod o+r /usr/share/keyrings/caddy-testing-archive-keyring.gpg
chmod o+r /etc/apt/sources.list.d/caddy-testing.list
sudo apt update
sudo apt install caddy

System Alternatives

If you ever add a module to caddy, chances are it will be lost when you next apt upgrade. This is because caddy phones home, requests a build for any non-standard module, and replaces itself. The devs don’t think this will be an issue. In any event, you should prepare for it, because you will be adding modules.

Because we’ve diverted, apt upgrade will not upgrade caddy. You must use caddy upgrade instead. Since this stops unattended-updates from any automation of vulnerability patching, maybe add a cron job if you like.

sudo dpkg-divert --divert /usr/bin/caddy.default --rename /usr/bin/caddy
sudo mv ./caddy /usr/bin/caddy.custom
sudo update-alternatives --install /usr/bin/caddy caddy /usr/bin/caddy.default 10
sudo update-alternatives --install /usr/bin/caddy caddy /usr/bin/caddy.custom 50
sudo systemctl restart caddy

2 - Configure

Configure

The easiest way to configure Caddy is by editing the Caddyfile

sudo vi /etc/caddy/Caddyfile

Sites

You define websites with a block that includes a root and the file_server directive. Once you reload, and assuming you already have the DNS in place, Caddy will reach out to Let’s Encrypt, acquire a certificate, and automatically redirect from port 80 to 443

site.your.org {        
    root * /var/www/site.your.org
    file_server
}

Authentication

You can add basic auth to a site by creating a hash and adding a directive to the site.

caddy hash-password

site.your.org {        
    root * /var/www/site.your.org
    file_server
    basic_auth { 
        allen SomeBigLongStringFromTheCaddyHashPasswordCommand
    }
}

Reverse Proxy

Caddy also makes a decent reverse proxy.

site.your.org {        
    reverse_proxy * http://some.server.lan:8080
}

You can also take advantage of path-based reverse proxy. Note the rewrite to accommodate the trailing-slash potentially missing.

site.your.org {
    rewrite /audiobooks /audiobooks/
    handle_path /audiobooks/* {
        uri strip_prefix /audiobooks/
        reverse_proxy * http://some.server.lan:8080
    }
}

Import

You can define common elements at the top (snippets) or in files and import them multiple times to save duplication. This helps when you have many sites.

# At the top in the global section of your Caddyfile
(logging) {
    log {
        output file /var/log/caddy/access.log
    }
}
site.your.org {
    import logging     
    reverse_proxy * http://some.server.lan:8080
}

Modules

Caddy is a single binary so when adding a new module (aka feature) you are actually downloading a new binary that has it compiled in. You can find the list of packages at their download page.

Do this at the command line with caddy itself.

sudo caddy add-package github.com/mholt/caddy-webdav
systemctl restart caddy

Now you can add the webdav directive to a site. You can have the same both serve up a directory listing and handle webdav with a litte extra config. The new module uses different ‘verbs’ when working, allowing it to cooexist as long as file_server gets the first shot.

# Global Option Block
{
    order webdav last
}

site.your.org {        

    root * /var/www/site.your.org

    file_server browse

    webdav *
}

Troubleshooting

You can test your config file and look at the logs like so

caddy validate --config /etc/caddy/Caddyfile
journalctl --no-pager -u caddy

3 - Wildcard Sites

Caddy has an individual cert for every virtual host you create. This is fine, but Let’s Encrypt publishes these as part of certificate transparency and the bad guys are watching. If you create a new site in caddy, you’ll see bots probing for weaknesses within 30 min - without you even having published the URL. There’s no security in anonymity, but need-to-know suggests we shouldn’t be informing the whole world about sites of limited scope.

One solution is a wildcard cert. It’s published as just ‘*.some.org’ so there’s no information disclosed. These are usally done with DNS integration. Caddy supports this and it requires a little extra work, but it does make it a lot easier to sort out the riff-raff later.

Installation

We’ll connect to CloudFlare’s DNS service. Check https://github.com/caddy-dns to see if your DNS provider is available. If you haven’t already configured a system-alternative for caddy, now is the time to do so.

sudo caddy add-package github.com/caddy-dns/cloudflare
sudo systemctl restart caddy.service    

DNS Provider Configuration

For Cloudflare, a decent example is below. Just use the ‘Getting the Cloudflare API Token’ part

https://roelofjanelsinga.com/articles/using-caddy-ssl-with-cloudflare/

Caddy Configuration

Use the acme-dns global option and then create a single site (used to determine the cert) and match the actual vhosts with subsites.

{
    acme_dns cloudflare alotcharactersandnumbershere
}

*.some.org, some.org {

    @site1 host site1.some.org
    handle @site1 {
        reverse_proxy * http://localhost:3200
    }

    @site2 host site2.some.org
    handle @site2 {
        root * /srv/www/site2
    }
}

4 - Log

Access Logs

Simple Logs

In general, you should create a snippet and import into each block.

#
# Global Options Block
#
{
    ...
    ...
}
#
# Importable Snippets
#
(logging) {
    log {
        output file /var/log/caddy/access.log
    }
}
#
# Host Blocks
#
site.your.org {
    import logging     
    reverse_proxy * http://some.server.lan:8080
}
other.your.org {
    import logging     
    reverse_proxy * http://other.server.lan:8080
}

Per VHost Logs

If you want separate logs for each vhosts you can still use a snippit by adding a parameter for the site name.

#
# Global Options Block
#
{
    ...
    ...
}
#
# Importable Snippets
#
(logging) {
    log {
        output file /var/log/caddy/{args[0]}.log        
}
#
# Host Blocks
#
site.your.org {
    # Add the argument "site.your.org" at the end sets the log file name to "site.your.org.log"
    import logging site.your.org
    reverse_proxy * http://some.server.lan:8080
}
other.your.org {
    import logging other.your.org    
    reverse_proxy * http://other.server.lan:8080
}

Wildcard Sites

Wildcard sites only have one block but you must use the hostname matcher to separate logs. Note the slight difference in the handling of log names.

#
# Global Options Block
#
{
    ...
    ...
}
#
# Importable Snippets
#
(logging) {
    log {
        output file /var/log/caddy/{args[0]}       
}
#
# Main Block
#
*.site.org, site.org {

        # Everything goes to the file named here (access.log) unless it's filtered out by a more specific log block
        import logging access.log

        # This site will write to the access.log 
        @site host other.site.org
        handle @site {
                reverse_proxy * http://other.site
        }

        # This site will write to just this log file. Make sure to import logging BEFORE the '@www host' matcher or requests don't get logged here.
        import logging www.site.org.log 
        @www host www.site.org                
        handle @www { 
                root * /var/www/www.site.org 
                file_server 
        }

Logging Credentials

If you want to track users that have authenticated, add a directive to the global headers so their usernames are included in the request log.

# 
# Global Options Block 
# 
{        
        servers { 
                log_credentials 
        }
}

File Permissions

By default, only caddy can read the log files. This is a problem when you have a log analysis package. In recent versions of caddy however, you can set the mode.

    log {
        output file /var/log/caddy/access.log {
                mode 644
        }
    }

If the log file doesn’t change modes, check the version of caddy. It must be newer than v2.8.4 for the change and you must restart.

Troubleshooting

You can have a case where the domain specific file never gets created. This usually happens when there us nothing to write to it. Check the hostname is correct.

5 - Secure

Web servers are constantly under attack. So much so some people way back before Y2K termed this “internet background radiation”.

There are two broad categories of attacks; ones that are just probing IPs, and ones that know a little about your web presence.

If an adversary just connects to your IP without knowing the domain, you can:

• Return an error
• Abort the request

The default configuration, and one that is consistent with the protocol, is to return an error. But the general consensus is to drop invalid requests. This prevents adversaries from profiling the system.

If they know your domains it more difficult. But they are usually asking for things that don’t exist, or trying to access a login page as fast as possible to guess passwords. In either case, you should apply a limit to how fast they can make requests.

So as a fist step, let’s drop or rate-limit every request.

Drop Unknown Domains

A default config will take connections to port 80, announce that it’s a Caddy web server and redirect you to https. Instead, let’s look at the host headers and SNI (Server Name Indicator) first. Then we can immediately abort any requests for things we don’t even have.

#
# Global Options Block
#
{
    ...
    ...
}
#
# Importable Sections
#
...
...

#
# Site Blocks
#

# Accept secure requests specific to our domain
https://*.some.org, https://some.org {

        # Serve any known host
        @some host some.site.org
        handle @some {
                encode
                reverse_proxy * http://some:8080
        }

        @other host other.site.org
        handle @other {
                import auth
                encode
                reverse_proxy * http://other:8080
        }

        # Abort any unknown host
        handle { 
                abort 
        }
}

# Accept any insecure request to our domain 
http://*.some.org, http://some.org {
        
        # Redirect only known host headers to https
        @secure {
                host some.site.org
                host other.site.org                
        }
        handle @secure {
                redir https://{host}{uri} permanent
        }

        # Abort anything else.
        handle {
                abort
        }
}

# Handle requests without the right domain or SNI
# so we can abort them early
:80, :443 {

        # Log it so we have a record of the probe
        import logging

        abort
}

Monitor Valid Domains

Other attackers will connect to a valid domain and making requests for potential exploits. You’ll send them back 401 Unauthorized or 404 Not Found, but these are bots and they’ll just keep chugging along, ignoring the errors.

Rate Limiting

A basic protection is to rate-limit your responses. You’re already aborting requests for invalid sites, but if you’re hosting a web form or basic-auth protected site it can be useful. While a web app using a form may have it’s own rate limiting mechanism, there’s no inherent limit to the BA prompt. The only way to handle this is apply a rate limit to the specific site.

sudo caddy add-package github.com/mholt/caddy-ratelimit

...
...
https://*.some.org, https://some.org {

    # A fairly generous over-all limit
    rate_limit { 
        zone descriptive-name-1 {
            key {http.request.remote_ip} 
                events 200
                window 5s 
            }
    }

    @some host some.site.org
    handle @some {
        # A conservative one for this site
        rate_limit { 
            zone descriptive-name-1 {
                key {http.request.remote_ip} 
                    events 20 
                    window 5s 
                } 
            }
        encode
        reverse_proxy * http://other:8080
    }
}

You’ll need to tune this as different sites may have radically different setups, some needing just one large request and another 50 small ones to fill a page load. It’s also mostly targeted at mitigating denial of service and credential stuffing. A distributed or patient bot makes it through just fine. Importantly, if you’re using a proxy service like Cloudflare, you’ll need to pull out the actual client IP.

Deploy a Protection System

For actual threat detection & response, consider adding Fail2Ban or CrowdSec

6 - WebDAV

Caddy can also serve WebDAV requests with the appropriate module. This is important because for many clients, such as Kodi, WebDAV is significantly faster.

sudo caddy add-package github.com/mholt/caddy-webdav
sudo systemctl restart caddy

{   # Custom modules require order of precedence be defined
    order webdav last
}
site.your.org {
    root * /var/www/site.your.org
    webdav * 
}

You can combine WebDAV and Directly Listing - highly recommended - so you can browse the directory contents with a normal web browser as well. Since WebDAV doesn’t use the GET method, you can use the @get filter to route those to the file_server module so it can serve up indexes via the browse argument.

site.your.org {
    @get method GET
    root * /var/www/site.your.org
    webdav *
    file_server @get browse        
}

Sources

https://github.com/mholt/caddy-webdav https://marko.euptera.com/posts/caddy-webdav.html

7 - MFA

The package caddy-security offers a suite of auth functions. Among these is MFA and a portal for end-user management of tokens.

Installation

# Install a version of caddy with the security module 
sudo caddy add-package github.com/greenpau/caddy-security
sudo systemctl restart caddy

Configuration

/var/lib/caddy/.local/caddy/users.json

caddy hash-password

Troubleshooting

journalctl –no-pager -u caddy

8 - Cloudflare

Cloudflare offers and excellent free service that proxies your web server, removing you from the front lines of the internet and blocking many known bad-actors. That deserves it’s own section, but we should make changes to caddy’s logging to accommodate the proxy. Most specifically, to log the actual IP address of the end user.

Add a Trusted Proxy

Cloudflare will inject the header CF-Connecting-IP into the request and we can use that to identify the end-user. This is preferred over other headers, which can be easily spoofed. But in order to make sure it’s Cloudfare doing the injecting, and not some internet rando, we need to explicitly know all the Cloudflare exit node IPs to trust them. That would be hard to manage, but happily, there’s a handy [caddy-cloudflare-ip] module for that. Many thanks to WeidiDeng!

sudo caddy add-package github.com/WeidiDeng/caddy-cloudflare-ip

sudo vi /etc/caddy/Caddyfile

#
# Global Options Block
#
{
        servers {             
                trusted_proxies cloudflare  
                client_ip_headers CF-Connecting-IP  
        }    
}

After restarting Caddy, we can see the header change

sudo head /var/log/caddy/access.log  | jq '.request'
sudo tail /var/log/caddy/access.log  | jq '.request'

Before

  "remote_ip": "172.68.15.223",
  "client_ip": "172.68.15.223",

After

  "remote_ip": "172.71.98.114",
  "client_ip": "109.206.128.45",

You can now update any rate_limit from the remote_ip to the client_ip

        rate_limit { 
            zone descriptive-name-1 {
                key {http.request.client_ip} # Uses the real client IP
                    events 20 
                    window 5s 
                } 
            }