On Caddy
There are two things you need to look at; detecting aborted connections and trusted proxies. Connections that you abort aren’t an established HTTP Status code, so you’ll need to tell CrowdSec about it. And trusted proxies require blocking at a higher level in the stack.
Detecting Aborted Connections From Probes
If you’ve secured caddy by aborting invalid requests they will be recorded in the log as “status 0”.
grep '"status":0' /var/log/caddy/access.log | tail -1 > aborts.log
sudo cscli explain -v --file ./aborts.log --type caddy
├ s01-parse
| └ 🟢 crowdsecurity/caddy-logs (+14 ~2)
| └ update evt.Stage : s01-parse -> s02-enrich
| └ ...
| └ ...
| └ create evt.Meta.http_status : 0
Crowdsec is parsing correctly, but none of the scenarios are looking for that status code.
sudo grep evt.Meta.http_status /etc/crowdsec/scenarios/*
...
...
# a bunch of thing looking for 401 and such
We just need to let CrowdSec know we’re doing this by adding a scenario.
sudo vi /etc/crowdsec/scenarios/custom-http-probe-aborted.yaml
type: leaky
name: custom/http-probe-aborted
description: "Detect http probes that caddy aborted"
filter: "evt.Meta.log_type == 'http_access-log' && evt.Meta.http_status == '0'"
groupby: evt.Meta.source_ip
leakspeed: 10s
capacity: 5
blackhole: 1m
labels:
remediation: true
classification:
- attack.T1595
behavior: "http:scan"
label: "HTTP Probing"
service: http
sudo systemctl restart crowdsec
Make a request and check metrics to see it.
# Make a request for just the IP address so it doesn't match a domain
curl -D - http://123.123.123.123
curl: (52) Empty reply from server
# Check metrics to see if it poured (matched a scenario)
sudo cscli metrics
...
...
╭────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ Scenario Metrics │
├──────────────────────────────────────┬───────────────┬───────────┬──────────────┬────────┬─────────┤
│ Scenario │ Current Count │ Overflows │ Instantiated │ Poured │ Expired │
├──────────────────────────────────────┼───────────────┼───────────┼──────────────┼────────┼─────────┤
│ crowdsecurity/http-crawl-non_statics │ 1 │ - │ 1 │ 1 │ - │
│ custom/http-probe-aborted │ 1 │ - │ 1 │ 1 │ - │
╰──────────────────────────────────────┴───────────────┴───────────┴──────────────┴────────┴─────────╯
# Check alerts - there won't be any with just one hit so far
sudo cscli alerts list
No active alerts
# Run a series of probes
for X in {1..10}; do curl -D - http://123.123.123.123/$X;done
sudo cscli alert list
╭────┬──────────────────┬───────────────────────────┬─────────┬────────────────────┬───────────┬──────────────────────╮
│ ID │ value │ reason │ country │ as │ decisions │ created_at │
├────┼──────────────────┼───────────────────────────┼─────────┼────────────────────┼───────────┼──────────────────────┤
│ 64 │ Ip:some.ip.addr │ custom/http-probe-aborted │ US │ 1234 SOME NAME │ ban:1 │ 2026-02-05T15:04:11Z │
╰────┴──────────────────┴───────────────────────────┴─────────┴────────────────────┴───────────┴──────────────────────╯
sudo cscli decision list
╭────────┬──────────┬──────────────────┬───────────────────────────┬────────┬─────────┬────────────────────┬────────┬────────────┬──────────╮
│ ID │ Source │ Scope:Value │ Reason │ Action │ Country │ AS │ Events │ expiration │ Alert ID │
├────────┼──────────┼──────────────────┼───────────────────────────┼────────┼─────────┼────────────────────┼────────┼────────────┼──────────┤
│ 709809 │ crowdsec │ Ip:some.ip.addr │ custom/http-probe-aborted │ ban │ US │ 1234 SOME NAME │ 6 │ 3h59m40s │ 64 │
╰────────┴──────────┴──────────────────┴───────────────────────────┴────────┴─────────┴────────────────────┴────────┴────────────┴──────────╯
# And un-ban yourself
sudo cscli decision delete --id 709809
Trusted Proxy
If you’re using a proxy, you’ve probably already configured caddy to log the correct end-user IP address. This will show up in your logs as the difference between remote_ip and client_ip. This is a good time to do that if you haven’t already.
sudo tail -1 /var/log/caddy/site.some.org.log | jq '.request'
# Just the relevant bits pasted in - the first two should be different if your trusted proxy setup is working
"remote_ip": "172.68.70.128",
"client_ip": "74.7.175.138",
"X-Forwarded-For": [
"74.7.175.138"
],
"Cf-Connecting-Ip": [
"74.7.175.138"
],
Cloudflare filters most bad actors for you, but many make it though. Here’s a sample of what it looks like if you haven’t set up trusted proxies. You can see that CrowdSec took action, but it’s a broad brush. It’s blocking the Cloudflare exit node and so removed everyone’s access, not just the attacker.
sudo cscli alert list
╭─────┬────────────────────┬───────────────────────────────────┬─────────┬────────────────────────┬───────────┬─────────────────────────────────────────╮
│ ID │ value │ reason │ country │ as │ decisions │ created_at │
├─────┼────────────────────┼───────────────────────────────────┼─────────┼────────────────────────┼───────────┼─────────────────────────────────────────┤
│ 221 │ Ip:162.158.49.136 │ crowdsecurity/jira_cve-2021-26086 │ IE │ 13335 CLOUDFLARENET │ ban:1 │ 2025-01-22 15:14:34.554328601 +0000 UTC │
│ 187 │ Ip:128.199.182.152 │ crowdsecurity/jira_cve-2021-26086 │ SG │ 14061 DIGITALOCEAN-ASN │ ban:1 │ 2025-01-19 20:50:45.822199509 +0000 UTC │
│ 186 │ Ip:46.101.1.225 │ crowdsecurity/jira_cve-2021-26086 │ GB │ 14061 DIGITALOCEAN-ASN │ ban:1 │ 2025-01-19 20:50:41.699518104 +0000 UTC │
│ 181 │ Ip:162.158.108.104 │ crowdsecurity/http-bad-user-agent │ SG │ 13335 CLOUDFLARENET │ ban:1 │ 2025-01-19 12:39:20.468268327 +0000 UTC │
│ 180 │ Ip:172.70.208.61 │ crowdsecurity/http-bad-user-agent │ SG │ 13335 CLOUDFLARENET │ ban:1 │ 2025-01-19 12:38:36.664997131 +0000 UTC │
╰─────┴────────────────────┴───────────────────────────────────┴─────────┴────────────────────────┴───────────┴─────────────────────────────────────────╯
But even though caddy identifies the correct client_ip, still have a problem. Crowdsec dutifully adds the client_ip to the block list, but their attacks continue. The bouncer can’t separate the good traffic from the bad because it’s layer 4 and only sees the Cloudflare IP.
for X in {1..100}; do curl -D - https://proxied.your.org/$X; done
allen@www:~$ sudo cscli alert list
# Even though you're listed, you can continue probing
╭────┬───────────────────┬───────────────────────────────────┬─────────┬──────────────────────────────┬───────────┬──────────────────────╮
│ ID │ value │ reason │ country │ as │ decisions │ created_at │
├────┼───────────────────┼───────────────────────────────────┼─────────┼──────────────────────────────┼───────────┼──────────────────────┤
│ 66 │ Ip:212.56.42.20 │ crowdsecurity/http-cve-2021-41773 │ US │ 40021 CONTABO-40021 │ ban:1 │ 2026-02-05T16:02:43Z │
│ 65 │ Ip:34.133.180.181 │ crowdsecurity/http-probing │ US │ 396982 GOOGLE-CLOUD-PLATFORM │ ban:1 │ 2026-02-05T15:38:37Z │
│ 64 │ Ip:some.ip.addr │ custom/http-probe-aborted │ US │ 1234 SOME NAME │ ban:1 │ 2026-02-05T15:04:11Z │
╰────┴───────────────────┴───────────────────────────────────┴─────────┴──────────────────────────────┴───────────┴──────────────────────╯
The ideal approach is to tell Cloudflare to stop forwarding traffic from the bad actors. There is a cloudflare-bouncer to do just that. It’s rate limited however, and only suitable for premium clients. There is also the CrowdSec Cloudflare Worker. It’s better, but still suffers from limits for non-premium clients.
Caddy Bouncer
Instead, we’ll use the caddy-crowdsec-bouncer. This is a layer 7 (protocol level) bouncer. It works inside Caddy and will block IPs based on the client_ip from the HTTP request.
The fist step is to tell the Local API that we’re going to add a bouncer and generate a key for it. If you’re running the LAPI on your router, you’d do this there.
# I named this after the hostname 'www' but you can use anything you like
sudo cscli bouncers add www-bouncer
Add the module to Caddy
sudo caddy add-package github.com/hslatman/caddy-crowdsec-bouncer
Configure Caddy
#
# Global Options Block
#
{
# Format like this or the syntax will fail.
crowdsec {
api_key 2WXxY440sLxjapyQKlvp6ILdLA/hn/dhoG1ML82+xiw
# Only needed if your LAPI is remote, like on your router
api_url http://192.168.1.1:8080
}
order crowdsec first
}
www.some.org {
crowdsec
root * /var/www/www.some.org
file_server
}
And restart. Make sure you restart as a reload will fail.
sudo systemctl restart caddy.service
#You should see something like this is your system journal
sudo journalctl | grep caddy | grep crowdsec
Feb 05 16:42:23 www caddy[10724]: {"level":"info","ts":1770309743.418141,"logger":"crowdsec","msg":"started","instance_id":"155c21c7"}
Testing Mitigation
Let’s test that probe again. Initially, you’ll get a 404 (not found) but after while of that, it should switch to 403 (access denied)
for X in {1..100}; do curl -D - --silent https://proxied.your.org/$X | grep HTTP; done
HTTP/2 404
HTTP/2 404
...
...
HTTP/2 403
HTTP/2 403
Conclusion
Congrats! after much work you’ve traded 404s for 403s. Was it worth it? Probably. If an adversary’s probe had a chance to find something, it has less of a chance now.
Bonus Spoofing Section
Back when we set up a trusted proxy for caddy we explicitly used the CF-Connecting-IP header. This is because the X-Forwarded-For header is easily spoofed. If you want to give that a test, comment out that line in your caddy config and reload.
sudo vi /etc/caddy/Caddyfile
# Comment out 'client_ip_headers CF-Connecting-IP' in the servers block
sudo systemctl reload caddy
for X in {1..100}; do curl -D - --silent "X-Forwarded-For: 192.168.0.2" https://www.some.org/$X | grep HTTP;done
HTTP/2 404
HTTP/2 404
...
...
HTTP/2 404
HTTP/2 404
No remediation happens. Turns out Cloudflare appends by default, giving you:
sudo tail -f /var/log/caddy/www.some.org.log | jq
"client_ip": "192.168.0.2",
"X-Forwarded-For": [
"192.168.0.2,109.206.128.45"
],
Caddy only uses the X-Forwarded-For when it trusts the upstream proxy, but it takes the first value, which is rather trusting but canonically correct. That’s why we use the CF-Connecting-IP header as francislavoie suggests, as it will always be replaced and not appended.
Adjusting Cloudflare
You don’t need to, but you can configure Cloudflare to “Remove visitor IP headers”. This is counterintuitive, but the notes say “…Cloudflare will only keep the IP address of the last proxy”. In testing, it keeps the last value in the X-Forwarded-For string, and that’s what we’re after. It works for normal and forged headers.
This adds an extra layer of protection - but it’s not clear if it’s worth it. It may add to the proxy time, so use at your own discretion.
- Log in to the Cloudflare dashboard and select your website
- Go to Rules > Overview
- Select “Manage Request Header Transform Rules”
- Select “Managed Transforms”
- Enable Remove visitor IP headers
The Overview page may look different depending on your plan, so you may have to hunt around for this setting.
Now when you test, you’ll get access denied regardless of your header
for X in {1..100}; do curl -D - --silent "X-Forwarded-For: 192.168.0.2" https://www.some.org/$X | grep HTTP;done
HTTP/2 404
HTTP/2 404
...
...
HTTP/2 403
HTTP/2 403
Troubleshooting
If you’re generating lots of overflows but no alerts, check your caddy logs for an error like this
level=error msg=“while pushing to api : failed sending alert to LAPI: API error: invalid character ‘\x1f’ looking for beginning of value”
This happens when your client is 1.7 but your LAPI is 1.6, as is the situation right now with OpenWrt if that’s what you’re using as the local hub. You can reverse the roles and use just a bouncer on the router, or install a true multi-server setup with a dedicated hub. You can also just copy the binaries around.
# On OpenWrt
mv /usr/bin/crowdsec /usr/bin/crowdsec.orig
mv /usr/bin/crowdsec-cli /usr/bin/crowdsec-cli
scp -T [email protected]:'/usr/bin/crowdsec /usr/bin/cscli' /usr/bin/
service crowdsec restart
Feedback
Was this page helpful?
Glad to hear it! Please tell us how we can improve.
Sorry to hear that. Please tell us how we can improve.