Whitelist
In the previous examples we’ve looked at the metrics and details of internal facing service like failed SSH logins. Those types aren’t prone to a lot of false positives. But other sources, like web access logs, can be.
False Positives
You’ll recall that when looking at metrics that a high number of ‘Lines unparsed’ is normal. They were simply entries that didn’t match any specific events the parser was looking for. Parsed lines however, are ‘poured’ to a bucket. A bucket being a potential attack type.
sudo cscli metrics
Acquisition Metrics:
╭────────────────────────────────┬────────────┬──────────────┬────────────────┬────────────────────────╮
│ Source │ Lines read │ Lines parsed │ Lines unparsed │ Lines poured to bucket │
├────────────────────────────────┼────────────┼──────────────┼────────────────┼────────────────────────┤
│ file:/var/log/auth.log │ 69 │ - │ 69 │ - │
│ file:/var/log/caddy/access.log │ 21 │ 21 │ - │ 32 │
│ file:/var/log/syslog │ 2 │ - │ 2 │ - │
╰────────────────────────────────┴────────────┴──────────────┴────────────────┴────────────────────────╯
Scenario Metrics:
╭──────────────────────────────────────┬───────────────┬───────────┬──────────────┬────────┬─────────╮
│ Scenario │ Current Count │ Overflows │ Instantiated │ Poured │ Expired │
├──────────────────────────────────────┼───────────────┼───────────┼──────────────┼────────┼─────────┤
│ crowdsecurity/http-crawl-non_statics │ - │ - │ 2 │ 17 │ 2 │
│ crowdsecurity/http-probing │ - │ 1 │ 2 │ 15 │ 1 │
╰──────────────────────────────────────┴───────────────┴───────────┴──────────────┴────────┴─────────╯
The 21 web requests where interesting to a couple scenarios. With 32 pours some where interesting to both.
The scenario http-crawl-non_statics is designed to allow some light web-crawling but http-probing is more sensitive and it overflowed into an alert.
Assuming this is related to a web application you’re trying to use, you just got blocked. Let’s take a look at the alert
allen@www:~$ sudo cscli alert list
╭─────┬────────────────────┬────────────────────────────────────────────┬─────────┬──────────────────────────────────┬───────────┬──────────────────────╮
│ ID │ value │ reason │ country │ as │ decisions │ created_at │
├─────┼────────────────────┼────────────────────────────────────────────┼─────────┼──────────────────────────────────┼───────────┼──────────────────────┤
│ 129 │ Ip:20.42.209.0 │ crowdsecurity/http-probing │ AU │ 8075 MICROSOFT-CORP-MSN-AS-BLOCK │ ban:1 │ 2026-02-06T20:33:26Z │
sudo cscli alerts inspect -d 129
...
...
- Context :
╭────────────┬───────────────────────────────────────────────────╮
│ Key │ Value │
├────────────┼───────────────────────────────────────────────────┤
│ method │ GET │
│ status │ 404 │
│ target_uri │ /0/icon/John │
│ target_uri │ /0/icon/Smith │
│ target_uri │ /0/icon/Adam │
│ target_uri │ /0/icon/Smith │
╰────────────┴───────────────────────────────────────────────────╯
...
...
target_fqdn │ app.some.org
You can see the application in question is trying to load some missing icons and just fails silently, That may be fine for the app, but it looks like someone crawling or probing the server. To fix it we’ll need to create a whitelist definition for the app.
Whitelist
To whitelist an app, we create a file with an expression that matches the behavior we see above, such as the apps attempts to load a file that doesn’t exist, and exempts it. You can only add these to the s02 stage folder and the name element but be unique for each.
sudo vi /etc/crowdsec/parsers/s02-enrich/some-app-whitelist.yaml
This example uses the startsWith expression and assumes that all requests start the same
name: you/some-app
description: "Whitelist 404s for icon requests"
whitelist:
reason: "icon request"
expression:
- evt.Parsed.request startsWith '/0/icon/'
If it’s less predictable, you can use a regular expression instead and combine with other expressions like a site match. In general, the more specific the better.
name: you/some-app-whitelist
description: "Whitelist 404s for icon requests"
whitelist:
reason: "icon request"
expression:
- evt.Parsed.request matches '^/[0-9]/icon/.*' && evt.Meta.target_fqdn == "some-app.you.org"
Now you can reload crowdsec and test
sudo systemctl restart crowdsec.service
sudo cscli explain -v --file ./test.log --type caddy
├ s00-raw
| ├ 🔴 crowdsecurity/syslog-logs
| └ 🟢 crowdsecurity/non-syslog (+5 ~8)
| └ update evt.ExpectMode : %!s(int=0) -> 1
| └ update evt.Stage : -> s01-parse
...
├ s02-enrich
| ├ 🟢 you/some-app-whitelist (~2 [whitelisted])
| ├ update evt.Whitelisted : %!s(bool=false) -> true
| ├ update evt.WhitelistReason : -> some icon request
| ├ 🟢 crowdsecurity/dateparse-enrich (+2 ~2)
...
...
| ├ 🟢 crowdsecurity/http-logs (+7)
| └ 🟢 crowdsecurity/whitelists (unchanged)
└-------- parser success, ignored by whitelist (icon request) 🟢
You’ll see in the above example, we successfully parsed the entry, but it was ‘ignored’ and didn’t go on to the Scenario section.
Troubleshooting
New Whitelist Has No Effect
If you have more than one whitelist, check the name you gave it on the first line. If that’s not unique, the whole thing will be silently ignored.
Regular Expression Isn’t Matching
CrowdSec uses the go-centric expr-lang. You may be used to unix regex where you’d escape slashes, for example. A tool like https://www.akto.io/tools/regex-tester is helpful.
Feedback
Was this page helpful?
Glad to hear it! Please tell us how we can improve.
Sorry to hear that. Please tell us how we can improve.