Inspect Parsing
The reason why things parse or don’t parse is sometimes worth getting into and you can have Crowdsec explain it to you.
Parsed vs Unparsed
SSH authentication is interesting, because not all lines parse.
sudo cscli metrics show acquisition
╭───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ Acquisition Metrics │
├─────────────────────────────────────────────────┬────────────┬──────────────┬────────────────┬────────────────────────┬───────────────────┤
│ Source │ Lines read │ Lines parsed │ Lines unparsed │ Lines poured to bucket │ Lines whitelisted │
├─────────────────────────────────────────────────┼────────────┼──────────────┼────────────────┼────────────────────────┼───────────────────┤
│ journalctl:journalctl-_SYSTEMD_UNIT=ssh.service │ 9 │ 5 │ 4 │ - │ 5 │
╰─────────────────────────────────────────────────┴────────────┴──────────────┴────────────────┴────────────────────────┴───────────────────╯
sudo cscli explain --dsn "journalctl://filters=_SYSTEMD_UNIT=ssh.service" --type syslog
# This line is just the server starting - it says 'success' but in the sense of 'successfully bailed out'
# before it completed parsing because it detected the 0.0.0.0 address. It will be counted as unparsed in the metrics.
line: Feb 02 15:04:22 www sshd[180]: Server listening on 0.0.0.0 port 22.
├ s00-raw
| ├ 🟢 crowdsecurity/syslog-logs (+12 ~8)
| └ 🔴 crowdsecurity/non-syslog
├ s01-parse
| ├ 🔴 crowdsecurity/caddy-logs
| ├ 🔴 crowdsecurity/sshd-logs
| └ 🔴 crowdsecurity/sshd-success-logs
└-------- parser success, ignored by whitelist (private ipv4/ipv6 ip/ranges) 🟢
# Here's me failing to log in. This was successfully parsed and made it to step 2. That's where crowdsec
# starts pulling out bits of the log message for further inspection.
#
# The numbers at the end of the line such as (+9 ~1) signify that the parser added 9 attributes
# (pieces of data) to the event, parsed from the raw log message, and updated 1.
#
# Now that it has more data, the step 2 'whitelists' get another go and see that I'm attacking
# from a private IP range. They flag it as such and it stops processing it before it gets further.
# Turns out private IPs are whitelisted by default so you can't lock yourself out.
#
# This entry is counted as parsed and whitelisted
line: Feb 06 18:12:30 www sshd-session[12643]: Failed password for allen from ::1 port 36082 ssh2
├ s00-raw
| ├ 🟢 crowdsecurity/syslog-logs (+12 ~8)
| └ 🔴 crowdsecurity/non-syslog
├ s01-parse
| ├ 🔴 crowdsecurity/caddy-logs
| ├ 🟢 crowdsecurity/sshd-logs (+9 ~1)
| └ 🔴 crowdsecurity/sshd-success-logs
├ s02-enrich
| ├ 🟢 crowdsecurity/dateparse-enrich (+2 ~1)
| ├ 🔴 crowdsecurity/geoip-enrich
| ├ 🔴 crowdsecurity/http-logs
| ├ 🟢 crowdsecurity/public-dns-allowlist (unchanged)
| └ 🟢 crowdsecurity/whitelists (~2 [whitelisted])
└-------- parser success, ignored by whitelist (private ipv4/ipv6 ip/ranges) 🟢
If that attempt was from the outside, a few of them would lead to an alert.
Parsing and Scenarios
Almost all HTTP access logs are parsed as they almost always have data interesting to CrowdSec. Take a look to see what a full parsing and scenario progression looks like.
sudo cscli metrics show acquisition
╭───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ Acquisition Metrics │
├─────────────────────────────────────────────────┬────────────┬──────────────┬────────────────┬────────────────────────┬───────────────────┤
│ Source │ Lines read │ Lines parsed │ Lines unparsed │ Lines poured to bucket │ Lines whitelisted │
├─────────────────────────────────────────────────┼────────────┼──────────────┼────────────────┼────────────────────────┼───────────────────┤
│ file:/var/log/caddy/access.log │ 71 │ 71 │ - │ 63 │ - │
╰─────────────────────────────────────────────────┴────────────┴──────────────┴────────────────┴────────────────────────┴───────────────────╯
sudo tail -1 /var/log/caddy/access.log > ./logs
sudo cscli explain -v --file ./logs --type caddy
...
...
├ Scenarios
├ 🟢 crowdsecurity/http-crawl-non_statics
└ 🟢 crowdsecurity/http-generic-bf
# At the very bottom you'll see this line was feed to two different scenarios
Feedback
Was this page helpful?
Glad to hear it! Please tell us how we can improve.
Sorry to hear that. Please tell us how we can improve.