Detailed Activity
Inspecting Metrics
Data comes in through the parsers. To see what they are doing, let’s take a look at the Acquisition and Parser metrics.
sudo cscli metrics
Most of the ‘Acquisition Metrics’ lines will be read and unparsed. This is because normal events are dropped. It only considers lines parsed if they were passed on to a scenario. The ‘bucket’ column refers to event scenarios and is also blank as there were no parsed lines to hand off.
Acquisition Metrics:
╭────────────────────────┬────────────┬──────────────┬────────────────┬────────────────────────╮
│ Source │ Lines read │ Lines parsed │ Lines unparsed │ Lines poured to bucket │
├────────────────────────┼────────────┼──────────────┼────────────────┼────────────────────────┤
│ file:/var/log/auth.log │ 216 │ - │ 216 │ - │
│ file:/var/log/syslog │ 143 │ - │ 143 │ - │
╰────────────────────────┴────────────┴──────────────┴────────────────┴────────────────────────╯
The ‘Parser Metrics’ will show the individual parsers - but not all of them. Only parsers that have at least one ‘hit’ are shown. In this example, only the syslog parser shows up. It’s a low-level parser that doesn’t look for matches, so every line is a hit.
Parser Metrics:
╭─────────────────────────────────┬──────┬────────┬──────────╮
│ Parsers │ Hits │ Parsed │ Unparsed │
├─────────────────────────────────┼──────┼────────┼──────────┤
│ child-crowdsecurity/syslog-logs │ 359 │ 359 │ - │
│ crowdsecurity/syslog-logs │ 359 │ 359 │ - │
╰─────────────────────────────────┴──────┴────────┴──────────╯
However, try a couple failed SSH login attemps and you’ll see them and how they feed up the the Acquistion Metrics.
Acquisition Metrics:
╭────────────────────────┬────────────┬──────────────┬────────────────┬────────────────────────╮
│ Source │ Lines read │ Lines parsed │ Lines unparsed │ Lines poured to bucket │
├────────────────────────┼────────────┼──────────────┼────────────────┼────────────────────────┤
│ file:/var/log/auth.log │ 242 │ 3 │ 239 │ - │
│ file:/var/log/syslog │ 195 │ - │ 195 │ - │
╰────────────────────────┴────────────┴──────────────┴────────────────┴────────────────────────╯
Parser Metrics:
╭─────────────────────────────────┬──────┬────────┬──────────╮
│ Parsers │ Hits │ Parsed │ Unparsed │
├─────────────────────────────────┼──────┼────────┼──────────┤
│ child-crowdsecurity/sshd-logs │ 61 │ 3 │ 58 │
│ child-crowdsecurity/syslog-logs │ 442 │ 442 │ - │
│ crowdsecurity/dateparse-enrich │ 3 │ 3 │ - │
│ crowdsecurity/geoip-enrich │ 3 │ 3 │ - │
│ crowdsecurity/sshd-logs │ 8 │ 3 │ 5 │
│ crowdsecurity/syslog-logs │ 442 │ 442 │ - │
│ crowdsecurity/whitelists │ 3 │ 3 │ - │
╰─────────────────────────────────┴──────┴────────┴──────────╯
Lines poured to bucket however, is still empty. That means the scenaros decided it wasn’t a hack attempt. With SSH timeouts it actually hard to do without a tool. Plus, you may notice the ‘whitelist` was triggered. Private IP ranges are whilelisted by default so you can’t lock yourself out from inside.
Let’s ask crowdsec to explain what’s going on
Detailed Parsing
To see which parsers got involved and what they did, you can ask.
sudo cscli explain --file /var/log/auth.log --type syslog
Here’s a ssh example of a failed login. The numbers, such as (+9 ~1), mean that the parser added 9 elements it parsed from the raw event, and updated 1. Notice the whitelists parser at the end. It’s catching this event and dropping it, hense the ‘parser failure’
line: Mar 1 14:08:11 www sshd[199701]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=192.168.1.16 user=allen
├ s00-raw
| └ 🟢 crowdsecurity/syslog-logs (first_parser)
├ s01-parse
| └ 🟢 crowdsecurity/sshd-logs (+9 ~1)
├ s02-enrich
| ├ 🟢 crowdsecurity/dateparse-enrich (+2 ~1)
| ├ 🟢 crowdsecurity/geoip-enrich (+9)
| └ 🟢 crowdsecurity/whitelists (~2 [whitelisted])
└-------- parser failure 🔴
Why exactly did it get whitelisted? Let’s ask for a verbose report.
sudo cscli explain -v --file /var/log/auth.log --type syslog
line: Mar 1 14:08:11 www sshd[199701]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=192.168.1.16 user=someGuy
├ s00-raw
| └ 🟢 crowdsecurity/syslog-logs (first_parser)
├ s01-parse
| └ 🟢 crowdsecurity/sshd-logs (+9 ~1)
| └ update evt.Stage : s01-parse -> s02-enrich
| └ create evt.Parsed.sshd_client_ip : 192.168.1.16
| └ create evt.Parsed.uid : 0
| └ create evt.Parsed.euid : 0
| └ create evt.Parsed.pam_type : unix
| └ create evt.Parsed.sshd_invalid_user : someGuy
| └ create evt.Meta.service : ssh
| └ create evt.Meta.source_ip : 192.168.1.16
| └ create evt.Meta.target_user : someGuy
| └ create evt.Meta.log_type : ssh_failed-auth
├ s02-enrich
| ├ 🟢 crowdsecurity/dateparse-enrich (+2 ~1)
| ├ create evt.Enriched.MarshaledTime : 2023-03-01T14:08:11Z
| ├ update evt.MarshaledTime : -> 2023-03-01T14:08:11Z
| ├ create evt.Meta.timestamp : 2023-03-01T14:08:11Z
| ├ 🟢 crowdsecurity/geoip-enrich (+9)
| ├ create evt.Enriched.Longitude : 0.000000
| ├ create evt.Enriched.ASNNumber : 0
| ├ create evt.Enriched.ASNOrg :
| ├ create evt.Enriched.ASNumber : 0
| ├ create evt.Enriched.IsInEU : false
| ├ create evt.Enriched.IsoCode :
| ├ create evt.Enriched.Latitude : 0.000000
| ├ create evt.Meta.IsInEU : false
| ├ create evt.Meta.ASNNumber : 0
| └ 🟢 crowdsecurity/whitelists (~2 [whitelisted])
| └ update evt.Whitelisted : %!s(bool=false) -> true
| └ update evt.WhitelistReason : -> private ipv4/ipv6 ip/ranges
└-------- parser failure 🔴
This shows the actual data and at the bottom, parser crowdsecurity/whitelists has updated the property ’evt.Whitelisted’ to true and gave it a reason. That property appears to be a built-in that flags events to be dropped.
If you want to change the ranges, you can edit the logic by editing the yaml file. A sudo cscli hub list will show you what file that is. Add or remove entries from the list it’s checking the ‘ip’ valie and ‘cidr’ value against. Any match cases whitelist to become true.
False Positives
You may see a high percent of ‘Lines poured to bucket’ relative to ‘Lines read’, like in this example where almost all are. Some lines triggering two scenareos when the ‘bucket’ is greater than the number of ‘parsed’
Acquisition Metrics:
╭────────────────────────────────┬────────────┬──────────────┬────────────────┬────────────────────────╮
│ Source │ Lines read │ Lines parsed │ Lines unparsed │ Lines poured to bucket │
├────────────────────────────────┼────────────┼──────────────┼────────────────┼────────────────────────┤
│ file:/var/log/auth.log │ 69 │ - │ 69 │ - │
│ file:/var/log/caddy/access.log │ 21 │ 21 │ - │ 32 │
│ file:/var/log/syslog │ 2 │ - │ 2 │ - │
╰────────────────────────────────┴────────────┴──────────────┴────────────────┴────────────────────────╯
Sometimes, that’s OK as not all scenarios are designed to take instant action. The ‘http-crawl-non_statics’ had 17 events and was considering action against 2 IPs, but never ‘Overflowed’ aka took action.
The http-probing did, however. And one of the two IPs had action take against them
Bucket Metrics:
╭──────────────────────────────────────┬───────────────┬───────────┬──────────────┬────────┬─────────╮
│ Bucket │ Current Count │ Overflows │ Instantiated │ Poured │ Expired │
├──────────────────────────────────────┼───────────────┼───────────┼──────────────┼────────┼─────────┤
│ crowdsecurity/http-crawl-non_statics │ - │ - │ 2 │ 17 │ 2 │
│ crowdsecurity/http-probing │ - │ 1 │ 2 │ 15 │ 1 │
╰──────────────────────────────────────┴───────────────┴───────────┴──────────────┴────────┴─────────╯
You can ask crowdsec to explain what’s going on with a -v and see that clients are asking for things that don’t exist.
├ s00-raw
| ├ 🟢 crowdsecurity/non-syslog (first_parser)
| └ 🔴 crowdsecurity/syslog-logs
├ s01-parse
| └ 🟢 crowdsecurity/caddy-logs (+19 ~2)
| └ update evt.Stage : s01-parse -> s02-enrich
| └ create evt.Parsed.request : /0/icon/Forman,%20M.L.%20
| ...
| └ create evt.Meta.http_status : 404
| ...
├-------- parser success 🟢
├ Scenarios
├ 🟢 crowdsecurity/http-crawl-non_statics
└ 🟢 crowdsecurity/http-probing
If you look at the rules (sudo cscli hub list) for http-probing, you’ll see it looks for 404s (file not found). If you get more than 10 in 10 seconds, it ‘overflows’ and the IP get baned.
Whitelist
The trouble is, some web apps generate a lot of 404s as they try and load page elements in case they exist. This generates lots of 404s and bans. In this case, we must whitelist the application with an expression that checks to see if it was an icon request, like above.
sudo vi /etc/crowdsec/parsers/s02-enrich/some-app-whitelist.yaml
name: crowdsecurity/whitelists
description: "Whitelist 404s for icon requests"
whitelist:
reason: "icon request"
expression:
- evt.Parsed.request startsWith '/0/icon/'
Feedback
Was this page helpful?
Glad to hear it! Please tell us how we can improve.
Sorry to hear that. Please tell us how we can improve.