Network Traffic

Recoding traffic on the network is critical for troubleshooting and compliance. For the latter, the most common strategy is to record the “flows”. These are the connections each host makes or accepts, and how much data is involved.

You can collect this information at the LAN on individual switches, but the WAN (at the router) is usually more important. And if the router is performing NAT, it’s the only place to record the mappings of internal to external IPs and ports.

Network Log System

A network flow log system usually has three main parts.

Exporter --> Collector --> Analyzer

The Exporter, which records the data, the Collector, which is where the data is stored, and the Analyzer which makes the data more human-readable.

Example

We’ll use a Palo Alto NG Firewall as our exporter, and an Elasticsearch back-end. The data we are collecting is essentially log data, and Elasticsearch is probably the best at handling unstructured information.

At small scale, you can combine all of the the collection and analysis parts on a single system. We’ll use windows servers in our example as well.

graph LR

A(Palo)
B(Beats)
C(ElasticSearch)
D(Kibana) 

subgraph Exporter
A
end

subgraph Collector and Analyzer
B --> C --> D
end

A --> B

Installation

Start with Elasticsearch and Kibana, then install Beats.

• Elastic Stack

Configuration

Beats and Palo have a couple of protocols in common. NetFlow is the traditional protocol, but when you’re using NAT the best choice is the syslog protocol as the Palo will directly tell you the NAT info all in one record and you don’t have to correlate multiple interface flows to see who did what.

Beats

On the Beats server, start an admin powershell session, change to the Filebeat directory, edit the config file and restart the server.

There is a bunch of example text in the config so tread carefully and keep in mind that indentation matters. Stick this block right under the filebeat.inputs: line and you should be OK.

This config stanza has a processor block that decodes the CVS content sent over in the message field, extracts a few select fields, then discards the rest. There’s quite a bit left over though, so see tuning below if you’d like to reduce the data load even more.

cd "C:\Program Files\Filebeat"
write.exe filebeat.yml

filebeat.inputs:
- type: syslog
  protocol.udp:
    host: ":9000"
  processors:
  - decode_csv_fields:
      fields:
        message: csv
  - extract_array:
      field: csv
      overwrite_keys: true
      omit_empty: true
      fail_on_error: false
      mappings:
        source.ip: 7
        destination.ip: 8
        source.nat.ip: 9
        network.application: 14
        source.port: 24
        destination.port: 25
        source.nat.port: 26
  - drop_fields:
        fields: ["csv", "message"]

A larger is example is under the beats documentation.

Palo Alto Setup

Perform steps 1 and 2 of the Palo setup guide with the notes below.

https://docs.paloaltonetworks.com/pan-os/10-0/pan-os-admin/monitoring/use-syslog-for-monitoring/configure-syslog-monitoring

• In step 1 - The panw module defaults to 9001
• In step 2 - Make sure to choose Traffic as the type of log

Tuning

You can reduce the amount of data even more by adding a few more Beats directives.

# At the very top level of the file, you can add this processor to affect global fields
processors:
- drop_fields:
   fields: ["agent.ephemeral_id","agent.id","agent.hostname","agent.type","agent.version","ecs.version","host.name"]

# You can also drop syslog fields that aren't that useful (you may need to put this under the syslog input)
- drop_fields:
    fields: ["event.severity","input.type","hostname","syslog.facility", "syslog.facility_label", "syslog.priority", "syslog.priority_label","syslog.severity_label"]

You may want even more data. See the Full Palo Syslog data to see what’s available. An example

Conclusion

At this point you can navigate to the Kibana web console and explore the logs. There is no dashboard as this is just for log retention and covers the minimum required. If you’re interested in more, check out the SIEM and Netflow dashboards Elasticsearch offers.

Sources

Palo Shipping

https://docs.logz.io/shipping/security-sources/palo-alto-networks.html