NXLOG

Background

There are several solutions for capturing logs in windows. A likely candidate is snare. But nxlog has some advantages; 
  • Cross-platform and Open Source
  • Captures windows events pre-parsed
  • Native windows installer and service
It's attractive at first brush to just run logstash everywhere. But in practice Logstash's memory requirements are several times nxlog and not everyone likes to install java everywhere.

Deploy on Windows

Download from http://nxlog.org/download. This will take you to the sourceforge site and the MSI you can install from. This installation is clean and the service installs automatically.

Configure on Windows

Nxlog uses a config file with blocks in the basic pattern of :

    Input Block       
    Output Block        
    Routing Block 

The latter being what ties together your inputs and outputs. You start out with one variable, called the $raw_event with everything in it. As you call modules, that variable gets parsed out to more useful individual variables.

Event Viewer Example
Here's an example of invoking the module that pulls in data from the windows event log entries associated . 
  • Navigate to C:\Program Files (x86)\nxlog\conf
  • Edit the security settings on the file nxlog.conf. Change the 'Users' to have modify rights. This allows you to actually edit the config file.  
  • Open that file in notepad and simply change it to look like so
# Set the ROOT to the folder your nxlog was installed into
define ROOT C:\Program Files (x86)\nxlog

## Default required locations based on the above
Moduledir %ROOT%\modules
CacheDir %ROOT%\data
Pidfile %ROOT%\data\nxlog.pid
SpoolDir %ROOT%\data
LogFile %ROOT%\data\nxlog.log

# Increase to DEBUG if needed for diagnosis
LogLevel INFO

# Input the windows event logs
<Input in>
  Module      im_msvistalog
</Input>


# Output the logs to a file for testing
<Output out>
    Module      om_file
    File        "C:/Program Files (x86)/nxlog/data/log-test-output.txt"
</Output>

# Define the route by mapping the input to an output
<Route 1>
    Path        in => out
</Route>

With any luck, you've now got some lines in your output file.

File Input Example

# Set the ROOT to the folder your nxlog was installed into
define ROOT C:\Program Files (x86)\nxlog

## Default required locations based on the above
Moduledir %ROOT%\modules
CacheDir %ROOT%\data
Pidfile %ROOT%\data\nxlog.pid
SpoolDir %ROOT%\data
LogFile %ROOT%\data\nxlog.log

# Increase to DEBUG if needed for diagnosis
LogLevel INFO

# Input a test file 
<Input in>
    Module      im_file
    File ""C:/Program Files (x86)/nxlog/data/test-in.txt"
    SavePos     FALSE   
    ReadFromLast FALSE
</Input>

# Output the logs to a file for testing
<Output out>
    Module      om_file
    File        "C:/Program Files (x86)/nxlog/data/log-test-output.txt"
</Output>

# Define the route by mapping the input to an output
<Route 1>
    Path        in => out
</Route>



Sending Events to a Remote Logstash Receiver

To be useful, you need to send your logs somewhere. Here's an example of sending them to a Logstash receiver.


# Set the ROOT to the folder your nxlog was installed into
define ROOT C:\Program Files (x86)\nxlog

## Default required locations based on the above
Moduledir %ROOT%\modules
CacheDir %ROOT%\data
Pidfile %ROOT%\data\nxlog.pid
SpoolDir %ROOT%\data
LogFile %ROOT%\data\nxlog.log

# Increase to DEBUG if needed for diagnosis
LogLevel INFO

# Load the JSON module needed by the output module
<Extension json>
    Module      xm_json
</Extension>

# Input the windows event logs
<Input in>
  Module      im_msvistalog
</Input>


# Output the logs out using the TCP module, convert to JSON format (important)
<Output out>
    Module      om_tcp
    Host        some.server
    Port        6379
    Exec to_json();
</Output>

# Define the route by mapping the input to an output
<Route 1>
    Path        in => out
</Route>
  • Restart the service in the windows services, and you are in business.

Note about JSON

You're probably shipping logs to a logstash broker (or similar json based tcp receiver). In that case, make sure to specify JSON on the way out, as in the example above or you'll spend hours trying to figure out why you're getting a glob of plain txt and loose all the pre-parsed windows event messages which are nearly impossible to parse back from plain text.

Using that to_json() will replace the contents  that variable we mentioned earlier, $raw_event, with all of the already parsed fields. If you hand't invoked a module to parse that data out, you'd just get a bunch of empty events as the data was replaced with a bunch of nothing.