NXLog

Background

There are several solutions for capturing logs in Windows, but NXLog has some advantages;

• Cross-platform and Open Source
• Captures windows events pre-parsed
• Native windows installer and service

You could just run logstash everywhere. But in practice, Logstash’s memory requirements are several times NXLog and not everyone likes to install java everywhere.

Deploy on Windows

Download from http://nxlog.org/download. This will take you to the sourceforge site and the MSI you can install from. This installation is clean and the service installs automatically.

Configure on Windows

NXLog uses a config file with blocks in the basic pattern of:

• Input Block
• Output Block
• Routing Block

The latter being what ties together your inputs and outputs. You start out with one variable, called the $raw_event with everything in it. As you call modules, that variable gets parsed out to more useful individual variables.

Event Viewer Example

Here’s an example of invoking the module that pulls in data from the windows event log entries associated.

• Navigate to C:\Program Files (x86)\nxlog\conf
• Edit the security settings on the file nxlog.conf. Change the ‘Users’ to have modify rights. This allows you to actually edit the config file.
• Open that file in notepad and simply change it to look like so

    # Set the ROOT to the folder your nxlog was installed into
    define ROOT C:\Program Files (x86)\nxlog

    ## Default required locations based on the above
    Moduledir %ROOT%\modules
    CacheDir %ROOT%\data
    Pidfile %ROOT%\data\nxlog.pid
    SpoolDir %ROOT%\data
    LogFile %ROOT%\data\nxlog.log

    # Increase to DEBUG if needed for diagnosis
    LogLevel INFO

    # Input the windows event logs
    <Input in>
      Module      im_msvistalog
    </Input>


    # Output the logs to a file for testing
    <Output out>
        Module      om_file
        File        "C:/Program Files (x86)/nxlog/data/log-test-output.txt"
    </Output>

    # Define the route by mapping the input to an output
    <Route 1>
        Path        in => out
    </Route>

With any luck, you’ve now got some lines in your output file.

File Input Example

    # Set the ROOT to the folder your nxlog was installed into
    define ROOT C:\Program Files (x86)\nxlog

    ## Default required locations based on the above
    Moduledir %ROOT%\modules
    CacheDir %ROOT%\data
    Pidfile %ROOT%\data\nxlog.pid
    SpoolDir %ROOT%\data
    LogFile %ROOT%\data\nxlog.log

    # Increase to DEBUG if needed for diagnosis
    LogLevel INFO

    # Input a test file 
    <Input in>
        Module      im_file
        File ""C:/Program Files (x86)/nxlog/data/test-in.txt"
        SavePos     FALSE   
        ReadFromLast FALSE
    </Input>

    # Output the logs to a file for testing
    <Output out>
        Module      om_file
        File        "C:/Program Files (x86)/nxlog/data/log-test-output.txt"
    </Output>

    # Define the route by mapping the input to an output
    <Route 1>
        Path        in => out
    </Route>

Sending Events to a Remote Logstash Receiver

To be useful, you need to send your logs somewhere. Here’s an example of sending them to a Logstash receiver.

    # Set the ROOT to the folder your nxlog was installed into
    define ROOT C:\Program Files (x86)\nxlog

    ## Default required locations based on the above
    Moduledir %ROOT%\modules
    CacheDir %ROOT%\data
    Pidfile %ROOT%\data\nxlog.pid
    SpoolDir %ROOT%\data
    LogFile %ROOT%\data\nxlog.log

    # Increase to DEBUG if needed for diagnosis
    LogLevel INFO

    # Load the JSON module needed by the output module
    <Extension json>
        Module      xm_json
    </Extension>

    # Input the windows event logs
    <Input in>
      Module      im_msvistalog
    </Input>


    # Output the logs out using the TCP module, convert to JSON format (important)
    <Output out>
        Module      om_tcp
        Host        some.server
        Port        6379
        Exec to_json();
    </Output>

    # Define the route by mapping the input to an output
    <Route 1>
        Path        in => out
    </Route>

    Restart the service in the windows services, and you are in business.

Note about JSON

You’re probably shipping logs to a logstash broker (or similar json based tcp receiver). In that case, make sure to specify JSON on the way out, as in the example above or you’ll spend hours trying to figure out why you’re getting a glob of plain txt and loose all the pre-parsed windows event messages which are nearly impossible to parse back from plain text.

Using that to_json() will replace the contents. The variable we mentioned earlier, $raw_event, with all of the already parsed fields. If you hand’t invoked a module to parse that data out, you’d just get a bunch of empty events as the data was replaced with a bunch of nothing.