How to send Windows event logs with S3 routing

In this tutorial, you learn how to process Windows event logs with Imply Lumi. This is particularly useful when your business operates Microsoft Windows machines and you use event logs for troubleshooting and analysis.

The following diagram illustrates the data flow wherein a Splunk® universal forwarder installed on Windows forwards event logs to a heavy forwarder. In turn, the heavy forwarder uses an S3 ingest action to send the event logs to Lumi.

This is one example of sending Windows event logs to Lumi. You can also configure the universal forwarder to send events directly to Lumi using S2S.

For more information on the S3 ingest action integration, see Send events with S3 ingest actions. For more information on Splunk forwarders, refer to the Splunk documentation.

Tutorial overview

The steps in this tutorial show you how to send Windows event logs to Lumi using Splunk S3 routing:

• Add a predefined pipeline in Lumi to perform Windows event log field transformations.
• Create a rule set for Windows event logs on the heavy forwarder to send events to Lumi.
• Enable TCP forwarding on the heavy forwarder.
• Configure the universal forwarder on Windows to send event logs to the heavy forwarder.
• Explore the Windows event logs in Lumi.

The following diagram summarizes the steps. Shaded boxes represent steps taken within Lumi, and unshaded boxes represent steps taken outside Lumi. Click any box in the diagram to jump to that step.

Prerequisites

To complete the steps in this tutorial, you need the following:

• Access to Lumi with the Data manager role or higher. For information on roles and permissions, see Manage roles.
• A Splunk heavy forwarder in your environment configured to route events to Lumi using an S3 ingest action.
• Administrator access to a Windows server running the Splunk universal forwarder with the Splunk Add-on for Microsoft Windows (Windows TA). For details, see Get Windows server data in the Splunk documentation.
• You have completed the steps in How to send events with an S3 ingest action and, as such, created an IAM key and set up a S3 desitnation in Splunk.

1. Add a predefined pipeline

First, add a predefined pipeline to extract metadata from the Windows event logs in Lumi.

1. From the Lumi navigation menu, click Pipelines > + Predefined pipelines.

2. Choose Windows Log Processing from the list of predefined pipelines.

3. Click Add pipeline.

   Lumi adds the pipeline to your list of pipelines.

4. The pipeline will work as-is. However, the expression filters all events by sourcetype. To refine the filter, click the ellipsis in the top right corner of the screen for the "Windows Log Processing" pipeline and select Edit.

5. Modify pipeline conditions to use the following expression that will filter on the index attribute:

   sourcetype=* AND index=wineventlog.

   

6. Save your changes.

Once created, the pipeline is enabled by default. It will process incoming events that match the expression criteria in the pipeline.

2. Create an S3 ruleset on the heavy forwarder

In this section, you create S3 ruleset on the Splunk heavy forwarder to route Windows event logs to Lumi. The ruleset uses the S3 destination you created in How to send events with S3. Refer to Use ingest actions to improve the data input process for details about ingest actions in Splunk.

1. In Splunk Web, go to Settings > Ingest actions > Rulesets > New ruleset.

2. Enter the following details:

   • Name: event-logs-tutorial
   • Description: Ruleset to send Windows event logs to Imply Lumi using S3

3. Under Event stream, set the sourcetype to WinEventLog:System. This is the source type for system-level Windows event logs. Leave all other fields unchanged.

   note

   This configuration only applies to system-level event logs. If you want to forward other types of logs, you must configure other source types. For example, WinEventLog:Application.

4. Click Add rule > Route to destination.

5. In the Immediately send to field, select the name of the S3 destination you created in the Create an S3 destination step of the "How to send events with S3" tutorial: lumi-s3-tutorial.

6. Click Apply and then Save.

7. The newly created ruleset is now listed in your S3 ruleset table.

note

The Splunk documentation for ingest actions recommends you create rulesets for the heavy forwarder in Splunk Web or using the REST API. For more information, see Interactions with TRANSFORMS.

Now that you've created the ruleset, you're ready to enable a receiver on the heavy forwarder.

3. Enable a TCP receiver on the heavy forwarder

In this section, you set up a receiver for the Windows event logs from the universal forwarder. The instructions set up the receiver on port 9998. The port number is arbitrary, but you need to ensure that your networking and firewall configurations allow communication over this port from the universal forwarder to the heavy forwarder.

Refer to Enable a receiver for details about using a receiver in Splunk.

1. In Splunk Web, go to Settings > Forwarding and receiving.

2. Under Receive data, click + Add new.

3. Set Listen on this port to 9998 and click Save.

For information on how to enable a receiver using the command line or configuration files, see Enable a receiver for Splunk Enterprise.

The universal forwarder can now connect on the configured port to send event logs.

4. Configure the universal forwarder on Windows

In this section, you verify that the Windows TA is monitoring Windows event logs and configure the Splunk universal forwarder to send the events to the heavy forwarder.

The example configuration uses the following values:

• IP address 192.0.2.0 and port 9998 for the heavy forwarder
• EC2AMAZ-EXAMPLE for the Windows host name

When configuring your system, substitute the IP address of your own heavy forwarder and the port to the receiver you configured previously. Modify the host to match your Windows host name.

1. To verify or enable monitoring of system-level Windows event logs in the Windows TA, confirm the settings in %SPLUNK_HOME%\etc\apps\Splunk_TA_windows\local\inputs.conf:

   [WinEventLog://System]
   disabled = 0
   start_from = oldest
   current_only = 0
   checkpointInterval = 5
   renderXml = 0
   info

   If you don't have %SPLUNK_HOME%\etc\apps\Splunk_TA_windows\local\inputs.conf:

   • Create the local directory if it doesn't exist.

   • Copy the file from %SPLUNK_HOME%\etc\apps\Splunk_TA_windows\default\inputs.conf to %SPLUNK_HOME%\etc\apps\Splunk_TA_windows\local\inputs.conf.

   • Verify monitoring is enabled in the configuration file. For example: disabled = 0

2. To configure monitoring of system-level event logs in the universal forwarder, add the following stanzas to %SPLUNK_HOME%\etc\system\local\inputs.conf:

   [WinEventLog] 
   _meta = hf_proxy::meta_test
   host = EC2AMAZ-EXAMPLE
   index = wineventlog
   
   [WinEventLog://System]
   disabled = 0 

   Modify the value of host to match the hostname of your Windows machine. Note that the value of index here must match the expression for your pipeline in Lumi.

3. To configure the universal forwarder to send the event logs to the heavy forwarder, add the following stanzas to %SPLUNK_HOME%\etc\system\local\outputs.conf:

   [tcpout]
   defaultGroup = linux_hf
   
   [tcpout:linux_hf]
   server =  192.0.2.0:9998

   Modify the value of server to match the IP address and port for the receiver on the heavy forwarder.

4. Restart the universal forwarder.

5. Preview data

In this section, you view the Windows events you sent to Lumi in the explore view.

1. In Lumi, click Explore in the navigation menu.

2. Enter index=wineventlog in the Search bar.

   Lumi displays the relevant events:

3. Select an event to view its attributes. For details on the attributes, see Event model.

   

Learn more

For more information, see the following topics:

• Send events with S3 ingest actions for details on the S3 ingest actions integration.
• Send events to Imply Lumi for other options to send events.
• How to search for events with Imply Lumi for example queries using Lumi search syntax.
• Search for events with Imply Lumi for information on searching events and filtering on event attributes.