How to query Lumi using a Splunk lookup

AI summary

About AI summaries.

Once you can query Imply Lumi with transparent federated search, you can add a lookup to Splunk® to enrich Lumi events with lookup data in your search results.

This tutorial uses a Splunk lookup table to enrich your Lumi events with extra fields at query time. It maps the user field in the example log file to the role, department, and risk_level fields defined in the lookup. You upload a lookup file and configure a lookup definition in Splunk, then run queries using the lookup command. Lumi performs the enrichment when the query runs and returns the enriched results to Splunk. Because the lookup lives in Splunk, you manage it in one place and can use it to query data from both Splunk and Lumi.

The following diagram shows where this tutorial fits in the end-to-end process of sending, transforming, and querying events. Yellow shaded boxes represent steps taken within Lumi, and blue shaded boxes represent steps taken outside Lumi. Click any box to go to that tutorial.

This tutorial builds on How to query Lumi using transparent federated search. Complete that tutorial first.

The steps assume that you've already:

• Uploaded sample web logs to Lumi.
• Configured a transparent federated provider in Splunk that points to a Lumi IAM key.
• Confirmed that transparent federated search returns events from Lumi.

This tutorial doesn't repeat the transparent federated search setup. It reuses the same IAM key and federated provider you already configured.

To complete the steps, you use the same sample web traffic data from a fictional online store. For background on the dataset and its format, see the tutorial data overview.

The following diagram summarizes the process of querying Lumi with a lookup. Click any box in the diagram to jump to that step.

Prerequisites

Before you begin, make sure you've completed How to query Lumi using transparent federated search. Download an up-to-date version of the sample log file and upload it to Lumi before you start this tutorial.

To complete the tutorial, you need the following:

• Access to Lumi with a role that can edit IAM keys. For information on roles and permissions, see Manage roles.
• The IAM key and transparent federated provider you configured in the transparent federated search tutorial.
• A Splunk user with permission to run lookup searches.

1. Upload a lookup file to Splunk

In this step, you upload a lookup file to Splunk.

1. Download the lookup file user_lookup.csv. This lookup file contains a list of users, their role, department, and their risk level.
2. In Splunk Web, go to Settings > Lookups.
3. Locate Lookup table files and click + Add new.
4. Verify that Destination app is set to search.
5. In Upload a lookup file, upload the lookup file you downloaded.
6. In Destination filename, name your file user_lookup.csv.
7. Click Save.

2. Update permissions

In this step, you update the permissions for your lookup file from Private to App.

1. In Lookup table files page, locate the Owner drop down and select your name.
2. Locate the lookup file you uploaded to Splunk, then click Permissions.
3. Under Object should appear in, select This app only (search).
4. For the purposes of this tutorial, under Permissions, check Read for everyone. In general, you need to allow read permissions in the Search & Reporting app for everyone who will query using the lookup.
5. Click Save. The sharing permissions for your lookup file are now set to App.

3. Define a lookup in Splunk

In this step, you define the lookup you created.

1. In Splunk Web, go to Settings > Lookups.
2. Locate Lookup definitions and click + Add new.
3. Verify that Destination app is set to search.
4. In Name, name your file user_lookup.
5. Verify that Type is set to File-based.
6. In the Lookup file dropdown, select user_lookup.csv.
7. Click Save.
8. Update the default permissions for your lookup definition by following the steps in the previous section.

4. Test a lookup query

In this step, you confirm that Lumi resolves a lookup query using the lookup file and definition you just added.

1. In Splunk Web, go to the Search & Reporting app.

2. Set the time range to the last 7 days.

3. Run the following query:

   index=main earliest=-7d@d
   | lookup user_lookup user OUTPUT role, department, risk_level 
   | table _time, clientip, user, role, department, risk_level, uri_path, status

   This query creates a table that contains data from the file you uploaded to Lumi based on the corresponding user attribute in the lookup file you uploaded to Splunk.

5. Run additional lookup queries

In this step, you run a series of lookup queries to explore traffic patterns in the sample data and enrich results with the lookup.

The example output may not exactly match yours. The queries use a relative 7-day time range, and the sample data has fixed timestamps, so results depend on when you downloaded the file.

1. Find all activity from high-risk users:

   index=main earliest=-7d@d
   | lookup user_lookup user OUTPUT role, department, risk_level
   | where risk_level="high"
   | table user, role, department

   user maps to the user key field in user_lookup and displays the role and department information for that user when risk_level=high.

   Example output:

   user	role	department
   jeanette928	admin	IT
   admin728	admin	IT

2. Count failed requests (status>=400) for each department in the lookup:

   index=main status>=400 earliest=-7d@d
   | lookup user_lookup user OUTPUT department
   | stats count BY department

   Example output:

   department	count
   IT	22
   automated	11
   external	68
   operations	36

3. Count bot traffic (role="bot") by HTTP status code:

   index=main earliest=-7d@d
   | lookup user_lookup user OUTPUT role
   | where role="bot"
   | stats count BY status

   Example output:

   status	count
   200	3
   204	5
   301	1
   302	1
   304	2
   400	2
   404	2

4. Count events for each role and risk-level combination:

   index=main earliest=-7d@d
   | lookup user_lookup user OUTPUT role, risk_level
   | stats count BY role, risk_level

   Example output:

   role	risk_level	count
   admin	high	19
   bot	medium	7
   contractor	medium	10
   customer	low	71
   internal_staff	medium	26

Learn more

To build on this tutorial, see:

• Enrich Lumi events with Splunk lookups for details on how to enrich search results in Lumi with Splunk lookups.
• Query Lumi events with data models to query Lumi events from Splunk using a data model.
• Federated search examples for more SPL queries you can run against the sample data.