Set up Splunk transparent federated search

AI summary

Explains how to set up a transparent federated provider in Splunk® and configure a Lumi IAM key so you can query Lumi data from Splunk using a data model.

About AI summaries.

Transparent mode federated search lets you query Imply Lumi events from Splunk® without the federated: prefix. This makes it easier to integrate Lumi data into existing Splunk queries and workflows.

Transparent mode requires field mapping on a Lumi IAM key to translate between Splunk data model fields and Lumi event fields. This topic provides details on configuring the mapping and adding a federated provider in Splunk.

For an overview of transparent federated search, see Transparent federated search. For an overview of the data model query workflow, see Query events with data models.

Prerequisites

Before you set up transparent federated search, you need the following:

• A Lumi user with the Viewer role or higher. For information on roles and permissions, see Manage roles.
• One or more data models configured in Splunk, such as the Common Information Model (CIM) Web data model.
  If you want to query Lumi through Splunk Pivot, ensure that the data model is accelerated. Acceleration isn't required for manually defined searches that reference a data model. See Data model acceleration for more information.
• Lumi events with extracted fields that correspond to data model fields. See Field extraction for more information.

info

Field mapping translates field names only—it doesn't extract or calculate fields. Use Lumi pipelines to transform raw data and perform calculations at index time. See the related Lumi requirement for details.

Configure a Lumi IAM key

You configure a Lumi IAM key to authenticate to Splunk and translate between Splunk data model fields and Lumi event fields.

To set up the IAM key:

1. From the Lumi navigation menu, click Integrations > Federated search.
2. Select Transparent.
3. Select or create an IAM key. See Create an IAM key for details.
4. Enter your field mapping configuration in the Data model attribute.
5. Save your changes.

Data model attribute

The data model attribute on the IAM key is a JSON object that maps Splunk data model fields to Lumi event fields.

The JSON structure is organized hierarchically: data models contain datasets, and datasets can contain child datasets.

The required JSON format for the attribute is as follows:

{
  "<dataModelName1>": {
    "<datasetName1>": {
      "filter": "<filter string>",
      "fields": {
        "<datasetFieldName1>": "<lumiFieldName1>",
        "<datasetFieldName2>": "<lumiFieldName2>"
      },
      "childDatasets": {
        "<childDatasetName1>": {
          "filter": "<filter string>",
          "fields": {
            "<datasetFieldName1>": "<lumiFieldName1>",
            "<datasetFieldName2>": "<lumiFieldName2>"
          }
        }
      }
    },
    "<datasetName2>": {}
  },
  "<dataModelName2>": {}
}

View JSON properties

• dataModelName

  • Name of the data model in Splunk.

• datasetName

  • Name of the dataset in Splunk.

• filter

  • Lumi filter to identify events for the data model mapping. Must include an index—for example, index=main. See Lumi query syntax for the filter format.
    Note that child dataset filters inherit the parent's filter and don't need to specify an index.

• fields

  • Key-value pairs mapping Splunk data model field names (keys) to Lumi field names (values). Include all data model fields in the mapping, even when field names match, since Lumi returns empty columns for unmapped fields.
    The special data model fields _time, source, sourcetype, and host are an exception—all Splunk data models inherit them so Lumi automatically maps them. You don't need to add them to your field mappings.

• childDatasets

  • Optional nested datasets that inherit from the parent dataset.

Field mapping example

This example shows field mappings for the CIM Web data model:

{
  "Web": {
    "Web": {
      "filter": "index=main sourcetype=access_combined",
      "fields": {
        "dest": "host",
        "http_method": "method",
        "http_user_agent": "useragent",
        "src": "clientip",
        "status": "status",
        "url": "uri_path"
      },
      "childDatasets": {
        "Proxy": {
          "filter": "source=proxy_access.log",
          "fields": {
            "http_content_type": "content_type",
            "http_referrer": "referer"
          }
        }
      }
    }
  }
}

This configuration tells Lumi to:

• Map incoming events with index=main and sourcetype=access_combined to the Web data model and Web dataset.
• Map Lumi's host field to the data model dest field, and Lumi's method field to the data model's http_method field.
• Apply additional mappings to child dataset Proxy for events matching the parent filter and source=proxy_access.log.

The tables later in this section list each mapping for the Web dataset and Proxy child dataset.

The screenshots below show where these data model field names (dest, http_method, src, etc.) appear in the Splunk CIM Web data model interface. The first screenshot shows the parent Web dataset, and the second shows the child Proxy dataset.

The filters and mappings work as follows.

Web data model: Web dataset

index=main AND sourcetype=access_combined

Lumi event field	Data model field
host	Web.dest
method	Web.http_method
useragent	Web.http_user_agent
clientip	Web.src
status	Web.status
uri_path	Web.url

Web data model: Web dataset: Proxy child dataset

Child datasets combine their own filter with the parent's filter using AND, and inherit the parent's field mappings while adding their own.

source=proxy_access.log AND index=main AND sourcetype=access_combined

Lumi event field	Data model field
content_type	Web.Proxy.http_content_type
referer	Web.Proxy.http_referrer

Add a federated provider

A transparent federated provider in Splunk routes queries to Lumi.

To configure the provider:

1. In Splunk Web, go to Settings ❯ Federation ❯ Federated Providers > Add federated provider.
2. Select Splunk and click Next.
3. Complete the following fields:
   • Provider mode: Transparent.
   • Provider name: Name for the provider.
   • Remote host: Host address provided by Lumi.
   • Service account username: IAM key ID.
   • Service account password: IAM key token.
4. Leave all other fields unchanged and click the Agree checkbox.
5. Test the connection.
6. Once the connection is successful, click Save.

Note that you don't need to create a federated index for a transparent provider. The provider routes queries to Lumi without the need to specify an index.

Search for events

Once you configure the transparent federated provider and Lumi integration, you can query Lumi events in Splunk with a data model.

To search for Lumi and Splunk events:

1. Open the Search & Reporting app in Splunk.
2. Enter your query into the search bar.
3. Use the time range selector next to the search bar to set a time range for the query.
4. Press Enter or click the search icon to execute the search.

The following example query finds bot activity on web-01:

| tstats count from datamodel=Web where nodename=Web Web.dest=web-01 Web.http_user_agent=*bot* by Web.http_user_agent, Web.url

In this query, nodename=Web specifies the Web dataset, and Web.dest (which maps to Lumi's host field) filters for web-01.

Example results:

Search child datasets

To query a child dataset, use the nodename parameter to specify the child dataset path. For example, to query the Proxy child dataset within the Web data model:

| tstats count from datamodel=Web where nodename=Web.Proxy Web.Proxy.http_referrer=*google* by Web.dest, Web.http_user_agent, Web.url

This query counts proxy web traffic from Google, grouping by destination server, user agent, and URL. Child dataset fields use the Web.Proxy. prefix, while inherited fields use the Web. prefix.

For complete tstats syntax and examples, see the Splunk tstats documentation.

Learn more

See the following topics for more information:

• Set up Splunk standard federated search to set up standard federated search of Lumi events without data models.
• Federated search reference for supported SPL commands, syntax, and operators in federated search.
• Federated search examples for queries that you can run against Lumi tutorial data.