Set up Splunk transparent federated search

AI summary

Explains how to configure transparent federated search to query Imply Lumi events from Splunk® using data models. Covers IAM key setup, field mapping, and federated provider configuration for seamless integration.

About AI summaries.

Transparent mode federated search lets you query Imply Lumi events from Splunk® without the federated: prefix. This makes it easier to integrate Lumi data into existing Splunk queries and workflows.

Transparent mode requires field mapping on a Lumi IAM key to translate between Splunk data model fields and Lumi event fields. This topic provides details on configuring the mapping and adding a federated provider in Splunk.

For an overview of transparent federated search, see Transparent federated search. For an overview of the data model query workflow, see Query events with data models.

Prerequisites

Before you set up transparent federated search, you need the following:

• A Lumi user with the Viewer role or higher. For information on roles and permissions, see Manage roles.
• One or more data models configured in Splunk, such as the Common Information Model (CIM) Web data model.
  If you want to query Lumi through Splunk Pivot, ensure that the data model is accelerated. Acceleration isn't required for manually defined searches that reference a data model. See Data model acceleration for more information.
• Lumi events with extracted fields that correspond to data model fields. See Field extraction for more information.

info

Field mapping translates field names only—it doesn't extract or calculate fields. Use Lumi pipelines to transform raw data and perform calculations at index time. See the related Lumi requirement for details.

Configure a Lumi IAM key

You configure a Lumi IAM key to authenticate to Splunk and translate between Splunk data model fields and Lumi event fields.

To set up the IAM key:

1. From the Lumi navigation menu, click Integrations > Federated search.
2. Select Transparent.
3. Select or create an IAM key. See Create an IAM key for details.
4. Configure the allowed indexes.
5. Enter your field mapping configuration in the Data model attribute.
6. Save your changes.

Configure allowed indexes

The allowed indexes setting on the IAM key controls which Lumi indexes can be queried for non-data model queries.

In the Allowed indexes field on the IAM key, set one of the following options:

• All: Query all indexes without restrictions. We recommend using Enter specific to control query scope and reduce load.
• None: Block direct queries to indexes. Data models can still reference indexes through field mapping filters.
• Enter specific: Enter the names of specific Lumi indexes to enable for queries.

info

The allowed indexes setting only applies to non-data model queries. Data model queries are not affected by the allowed indexes setting.

When you select None or Enter specific for allowed indexes, you must include an explicit index in non-data model queries. When you select All, you can query without specifying an index.

For example, if main is explicitly allowed, sourcetype=access_combined status=500 returns empty results. Instead, use index=main sourcetype=access_combined status=500.

Data model attribute

The data model attribute on the IAM key is a JSON object that maps Splunk data model fields to Lumi event fields.

The JSON structure is organized hierarchically: data models contain datasets, and datasets can contain child datasets.

The required JSON format for the attribute is as follows:

{
  "<dataModelName1>": {
    "<datasetName1>": {
      "filter": "<filter string>",
      "fields": {
        "<datasetFieldName1>": "<lumiFieldName1>",
        "<datasetFieldName2>": "<lumiFieldName2>"
      },
      "childDatasets": {
        "<childDatasetName1>": {
          "filter": "<filter string>",
          "fields": {
            "<datasetFieldName1>": "<lumiFieldName1>",
            "<datasetFieldName2>": "<lumiFieldName2>"
          }
        }
      }
    },
    "<datasetName2>": {}
  },
  "<dataModelName2>": {}
}

View JSON properties

• dataModelName

  • Name of the data model in Splunk.

• datasetName

  • Name of the dataset in Splunk.

• filter

  • Lumi filter to identify events for the data model mapping. Optionally include an index—for example, index=main. See Lumi query syntax for the filter format.
    Note that child dataset filters inherit the parent's filter and don't need to specify an index.

• fields

  • Key-value pairs mapping Splunk data model field names (keys) to Lumi field names (values). Include all data model fields in the mapping, even when field names match, since Lumi returns empty columns for unmapped fields.
    The special data model fields _time, source, sourcetype, and host are an exception—all Splunk data models inherit them so Lumi automatically maps them. You don't need to add them to your field mappings.

• childDatasets

  • Optional nested datasets that inherit from the parent dataset.

Field mapping example

This example shows field mappings for the CIM Web data model:

{
  "Web": {
    "Web": {
      "fields": {
        "src": "clientip",
        "url": "uri_path",
        "dest": "host",
        "source": "source",
        "status": "status",
        "http_method": "method",
        "http_user_agent": "useragent"
      },
      "filter": "index=main sourcetype=access_combined",
      "childDatasets": {
        "Proxy": {
          "fields": {
            "http_referrer": "referer",
            "http_content_type": "content_type"
          },
          "filter": "source=proxy_visitors.json"
        }
      }
    }
  }
}

This configuration tells Lumi to:

• Map incoming events with index=main and sourcetype=access_combined to the Web data model and Web dataset.
• Map Lumi's host field to the data model dest field, and Lumi's method field to the data model's http_method field.
• Apply additional mappings to child dataset Proxy for events matching the parent filter and source=proxy_visitors.log.

View Splunk data model interface

The screenshots below show where the CIM Web data model field names (dest, http_method, src, etc.) appear in the Splunk interface. The first screenshot shows the parent Web dataset, and the second shows the child Proxy dataset.

View detailed field mappings

The filters and mappings work as follows.

Web data model: Web dataset

index=main AND sourcetype=access_combined

Lumi event field	Data model field
host	Web.dest
method	Web.http_method
useragent	Web.http_user_agent
clientip	Web.src
status	Web.status
uri_path	Web.url

Web data model: Web dataset: Proxy child dataset

Child datasets combine their own filter with the parent's filter using AND, and inherit the parent's field mappings while adding their own.

source=proxy_visitors.log AND index=main AND sourcetype=access_combined

Lumi event field	Data model field
content_type	Web.Proxy.http_content_type
referer	Web.Proxy.http_referrer

Add a federated provider

A transparent federated provider in Splunk routes queries to Lumi.

To configure the provider:

1. In Splunk Web, go to Settings ❯ Federation ❯ Federated Providers > Add federated provider.
2. Select Splunk and click Next.
3. Complete the following fields:
   • Provider mode: Transparent.
   • Provider name: Name for the provider.
   • Remote host: Host address provided by Lumi.
   • Service account username: IAM key ID.
   • Service account password: IAM key token.
4. Leave all other fields unchanged and click the Agree checkbox.
5. Test the connection.
6. Once the connection is successful, click Save.

Note that you don't need to create a federated index for a transparent provider. The provider routes queries to Lumi based on the allowed indexes configuration.

Search for events

Once you configure the transparent federated provider and Lumi integration, you can query Lumi events in Splunk with a data model. The following example finds bot activity on web-01:

| tstats count from datamodel=Web where nodename=Web Web.dest=web-01 Web.http_user_agent=*bot* by Web.http_user_agent, Web.url

In this query, nodename=Web specifies the Web dataset, and Web.dest (which maps to Lumi's host field) filters for web-01.

Example results:

Search child datasets

To query a child dataset, use the nodename parameter to specify the child dataset path. For example, to query the Proxy child dataset within the Web data model:

| tstats count from datamodel=Web where nodename=Web.Proxy Web.Proxy.http_referrer=*google* by Web.dest, Web.http_user_agent, Web.url

This query counts proxy web traffic from Google, grouping by destination server, user agent, and URL. Child dataset fields use the Web.Proxy. prefix, while inherited fields use the Web. prefix.

For complete tstats syntax and examples, see the Splunk tstats documentation.

Learn more

See the following topics for more information:

• Query events with data models for an overview of the data model query workflow.
• Federated search reference for supported SPL commands and syntax.
• Federated search examples for sample queries with example output.
• Set up Splunk standard federated search to set up standard federated search.