Skip to main content

Lumi concepts for Splunk users

AI summary
This guide helps Splunk administrators understand how to integrate Lumi with Splunk by mapping familiar Splunk concepts like inputs, outputs, props.conf, and indexes to equivalent Lumi features, ensuring smooth data flow and compatibility between systems.

About AI summaries.

Imply Lumi is designed for compatibility with the Splunk® platform in your observability applications. If you use Lumi in conjunction with Splunk, it's important to understand how their similar concepts align and translate functionality from one system to another. You can ensure compatibility and continuity when making changes to your observability workflow.

This topic describes analogous concepts in Lumi and Splunk, focusing on the technical aspects of sending and searching events. The information is geared towards a Splunk administrator who's familiar with configuring universal or heavy forwarders in Splunk to ingest, process, and forward data.

Refer to the Splunk documentation for more information on configuring Splunk.

How to use this topic​

This topic is organized by the flow of data through the Splunk pipeline. Depending on the topology of your deployment, you can apply the configuration settings on different Splunk processing components, whether the forwarder, indexer, or search head.

  • When you send events to Lumi, your forwarder configuration largely remains the same with the addition of defining a destination for Lumi. See Inputs and Outputs in this topic.

  • If you define event processing on the indexers, be sure to apply the equivalent processing settings in Lumi. For details, see Props and transforms and Splunk default fields.

  • You can search events in Lumi either directly in the explore view or through a federated search from Splunk. Commands in the query are processed by the federated search head (Splunk) or the remote search head (Lumi). For more information, see Search.

See the Splunk documentation on how the data pipeline relates to Splunk processing components.

The following diagram presents an overview of the relationship between Splunk and Lumi concepts:

Splunk and Lumi concepts

Inputs​

Inputs define the source of events you plan to ingest. You set these in the Splunk configuration file inputs.conf. In many cases, you don't need to update your existing inputs.

The batch input source in Splunk is similar to the file upload integration in Lumi. File upload in Lumi is intended for you to ingest data quickly to evaluate how to use the product in your observability workflows. Note that it's not intended for backfill scenarios to retroactively process historical data.

Receiver​

The inputs.conf file also defines the Splunk receivers—how Splunk listens for incoming data. You don't need to configure receivers in Lumi, but take note of the port Lumi uses to listen for incoming data. For the port listed in an integration's prerequisites, ensure that it's open for outbound traffic on the machine forwarding the events to Lumi.

Outputs​

In Splunk, the outputs.conf configuration file determines where to forward events, whether to a Splunk instance or a third-party system. In Lumi, ingestion integrations are the routes available for event forwarding.

For example, if you already use the S2S data protocol from Splunk, you might have configured processors of type [tcpout] or [httpout] in your outputs configuration. To integrate with Lumi, you add a target group to the existing output processor with configuration details for one of the S2S integrations in Lumi.

For the list of ingestion integrations in Lumi, see Send events to Lumi.

To configure event forwarding destinations in Splunk, see Configure forwarding with outputs.conf.

Props and transforms​

Before you store or analyze observability data, you might choose to perform field extractions or mask sensitive data. In Splunk, you configure this processing in the props.conf and transforms.conf configuration files.

In Lumi, you can apply some of the props.conf and transforms.conf settings using pipelines and IAM key settings.

Pipelines​

Pipelines transform events entering Lumi. You can create a pipeline to parse events and to add, replace, or remove metadata associated with the events. For an example of how a Splunk configuration from props.conf and transforms.conf maps to a pipeline, see Compare to Splunk configuration.

IAM key attributes​

For events in the S2S protocol, you configure event parsing in the props.conf configuration file. In Lumi, you apply event parsing settings on the IAM key. For more information, see S2S attributes and the S2S tutorial.

Splunk default fields​

Splunk default fields are metadata fields on the events, such as index, source, and sourcetype. They provide context that you can use to drill down in your observability analysis.

In Splunk, the forwarders assign these metadata fields, which you can set in the configuration file inputs.conf. For the HTTP event collector (HEC) in Splunk, you can assign these fields on the HEC token using Splunk Web.

In Lumi, you can set the default values for these fields with Splunk HEC, S2S over HTTP, and S3 pull. You assign the default values as IAM key attributes. You can also specify them for CSV file upload in the Lumi UI, although any values specified in the CSV file take precedence.

Lumi stores the Splunk default fields as user attributes on the incoming events. Note that you can also set these fields using a forwarding agent or a pipeline in Lumi. These take precedence over any default values stored on the IAM key. For details on how Lumi prioritizes assignment of user attributes, see Event model.

Index​

An index in Splunk is the repository that stores the data. It's similar to a database table in terms of how you organize and scope your data. An index stores structured events and associated metadata after Splunk parses the events and applies any specified filters or transforms. In Splunk you also use an index for:

  • Maintenance purposes, such as to define a retention rule or to limit user access.
  • Filtering your searches. This improves query performance when you only want to return events from the relevant index.

In Lumi, the index is a user attribute on an event. Treat it as any other metadata field to understand an event's context or to filter your searches.

The index user attribute also serves a purpose in Splunk federated search. When you create a federated index, you provide the name of a Lumi index. The federated search results only include events with the specified Lumi index attribute.

For example, your events in Lumi store demo, test, or main for the index. You create a federated index in Splunk and enter demo for the remote dataset. When you search the federated index, Splunk only returns events that have the Lumi user attribute index: demo.

For details on assigning the index user attribute in Lumi, see Index user attribute.

In both Splunk and Lumi, the default index name is main.

You can issue queries on Lumi events in Splunk through Splunk federated search. In federated searches, you use the Splunk Search Processing Language (SPL) commands and functions supported for use in Lumi.

When you perform a federated search in Splunk, some commands run on the federated search head (Splunk) and others run on the remote search head (Lumi). See Federated search reference for the commands that run on the federated search head by design and commands Lumi has implemented on the remote search head.

To learn how to search Lumi events in Splunk, see Set up federated search.

Learn more​

See the following topics for more information: