Skip to main content

Lumi concepts for Splunk users

Imply Lumi is designed for compatibility with the Splunk® platform in your observability applications. If you use Lumi in conjunction with Splunk, it's important to understand how their similar concepts align and translate functionality from one system to another. You can ensure compatibility and continuity when making changes to your observability workflow.

This topic describes analogous concepts in Lumi and Splunk, focusing on the technical aspects of sending and searching events. The information is geared towards a Splunk administrator who's familiar with configuring universal or heavy forwarders in Splunk to ingest, process, and forward data.

Refer to the Splunk documentation for more information on configuring Splunk.

Topic structure​

This topic is organized by Splunk function. Depending on the topology of your deployment, you can apply the configuration settings on different Splunk processing components, whether the forwarder, indexer, or search head.

  • When you send events to Lumi, your forwarder configuration largely remains the same with the addition of defining a destination for Lumi. See Inputs and Outputs in this topic.

  • If you define event processing on the indexers, be sure to apply the equivalent processing settings in Lumi. For details, see Props and transforms and Splunk metadata.

  • You can search events in Lumi either directly in the explore view or through a federated search from Splunk. Commands in the query are processed by the federated search head (Splunk) or the remote search head (Lumi). For more information, see Search.

See the Splunk documentation on how the data pipeline relates to Splunk processing components.

Inputs​

Inputs define the source of events you plan to ingest. You set these in the Splunk configuration file inputs.conf. In many cases, you don't need to update your existing inputs.

The batch input source in Splunk is similar to the file upload integration in Lumi. File upload in Lumi is intended for you to ingest data quickly to evaluate how to use the product in your observability workflows. Note that it's not intended for backfill scenarios to retroactively process historical data.

Receiver​

The inputs.conf file also defines the Splunk receivers—how Splunk listens for incoming data. You don't need to configure receivers in Lumi, but take note of the port Lumi uses to listen for incoming data. For the port listed in an integration's prerequisites, ensure that it's open for outbound traffic on the machine forwarding the events to Lumi.

Outputs​

In Splunk, the outputs.conf configuration file determines where to forward events, whether to a Splunk instance or a non-Splunk system. In Lumi, ingestion integrations are the routes available for event forwarding.

For example, if you already use the S2S data protocol from Splunk, you might have configured processors of type [tcpout] or [httpout] in your outputs configuration. To integrate with Lumi, you add a target group to the existing output processor with configuration details for the S2S integration in Lumi.

For the list of ingestion integrations in Lumi, see Send events to Lumi.

To configure event forwarding destinations in Splunk, see Configure forwarding with outputs.conf.

Props and transforms​

Before you store or analyze observability data, you might choose to perform field extractions or mask sensitive data. In Splunk, you configure this processing in the props.conf and transforms.conf configuration files.

In Lumi, you can apply some of the props.conf and transforms.conf settings using pipelines and IAM key settings.

Pipelines transform events entering Lumi. You can create a pipeline to parse events and to add, replace, or remove metadata associated with the events. For an example of how a Splunk configuration from props.conf and transforms.conf maps to a pipeline, see Compare to Splunk configuration.

For events in the S2S protocol, you configure event parsing in the props.conf configuration file. In Lumi, you apply event parsing settings on the IAM key that you use with the S2S integration. For more information, see Set event parsing rules.

Splunk metadata​

Splunk default fields are metadata fields on the events, such as index, source, and sourcetype. They provide context that you can use to drill down in your observability analysis.

In Splunk, use the configuration file inputs.conf to assign these metadata fields. For the HTTP event collector (HEC) in Splunk, you can assign these fields on the HEC token using Splunk Web.

In Lumi, you can specify defaults for the source, source type, and index for the HEC integration. You assign them on the IAM key you use with HEC. Lumi stores those values as user attributes on the incoming events, provided those attributes were not already set on the events. Regardless of integration, you can use pipelines in Lumi to enrich incoming events with these attributes as well as any other user-defined attributes.

For details on how Lumi prioritizes assignment of user attributes, see Event model.

Index​

An index in Splunk names the repository that stores the data. It's similar to a database table in terms of how you organize and scope your data. The index stores structured events and associated metadata after Splunk parses the events and applies any specified filters or transforms.

You specify the index name when you perform a search in Splunk. This improves query performance when you only want to return events from the relevant index. In Splunk, you can also use indexes for maintenance purposes, such as to apply retention rules or access to different indexes.

The index assignment in Splunk depends on your Splunk setup. For example, you might assign the index in inputs.conf or set up rules in props.conf and transforms.conf to route events to different indexes.

In Lumi, the index is a user attribute on an event. Treat it as any other metadata field to understand an event's context or to filter your searches. When you use the HEC integration, you can assign defaults for the index and other HEC attributes on the Lumi IAM key used to send the events.

With other integrations in Lumi, you don't set the index explicitly, but you can send the metadata field with the event. For example, when you send an event with the S2S integration, the event in Lumi stores the index attribute from your Splunk input configuration in inputs.conf. You can also add the attribute to incoming events using a pipeline.

In both Splunk and Lumi, the default index name is main.

You can issue queries on Lumi events in Splunk through Splunk federated search. In federated searches, you use the Splunk Search Processing Language (SPL) commands and functions supported for use in Lumi.

When you perform a federated search in Splunk, some commands run on the federated search head (Splunk) and others run on the remote search head (Lumi). See Federated search reference for the commands that run on the federated search head by design and commands Lumi has implemented on the remote search head.

To learn how to search Lumi events in Splunk, see Set up federated search.

Learn more​

See the following topics for more information: