Skip to main content

Transform events

The data you send to Imply Lumi varies based on your observability needs. You might instrument applications to generate telemetry about performance, requests, and service interactions. In other scenarios, you might work with existing event logs from systems like Windows or Amazon VPC.

Given different structures and sources of events, you can transform events to make them more searchable and useful. For example, you might standardize field names across different log sources so you can query them consistently. You might filter high-volume events into separate datasets to improve query performance. You can also extract specific fields—like error codes or user IDs—to make critical information easier to find and analyze.

Lumi offers several ways to enrich and parse your incoming events. Choose your approach based on your event format and forwarding agent. This topic introduces these options and provides on guidance on when to use each.

Tools for event transformation​

Pipelines are channels that process incoming events that meet a user-specified condition. A pipeline contains one or more processors which define the data processing tasks.

An IAM key authenticates requests for sending data to Lumi. It also stores parsing settings and default values for user attributes. These are collectively known as IAM key attributes. IAM key attributes only apply to specific ingestion integrations. Lumi uses the IAM key attributes when a forwarding agent meets both conditions:

  • Authenticates with the specified IAM key
  • Uses an integration associated with the attributes

Pipelines​

Pipelines process events before Lumi stores them. Pipelines make it easy for you to automatically extract details from log data into searchable attributes. When you explore events, you can filter on those attributes.

Lumi provides a library of predefined pipelines, which contain a set of standard processors to parse and transform events with a specific data structure or format. You can also define your own pipeline to transform any kind of data you send to Lumi.

To learn more about pipelines, see Create pipelines to transform events.

For a tutorial, see How to transform events.

Overview of pipeline processing

IAM key attributes​

IAM key settings are available to specified integrations. You can enrich events with select Splunk® default fields for the HEC integration. Furthermore, you can parse events with Splunk props configuration for the S2S integrations.

These attributes only apply to their specified integration. For example, consider a scenario where you're using the same IAM key to send events in HTTP requests using HEC and send events from an OTel collector with the OTLP protocol. Lumi only applies the HEC attributes in the first case, not for incoming OTLP events. You can still add or modify these attributes using a pipeline.

For more information, see the IAM keys reference.

Selection criteria​

Pipelines and IAM key attributes are fundamentally separate concepts, but they have some overlap.

Event enrichment​

Pipelines and IAM key attributes both can enrich events with user attributes, although pipelines have more flexibility. Only pipelines perform more advanced processing, and only IAM key attributes configure global system attributes.

  • Pipelines: A pipeline can assign or modify user attributes independent of integration or the attributes themselves. You can generally always use pipelines to enrich events with user attributes.

    Use pipelines when you want to ensure specific user attribute values, since values set by pipelines override any values set by an IAM key or upstream agent. To learn about how Lumi prioritizes and assigns user attributes, see Event model.

    Also use pipelines when you need more options for event enrichment, such as using basic arithmetic formulas or conditional assignment with lookups.

  • IAM key attributes: An IAM key can enrich events with select user attributes when you use the HEC receiver. When you only need to set the Splunk default fields, use IAM key attributes.

    Also use IAM key attributes to set the system attributes for environment and team. You can't assign system attribute values using pipelines.

Event parsing​

Pipelines and IAM key attributes define different parsing methods to process events.

  • Pipelines: A pipeline can parse events independent of integration using regex or grok. When you need to define regex or grok rules to extract details from your events, use pipelines.

  • IAM key attributes: An IAM key can define parsing settings for events that use the Splunk-to-Splunk (S2S) protocol. When you configure one of the S2S integrations, use IAM key attributes.

You can combine both with the S2S protocol. For example, use IAM key attributes to extract timestamps and pipelines to parse event data into user attributes.

To learn about Lumi as it relates to the Splunk ecosystem, see Lumi concepts for Splunk users.

Indexing configuration​

In addition to event transformation, IAM key attributes configure how Lumi ingests events for specific integrations.

For HEC, you can set allowed indexes and enable indexer acknowledgment. For S3 pull, you provide your AWS authentication details so Lumi can access your S3 bucket.

Indexing configuration doesn't apply to pipelines.

Learn more​

For more information, see the following topics: