Skip to main content

Transform events with pipelines

Event transformation is essential for observability applications to ensure proper timestamp handling, improve searchability, and optimize storage costs and query performance. Through parsing, enrichment, and other metadata management, you can extract critical information from raw logs, add contextual data, and maintain data quality and consistency throughout your observability pipeline.

For example, you might carry out the following transformations:

  • Add metadata to describe the source of an event
  • Remove metadata to avoid storing personally identifiable information
  • Parse the event message to extract IP traffic details for easier troubleshooting

In Imply Lumi, you transform events using pipelines. Pipelines are channels that process events before Lumi writes them to storage. You define the conditions to filter events that enter the pipeline. Events that satisfy the condition go through the pipeline, and events that don't satisfy the condition get stored without transformation.

Pipeline diagram

In this topic, you'll learn about the components of a pipeline and how to create and manage pipelines.

For a tutorial on working with pipelines, see How to transform events.

Processors​

Pipelines contain processors, which perform the event transformation tasks. Processors can perform field extractions using regular expressions as well as add or remove attributes.

The following behavior applies to processors:

  • Each processor is unique to a pipeline. That is, you can't reuse a processor in multiple pipelines.
  • Processors operate on an event in numerical order. When one processor adds an attribute to an event, a subsequent processor can refer to the attribute.
  • If processing fails for any reason, the event continues without changes to the next processor if applicable. Otherwise Lumi stores the event as is.

The following diagram shows an example pipeline that does the following:

  1. Extracts text from the event message using a regular expression and assigns it to a user attribute named realm.
  2. Renames the incoming metadata field named status to http_status.
  3. Deletes the userid and status event metadata.

Example pipeline

To follow along interactively with this example, see the pipelines tutorial. For reference and examples of all processor types, see Processors.

Compare to Splunk® configuration​

Think of a pipeline in Lumi as conceptually similar to a combination of props.conf and transforms.conf in Splunk.

Here’s an example Splunk configuration that extracts text inside realm [ ... ] as the realm field, which for the example message in the above diagram results in realm=demo:

transforms.conf

[extract_realm]
REGEX = realm\s\[([^\]]+)\]
FORMAT = realm::$1

props.conf

[source::...]
TRANSFORMS-extract_realm = extract_realm

In Lumi, this logic is handled in a pipeline using a regex processor that extracts realm from the message. For more information, see Lumi concepts for Splunk users.

Create and update pipelines​

This section walks you through the process of creating and managing pipelines.

Prerequisites​

To create and manage pipelines in Lumi, you need the Data manager role or higher. For information on roles and permissions, see Manage roles.

Create a pipeline​

To process events in a pipeline, you create the pipeline, then add processors to the pipeline. Once you create new pipelines and processors, Lumi enables them by default. Disabled elements appear dimmed in the pipelines list.

To create a pipeline:

  1. Click Pipelines from the navigation menu.
  2. Click + Create pipeline.
  3. Enter pipeline details:
    • Position: Place in the sequential order.
    • Name: Name to identify the pipeline.
    • Description: Optional description for the pipeline.
    • Expression: Query that qualifies events for the pipeline. For examples, see Pipeline conditions.
  4. Click Create.

Create a processor​

Add functionality to a pipeline by creating one or more processors. To create a processor in a pipeline:

  1. Click Pipelines from the navigation menu.
  2. Select a pipeline, then click Create processor.
  3. Select its position in the pipeline and the processor type.
  4. Enter a name and optional description for the processor.
  5. Fill in the processing rules. The available fields depend on the processor type.
    For examples, see Processors. For guidance on how to specify attributes, see Processor settings.
  6. Click Create.

Update a pipeline or processor​

To update a pipeline or processor:

  1. Click Pipelines from the navigation menu.
  2. For the processor you want to update, click the ellipsis and select Edit.
  3. Make the desired changes then click Save.

From the ellipsis menu, you can also complete the following tasks:

  • Reorder its position
  • Duplicate it
  • Enable or disable it
  • Delete it
  • Add a processor (for pipelines)

View a pipeline or processor​

The Pipelines page shows a list of all pipelines. Expand any pipeline to see the processors in the pipeline. Click the information icon next to a pipeline or processor to view its description and the date it was last modified.

Pipelines page

Pipeline conditions​

Each pipeline requires a filter expression to identify events to process. If an event doesn't satisfy the conditions for any pipelines, Lumi stores it without processing. Use Lumi query syntax to define filters on incoming event metadata or system attributes.

For an event that satisfies multiple pipelines, Lumi processes the event through each pipeline in numerical order. If you add an attribute in one pipeline, a subsequent pipeline can use that attribute.

The following syntax behavior applies to event conditions:

  • Use AND or OR to join multiple search criteria.
  • A standalone string matches an event when the event message contains the string.
  • For system attributes such as env, preface the name with #.
  • You can't use the equality operator = with the event message or timestamp.

The following examples are valid event conditions:

# filter events labeled with the main index
index=main

# filter events from the otel source and access_combined source type
source=otel AND sourcetype=access_combined

# filter events that contain the text "hello world" and labeled with the prod environment
"hello world" AND #env=prod

# filter events sent with a specific integration and IAM key
#iamKeyId=3e99daf3-xxxx-xxxx-xxxx-1aa2b41d62ba #receiver=splunk.hec

To designate a pipeline for a particular IAM key and integration, go to the integration page, select the key, and click Explore events. For an example of how to find the query for a given integration and key, see Check Imply Lumi for events.

View which pipelines processed an event​

Lumi records the pipelines that processed an event. You can view the processing on an event as follows:

  1. Select an event from the explore view.
  2. In the event details pane, go to the System attributes tab.
  3. If the event was processed, Lumi displays a section called Processed by pipelines. This section displays pipeline names in order of processing.
  4. Click a pipeline to view it in the Pipelines page. Lumi expands the specific pipeline you selected.

Event processed by a pipeline

Lumi retains the ID of a deleted pipeline that processed an event. The Processed by pipelines section shows the pipeline ID and marks it as deleted.

Learn more​

See the following topics for more information: