Skip to main content

Manage pipelines and processors

AI summary
Explains how to create and manage pipelines in Imply Lumi to transform events before storage. Covers creating pipelines and processors and how to view when events are processed by pipelines.

About AI summaries.

Pipelines transform events you send to Imply Lumi. A pipeline contains one or more processors that define the data processing tasks. Pipelines don't process events already stored in Lumi, and they don't create or delete events.

This topic walks you through the process to create and manage pipelines and processors. You'll learn how to:

  • Create a pipeline and define conditions for events to enter the pipeline
  • Add processors to perform operations such as parsing text or assigning user attributes
  • Update pipelines and processors to reorder, disable, or delete them
  • View the pipelines that processed an event

In addition to your own custom processing rules, Lumi offers several predefined pipelines. A predefined pipeline has a preset list of processors tailored to a specific event type, such as Windows event logs. Predefined pipelines are enabled by default. For more information, see Work with predefined pipelines.

To follow along with a tutorial, see How to build pipelines.

Prerequisites​

To create and manage pipelines in Lumi, you need the Data manager role or higher. For information on roles and permissions, see Manage roles.

View pipelines​

The Pipelines page shows a list of all pipelines. Expand any pipeline to see the processors in the pipeline. Click the information icon next to a pipeline or processor to view its description and the date it was last modified.

Pipelines page

Create a pipeline​

To create your own pipeline:

  1. Click Pipelines from the navigation menu.
  2. Click + Create pipeline.
  3. Enter pipeline details:
    • Name: Name to identify the pipeline.
    • Description: Optional description for the pipeline.
    • Expression: Query that qualifies events for the pipeline. For example sourcetype=access_combined. For more details, see Pipeline conditions.
  4. Click Create.
  5. Optionally, update the pipeline to reorder its position.
  6. Create one or more processors to define the pipeline functionality.

Create a pipeline

Duplicate a pipeline​

You can duplicate the pipeline:

  • If you want to reuse many of the processors, but want to make some modifications.
  • To apply the same processors to a different set of conditions.

If the conditions are a subset of your original expression, such as sourcetype=access_combined AND index=dev, create a nested pipeline instead.

To duplicate a pipeline:

  1. Click the ellipsis next to the date added.
  2. Select Duplicate.
  3. Supply the name and conditions for the new pipeline.

Import a pipeline​

If you previously exported a pipeline configuration as a yaml file, you can import the file as a new pipeline in your environment. For example, you might not have the original pipeline that you can duplicate, or the original pipeline deviated from the exported configuration.

You can also generate a pipeline from a Splunk® technology add-on (TA). For a detailed guide, see Build pipelines from Splunk technology add-ons.

  1. Click + Create pipeline.
  2. Click Import.
  3. Select a Lumi pipeline YAML or a Splunk TA file.
  4. Click Next.
    1. If you selected a pipeline YAML, Lumi creates the pipeline in a disabled state.
    2. If you selected a Splunk TA file, select the source types that you want to transform, then click Create pipeline.
  5. Optionally, review and simulate the pipeline then enable it.

Create a processor​

To create a processor in a pipeline:

  1. Click Pipelines from the navigation menu.
  2. Select a pipeline, then click Create > Processor.
  3. Select the processor type.
  4. Enter a name and optional description for the processor.
  5. Fill in the processing rules. The available fields depend on the processor type.
  6. Optionally, try out the processor.
  7. Click Create.
  8. Optionally, update the processor to reorder its position.

Before sending events to Lumi, you can check how the pipeline, or the entire set of pipelines, will operate on your events. See Simulate pipelines and processors for more information.

Duplicate a processor​

You can duplicate a processor to modify its functionality or to use it with a different set of attributes. To duplicate a processor, click the ellipsis on the right, and select Duplicate. If you use the same source attribute for two processors, change the input value for the second processor to match the overridden value set by the first processor.

Update a pipeline or processor​

You can update a pipeline or processor to change its configuration, reorder it, enable or disable it, or delete it. To update a pipeline or processor:

  1. Click Pipelines from the navigation menu.
  2. For the processor you want to update, click the ellipsis and select Edit.
    You can also select options to reorder, enable or disable, or delete.
  3. Make the desired changes then click Save.

When you create a new pipeline or processor, Lumi enables it by default. Disabled elements appear dimmed in the pipelines list.

View pipelines for a processed event​

To view the pipelines that processed an event:

  1. Go to the explore view and find the event.
  2. Select the event to open its event details pane.
  3. If the event was processed, Lumi displays a tab called Pipelines. The list displays pipelines in order of processing.
  4. Click a pipeline to view it in the Pipelines page. Lumi expands the specific pipeline you selected.

Event processed by a pipeline

After you delete a pipeline, Lumi retains the pipeline ID. If an event was processed by a deleted pipeline, the list shows the pipeline ID and its deleted status. You can't view the conditions and processors for a deleted pipeline.

Export a pipeline​

You can export a pipeline to save its configuration in the form of a YAML file. You can later import this configuration to create a new pipeline.

To export a pipeline, click the ellipsis next to it, and select Export. A download automatically begins for NAME.yaml where NAME is the pipeline name.

Learn more​

See the following topics for more information: