Skip to main content

Manage pipelines and processors

In many cases you might want to transform the events you send to Lumi before storing them. For example, you want to extract key details from a raw event message into new attributes for your observability analyses. In Lumi, you process incoming events using pipelines. A pipeline contains one or more processors that define the data processing tasks. For a conceptual overview of pipelines, see Transform events with pipelines.

This topic walks you through the process to create and manage pipelines and processors. You'll learn how to:

  • Create a pipeline and define conditions for events to enter the pipeline
  • Add processors to perform operations such as parsing text or assigning user attributes
  • Update pipelines and processors to reorder, disable, or delete them
  • View the pipelines that processed an event

In addition to your own custom processing rules, Lumi offers several predefined pipelines. A predefined pipeline has a preset list of processors tailored to a specific event type, such as Windows event logs. To use a predefined pipeline, all you need to do is add the pipeline and confirm the conditions. For more information, see Work with predefined pipelines.

To follow along with a tutorial, see How to transform events with pipelines.

Prerequisites​

To create and manage pipelines in Lumi, you need the Data manager role or higher. For information on roles and permissions, see Manage roles.

Create a pipeline​

You can create a pipeline with your own processors or use a predefined pipeline provided by Lumi. To use a predefined pipeline, see Work with predefined pipelines.

If you already have an existing pipeline and want to create a nested pipeline within it, see Work with nested pipelines.

To create your own pipeline:

  1. Click Pipelines from the navigation menu.
  2. Click + Create pipeline.
  3. Enter pipeline details:
    • Name: Name to identify the pipeline.
    • Description: Optional description for the pipeline.
    • Expression: Query that qualifies events for the pipeline. For more details and examples, see Pipeline conditions.
  4. Click Create.
  5. Optionally, update the pipeline to reorder its position.

Create a pipeline

Create a processor​

Add functionality to a pipeline by creating one or more processors.

To create a processor in a pipeline:

  1. Click Pipelines from the navigation menu.
  2. Select a pipeline, then click Create > Processor.
  3. Select the processor type.
  4. Enter a name and optional description for the processor.
  5. Fill in the processing rules. The available fields depend on the processor type.
  6. Optionally, try out the processor.
    • For parser processors, provide the event message in the Sample field, and view the result in Expected output.
    • For other processors, provide a JSON object with the input user attributes in Sample user attributes. View the processed result in Expected output.
  7. Click Create.
  8. Optionally, update the processor to reorder its position.

Update a pipeline or processor​

To update a pipeline or processor:

  1. Click Pipelines from the navigation menu.
  2. For the processor you want to update, click the ellipsis and select Edit.
  3. Make the desired changes then click Save.

From the ellipsis menu, you can also take the following actions:

  • Reorder
  • Duplicate
  • Enable or disable
  • Delete
  • Add a processor (for pipelines)

When you create a new pipeline or processor, Lumi enables it by default. Disabled elements appear dimmed in the pipelines list.

View a pipeline or processor​

The Pipelines page shows a list of all pipelines. Expand any pipeline to see the processors in the pipeline. Click the information icon next to a pipeline or processor to view its description and the date it was last modified.

Pipelines page

View pipelines for a processed event​

To view the pipelines that processed an event:

  1. Go to the explore view and find the event.
  2. Select the event to open its event details pane.
  3. If the event was processed, Lumi displays a tab called Pipelines. The list displays pipelines in order of processing.
  4. Click a pipeline to view it in the Pipelines page. Lumi expands the specific pipeline you selected.

Event processed by a pipeline

After you delete a pipeline, Lumi retains the pipeline ID. If an event was processed by a deleted pipeline, the list shows the pipeline ID and its deleted status. You can't view the conditions and processors for a deleted pipeline.

Learn more​

See the following topics for more information: