Skip to main content

Manage pipelines and processors

In many cases you might want to transform the events you send to Lumi before storing them. For example, you want to extract key details from a raw event message into new attributes for your observability analyses. In Lumi, you process incoming events using pipelines. A pipeline contains one or more processors that define the data processing tasks. For a conceptual overview of pipelines, see Transform events with pipelines.

This topic walks you through the process to create and manage pipelines and processors. You'll learn how to:

  • Create a pipeline and define conditions for events to enter the pipeline
  • Add processors to perform operations such as parsing text or assigning user attributes
  • Update pipelines and processors to reorder, disable, or delete them
  • View the pipelines that processed an event

In addition to your own custom processing rules, Lumi offers several predefined pipelines. A predefined pipeline has a preset list of processors tailored to a specific event type, such as Windows event logs. To use a predefined pipeline, all you need to do is add the pipeline and confirm the conditions. For more information, see Work with predefined pipelines.

To follow along with a tutorial, see How to transform events with pipelines.

Prerequisites​

To create and manage pipelines in Lumi, you need the Data manager role or higher. For information on roles and permissions, see Manage roles.

View pipelines​

The Pipelines page shows a list of all pipelines. Expand any pipeline to see the processors in the pipeline. Click the information icon next to a pipeline or processor to view its description and the date it was last modified.

Pipelines page

Create a pipeline​

You can create a pipeline with your own processors or use a predefined pipeline provided by Lumi. To use a predefined pipeline, see Work with predefined pipelines.

If you already have an existing pipeline and want to create a nested pipeline within it, see Work with nested pipelines.

To create your own pipeline:

  1. Click Pipelines from the navigation menu.
  2. Click + Create pipeline.
  3. Enter pipeline details:
    • Name: Name to identify the pipeline.
    • Description: Optional description for the pipeline.
    • Expression: Query that qualifies events for the pipeline. For more details and examples, see Pipeline conditions.
  4. Click Create.
  5. Optionally, update the pipeline to reorder its position.

Create a pipeline

Create a processor​

Add functionality to a pipeline by creating one or more processors.

To create a processor in a pipeline:

  1. Click Pipelines from the navigation menu.
  2. Select a pipeline, then click Create > Processor.
  3. Select the processor type.
  4. Enter a name and optional description for the processor.
  5. Fill in the processing rules. The available fields depend on the processor type.
  6. Optionally, try out the processor.
  7. Click Create.
  8. Optionally, update the processor to reorder its position.

Try out a processor​

Each processor has a Try it out section you can use to provide sample input and preview the expected output. The sample input you enter depends on the processor type:

  • For the grok and regex parsers, provide the event message in the Sample field. View the result in Expected output. The result shows the parsed user attributes in user_attributes and the event message in _log.

    Try regex

  • For other processors, provide a JSON object with the input user attributes in Sample user attributes. View the processed result in Expected output.

    Try attribute mapper

    To provide the event message as part of the sample, include it in _log, like in this message mapper:

    Try message mapper

    Note that the expected output doesn't include the event timestamp.

Update a pipeline or processor​

To update a pipeline or processor:

  1. Click Pipelines from the navigation menu.
  2. For the processor you want to update, click the ellipsis and select Edit.
  3. Make the desired changes then click Save.

From the ellipsis menu, you can also take the following actions:

  • Reorder
  • Duplicate
  • Enable or disable
  • Delete
  • Add a processor (for pipelines)

When you create a new pipeline or processor, Lumi enables it by default. Disabled elements appear dimmed in the pipelines list.

View pipelines for a processed event​

To view the pipelines that processed an event:

  1. Go to the explore view and find the event.
  2. Select the event to open its event details pane.
  3. If the event was processed, Lumi displays a tab called Pipelines. The list displays pipelines in order of processing.
  4. Click a pipeline to view it in the Pipelines page. Lumi expands the specific pipeline you selected.

Event processed by a pipeline

After you delete a pipeline, Lumi retains the pipeline ID. If an event was processed by a deleted pipeline, the list shows the pipeline ID and its deleted status. You can't view the conditions and processors for a deleted pipeline.

Learn more​

See the following topics for more information: