Work with predefined pipelines

AI summary

Explains how to use Imply Lumi's built-in pipelines for common event formats without creating custom processors. Covers adding predefined pipelines and customizing them by duplicating or editing conditions for your specific requirements.

About AI summaries.

Predefined pipelines are built-in pipelines that Imply Lumi provides for processing specific event formats. They apply standard parsing and enrichment rules so your data is structured and ready to use as soon as it’s ingested. This saves you from creating custom processors for common event structures.

You can use predefined pipelines as they are, or duplicate and customize them to fit your requirements.

In this topic, you’ll learn how to use and manage predefined pipelines.

For a list of all predefined pipelines, see Predefined pipelines reference.

How predefined pipelines work

By default, predefined pipelines are enabled in a standby mode, meaning that they are ready to process events. When Lumi detects matching events, it activates the corresponding pipeline and adds it to the Pipelines page. Note that there can be a slight delay for Lumi to activate a predefined pipeline across all event collectors, which can cause some initial events to skip processing. If you prefer that all initial events get processed, manually add the pipeline before sending events.

A predefined pipeline has a fixed definition of its conditions and processors. If you want to make changes, you can duplicate the predefined pipeline and update your copy.

You can't delete predefined pipelines. To avoid processing, manually disable the predefined pipeline.

Workflows

Predefined pipelines work as is, so you can use them without interruption in your event forwarding workflows. For example:

1. Check that your events match the predefined pipeline conditions, such as assigning the correct sourcetype.
   To see pipeline details, see View predefined pipelines.
2. Send events to Lumi. For details, see Send events to Lumi.
3. Explore events. For details, see Search events in Lumi.
4. From the explore view, click an event to view pipelines that processed it.

The following sections describe additional example workflows for using predefined pipelines.

Simulate pipeline

To preview how a predefined pipeline operates:

1. On the Pipelines page, find the predefined pipeline, marked with a box icon.
   If you don't see the predefined pipeline, manually add it to the list.
2. Follow the steps to simulate a pipeline.

You can also send a test event, such as with the HEC API, and view the processed event in the explore view.

Change order

Pipelines operate in the order displayed. A newly added predefined pipeline goes to the bottom of the list.

To update the pipeline order:

1. On the Pipelines page, find the predefined pipeline, marked with a box icon.
   If you don't see the predefined pipeline, manually add it to the list.
2. On the Pipelines page, click the ellipsis next to the predefined pipeline, and select Reorder.

Customize processing

The predefined pipeline is a template for processing events. You can't make changes to the predefined pipeline itself, but you can duplicate and update the copy. For example, you might want to:

• Change the pipeline conditions to match your forwarding setup.
• Customize processors to add or remove fields.
• Create two versions of the same pipeline with different conditions to handle similar events for different use cases.

To customize processing from a predefined pipeline:

1. Create a pipeline copy:

   • If you see the predefined pipeline on the Pipelines page, click the ellipsis menu and select Duplicate.
   • Otherwise, view the predefined pipeline in the library, click the ellipsis menu, and select Create editable copy.

2. The pipeline copy appears in the list with (Copy) appended to the name. The original predefined pipeline is marked with a box icon. Click the ellipsis next to the pipeline copy and select Edit. You can:

   • Edit the name and description.
   • Update the pipeline condition for how Lumi determines events to process. See Pipeline conditions for more information.

3. Customize the processors:

   • To add processors, click the ellipsis next to the pipeline copy and select Create processors.
     See Create a processor for more information.
   • To edit processors, expand the pipeline and click the ellipsis next to a processor. You can reorder, edit, duplicate, disable, or delete it.

4. If your pipeline copy has the same conditions as the predefined pipeline, consider if you want both pipelines to operate. If so, confirm the ordering of the pipelines. If not, you can disable the predefined pipeline.

The following example shows a VPC flow log processing pipeline and a duplicate:

To learn more about updating pipelines, see Update pipelines and processors.

View predefined pipelines

To view predefined pipelines:

1. In the Lumi navigation menu, click Pipelines.

2. Click Predefined pipelines.

3. In the Predefined pipelines dialog, select a predefined pipeline to view its details. A gray checkmark indicates that Lumi identified matching events and activated the predefined pipeline.

   

You can also view details for a predefined pipeline on the Pipelines page. Lumi automatically adds it to the page when it detects matching events, or you can add it yourself. A predefined pipeline is denoted with a box icon next to the pipeline name. For more details, see View pipelines.

Add a predefined pipeline

Since predefined pipelines are enabled by default, you typically don't need to add it manually. You can choose to add a predefined pipeline when you want to control its ordering or in case of an activation delay.

To add a predefined pipeline:

1. In the Lumi navigation menu, click Pipelines.

2. Click Predefined pipelines.

3. Find the predefined pipeline you want to add. You can use the search function in the Predefined pipelines dialog.

4. Click the ellipsis, and select Add predefined pipeline.

   

Disable a predefined pipeline

By default, predefined pipelines are enabled in a standby manner. You might need to disable a predefined pipeline if you duplicate and update the copy, so you don't have two sets of processing.

To disable a predefined pipeline:

1. In the Lumi navigation menu, click Pipelines.
2. Find the predefined pipeline, marked with a box icon.
   If you don't see the predefined pipeline, manually add it to the list.
3. Click the ellipsis and select Disable.

You can't delete predefined pipelines.

Learn more

See the following topics for more information:

• How to transform events with pipelines for a tutorial on using pipelines.
• Transform events using pipelines for an overview of pipelines and processors.
• Manage pipelines and processors for how to create and manage pipelines and processors.