Work with predefined pipelines

AI summary

Explains how to use Imply Lumi's built-in pipelines for common event formats without creating custom processors. Covers adding predefined pipelines and customizing them by duplicating or editing conditions for your specific requirements.

About AI summaries.

Predefined pipelines are built-in pipelines that Imply Lumi provides for processing specific event formats. They apply standard parsing and enrichment rules so your data is structured and ready to use as soon as it’s ingested. This saves you from creating custom processors for common event structures. You can use predefined pipelines as they are, or duplicate and customize them to fit your requirements.

In this topic, you’ll learn how to use and manage predefined pipelines.

For a list of all predefined pipelines, see Predefined pipelines reference.

How predefined pipelines work

A predefined pipeline has a fixed definition of conditions and processors to process specific log types. You can't change a predefined pipeline, but you can use it as a template to create a copy and customize it.

Predefined pipelines are automatically ready to process events. When Lumi detects a matching event based on its source type, it activates the corresponding pipeline and adds it to the Pipelines page. To prevent automatic processing, manually disable the predefined pipeline.

Note that there can be a slight delay for Lumi to activate a predefined pipeline across all event collectors, which can cause some initial events to skip processing. If you prefer that all initial events get processed, manually add the pipeline before sending events.

Source type assignment

Each predefined pipeline has a condition based on an incoming event's source or source type attribute. For example, sourcetype=aws:cloudtrail. Lumi evaluates existing metadata on incoming events, which you might assign from a forwarding agent, such as in fields for a Splunk HEC request.

If your incoming events don't have the relevant condition, you can assign a fallback default value using the IAM key. You set IAM key attributes specific to the integration used to send events, supported for Splunk HEC and S3 pull. You can also set source and source type in the file upload UI. The default value only applies when incoming events don't already have the attribute.

If you can't assign the condition using a forwarding agent or IAM key attribute, you can create a preliminary pipeline to create the attribute. If incoming events have the attribute, the pipeline overwrites them.

Workflows

Predefined pipelines work as is, so you can use them without interruption in your event forwarding workflows. For example:

1. Check that your events match the predefined pipeline conditions, such as assigning the correct sourcetype.
   To see pipeline details, see View predefined pipelines.
2. Send events to Lumi. For details, see Send events to Lumi.
3. Explore events. For details, see Search events in Lumi.
4. From the explore view, click an event to view pipelines that processed it.

The following sections describe additional example workflows for using predefined pipelines.

Simulate pipeline

To preview how a predefined pipeline operates:

1. On the Pipelines page, find the predefined pipeline, marked with a box icon.
   If you don't see the predefined pipeline, manually add it to the list.
2. Follow the steps to simulate a pipeline.

You can also send a test event, such as with the HEC API, and view the processed event in the explore view.

Change order

Pipelines operate in the order displayed. A newly added predefined pipeline goes to the bottom of the list.

To update the pipeline order:

1. On the Pipelines page, find the predefined pipeline, marked with a box icon.
   If you don't see the predefined pipeline, manually add it to the list.
2. On the Pipelines page, click the ellipsis next to the predefined pipeline, and select Reorder.

Customize processing

The predefined pipeline is a template for processing events. You can't make changes to the predefined pipeline itself, but you can duplicate and update the copy. For example, you might want to:

• Change the pipeline conditions to match your forwarding setup.
• Customize processors to add or remove fields.
• Create two versions of the same pipeline with different conditions to handle similar events for different use cases.

To customize processing from a predefined pipeline:

1. Create a pipeline copy:

   • If you see the predefined pipeline on the Pipelines page, click the ellipsis menu and select Duplicate.
   • Otherwise, view the predefined pipeline in the library, click the ellipsis menu, and select Create editable copy.

   The pipeline copy appears in the list with (Copy) appended to the name. The original predefined pipeline is marked with a box icon.

2. Click the ellipsis next to the pipeline copy and select Edit. You can:

   • Edit the name and description.
   • Update the pipeline condition for how Lumi determines events to process.

3. Customize the processors:

   • To add a processor, click the ellipsis and select Create processors.
   • To edit a processor, expand the pipeline and click the ellipsis next to the processor. You can reorder, edit, duplicate, disable, or delete it.

4. If your pipeline copy has the same conditions as the predefined pipeline, confirm the ordering of the pipelines or disable the predefined pipeline.

The following example shows a VPC flow log processing pipeline and a duplicate:

To learn more about updating pipelines, see Update pipelines and processors.

View predefined pipelines

To view predefined pipelines:

1. In the Lumi navigation menu, click Pipelines.

2. Click Predefined pipelines.

3. In the Predefined pipelines dialog, select a predefined pipeline to view its details. A gray checkmark indicates that Lumi identified matching events and activated the predefined pipeline.

   

You can also view details for a predefined pipeline on the Pipelines page. Lumi adds it to the page when it detects matching events, or you can add it yourself. A predefined pipeline is denoted with a box icon next to the pipeline name. For more details, see View pipelines.

Add a predefined pipeline

Since predefined pipelines are enabled by default, you typically don't need to add it manually. You can choose to add a predefined pipeline when you want to control its ordering or in case of an activation delay.

To add a predefined pipeline:

1. In the Lumi navigation menu, click Pipelines.

2. Click Predefined pipelines.

3. Find the predefined pipeline you want to add. You can use the search function in the Predefined pipelines dialog.

4. Click the ellipsis, and select Add predefined pipeline.

   

Disable a predefined pipeline

You might need to disable a predefined pipeline if you duplicate and update the copy, so you don't have two sets of processing.

To disable a predefined pipeline:

1. In the Lumi navigation menu, click Pipelines.
2. Find the predefined pipeline, marked with a box icon.
   If you don't see the predefined pipeline, manually add it to the list.
3. Click the ellipsis and select Disable.

You can't delete predefined pipelines.

Learn more

See the following topics for more information:

• How to transform events with pipelines for a tutorial on using pipelines.
• Transform events using pipelines for an overview of pipelines and processors.
• Manage pipelines and processors for how to create and manage pipelines and processors.