Work with predefined pipelines

AI summary

Explains predefined pipelines in Imply Lumi for processing common event formats automatically. Covers how Lumi activates pipelines based on source type conditions. Describes workflows for simulating, reordering, duplicating, and customizing these built-in pipelines for your needs.

About AI summaries.

Predefined pipelines are built-in pipelines that Imply Lumi provides for processing specific event formats. They apply standard parsing and enrichment rules so your data is structured and ready to use as soon as it’s ingested. This saves you from creating custom processors for common event structures. You can use predefined pipelines as they are, or duplicate and customize them to fit your requirements.

In this topic, you’ll learn how to use and manage predefined pipelines.

For a list of all predefined pipelines, see Predefined pipelines reference.

How predefined pipelines work

A predefined pipeline has a fixed definition of conditions and processors to process specific log types. You can't change a predefined pipeline, but you can use it as a template to create a copy and customize it.

Predefined pipelines are automatically ready to process events. When Lumi detects a matching event based on its source type, it activates the corresponding pipeline and adds it to the Pipelines page. To prevent automatic processing, manually disable the predefined pipeline.

Note that there can be a slight delay for Lumi to activate a predefined pipeline across all event collectors, which can cause some initial events to skip processing. If you prefer that all initial events get processed, manually add the pipeline before sending events.

Source type assignment

Each predefined pipeline has a condition based on an incoming event's source or source type attribute. For example, sourcetype=aws:cloudtrail.

Lumi reads an event's sourcetype attribute from the following sources:

• Metadata on incoming events that you assign from a forwarding agent, such as in fields for a Splunk HEC request.
  For file upload, you can set source and source type in the UI.
• Default value you set on the IAM key. Only for the integrations Splunk HEC or S3 pull.
• Pipeline you create in Lumi. Order this pipeline before the predefined pipeline, so that the value is in place when Lumi evaluates the pipeline condition.

To learn how to use these approaches to assign the source type, see Assign Splunk default fields.

Choose an approach based on whether the incoming source type matches the pipeline condition and which processing option you want:

Incoming source type	Processing option	Approach
None	Predefined	Set a default value using the IAM key attribute
Doesn't match, and you don't want to keep your value	Predefined	Create a pipeline to overwrite the value, ordered before the predefined pipeline
Doesn't match, and you want to keep your value	Predefined	Duplicate the pipeline and change the condition to match your value
Doesn't match, and you want to keep your value	Your own	Create your own pipeline

Workflows

Predefined pipelines work as is, so you can use them without interruption in your event forwarding workflows. For example:

1. Check that your events match the predefined pipeline conditions, such as assigning the correct sourcetype.
   To see pipeline details, see View predefined pipelines.
2. Send events to Lumi. For details, see Send events to Lumi.
3. Explore events. For details, see Search events in Lumi.
4. From the explore view, click an event to view pipelines that processed it.

The following sections describe additional example workflows for using predefined pipelines.

Simulate pipeline

To preview how a predefined pipeline operates:

1. On the Pipelines page, find the predefined pipeline, marked with a box icon.
   If you don't see the predefined pipeline, manually add it to the list.
2. Follow the steps to simulate a pipeline.

You can also send a test event, such as with the HEC API, and view the processed event in the explore view.

Change order

Pipelines operate in the order displayed. A newly added predefined pipeline goes to the bottom of the list.

To update the pipeline order:

1. On the Pipelines page, find the predefined pipeline, marked with a box icon.
   If you don't see the predefined pipeline, manually add it to the list.
2. On the Pipelines page, click the ellipsis next to the predefined pipeline, and select Reorder.

Customize processing

The predefined pipeline is a template for processing events. You can't make changes to the predefined pipeline itself, but you can duplicate and update the copy. For example, you might want to:

• Change the pipeline conditions to match your forwarding setup.
• Customize processors to add or remove fields.
• Create two versions of the same pipeline with different conditions to handle similar events for different use cases.

To customize processing from a predefined pipeline:

1. Create a pipeline copy:

   • If you see the predefined pipeline on the Pipelines page, click the ellipsis menu and select Duplicate.
   • Otherwise, view the predefined pipeline in the library, click the ellipsis menu, and select Create editable copy.

   The pipeline copy appears in the list with (Copy) appended to the name. The original predefined pipeline is marked with a box icon.

2. Click the ellipsis next to the pipeline copy and select Edit. You can:

   • Edit the name and description.
   • Update the pipeline condition for how Lumi determines events to process.

3. Customize the processors:

   • To add a processor, click the ellipsis and select Create processors.
   • To edit a processor, expand the pipeline and click the ellipsis next to the processor. You can reorder, edit, duplicate, disable, or delete it.

4. If your pipeline copy has the same conditions as the predefined pipeline, confirm the ordering of the pipelines or disable the predefined pipeline.

The following example shows a VPC flow log processing pipeline and a duplicate:

To learn more about updating pipelines, see Update pipelines and processors.

View predefined pipelines

To view predefined pipelines:

1. In the Lumi navigation menu, click Pipelines.

2. Click Predefined pipelines.

3. In the Predefined pipelines dialog, select a predefined pipeline to view its details. A gray checkmark indicates that Lumi identified matching events and activated the predefined pipeline.

   

You can also view details for a predefined pipeline on the Pipelines page. Lumi adds it to the page when it detects matching events, or you can add it yourself. A predefined pipeline is denoted with a box icon next to the pipeline name. For more details, see View pipelines.

Add a predefined pipeline

Since predefined pipelines are enabled by default, you typically don't need to add it manually. You can choose to add a predefined pipeline when you want to control its ordering or in case of an activation delay.

To add a predefined pipeline:

1. In the Lumi navigation menu, click Pipelines.

2. Click Predefined pipelines.

3. Find the predefined pipeline you want to add. You can use the search function in the Predefined pipelines dialog.

4. Click the ellipsis, and select Add predefined pipeline.

   

Disable a predefined pipeline

You might need to disable a predefined pipeline if you duplicate and update the copy, so you don't have two sets of processing.

To disable a predefined pipeline:

1. In the Lumi navigation menu, click Pipelines.
2. Find the predefined pipeline, marked with a box icon.
   If you don't see the predefined pipeline, manually add it to the list.
3. Click the ellipsis and select Disable.

You can't delete predefined pipelines.

Learn more

See the following topics for more information:

• How to transform events with pipelines for a tutorial on using pipelines.
• Transform events using pipelines for an overview of pipelines and processors.
• Manage pipelines and processors for how to create and manage pipelines and processors.