Work with predefined pipelines

AI summary

Predefined pipelines are ready-to-use data processing templates in Imply Lumi that automatically parse and structure common data types during ingestion, eliminating the need to build custom processors from scratch for standard sources.

About AI summaries.

Predefined pipelines are built-in pipelines that Imply Lumi provides for processing specific event formats. They apply standard parsing and enrichment rules so your data is structured and ready to use as soon as it’s ingested. This saves you from creating custom processors for common event structures.

You can use predefined pipelines as they are, or customize them to fit your requirements.

In this topic, you’ll learn how to add and manage predefined pipelines.

For a list of all predefined pipelines, see Predefined pipelines reference.

Workflow

The following workflow describes how you can send and process incoming events with a standardized log format:

1. Add the predefined pipeline that matches your log format.

2. Review the processors in the pipeline definition. If you need to make changes to the processors, duplicate the predefined pipeline and edit as necessary.

3. Review the pipeline conditions, such as sourcetype=LOG_TYPE, and ensure that it matches your event forwarding setup.

4. Confirm or update the order of the pipeline. Pipelines operate in the order displayed.

5. Send logs to Lumi using your chosen ingestion integration.

6. Check the Lumi integration to preview incoming data. In this example, an S3 ingest action added VPC flow logs:

   

7. Query Lumi to see the enriched events. The event details pane shows the pipeline that processed the event:

   

Add a predefined pipeline

To add a predefined pipeline:

1. In the Lumi navigation menu, click Pipelines.
2. Click Predefined pipelines.
3. Select a predefined pipeline and click Add pipeline. The pipeline appears at the bottom of the list.
4. Click the ellipsis next to the pipeline and select Edit. You can:
   • Edit the name and description.
   • Update the pipeline condition if you want to change how Lumi filters events for the pipeline. See Pipeline conditions for more information.
5. Use the ellipsis menu to reorder, duplicate, or disable the pipeline.

Click the arrow next to the pipeline name to expand it and view its processors. To edit the processors, see Customize a predefined pipeline.

Customize a predefined pipeline

To customize a predefined pipeline, first add the pipeline and then duplicate it. You can then make changes to the copy.

You might customize processors to add or remove fields. You might also create two versions of the same pipeline with different conditions to handle similar events for different use cases.

If you only need to change the conditions for a predefined pipeline, you can edit the pipeline directly.

To customize processors in a predefined pipeline:

1. Add a predefined pipeline and create a duplicate. The pipeline copy appears in the list with (Copy) appended to the name. The original predefined pipeline is marked with a box icon.
2. Click the ellipsis next to the pipeline copy and edit its details as required.
3. Customize the pipeline:
   • To add processors, click the ellipsis and select Add processors.
     See Create a processor for more information.
   • To edit processors, expand the pipeline and click the ellipsis next to a processor. Reorder, edit, duplicate, disable, or delete a processor.

The following example shows a VPC flow log processing pipeline and a duplicate:

Learn more

See the following topics for more information:

• How to transform events with pipelines for a tutorial on using pipelines.
• Transform events using pipelines for an overview of pipelines and processors.
• Manage pipelines and processors for how to create and manage pipelines and processors.