Skip to main content

Work with nested pipelines

AI summary
Explains how to nest Imply Lumi pipelines to organize event processing logic, apply conditional filtering based on source types, and perform sequential transformations.

About AI summaries.

You can nest pipelines to organize them, group related logic, and apply extra filtering before transforming events.

For example, you might have a single source of events containing logs from different source types. You can add a top-level condition on the pipeline to store metadata on the source. Then you can filter events based on their source type to perform transformations specific to that type.

A pipeline can contain up to three levels of nesting. If a parent pipeline includes processors positioned after nested pipelines, those processors run on the results of the nested processing.

Create a nested pipeline​

Create a nested pipeline as follows:

  1. Click Pipelines from the navigation menu.
  2. Select a pipeline, then click Create > Nested pipeline.
  3. Follow the same steps as creating a pipeline to define its name, description, and conditions.

Example​

The following diagram shows an example of sequential processing by a pipeline with nested elements:

  • Parent pipeline filters incoming events by source=var.
  • Value mapper [1] adds metadata to events.
  • Access combined path: nested pipeline filters by sourcetype=access_combined for regex parsing [2A].
  • VPC flow path: nested pipeline events filters by sourcetype=vpcflow for grok parsing [3A] and timestamp mapping [3B].
  • Attribute remover [4] performs final cleanup before storage.

Example flow of pipeline nesting

Learn more​

See the following topics for more information: