Skip to main content

Work with nested pipelines

AI summary
Explains how to create nested pipelines in Imply Lumi to organize logic and filter events. Covers nesting up to three levels deep. Describes how parent pipeline processors handle results from nested processing.

About AI summaries.

You can nest pipelines to organize them, group related logic, and apply extra filtering before transforming events.

For example, you might have a single source of events containing logs from different source types. You can add a top-level condition on the pipeline to store metadata on the source. Then you can filter events based on their source type to perform transformations specific to that type.

A pipeline can contain up to three levels of nesting. If a parent pipeline includes processors positioned after nested pipelines, those processors run on the results of the nested processing.

Create a nested pipeline​

Create a nested pipeline as follows:

  1. Click Managed data > Pipelines from the navigation menu.
  2. Select a pipeline, then click Create > Nested pipeline.
  3. Follow the same steps as creating a pipeline to define its name, description, and conditions.

Example​

The following diagram shows an example of sequential processing by a pipeline with nested elements:

  • Parent pipeline filters incoming events by source=var.
  • Value mapper [1] adds metadata to events.
  • Access combined path: nested pipeline filters by sourcetype=access_combined for regex parsing [2A].
  • VPC flow path: nested pipeline events filters by sourcetype=vpcflow for grok parsing [3A] and timestamp mapping [3B].
  • Attribute remover [4] performs final cleanup before storage.

Example flow of pipeline nesting

Learn more​

See the following topics for more information: