Manage pipelines with Terraform
This topic describes how to use the Imply Lumi Terraform provider to manage pipelines, which transform incoming events. With Terraform, you can define pipelines in configuration files, automate pipeline creation, and track changes in version control.
The provider supports two pipeline resources:
lumi_pipeline: Custom pipelines that you build from your own processors. See Manage custom pipelines.lumi_predefined_pipeline: Predefined pipelines that Lumi provides for standard log formats. See Manage predefined pipelines.
You can automate pipeline creation from Splunk technology add-ons (TAs). For details, see Automate pipeline creation for Splunk add-ons with Terraform.
Prerequisites
Before you proceed, complete the steps in Install the Lumi Terraform provider.
The provider requires you to set endpoint or the environment variable LUMI_ENDPOINT.
See Provider configuration for details.
Manage custom pipelines
The resource lumi_pipeline manages a custom pipeline.
Example
The following resource creates a pipeline with a single processor to parse NGINX logs with a grok expression:
resource "lumi_pipeline" "nginx" {
pipeline = jsonencode({
name = "Nginx"
searchQuery = "sourcetype=\"nginx\""
enabled = true
processors = [
{
type = "grok_parser"
name = "Parse access log"
enabled = true
parsingRules = [
"%%{IPORHOST:client_ip} - %%{NOTSPACE:user} \\[%%{HTTPDATE:timestamp}\\] \"%%{WORD:method} %%{NOTSPACE:path} HTTP/%%{NUMBER:http_version}\" %%{INT:status:int} %%{INT:bytes:int}",
]
},
]
})
}
In a Terraform string, %{ starts a template directive, so you need to escape grok patterns using %%{.
Terraform reads %%{ as a literal %{.
A definition that you load with file() needs no escaping.
Resource schema
| Name | Type | Description |
|---|---|---|
| Required | ||
pipeline | JSON string | Pipeline definition, including its processors. Requires name, searchQuery, and enabled. |
| Read-only | ||
id | String | Lumi ID for the pipeline. Use this ID to import the resource. |
name | String | Name of the pipeline from pipeline.name. |
description | String | Description of the pipeline from pipeline.description. |
enabled | Boolean | Whether the pipeline processes events from pipeline.enabled. |
search_query | String | Pipeline condition from pipeline.searchQuery. |
version | Number | Current version of the pipeline. Increments at every change, including changes made in the Lumi UI. |
applied_version | Number | Version produced by the last Terraform apply. When it differs from version, someone changed the pipeline outside Terraform so the next apply restores the configured pipeline. |
The read-only attributes repeat fields from the pipeline JSON.
Fields in the JSON use camel case, such as searchQuery, while Terraform attributes use snake case, such as search_query.
Pipeline schema
For pipeline, provide a JSON-formatted string that defines the pipeline as follows:
| Name | Type | Description |
|---|---|---|
| Required | ||
name | String | Name of the pipeline. |
searchQuery | String | Pipeline condition in Lumi query syntax. |
enabled | Boolean | Whether the pipeline processes events. |
| Optional | ||
description | String | Description of the pipeline. |
processors | Array of objects | Processors to apply to each matching event, in order. Each processor object requires the fields type, name, enabled, and any settings specific to the processor type. Optionally provide description for the processor. |
Processor schema
You can use an existing pipeline as the basis to build and modify your own processors. To do this:
- Export the pipeline from the Lumi UI.
- Edit the processors for your intended transformations. For example, updating
parsingRulesin the grok parser. - Create a resource in your Terraform configuration that references the exported YAML.
- Run
terraform planand review the proposed pipeline configuration. - Create or update the pipeline with
terraform apply.
Manage predefined pipelines
The resource lumi_predefined_pipeline manages a predefined pipeline.
You can only change its description or enable or disable it.
You can only have one instance of each predefined pipeline. If you get a 409 error when trying to create a predefined pipeline, it means the pipeline already exists. Import the resource instead of creating it.
Example
The following resource disables the Zscaler NSS predefined pipeline:
resource "lumi_predefined_pipeline" "zscaler" {
template_id = "zscaler-nss"
enabled = false
}
Resource schema
| Name | Type | Description |
|---|---|---|
| Required | ||
template_id | String | Identifier for the predefined pipeline, described below. |
| Optional | ||
enabled | Boolean | Whether the pipeline processes events. Defaults to true. |
description | String | Description of the pipeline. Omit to keep the pipeline's current description. To clear it, set to "". |
| Read-only | ||
id | String | Lumi ID for the pipeline. Use this ID to import the resource. |
name | String | Name of the predefined pipeline. |
search_query | String | Pipeline condition, such as sourcetype="fgt_event". |
For template_id, you can assign one of the following:
aws-cloudtrail: AWS CloudTrail logscrowdstrike-fdr: CrowdStrike FDR logsf5-bigip: F5 BIG-IP logsfortigate-event: FortiGate event logsfortigate-traffic: FortiGate traffic logsfortigate-utm: FortiGate UTM logslumi-ta-pan: PAN firewall logslumi-ta-pan-traps: PAN Traps logsunix-linux-ta: Unix and Linux logsvpc-flow-processing: AWS VPC flow logswindows-log-processing: Windows event logszscaler-nss: Zscaler NSS logs
If you misspell a template ID, terraform apply fails with the error Could not read the pipeline template.
To use a different template, add a new resource.
If you change template_id instead, the existing pipeline stays in Lumi as it is, and Terraform no longer manages it.
Before you change the ID, consider disabling the current resource: add enabled = false and run terraform apply.
Load definitions from files
Instead of writing the pipeline definition inline, you can store it in a separate file and reference it from your Terraform configuration.
This keeps your .tf files short and lets you review pipeline changes on their own.
You can load files from Splunk TA imports or an exported pipeline configuration from Lumi.
Update the name, description, and processors for the new pipeline as needed.
To create a new pipeline from the YAML file, extract fields from metadata and spec:
locals {
export = yamldecode(file("${path.module}/access_combined_pipeline.yaml"))
}
resource "lumi_pipeline" "access_combined_pipeline" {
pipeline = jsonencode({
name = local.export.metadata.name
description = local.export.metadata.description
enabled = false
searchQuery = local.export.spec.searchQuery
# Drop the processor IDs, which belong to the exported pipeline. Lumi assigns new ones.
processors = [
for p in local.export.spec.processors : { for k, v in p : k => v if k != "id" }
]
})
}
This creates a disabled pipeline that you can review. If you're ready to process events, set enabled = true.
Import pipelines
To import a pipeline that exists in Lumi into your Terraform state:
-
Write an empty resource block to your configuration. For example:
resource "lumi_pipeline" "mypipeline" {} -
Obtain the pipeline ID from the Lumi UI:
- Click Managed data > Pipelines from the navigation menu.
- Find your pipeline and click the information icon.
- Copy the ID value.
-
Run
terraform importwith your resource name and ID. For example:terraform import lumi_pipeline.mypipeline 01KZS47BMYT6CQYBS2EES9D3XP -
Update the resource definition to align with the current or intended state of the pipeline.
For
lumi_pipeline, Lumi doesn't return the pipeline definition, so you must write thepipelineattribute yourself. To start from the current pipeline, export it and load it from a file.
For more details on importing resources, see the HashiCorp documentation.
Delete pipelines
When you delete a pipeline from Lumi, you won't be able to use the pipeline again in configuration or the Lumi UI.
This is a destructive action.
Remove the pipeline block from your configuration, then run terraform apply.
When you stop managing a pipeline in Terraform without deleting it from Lumi, the pipeline stays in Lumi with its current settings. Do one of the following:
- Use a
removedblock withdestroy = false. Available in Terraform 1.7 or later. - Run
terraform state rmfor your resource type and name.
Note the following behavior for predefined pipelines:
- Removing a predefined pipeline resource doesn't delete the pipeline. Terraform stops managing it, and the pipeline stays in Lumi as Terraform last set it.
- To stop predefined processing, set
enabled = falseon the resource, and runterraform applybefore you remove the resource.
Unsupported settings
The following settings can only be configured in the Lumi UI and aren't available as resource arguments.
-
Pipeline order. Terraform doesn't read, manage, or change the pipeline order. Reordering in the UI doesn't change a pipeline's version, so no apply moves it back. Lumi adds each new pipeline after any existing pipelines. See Update a pipeline or processor to reorder it in the UI.
-
Predefined pipeline name and condition. The name and condition of a predefined pipeline is read-only in Terraform. If you need to make changes, use the Lumi UI to duplicate the pipeline and make changes to your own copy. You can then import the pipeline into Terraform management.
Learn more
For more information, see the following topics:
- Transform events using pipelines for an overview of pipelines and processors.
- Work with predefined pipelines to learn about predefined pipelines.
- Predefined pipelines reference to see the available predefined pipelines in Imply Lumi.
- Processors reference for the types of processors available in Lumi.