Skip to main content

Manage pipelines with Terraform

This topic describes how to use the Imply Lumi Terraform provider to manage pipelines, which transform incoming events. With Terraform, you can define pipelines in configuration files, automate pipeline creation, and track changes in version control.

The provider supports two pipeline resources:

info

You can automate pipeline creation from Splunk technology add-ons (TAs). For details, see Automate pipeline creation for Splunk add-ons with Terraform.

Prerequisites​

Before you proceed, complete the steps in Install the Lumi Terraform provider.

The provider requires you to set endpoint or the environment variable LUMI_ENDPOINT. See Provider configuration for details.

Manage custom pipelines​

The resource lumi_pipeline manages a custom pipeline.

Example​

The following resource creates a pipeline with a single processor to parse NGINX logs with a grok expression:

resource "lumi_pipeline" "nginx" {
pipeline = jsonencode({
name = "Nginx"
searchQuery = "sourcetype=\"nginx\""
enabled = true
processors = [
{
type = "grok_parser"
name = "Parse access log"
enabled = true
parsingRules = [
"%%{IPORHOST:client_ip} - %%{NOTSPACE:user} \\[%%{HTTPDATE:timestamp}\\] \"%%{WORD:method} %%{NOTSPACE:path} HTTP/%%{NUMBER:http_version}\" %%{INT:status:int} %%{INT:bytes:int}",
]
},
]
})
}

In a Terraform string, %{ starts a template directive, so you need to escape grok patterns using %%{. Terraform reads %%{ as a literal %{. A definition that you load with file() needs no escaping.

Resource schema​

NameTypeDescription
Required
pipelineJSON stringPipeline definition, including its processors.
Requires name, searchQuery, and enabled.
Read-only
idStringLumi ID for the pipeline.
Use this ID to import the resource.
nameStringName of the pipeline from pipeline.name.
descriptionStringDescription of the pipeline from pipeline.description.
enabledBooleanWhether the pipeline processes events from pipeline.enabled.
search_queryStringPipeline condition from pipeline.searchQuery.
versionNumberCurrent version of the pipeline.
Increments at every change, including changes made in the Lumi UI.
applied_versionNumberVersion produced by the last Terraform apply.
When it differs from version, someone changed the pipeline outside Terraform so the next apply restores the configured pipeline.

The read-only attributes repeat fields from the pipeline JSON. Fields in the JSON use camel case, such as searchQuery, while Terraform attributes use snake case, such as search_query.

Pipeline schema​

For pipeline, provide a JSON-formatted string that defines the pipeline as follows:

NameTypeDescription
Required
nameStringName of the pipeline.
searchQueryStringPipeline condition in Lumi query syntax.
enabledBooleanWhether the pipeline processes events.
Optional
descriptionStringDescription of the pipeline.
processorsArray of objectsProcessors to apply to each matching event, in order.
Each processor object requires the fields type, name, enabled, and any settings specific to the processor type. Optionally provide description for the processor.

Processor schema​

You can use an existing pipeline as the basis to build and modify your own processors. To do this:

  1. Export the pipeline from the Lumi UI.
  2. Edit the processors for your intended transformations. For example, updating parsingRules in the grok parser.
  3. Create a resource in your Terraform configuration that references the exported YAML.
  4. Run terraform plan and review the proposed pipeline configuration.
  5. Create or update the pipeline with terraform apply.

Manage predefined pipelines​

The resource lumi_predefined_pipeline manages a predefined pipeline. You can only change its description or enable or disable it.

You can only have one instance of each predefined pipeline. If you get a 409 error when trying to create a predefined pipeline, it means the pipeline already exists. Import the resource instead of creating it.

Example​

The following resource disables the Zscaler NSS predefined pipeline:

resource "lumi_predefined_pipeline" "zscaler" {
template_id = "zscaler-nss"
enabled = false
}

Resource schema​

NameTypeDescription
Required
template_idStringIdentifier for the predefined pipeline, described below.
Optional
enabledBooleanWhether the pipeline processes events.
Defaults to true.
descriptionStringDescription of the pipeline.
Omit to keep the pipeline's current description. To clear it, set to "".
Read-only
idStringLumi ID for the pipeline.
Use this ID to import the resource.
nameStringName of the predefined pipeline.
search_queryStringPipeline condition, such as sourcetype="fgt_event".

For template_id, you can assign one of the following:

  • aws-cloudtrail: AWS CloudTrail logs
  • crowdstrike-fdr: CrowdStrike FDR logs
  • f5-bigip: F5 BIG-IP logs
  • fortigate-event: FortiGate event logs
  • fortigate-traffic: FortiGate traffic logs
  • fortigate-utm: FortiGate UTM logs
  • lumi-ta-pan: PAN firewall logs
  • lumi-ta-pan-traps: PAN Traps logs
  • unix-linux-ta: Unix and Linux logs
  • vpc-flow-processing: AWS VPC flow logs
  • windows-log-processing: Windows event logs
  • zscaler-nss: Zscaler NSS logs

If you misspell a template ID, terraform apply fails with the error Could not read the pipeline template.

To use a different template, add a new resource. If you change template_id instead, the existing pipeline stays in Lumi as it is, and Terraform no longer manages it. Before you change the ID, consider disabling the current resource: add enabled = false and run terraform apply.

Load definitions from files​

Instead of writing the pipeline definition inline, you can store it in a separate file and reference it from your Terraform configuration. This keeps your .tf files short and lets you review pipeline changes on their own. You can load files from Splunk TA imports or an exported pipeline configuration from Lumi.

Update the name, description, and processors for the new pipeline as needed. To create a new pipeline from the YAML file, extract fields from metadata and spec:

locals {
export = yamldecode(file("${path.module}/access_combined_pipeline.yaml"))
}

resource "lumi_pipeline" "access_combined_pipeline" {
pipeline = jsonencode({
name = local.export.metadata.name
description = local.export.metadata.description
enabled = false
searchQuery = local.export.spec.searchQuery
# Drop the processor IDs, which belong to the exported pipeline. Lumi assigns new ones.
processors = [
for p in local.export.spec.processors : { for k, v in p : k => v if k != "id" }
]
})
}

This creates a disabled pipeline that you can review. If you're ready to process events, set enabled = true.

Import pipelines​

To import a pipeline that exists in Lumi into your Terraform state:

  1. Write an empty resource block to your configuration. For example:

    resource "lumi_pipeline" "mypipeline" {
    }
  2. Obtain the pipeline ID from the Lumi UI:

    1. Click Managed data > Pipelines from the navigation menu.
    2. Find your pipeline and click the information icon.
    3. Copy the ID value.
  3. Run terraform import with your resource name and ID. For example:

    terraform import lumi_pipeline.mypipeline 01KZS47BMYT6CQYBS2EES9D3XP
  4. Update the resource definition to align with the current or intended state of the pipeline.

    For lumi_pipeline, Lumi doesn't return the pipeline definition, so you must write the pipeline attribute yourself. To start from the current pipeline, export it and load it from a file.

For more details on importing resources, see the HashiCorp documentation.

Delete pipelines​

When you delete a pipeline from Lumi, you won't be able to use the pipeline again in configuration or the Lumi UI. This is a destructive action. Remove the pipeline block from your configuration, then run terraform apply.

When you stop managing a pipeline in Terraform without deleting it from Lumi, the pipeline stays in Lumi with its current settings. Do one of the following:

  • Use a removed block with destroy = false. Available in Terraform 1.7 or later.
  • Run terraform state rm for your resource type and name.

Note the following behavior for predefined pipelines:

  • Removing a predefined pipeline resource doesn't delete the pipeline. Terraform stops managing it, and the pipeline stays in Lumi as Terraform last set it.
  • To stop predefined processing, set enabled = false on the resource, and run terraform apply before you remove the resource.

Unsupported settings​

The following settings can only be configured in the Lumi UI and aren't available as resource arguments.

  • Pipeline order. Terraform doesn't read, manage, or change the pipeline order. Reordering in the UI doesn't change a pipeline's version, so no apply moves it back. Lumi adds each new pipeline after any existing pipelines. See Update a pipeline or processor to reorder it in the UI.

  • Predefined pipeline name and condition. The name and condition of a predefined pipeline is read-only in Terraform. If you need to make changes, use the Lumi UI to duplicate the pipeline and make changes to your own copy. You can then import the pipeline into Terraform management.

Learn more​

For more information, see the following topics: