Automate pipeline creation for Splunk add-ons with Terraform
This topic describes how to convert Splunk® technology add-ons (TAs) into Imply Lumi pipelines and manage them with the Lumi Terraform provider.
When you import a Splunk TA in the Lumi UI, Lumi translates the transformations in props.conf and transforms.conf into a pipeline.
You can use a Terraform workflow for this process, where you use the Lumi API to convert a Splunk TA.
This creates a pipeline definition file the Terraform provider reads to build the pipeline.
To update the pipeline, you edit the file and trigger a workflow to plan and apply the changes.
In the context of infrastructure as code, CI/CD automates the process to create and update pipelines, and version control lets you view previous states of the pipeline and track or revert changes.
Before you convert a TA, check whether Lumi already offers a predefined pipeline for your source type.
You can manage a predefined pipeline with the lumi_predefined_pipeline resource instead.
See Manage pipelines with Terraform.
Prerequisites
Before you begin, make sure you have the following:
- Access to Lumi with the Data manager role or higher. For details, see Use roles to control access.
- Lumi Terraform provider. From the installation, you should also have the following:
- Lumi IAM key token, stored in the environment variable
LUMI_IAM_KEY_TOKEN. - Lumi endpoint for pipeline management, stored in the environment variable
LUMI_ENDPOINT.
- Lumi IAM key token, stored in the environment variable
- Splunk TA in
.spl,.tgz, or.tar.gzformat. You can use any of your existing TAs or download one from Splunk.
1. Convert the TA
Use the Lumi API /v0/pipelines/convert/splunk-ta to convert a Splunk TA to a Lumi pipeline definition.
You do this outside of Terraform.
The pipeline is only created when you load the definition file into your Terraform configuration and apply changes.
Convert the Splunk TA prior to terraform plan so that you can store the pipeline definition in version control.
You only need to convert the Splunk TA one time to generate and manage your pipeline.
With minor updates, you can directly edit the definition file, but if there are major changes to the TA or Lumi conversion process, you can rerun the process to overwrite the definition.
Convert the TA using the following command. Replace these variables with your own values:
SPLUNK_TA_FILE: name of your TA, such asSplunk_TA_cisco.tgzPIPELINE_FILE: name of the JSON file that stores the pipeline definition, such ascisco.json
curl -sS -H "Authorization: Bearer $LUMI_IAM_KEY_TOKEN" \
-F file=@SPLUNK_TA_FILE \
"$LUMI_ENDPOINT/v0/pipelines/convert/splunk-ta" \
| jq '{name: .template.metadata.name,
description: .template.metadata.description,
enabled: false,
searchQuery: .template.spec.searchQuery,
processors: .template.spec.processors}' \
> PIPELINE_FILE
To only convert a specific source type, add the following line to the curl command before the endpoint.
Replace the SOURCETYPE values with Splunk source types, such as cisco:asa.
-F 'config={"sourcetypes":["SOURCETYPE1","SOURCETYPE2"]};type=application/json' \
The API response is a JSON body with the following structure:
- Top-level fields:
template: Pipeline converted from the TA. Not returned if Lumi couldn't convert the TA into a pipeline.status: Overall result of the conversion. One of the following:OK,WARNINGS,INVALID,EMPTY.summary: Statistics about the conversion, such as the number of source types converted and processors generated.sourcetypes: Status, statistics, and warnings grouped by each source type in the TA.warnings: Warnings that don't apply to a single source type, such as the reason a pipeline isn't valid.
- Pipeline definition fields:
template.metadata.name: Name of the pipeline. Defaults to the name of the TA.template.metadata.description: Description of the pipeline. Defaults to a list of the converted source types.template.spec.searchQuery: Pipeline condition that matches the imported TA.template.spec.processors: Processors converted from the TA, with one nested pipeline for each source type
The jq command extracts components from the template field of the JSON response to build the pipeline schema.
It also adds the field enabled: false to disable pipeline processing.
You can update it to true after you review any warnings, check the pipeline configuration, and confirm the plan.
If you want to store the other conversion metadata, such as the summary statistics, store the entire API response in a file and filter the definition elements from it into a separate file.
Example pipeline definition JSON
The following JSON is the converted pipeline from the Splunk Add-on for Cisco ASA for the cisco:asa source type.
In searchQuery, Lumi assigns the pipeline condition based on the imported TA as a whole, but you can update it to be more specific to your source type.
The example truncates the processors list to only the first and last processors.
{
"name": "Splunk_TA_cisco-asa",
"description": "Process events that match sourcetype IN (\"syslog\", \"cisco_asa\", \"cisco:*\", \"cisco:asa\") OR source=\"tcp:514\" OR source=\"udp:514\".",
"enabled": false,
"searchQuery": "sourcetype IN (\"syslog\", \"cisco_asa\", \"cisco:*\", \"cisco:asa\") OR source=\"tcp:514\" OR source=\"udp:514\"",
"processors": [
{
"type": "attribute_mapper",
"sourceAttribute": {
"value": "source",
"isJsonPath": false
},
"targetAttribute": {
"value": "__lumi_arrival_source__"
},
"enabled": true,
"name": "Snapshot arrival source",
"description": "Freezes the arrival source so a later TRANSFORMS- rewriting it cannot re-trigger this or another source:: stanza's own index-time processing",
"writeMode": "OVERRIDE"
},
...
{
"type": "attribute_remover",
"targetAttributes": [
{
"value": "__lumi_arrival_sourcetype__"
},
{
"value": "__lumi_final_sourcetype__"
},
{
"value": "__lumi_arrival_source__"
},
{
"value": "__lumi_arrival_host__"
}
],
"enabled": true,
"name": "Remove snapshot attributes"
}
]
}
2. Check the conversion
After you convert the TA, search for any processors with warnings.
Lumi flags any transforms it couldn't convert exactly, such as an unsupported EVAL function, and creates the associated processor in a disabled state.
In the processor list, search for the [WARNING] description prefix or for disabled processors.
If you don't need the attributes it creates, you can leave the processor disabled or delete it. Otherwise rewrite the processor to replicate the Splunk transform. For details and a step-by-step example, see Build unconverted transforms.
3. Define the pipeline resource
In your Terraform configuration, create a pipeline using the lumi_pipeline resource.
You only need to complete this step once.
The resource takes a pipeline argument that expects a JSON string. Load it from a file as follows:
resource "lumi_pipeline" "cisco_pipeline" {
pipeline = file("${path.module}/pipelines/splunk_ta/cisco.json")
}
This example creates a custom pipeline with the resource name cisco_pipeline.
The name of the pipeline itself comes from the name field in cisco.json.
4. Plan and apply
Use a workflow like the following to create the pipeline. You can set your own conditions and automation for when to plan, enable, and apply changes.
- Commit the Terraform and pipeline files into version control.
- Run
terraform planand review the proposed changes. - If you're ready for it to transform events, set
enabled: truein the pipeline configuration and commit the change. - Apply the configuration with
terraform apply. The pipeline is now active. - Review and update the pipeline order as needed in the Lumi UI.
Pipeline edits
With CI/CD, you can track and automate changes to the pipeline:
- Edit the pipeline definition file. You can add, remove, or update processors manually, or you can regenerate the file with the Lumi API.
- Manually update the file when you want to customize existing transforms, such as to add new attributes or modify processor configuration.
- Regenerate the file when Lumi changes or adds baseline functionality that improves parity with Splunk transforms. For example, handling delimiters in key-value parsing. See the release notes for updates.
- Finally, complete the steps to commit, review, and apply the changes.
Manage multiple TAs
To manage pipelines for several TAs, convert each TA separately and save each pipeline definition to its own file.
Then define one lumi_pipeline resource for each file.
For example:
resource "lumi_pipeline" "aws_pipeline" {
pipeline = file("${path.module}/pipelines/splunk_ta/aws.json")
}
resource "lumi_pipeline" "cisco_pipeline" {
pipeline = file("${path.module}/pipelines/splunk_ta/cisco.json")
depends_on = [lumi_pipeline.aws_pipeline]
}
Each pipeline is managed independently, so you can enable, update, or remove one without affecting the others.
By default, Terraform creates resources in parallel. If Lumi doesn't have any pipelines yet, creating two pipelines at the same time can fail with the following error:
HTTP 500: Record has changed since last read in table 'pipeline_configs'
To avoid the error, do one of the following:
- Add
depends_onto create the pipelines one at a time, as shown in the preceding example. - Run
terraform apply -parallelism=1.
Creating the pipelines one at a time also controls their order. Pipelines are added in the order Terraform creates them.
Learn more
For more information, see the following topics:
- Manage pipelines with Terraform to learn how to manage pipeline resources with the Lumi Terraform provider.
- Build pipelines from Splunk technology add-ons to learn how to convert Splunk TAs in the Lumi UI.
- Lumi concepts for Splunk users to learn about how Splunk TAs conceptually map to Lumi pipelines.