Skip to main content

Automate pipeline creation for Splunk add-ons with Terraform

This topic describes how to convert Splunk® technology add-ons (TAs) into Imply Lumi pipelines and manage them with the Lumi Terraform provider.

When you import a Splunk TA in the Lumi UI, Lumi translates the transformations in props.conf and transforms.conf into a pipeline. You can use a Terraform workflow for this process, where you use the Lumi API to convert a Splunk TA. This creates a pipeline definition file the Terraform provider reads to build the pipeline. To update the pipeline, you edit the file and trigger a workflow to plan and apply the changes.

In the context of infrastructure as code, CI/CD automates the process to create and update pipelines, and version control lets you view previous states of the pipeline and track or revert changes.

info

Before you convert a TA, check whether Lumi already offers a predefined pipeline for your source type. You can manage a predefined pipeline with the lumi_predefined_pipeline resource instead. See Manage pipelines with Terraform.

Prerequisites​

Before you begin, make sure you have the following:

  • Access to Lumi with the Data manager role or higher. For details, see Use roles to control access.
  • Lumi Terraform provider. From the installation, you should also have the following:
    • Lumi IAM key token, stored in the environment variable LUMI_IAM_KEY_TOKEN.
    • Lumi endpoint for pipeline management, stored in the environment variable LUMI_ENDPOINT.
  • Splunk TA in .spl, .tgz, or .tar.gz format. You can use any of your existing TAs or download one from Splunk.

1. Convert the TA​

Use the Lumi API /v0/pipelines/convert/splunk-ta to convert a Splunk TA to a Lumi pipeline definition. You do this outside of Terraform. The pipeline is only created when you load the definition file into your Terraform configuration and apply changes.

Convert the Splunk TA prior to terraform plan so that you can store the pipeline definition in version control. You only need to convert the Splunk TA one time to generate and manage your pipeline. With minor updates, you can directly edit the definition file, but if there are major changes to the TA or Lumi conversion process, you can rerun the process to overwrite the definition.

Convert the TA using the following command. Replace these variables with your own values:

  • SPLUNK_TA_FILE: name of your TA, such as Splunk_TA_cisco.tgz
  • PIPELINE_FILE: name of the JSON file that stores the pipeline definition, such as cisco.json
curl -sS -H "Authorization: Bearer $LUMI_IAM_KEY_TOKEN" \
-F file=@SPLUNK_TA_FILE \
"$LUMI_ENDPOINT/v0/pipelines/convert/splunk-ta" \
| jq '{name: .template.metadata.name,
description: .template.metadata.description,
enabled: false,
searchQuery: .template.spec.searchQuery,
processors: .template.spec.processors}' \
> PIPELINE_FILE

To only convert a specific source type, add the following line to the curl command before the endpoint. Replace the SOURCETYPE values with Splunk source types, such as cisco:asa.

-F 'config={"sourcetypes":["SOURCETYPE1","SOURCETYPE2"]};type=application/json' \

The API response is a JSON body with the following structure:

  • Top-level fields:
    • template: Pipeline converted from the TA. Not returned if Lumi couldn't convert the TA into a pipeline.
    • status: Overall result of the conversion. One of the following: OK, WARNINGS, INVALID, EMPTY.
    • summary: Statistics about the conversion, such as the number of source types converted and processors generated.
    • sourcetypes: Status, statistics, and warnings grouped by each source type in the TA.
    • warnings: Warnings that don't apply to a single source type, such as the reason a pipeline isn't valid.
  • Pipeline definition fields:
    • template.metadata.name: Name of the pipeline. Defaults to the name of the TA.
    • template.metadata.description: Description of the pipeline. Defaults to a list of the converted source types.
    • template.spec.searchQuery: Pipeline condition that matches the imported TA.
    • template.spec.processors: Processors converted from the TA, with one nested pipeline for each source type

The jq command extracts components from the template field of the JSON response to build the pipeline schema. It also adds the field enabled: false to disable pipeline processing. You can update it to true after you review any warnings, check the pipeline configuration, and confirm the plan.

If you want to store the other conversion metadata, such as the summary statistics, store the entire API response in a file and filter the definition elements from it into a separate file.

Example pipeline definition JSON

The following JSON is the converted pipeline from the Splunk Add-on for Cisco ASA for the cisco:asa source type. In searchQuery, Lumi assigns the pipeline condition based on the imported TA as a whole, but you can update it to be more specific to your source type. The example truncates the processors list to only the first and last processors.

{
"name": "Splunk_TA_cisco-asa",
"description": "Process events that match sourcetype IN (\"syslog\", \"cisco_asa\", \"cisco:*\", \"cisco:asa\") OR source=\"tcp:514\" OR source=\"udp:514\".",
"enabled": false,
"searchQuery": "sourcetype IN (\"syslog\", \"cisco_asa\", \"cisco:*\", \"cisco:asa\") OR source=\"tcp:514\" OR source=\"udp:514\"",
"processors": [
{
"type": "attribute_mapper",
"sourceAttribute": {
"value": "source",
"isJsonPath": false
},
"targetAttribute": {
"value": "__lumi_arrival_source__"
},
"enabled": true,
"name": "Snapshot arrival source",
"description": "Freezes the arrival source so a later TRANSFORMS- rewriting it cannot re-trigger this or another source:: stanza's own index-time processing",
"writeMode": "OVERRIDE"
},
...
{
"type": "attribute_remover",
"targetAttributes": [
{
"value": "__lumi_arrival_sourcetype__"
},
{
"value": "__lumi_final_sourcetype__"
},
{
"value": "__lumi_arrival_source__"
},
{
"value": "__lumi_arrival_host__"
}
],
"enabled": true,
"name": "Remove snapshot attributes"
}
]
}

2. Check the conversion​

After you convert the TA, search for any processors with warnings. Lumi flags any transforms it couldn't convert exactly, such as an unsupported EVAL function, and creates the associated processor in a disabled state. In the processor list, search for the [WARNING] description prefix or for disabled processors.

If you don't need the attributes it creates, you can leave the processor disabled or delete it. Otherwise rewrite the processor to replicate the Splunk transform. For details and a step-by-step example, see Build unconverted transforms.

3. Define the pipeline resource​

In your Terraform configuration, create a pipeline using the lumi_pipeline resource. You only need to complete this step once. The resource takes a pipeline argument that expects a JSON string. Load it from a file as follows:

resource "lumi_pipeline" "cisco_pipeline" {
pipeline = file("${path.module}/pipelines/splunk_ta/cisco.json")
}

This example creates a custom pipeline with the resource name cisco_pipeline. The name of the pipeline itself comes from the name field in cisco.json.

4. Plan and apply​

Use a workflow like the following to create the pipeline. You can set your own conditions and automation for when to plan, enable, and apply changes.

  1. Commit the Terraform and pipeline files into version control.
  2. Run terraform plan and review the proposed changes.
  3. If you're ready for it to transform events, set enabled: true in the pipeline configuration and commit the change.
  4. Apply the configuration with terraform apply. The pipeline is now active.
  5. Review and update the pipeline order as needed in the Lumi UI.

Pipeline edits​

With CI/CD, you can track and automate changes to the pipeline:

  1. Edit the pipeline definition file. You can add, remove, or update processors manually, or you can regenerate the file with the Lumi API.
    • Manually update the file when you want to customize existing transforms, such as to add new attributes or modify processor configuration.
    • Regenerate the file when Lumi changes or adds baseline functionality that improves parity with Splunk transforms. For example, handling delimiters in key-value parsing. See the release notes for updates.
  2. Finally, complete the steps to commit, review, and apply the changes.

Manage multiple TAs​

To manage pipelines for several TAs, convert each TA separately and save each pipeline definition to its own file. Then define one lumi_pipeline resource for each file. For example:

resource "lumi_pipeline" "aws_pipeline" {
pipeline = file("${path.module}/pipelines/splunk_ta/aws.json")
}

resource "lumi_pipeline" "cisco_pipeline" {
pipeline = file("${path.module}/pipelines/splunk_ta/cisco.json")

depends_on = [lumi_pipeline.aws_pipeline]
}

Each pipeline is managed independently, so you can enable, update, or remove one without affecting the others.

By default, Terraform creates resources in parallel. If Lumi doesn't have any pipelines yet, creating two pipelines at the same time can fail with the following error:

HTTP 500: Record has changed since last read in table 'pipeline_configs'

To avoid the error, do one of the following:

  • Add depends_on to create the pipelines one at a time, as shown in the preceding example.
  • Run terraform apply -parallelism=1.

Creating the pipelines one at a time also controls their order. Pipelines are added in the order Terraform creates them.

Learn more​

For more information, see the following topics: