Skip to main content

How to send events through pipelines

AI summary
Explains how to transform events in Imply Lumi using pipelines with processors. Covers adding processors to mask sensitive user information and categorize events by status codes.

About AI summaries.

After you send events to Lumi, you can process them with a pipeline to extract and transform event data.

The following diagram shows where this tutorial fits in the end-to-end process of sending, transforming, and querying events. Yellow shaded boxes represent steps taken within Lumi, and blue shaded boxes represent steps taken outside Lumi. Click any box to go to that tutorial.

In this tutorial, you learn how to transform events coming into Imply Lumi using a pipeline. A pipeline is an ordered list of processors that operate on events that meet specific search criteria. A processor performs a distinct task based on its type and processing rules.

The steps show you how to:

  • Upload events to Lumi with and without using a pipeline.
  • Configure processors to redact values and categorize attributes with conditionals.

You'll then compare user attributes with and without pipeline processing.

The following diagram summarizes the end-to-end process of processing events in Lumi. Click any box in the diagram to jump to that step.

Prerequisites​

To complete the tutorial, you need the following:

  • Access to Lumi with the Data manager role or higher.
    For information on roles and permissions, see Manage roles.
  • A pipeline that processes Apache access combined logs. For instructions on how to build it, see How to build pipelines.

1. Upload data to Imply Lumi​

In this section you upload a sample data file to Lumi. The file contains a hundred lines of logs in a Lumi-compliant Apache access combined log file. See Example data for Lumi tutorials for a full description of the data.

Download the sample data file: site_visitors.log

To upload the file:

  1. Select Integrations in the navigation menu.
  2. Select File upload. Upload file tile
  3. Click Select file and select the file you downloaded. Lumi detects the format of the file as Plain text. Lumi loads a preview of the file: Uploaded file
  4. Click Upload.
  5. In the dialog that reads "File upload successful," click Go to Explore. Successful file upload
  6. On the Explore page, you will see the events your uploaded events.
  7. Click an event. Since the event is unprocessed, it has zero user attributes. Unprocessed event

2. Add processors​

In this section, you add processors, which are components of a pipeline that perform processing tasks.

This section continues from the How to build pipelines tutorial, where you build a pipeline and add a regex parser and timestamp mapper to extract and normalize timestamps from event data.

To keep the pipeline from the previous tutorial unchanged, skip to Observe the difference.

Redaction processor​

First, add a redaction processor to the pipeline to mask the IP address from an event message.

  1. On the Pipelines page, select the Apache access combined logs pipeline that you created previously.

  2. Click Create > Processor. Pipeline menu

  3. Enter the following processor details:

    • Type: Redaction processor
    • Name: Mask IP
    • Source attribute: Select Extract from log body
    • Regular expression: (?:\d{1,3}\.){3}\d{1,3}
      This regular expression looks for the IPv4 IP address format.
    • Strategy: String
    • Replacement text: [REDACTED]

    Create redaction processor

  4. Scroll down to try out the regular expression. In Sample, enter the following event message:

    112.153.199.175 - willowlim [16/Jul/2026:08:26:19 +0000] "GET /products HTTP/1.1" 200 1436 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Vivaldi/4.0.2312.27 Chrome/91.0.4472.124 Safari/537.36"

  5. Verify that Expected output replaces the IP address with [REDACTED].

    Try redaction processor

  6. Click Create.

Conditional mapper​

Next, add a conditional mapper to categorize event messages by their status code, using three conditional statements: one to target 200-level status codes, one to target 400-level status codes, and one to target 500-level status codes.

  1. On the Pipelines page, select your pipeline.

  2. Click Create > Processor.

  3. For Type, select Conditional mapper.

  4. In Name, type Categorize failed status.

  5. Enter the details for the first condition to target 200-level status codes:

    • Condition: status>=200 AND status<300
    • Mapper type: Value
    • Value/Attribute: success

  6. Click + Add condition.

  7. Enter the details for the second condition to target 400-level status codes:

    • Condition: status>=400 AND status<500
    • Mapper type: Value
    • Value/Attribute: client_error

  8. Click + Add condition.

  9. Enter the details for the final condition to target 500-level status codes:

    • Condition: status>=500
    • Mapper type: Value
    • Value/Attribute: server_error

    Conditional mapper processing rules

  10. In Output attribute, type status_type.

  11. In the Try it out section, copy and paste the following JSON in Sample attributes:

    {"method": "GET", "status": 200}

  12. In Expected output, verify that success is output for the 200-level status.

    Try conditional mapper

  13. Click Create.

3. Observe the difference​

In this section, you upload more data to Lumi to process with your pipeline and compare the results to unprocessed data.

  1. Follow the steps in Upload data to Imply Lumi to upload more data. In the Attributes pane, fill out the following attributes to target your pipeline's conditions:

    • Source type: access_combined
    • Index: tutorial Filled out event information

  2. Click Upload.

  3. In the dialog that reads "File upload successful," click Go to Explore.

  4. Adjust the time filter if needed to display the processed events.

  5. Select an event and view its attributes. Since the event has been processed by your pipeline, user attributes have been extracted by its regex parser and the timestamp has been normalized by its timestamp mapper.

    Processed event Notice that index and sourcetype are also present as user attributes since they were specified during file upload.

    If you added the redaction process and conditional mapper to your pipeline, also notice that the client's IP address in the event message has been redacted and status_type has been added as a user attribute.

Learn more​

See the following topics for more information: