Skip to main content

Processors reference

Within pipelines, processors perform the tasks to transform events. Each processor performs a unique action, such as parsing the event message or removing incoming metadata. Lumi stores unprocessed metadata from incoming events as user attributes.

This topic describes the available processors in Lumi.

To learn how to create pipelines and processors, see Transform events with pipelines.

Processor settings​

Use the following guidelines when configuring processors.

Source and output attributes​

Specify your source and output attributes following these rules:

  • Assign a source attribute from incoming event metadata or an attribute created by a previous processor.
  • Refer to the source attribute, such as clientip.
  • You can use the event message as the source attribute in select processors when specified.
  • You can't use system attributes as source or output attributes.
    Note that you can use them in the pipeline conditions.

Override output attributes​

Some mapper processors allow you to override an existing user attribute. This overwrites the original attribute and assigns it a new value. For example, for incoming metadata source: httpevent, you can choose whether to preserve the original value or reassign its value in a mapping.

When available on the processor, select the toggle Override value when output attribute exists. The override applies even when the input value is an empty string or one or more whitespace characters. An exception is when the source attribute is missing or its value is null, in which case processing is skipped.

If you don't select the override option, no processing occurs when the output attribute already exists.

Remove mapped attributes​

When you map an attribute, the processor doesn't remove the source. To remove it, use the attribute remover.

Arithmetic processor​

Evaluates an arithmetic formula and outputs the result to an attribute.

You can reference existing attributes as variables in the formula. The formula supports the basic operators for addition (+), subtaction (-), multiplication (*), and division (/). Parentheses (()) control the order of operations.

In the arithmetic formula, surround operators with space characters. For example, val1 - val2 is a valid subtraction formula. Without the space characters, the processor treats val1-val2 as a single attribute.

If you reference a nonexistent attribute, Lumi either replaces it with zero or skips processing. To replace with zero, select the toggle Replace invalid values with zero. Otherwise, Lumi skips processing and doesn't evaluate the formula.

The processor stores the formula result in the specified output user attribute. If there’s already an attribute with the same name, you can override its value or leave it unchanged.

Example
Processor configuration
Arithmetic formula: (val1 + val2) / (val4 - val3)
Output attribute: computed
Event input
Event metadata: 
val1: 5
val2: 8
val3: 11
val4: 14
Event output
User attribute: computed: 4.333

Attribute mapper​

Maps the value of a source attribute to an output user attribute.

The processor creates a new attribute when it doesn't exist. If there's already an attribute with the same name, you can override its value or leave it unchanged.

Example
Processor configuration
Source attribute: status
Output attribute: http_status
Event input
Event metadata: status: 401
Event output
User attribute: http_status: 401

Attribute remover​

Removes one or more source attributes.

Use this processor to drop unneeded fields to reduce storage size and improve query performance. You can also use the attribute remover to drop personally identifiable information, whether to remove it completely or to remove the source metadata after redaction.

Example
Processor configuration
Attributes to remove: userid
Event input
Event metadata: userid: wilma
Event output
User attribute: none

Conditional mapper​

Evaluates one or more conditions, and maps a source attribute or value to an output user attribute. When no conditions are satisifed, no mapping occurs.

Specify one or more conditions to evaluate to determine the mapping behavior. The processor evaluates conditions from highest to lowest priority and applies the mapping for the first condition that's satisfied. Create a separate processor for each output user attribute.

A condition takes the following components:

  • Search expression in Lumi query syntax
  • Type of mapping to perform, whether a value mapper or attribute mapper
  • Configuration based on the mapper type:
    • For a value mapper, a static value
    • For an attribute mapper, the name of the source attribute

After you specify the conditions, provide the name of the attribute to store the mapped value. You can select whether to override the attribute if it already exists.

Example

Consider a static value replacement only for events that have a specific source type.

Processor configuration
Condition: sourcetype=access_combined
Mapper type: Value
Value / Attribute: redacted
Output attribute: user
Event input
Event metadata: 
sourcetype: access_combined
user: wilma
Event output
User attributes: 
sourcetype: access_combined
user: redacted

This configuration ensures that events store the user attribute user: redacted when the event satisfies the pipeline condition as well as the condition sourcetype=access_combined.

Grok parser​

Parses a source attribute into one or more output attributes using a grok expression. You can use the event message as the source attribute.

The grok parser extracts structured data when it matches the specified expression, similar to the regex parser. Grok expressions tend to be more human-readable than regex because they use preset templates to represent common string patterns. For example, instead of writing an ISO 8601 format in regex, you can specify the pattern TIMESTAMP_ISO8601.

A grok pattern stores the name of the output attribute directly in the pattern. A grok expression is made up of one or more grok patterns in the following format:

%{PATTERN_NAME:OUTPUT}

PATTERN_NAME identifies a preset pattern, and OUTPUT is the label you assign to the output value that Lumi stores as a user attribute.

For a reference on the available patterns, see Grok patterns. Note that you can test your grok patterns using an online parser such as Grok Debugger before you add them to a processor.

Example
Processor configuration
Source attribute: Select the option to Extract from log body
Grok expression: %{TIMESTAMP_ISO8601:time} %{LOGLEVEL:status}: %{GREEDYDATA:message}
Event input
Event message: 2025-08-05 15:45:00 INFO: Starting application...
Event output
User attributes: 
time: 2025-08-05 15:45:00
status: INFO
message: Starting application...

For examples of how to map the extracted values to other event components, see the timestamp mapper and message mapper.

Example with Apache combined log format

This example parses a log in Apache combined log format as represented in the tutorial data.

Processor configuration
Source attribute: Select the option to Extract from log body
Grok expression: 
%{IP:clientip} %{DATA:identd} %{DATA:user} \[%{HTTPDATE:time}\] "%{WORD:method} %{DATA:uri} %{DATA:protocol}" %{NUMBER:status} %{NUMBER:bytes} "%{URI:referer}" "%{GREEDYDATA:useragent}"
Event input
Event message: 
830:1e0e:525:e6a0:6479:cd69:c364:23c3 - - [24/Mar/2025:16:25:29 -0500] "POST /products/23394 HTTP/1.1" 200 1027 "https://techcrunch.com/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:110.0) Gecko/20100101 Firefox/110.0"
Event output
User attributes: 
bytes: 1027
clientip: 830:1e0e:525:e6a0:6479:cd69:c364:23c3
identd: -
method: POST
protocol: HTTP/1.1
referer: https://techcrunch.com/
status: 200
time: 24/Mar/2025:16:25:29 -0500
uri: /products/23394
user: -
useragent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:110.0) Gecko/20100101 Firefox/110.0

Lookup mapper​

Looks up source attributes in a user-provided CSV lookup table, and creates one or more output attributes based on the table columns.

You can set the delimiter to another character, such as ;. Be sure to match the delimiter to your lookup table. For example, the delimiter , is different from , .

Designate one or more source attributes as the lookup IDs in the table. The processor uses the ID columns to look up the matching row and creates output user attributes from the specified columns. The source attributes are also user attributes on the event.

The source and output attributes must match the names of the provided headers. You can provide the column headers as part of the lookup CSV or as comma-separated values in the Headers field. If your events already contain the output attributes, you can designate whether to overwrite existing values.

Consider an example lookup table:

product_idcategorydescription
23394FurnitureLeather Sectional Sofa
32729ElectronicsRaspberry Pi 5
23002BooksMan's Search for Meaning
23394InstrumentsAnalog Theremin
78905JewelryArt Deco Diamond and Silver Bracelet

If product_id is the source attribute, the processor can create user attributes for category and description when it identifies a row matching the product ID. You can specify category, description, or both for the output attributes. The processor doesn't create user attributes when it doesn't identify a match.

Example

This example adds the description user attribute for events that store a specific product ID and category.

Processor configuration
Headers: Lookup CSV includes header line
Lookup CSV: 
product_id,category,description
23394,Furniture,Leather Sectional Sofa
32729,Electronics,Raspberry Pi 5
23002,Books,Man's Search for Meaning
23394,Instruments,Analog Theremin
28201,Jewelry,Art Deco Diamond and Silver Bracelet
Delimiter: ,
Source attributes: product_id,category
Output attribute: description
Event input
Event metadata: 
product_id: 23394
category: Instruments
Event output
User attributes: 
product_id: 23394
category: Instruments
description: Analog Theremin

Note that if you only select product_id as the source attribute, the resulting user attribute would be description: Leather Sectional Sofa, since it's the first matched row for product ID 23394.

Message mapper​

Maps the value of a source attribute to the event message.

You have the option to overwrite the event message with an empty string when the source attribute is missing or empty.

Example
Preceding processor
Grok parser to extract message: Starting application...
Processor configuration
Source attribute: message
Event input
Event message: 2025-08-05 15:45:00 INFO: Starting application...
Event output
Event message: Starting application...

Regex parser​

Parses a source attribute into one or more output attributes using a regular expression. You can use the event message as the source attribute.

The number of capturing groups in the regular expression determines the number of output attributes. If a capturing group matches more than one result, the processor only uses the first result. For example, for the regex pattern (abc), the test string abc abc would only return the first abc.

If the incoming metadata already includes any of the output attributes, the processor overrides the existing metadata when it matches the regex pattern. This behavior applies even if the match is an empty string or whitespace character.

Note that you can use a free regex parser such as Regex101 to test regular expressions before you add them to a processor.

Example
Processor configuration
Source attribute: Select the option to Extract from log body
Regular expression: status: \[(\w*)\]
Output attributes: status
Event input
Event message: Deployment successful. System 1 status: [ok] System 2 status: [alert]
Event output
User attribute: status: ok
Example with Apache combined log format

This example parses a log in Apache combined log format as represented in the tutorial data.

Processor configuration
Source attribute: Select the option to Extract from log body
Regular expression: 
([^ ]*) ([^ ]*) ([^ ]*) \[([^\]]*)\] "(\S+)(?: +([^\"]*?)(?: +(\S+))?)?" ([^ ]*) ([^ ]*)(?: "([^\"]*)" "([^\"]*)")?
Output attributes: 
clientip, identd, user, time, method, uri, protocol, status, bytes, referer, useragent
Event input
Event message: 
830:1e0e:525:e6a0:6479:cd69:c364:23c3 - - [24/Mar/2025:16:25:29 -0500] "POST /products/23394 HTTP/1.1" 200 1027 "https://techcrunch.com/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:110.0) Gecko/20100101 Firefox/110.0"
Event output
User attributes: 
bytes: 1027
clientip: 830:1e0e:525:e6a0:6479:cd69:c364:23c3
identd: -
method: POST
protocol: HTTP/1.1
referer: https://techcrunch.com/
status: 200
time: 24/Mar/2025:16:25:29 -0500
uri: /products/23394
user: -
useragent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:110.0) Gecko/20100101 Firefox/110.0

Status mapper​

Maps the value of a source attribute to the event status.

Lumi attempts to map status codes to human-readable values. For example, the HTTP status code 500 maps to Error. For more information, see the system attribute for status.

You can optionally include a fallback value that Lumi sets for the status when the source attribute doesn't exist or if it's unable to be interpreted.

Example
Processor configuration
Source attribute: http_code
Event input
Event metadata: http_code: 200
Event output
System attribute: status: ok

Note that status here is a system attribute, not a user attribute. If you want to remove the source attribute after the status mapping, use the attribute remover.

Timestamp mapper​

Maps the value of a source attribute to the event timestamp.

By default, Lumi assigns the event timestamp from when it received the event. When you want to assign the timestamp from part of the event message, use another processor to parse the timestamp portion from the message then assign it to the event timestmap with the timestamp mapper.

In the configuration, provide the name of the source attribute, format of the timestamp, and optionally the time zone. When the timestamp includes the time zone, such as 2023-10-26T15:30:00-05:00, Lumi uses that value. If the timestamp doesn't have a time zone, you can select a different location (UTC by default).

Example
Preceding processor
Grok parser to extract time: 2025-08-05 15:45:00
Processor configuration
Source attribute: time
Time format: Custom: yyyy-MM-dd HH:mm:ss
Time zone ID: supply your time zone
Event input
Event message: 2025-08-05 15:45:00 INFO: Starting application...
Event output
Event timestamp: Aug 05, 03:45:00.000 PM
Example with Apache combined log format
Preceding processor
Regex parser to extract time: 24/Mar/2025:16:25:29 -0500
Processor configuration
Source attribute: time
Time format: CLF (Common Log Format)
Time zone ID: leave empty
Event input
Event message: 29.182.147.96 - - [24/Mar/2025:16:25:29 -0500] "POST /products/23394 ...
Event output
Event timestamp, viewed from PDT time: Mar 24, 02:25:29.000 PM

In this example, the event message recorded the time as 4:25 PM CDT (denoted by the -0500 time zone specification). The user observed the event from the America/Los_Angeles time zone (PDT). As a result, the event displays the timestamp in Lumi as two hours prior.

Value mapper​

Maps a static value to an output user attribute.

The processor creates a new attribute when it doesn't exist. If there's already an attribute with the same name, you can override its value or leave it unchanged.

Example
Processor configuration
Static value: example.com
Event input
Event metadata: host: 23.192.228.84
Event output
User attribute: host: example.com

Limitations​

Lumi doesn't currently support extractions on time fields.

Learn more​

See the following topics for more information: