Skip to main content

Processors reference

AI summary
Explains how to configure processors in Imply Lumi pipelines to transform event data. Covers processors for parsing, mapping, redacting, and enriching attributes. Details configuration options, regular expression support, and testing features.

About AI summaries.

Pipelines transform incoming events in Imply Lumi. A pipeline is made up of processors that perform discrete transformation tasks, such as creating or redacting a user attribute. This topic describes all supported processors, how to configure them, and examples of how to use them.

To learn how to create a pipeline and add processors, see Manage pipelines and processors.

Processor settings​

Use the following guidelines when configuring processors.

Source and output attributes​

When defining source and output attributes, keep the following in mind:

  • You can use incoming event metadata as source attributes. For example, sourcetype.
  • An attribute created by one processor can be the source attribute for a later processor in the same or different pipeline.
  • Select processors let you use the event message (log body) as the source attribute. For example, the redaction processor.
  • You can write to the event message using the message mapper.
  • You can't use system attributes as source or output attributes.
    Note that system attributes are allowed in the pipeline conditions.

Override output attributes​

Some processors have the option to Override value when output attribute exists. This applies to situations in which you specify an output attribute with the same name as a previously existing user attribute or event metadata. When you choose to override, the processor replaces the original attribute value with the newly processed result.

For example, suppose you send an event with incoming metadata key: value1, and a processor computes the output attribute key: value2. You can choose whether to preserve value1 or override it to value2.

The override applies even when the new value is an empty string or one or more whitespace characters. No processing occurs when the new value is null, such as if you provide a nonexistent source attribute for the attribute mapper.

The grok and regex parsers override any existing user attributes by default. You can't disable override for these processors.

Remove source attribute​

When you map an attribute, the processor doesn't remove the source. To remove it, use the attribute remover.

Removing unused attributes can lead to better query performance and more efficient storage. It also simplifies your search experience and reduces complexity for any data maintenance tasks.

Regular expressions​

Processors can take regular expressions (regex) to parse logs, redact content, or find and replace text. Lumi uses Perl-compatible regular expressions, version 2 (PCRE2), which is compatible with Splunk® regular expressions. When testing regex such as in Regex101, be sure to use the PCRE2 flavor.

Both the regex parser and key-value parser use regular expressions to parse events. In most cases, the regex parser provides the most extensible parsing capabilities, including support for backreferences and named capture groups. The key-value parser is the simplest option when you have a repeating key-value pattern, such as a=b c=d, and the key is the user attribute name.

If you need to preserve existing output attributes, use the key-value parser and uncheck the option to Override value when output attribute exists.

Try out a processor​

For any processor, use the Try it out feature to preview the expected output for your test case. You can also simulate all processors in a pipeline. For details, see Simulate pipelines.

Attribute mapper​

Maps the value of a source attribute to an output user attribute.

The processor creates a new attribute when it doesn't exist. If an attribute with the same name already exists, you choose to override its value or leave it unchanged.

If you use the attribute mapper to rename a user attribute, create a subsequent attribute remover to remove the source.

Example
Processor configuration
Source attribute: status
Output attribute: http_status
Event input
Event metadata: status: 401
Event output
User attribute: http_status: 401

Attribute remover​

Removes one or more source attributes. Specify multiple attributes as a comma-separated list.

Use this processor to drop unneeded fields to reduce storage size and improve query performance. You can also use the attribute remover to drop personally identifiable information, whether to remove it completely or to remove the source metadata after redaction.

Example
Processor configuration
Attributes to remove: userid
Event input
Event metadata: userid: wilma
Event output
User attribute: none

Conditional mapper​

Maps a static value or source attribute to an output user attribute when a user-specified condition passes.

A conditional mapper can take one or more conditions. The processor evaluates them from highest to lowest priority and applies the mapping for the first condition that's met. If no conditions are satisfied, no mapping occurs.

A condition takes the following components:

  • Search expression in Lumi query syntax
  • Value to map, whether a static value or source attribute

In the search expression, you can use the comparison operators =, !=, <, <=, >, >=. The left-hand side of the comparison must refer to an attribute, and the right-hand side must be a string or numeric literal. If the attribute value is null, the result is false. Attribute names are case-sensitive; attribute values are compared case-insensitively.

The processor only creates a single output attribute. You can choose whether to override the attribute if it already exists. Create a separate processor for each output user attribute.

Note the following similar processors:

  • A value mapper maps from a static value unconditionally.
  • An attribute mapper maps from a source attribute unconditionally.
  • A lookup mapper evaluates source attributes against a lookup table and creates one or more output attributes.
Example

Consider a static value replacement only for events that have a specific source type.

Processor configuration
Condition: sourcetype=access_combined
Mapper type: Value
Value / Attribute: redacted
Output attribute: user
Event input
Event metadata: 
sourcetype: access_combined
user: wilma
Event output
User attributes: 
sourcetype: access_combined
user: redacted

This configuration only redacts user information when the event satisfies the processor condition for sourcetype and the pipeline condition.

Expression processor​

Computes an expression and maps the result to an output user attribute. Supports arithmetic, string processing, conditional branching, comparisons, and predicate tests. Optionally overrides the output attribute if it already exists.

When defining an expression, note the following:

  • Syntax: Follows Splunk® Search Processing Language (SPL) and supported functions, unless otherwise noted.

    • Attribute names are case-sensitive; values and function names are not.

    • Reserved keywords include function names, Boolean operators, TRUE, FALSE, and NULL in any case.
      Wrap conflicting attribute names in single quotes (''), such as 'match'.

    • String literals accept the following escape sequences. Unknown sequences return the literal backslash and character.

      Escape sequences
      SequenceResult
      \'Single quote
      \"Double quote
      \\Backslash
      \nNewline
      \tTab
      \rCarriage return
      \bBackspace
      \fForm feed
      \XLiteral \X

  • Type coercion: Expressions parse values as numbers when possible and use three-valued logic for logical operations.

    • Numeric coercion: Lumi coerces values to numbers in comparisons and with the addition (+) operator, otherwise falls back to lexicographic comparison and string concatenation. For example, "9" < "10" returns true, whereas "9a" < "10a" returns false. You can use ISNUM to detect whether an argument is a valid number.

    • Null coercion: Predicate expressions can yield true, false, or null values. Null values propagate through expressions and resolve to false in Boolean operations, IF, and CASE.

  • Output requirements: The expression must result in a numeric or string value. The output can be a multi-value string, where multiple values are separated by the pipe character | in a single string.

    • Attribute override: You can select whether to override an existing value when the output attribute already exists.

    • Boolean result: You can't create the processor when its expression yields a Boolean result, in which case Lumi returns a parse error. This happens if you use a predicate function or a Boolean operation as a top-level expression. To generate a string or numeric value, wrap the expression in IF or CASE.

    • Null result: If the expression yields a null result, the processor doesn't create a new attribute. If the output attribute already exists, the processor leaves it unchanged unless you selected the override option, in which case the processor removes the attribute from the event.

      A null result can occur when the expression references a nonexistent attribute, such as rounding a missing value, or when the expression is invalid, such as rounding a string value. To set a default result, wrap the entire expression in COALESCE.

      Null can also result from LIKE, MATCH, or REPLACE if you supply a complicated regex that exceeds resource limits. This is a safeguard to prevent pathological patterns from running indefinitely.

Example
Processor configuration
Expression: raw_bytes/1000 + ' ' + 'kB'
Output attribute: file_size
Event input
Event metadata: raw_bytes: 84352
Event output
User attributes: 
file_size: 84.352 kB
raw_bytes: 84352

Supported functions​

You can use the following functions in the expression processor:

  • Arithmetic functions: Perform mathematical calculations on numeric values.

  • String functions: Manipulate and transform string values.

  • Conditional functions: Return a value based on a Boolean condition.
    A null condition is treated as false. You can nest conditions, such as CASE(<condition1>, IF(<condition2>, ...)).

  • Predicate functions: Evaluate a condition and return a Boolean or null result.
    Predicate functions can't be used as top-level expressions. All comparison operations are case-sensitive.

  • Null-handling functions: Produce or respond to null values.

  • Multi-value functions: Create, transform, and query multi-value fields.

See Expression functions reference for details on each function as well as supported operators.

ArithmeticStringConditionalPredicateNull-handlingMulti-value
CEILCOALESCECASEINNULLMVAPPEND
CEILINGCONCATIFLIKENULLIFMVCOUNT
FLOORLENMATCHMVDEDUP
POWLENGTHCIDRMATCHMVFILTER
ROUNDTRIMCONTAINSMVFIND
LTRIMSTARTS_WITHMVINDEX
RTRIMENDS_WITHMVJOIN
LOWERISNUMMVMAP
UPPERISNULLMVRANGE
REPLACEISNOTNULLMVSORT
SUBSTRMVZIP
SUBSTRINGMV_TO_JSON_ARRAY
TONUMBERSPLIT

Grok parser​

Parses a source attribute into one or more output attributes using a grok expression. You can use the event message as the source attribute.

In the parser configuration, provide the source attribute to parse and a grok expression. A grok expression is made up of one or more grok patterns in the following format:

%{PATTERN_NAME:OUTPUT}

PATTERN_NAME identifies a preset pattern, and OUTPUT is the label you assign to the output value that Lumi stores as a user attribute.

The grok parser extracts structured data when it matches the specified expression, similar to the regex parser. Grok expressions tend to be more human-readable than regex because they use preset templates for common patterns, such as TIMESTAMP_ISO8601. Unlike the regex parser, you don't supply the output attribute names in a separate field; you include them directly in the grok expression.

For a reference on the available patterns, see Grok patterns. Note that you can test your grok patterns using an online parser such as Grok Debugger before you add them to a processor.

Example
Processor configuration
Source attribute: Select the option to Extract from log body
Grok expression: %{TIMESTAMP_ISO8601:time} %{LOGLEVEL:status}: %{GREEDYDATA:message}
Event input
Event message: 2025-08-05 15:45:00 INFO: Starting application...
Event output
User attributes: 
time: 2025-08-05 15:45:00
status: INFO
message: Starting application...

For examples of how to map the extracted values to other event components, see the timestamp mapper and message mapper.

Example with Apache combined log format

This example parses a log in Apache combined log format as represented in the tutorial data.

Processor configuration
Source attribute: Select the option to Extract from log body
Grok expression: 
%{IP:clientip} %{DATA:ident} %{DATA:user} \[%{HTTPDATE:req_time}\] "%{WORD:method} %{DATA:uri} %{DATA:version}" %{NUMBER:status} %{NUMBER:bytes} "%{URI:referer}" "%{GREEDYDATA:useragent}"
Event input
Event message: 
830:1e0e:525:e6a0:6479:cd69:c364:23c3 - - [24/Mar/2025:16:25:29 -0500] "POST /products/23394 HTTP/1.1" 200 1027 "https://techcrunch.com/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:110.0) Gecko/20100101 Firefox/110.0"
Event output
User attributes: 
bytes: 1027
clientip: 830:1e0e:525:e6a0:6479:cd69:c364:23c3
ident: -
method: POST
version: HTTP/1.1
referer: https://techcrunch.com/
status: 200
req_time: 24/Mar/2025:16:25:29 -0500
uri: /products/23394
user: -
useragent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:110.0) Gecko/20100101 Firefox/110.0

Key-value parser​

Parses key-value pairs from a source attribute into one or more output attributes.

Each key-value pair parsed from the source attribute becomes its own output attribute. For example, the value index=main, source=/var/log/messages, sourcetype=access_combined creates user attributes for index, source, and sourcetype when parsed using the equality pattern.

The processor takes the following configuration:

  • Source attribute: Log body or user attribute

  • Prefix: Optional value to prepend to the output attribute name. Lumi names output attributes using the format PREFIX.KEY, where KEY is parsed from the source attribute. If you don't specify a prefix, Lumi only uses the key for the output attribute name. When you select an existing user attribute as the source, the UI pre-populates the source attribute name.

    The prefix can be helpful to trace back the source of parsed attributes, especially if you remove the source attribute to avoid storing the same information twice. It can also prevent naming conflicts with other attributes, such as if your incoming event contains the JSON fields request.instanceId and response.instanceId.

  • Key-value pattern: Type of key-value format: CSV, equality, JSON, regex, XML

  • Override value when output attribute exists: Whether to overwrite any existing user attributes

The following sections describe supported key-value patterns and any additional required configuration. Unless otherwise specified, the examples don't provide a prefix and override any existing output attributes.

CSV​

Example input: value, where the processor uses the comma delimiter, multiple attributes output, and key header

Parses delimiter-separated values, including CSV and TSV.

Additional configuration:

  • Delimiter: Select comma for CSV, tab for TSV, or custom to provide your own delimiter.
  • Output: Select whether to store parsed results as multiple attributes or a single one.
    • Multiple attributes: Creates a user attribute for each parsed field. For example, a,b,c yields three fields.
      Supply Headers for the respective keys (output attribute names) as a list separated by the same delimiter.
    • Single attribute: Creates a single user attribute that stores all parsed values in an array.
      Supply the name for Output attribute.
Key-value example with CSV parsing, multiple attributes
Processor configuration
Source attribute: Select the option to Extract from user attribute csv_attr
Key-value pattern: CSV
Delimiter: Custom ;
Output: Multiple attributes
Headers: key1;key2
Event input
Event metadata: csv_attr: value1;value2
Event output
User attributes: 
csv_attr: value1;value2
key1: value1
key2: value2
Key-value example with CSV parsing, single attribute
Processor configuration
Source attribute: Select the option to Extract from user attribute csv_attr
Key-value pattern: CSV
Delimiter: Custom ;
Output: Single attribute
Output attribute: key
Event input
Event metadata: csv_attr: value1;value2
Event output
User attributes: 
csv_attr: value1;value2
key: [value1, value2]

Equality​

Example input: key=value

Parses text joined by an equal sign and doesn't have space characters on either side of the sign. If an input contains duplicate keys, such as a=b a=c, the parser only takes the last match: a=c.

Logs can contain key-value pairs separated by whitespace (including tab), comma, semicolon, pipe, and ampersand characters, such as a=b | c=d & e=f. Key-value pairs can also be detected from a multi-line event separated by newlines (\n) or carriage returns (\r). Take note of the following behavior for characters that surround the equal sign:

  • Delimiter within value: key=val1,val2 extracts key: val1
    The parser doesn't extract individual values when the delimiter separates multiple values on a single key.

  • Space after equality: key= val extracts nothing
    The parser doesn't store a key-value pair when a space follows the equal sign.

  • Space within values: key1=Multiword value|key2=Another value extracts key1: Multiword value and key2: Another
    When a non-space delimiter is present, the parser retains the space in the value for non-terminal pairs.

Key-value example with equality parsing
Processor configuration
Source attribute: Select the option to Extract from log body
Key-value pattern: Equality
Event input
Event message: index=main, source=/var/log/messages, sourcetype=access_combined
Event output
User attributes: 
index: main
sourcetype: access_combined
source: /var/log/messages

JSON​

Example input: {"key": "value"}

Parses key-value pairs from JSON objects.

Select the checkbox Flatten into a single level to flatten the JSON structure, else preserve original nesting. For example, when parsing the input value {"key1": {"key2": "value"}}, you can store "key1": {"key2": "value"} (unchecked) or key1.key2: value (checked).

Key-value example with JSON parsing
Processor configuration
Source attribute: Select the option to Extract from user attribute request
Key-value pattern: JSON
Flatten into a single level: true
Subsequent processor
Attribute remover to remove request
Event input
Event metadata: request: {"customerId": "abc123", "device": {"id": "i714"}}
Event output
User attributes: 
customerId: abc123
device.id: i714
Key-value example with JSON parsing, with prefix
Processor configuration
Source attribute: Select the option to Extract from user attribute request
Prefix: request
Key-value pattern: JSON
Flatten into a single level: true
Subsequent processor
Attribute remover to remove request
Event input
Event metadata: request: {"customerId": "abc123", "device": {"id": "i714"}}
Event output
User attributes: 
request.customerId: abc123
request.device.id: i714

Regex​

Example input: key_value with regex (\w*)_(\w*)

Parses text using a regular expression with two capture groups, where the first group is the key and the second group is the value. See also regex parser. For details on the regex dialect and how the processors compare, see Regular expressions.

Select the checkbox Combine values from duplicate keys to capture multiple values for a recurring key, separated by a pipe (|) character. When unselected, the processor only stores the first match. For example, you use the regex (\w*):(\w*) to parse the source value key:val1 key:val2 key:val3. By default, the parser returns key:val1. When you combine values, the parser returns key:val1|val2|val3.

Key-value example with regex parsing
Processor configuration
Source attribute: Select the option to Extract from log body
Key-value pattern: Regex
Regular expression: (\w+): ((?:.(?!\w+:))+)
Event input
Event message: 
Apr 23 08:45:11 192.168.56.1 Apr 23 08:45:10 HERMES/192.168.56.22 THOR: Info: MODULE: UserAccounts USER: neo PRIV: 0 LAST_LOGON: 04/22/2026 17:33:01 BADPWCOUNT: 2 NUM_LOGONS: 341 PASS_AGE: 12.00 days
Event output
User attributes: 
LAST_LOGON: 04/22/2026
MODULE: UserAccounts
PASS_AGE: 12.00 days
NUM_LOGONS: 341
PRIV: 0
USER: neo
BADPWCOUNT: 2

XML​

Example input: <root><key>value</key></root>

Parses text within an XML document's root element. Each input requires a single root element. The key-value parser doesn't extract the root name.

Select the checkbox Flatten into a single level to flatten the XML structure, else preserve original nesting.

Key-value example with XML parsing
Processor configuration
Source attribute: Select the option to Extract from log body
Key-value pattern: XML

Event input
Event message:

<guestbook><guestcount>123</guestcount><guest rsvp="true">Wilma Rudolph</guest><venue><reception>courtyard</reception></venue></guestbook>

Event output with flattening
User attributes:

guest: Wilma Rudolph
guestcount: 123
venue.reception: courtyard
guest.rsvp: true

Event output retaining nested structure
User attributes:

guest: {
  rsvp: true
  Wilma Rudolph
}
guestcount: 123
venue: {
  reception: courtyard
}

Lookup mapper​

Looks up source attributes in a user-provided lookup table, and creates output attributes from the table values.

A lookup takes a search key (lookup ID) from source attributes, finds the corresponding row, then creates output attributes for the values in that row. Source attributes are metadata sent with the incoming event or attributes assigned by a previous processor. You can use one or more source attributes such as processId or firstName,lastName. If multiple rows satisfy the lookup condition, the processor uses the first matched row. If no rows match the condition, the processor doesn't create any output attributes.

A lookup table contains a delimiter-separated set of values. You can provide the column headers as part of the table itself or in the Headers field as comma-separated values. Denote the type of character separation in Delimiter. Be sure to match the delimiter to your lookup table. For example, the delimiter comma , is different from comma with a space , .

From the set of column headers, list source and output attributes for the lookup ID and output values:

  • Source attributes: Comma-separated list of incoming attributes to use as the lookup ID.
  • Output attributes: Comma-separated list of attributes to create from the lookup. If your events already contain the output attributes, you can designate whether to override existing values.

Consider the following lookup table, where you use product_id as the source attribute and category, description as the output attributes.

product_idcategorydescription
23394FurnitureLeather Sectional Sofa
32729ElectronicsRaspberry Pi 5
23002BooksMan's Search for Meaning
23394InstrumentsAnalog Theremin
78905JewelryArt Deco Diamond Bracelet

The following examples illustrate the lookup mapping behavior:

  • When an incoming event contains product_id: 23002, the processor creates category: Books and description: Man's Search for Meaning.
  • When an incoming event contains product_id: 23394, the processor creates category: Furniture and description: Leather Sectional Sofa.
  • When an incoming event contains product_id: 1234, the processor doesn't create any attributes.
Example

This example adds the description user attribute for events that store a specific product ID and category.

Processor configuration
Headers: Lookup CSV includes header line
Lookup CSV: 
product_id,category,description
23394,Furniture,Leather Sectional Sofa
32729,Electronics,Raspberry Pi 5
23002,Books,Man's Search for Meaning
23394,Instruments,Analog Theremin
28201,Jewelry,Art Deco Diamond Bracelet
Delimiter: ,
Source attributes: product_id,category
Output attribute: description
Event input
Event metadata: 
product_id: 23394
category: Instruments
Event output
User attributes: 
product_id: 23394
category: Instruments
description: Analog Theremin

Message mapper​

Maps the value of a source attribute to the event message.

You have the option to overwrite the event message with an empty string when the source attribute is missing or empty.

Example
Preceding processor
Grok parser to extract message: Starting application...
Processor configuration
Source attribute: message
Event input
Event message: 2025-08-05 15:45:00 INFO: Starting application...
Event output
Event message: Starting application...

Redaction processor​

Redacts a source attribute using a regular expression. You can use the event message as the source attribute.

The processor overrides the source attribute to store the redacted content. You can search on the redacted content such as user!=*redacted*.

Take note of the following for regex:

  • Must have at least one capture group.
  • Can match zero or more times in the source attribute. Each match is redacted.
  • Captures the entire value when the pattern is (.+). However, if you're redacting the entire value, consider the value mapper with override enabled.

In the processor configuration, you specify the redaction strategy, which determines how Lumi identifies and performs the replacement. The following sections describe these strategies in more detail.

String redaction​

Replaces the entirety of every regex match. Capture groups let you optionally retain content from the match.

To keep a capture group, backreference it in the replacement text using the syntax $N, where N is the one-based index of the group. For example, $2 references the second capture group. You can specify the capture groups in any order, for example, $3-$1-$2. When the replacement text doesn't reference a capture group, any specified capture group is ignored.

Consider the email address username@example.org that you want to redact to u***@example.org. The following configuration generates the redacted output:

  • Regex: (\w)\w*(@\w+\.\w+)
    \w matches word characters [a-zA-Z0-9_]. The first capture group matches the first character u, and the second capture group matches the email domain @example.org.
  • Replacement text: $1***$2
    The replacement text retains the two capture groups with asterisk characters in between.

See the following for additional examples.

Example of string redaction for a user identification number

This example redacts a Social Security number and retains the last four digits.

Processor configuration
Source attribute: Select the option to Extract from log body
Regular expression: (\d{3})-(\d{2})-(\d{4})
Strategy: String
Replacement text: xxx-xx-$3
Event input
Event message: 
2023-10-27 10:01:05 INFO UserID: 88421 - Username: jdoe - SSN: 999-00-1111 - Action: UpdateRecord
Event output
Event message: 
2023-10-27 10:01:05 INFO UserID: 88421 - Username: jdoe - SSN: xxx-xx-1111 - Action: UpdateRecord
Example of string redaction on multiple matches

When there are multiple regex matches against the source attribute, the processor operates on each match. Consider a log that contains two phone numbers.

Processor configuration
Source attribute: Select the option to Extract from log body
Regular expression: (phone=)"\d{3}-\d{3}-\d{4}"
Strategy: String
Replacement text: $1[REDACTED]
Event input
Event message: 
user=wilma, phone="800-555-0100", phone="800-555-0100"
Event output
Event message: 
user=wilma, phone=[REDACTED], phone=[REDACTED]

Hash redaction​

Replaces only the capture group for every regex match. The purpose of the capture group is to define the content to redact.

The processor replaces each capture group with its own cryptographic hash. Note the difference from the string strategy, where the entire regex is replaced and not just the capture groups. You can't backreference capture groups in the hash strategy.

In the processor configuration, select a hash algorithm such as MD5 or SHA-256. You can also supply a salt, such as a random string, to combine with the source attribute before hashing. Select whether to prepend or append the salt to the source attribute.

Example of hash redaction of a password

This example redacts the password from a log and replaces it with a hash.

Processor configuration
Source attribute: Select the option to Extract from log body
Regular expression: password=(.+);
Strategy: Hash
Algorithm: SHA-512
Salt: True
    Value: jv4w7m
    Position: Append
Event input
Event message: 
connection=db;user=admin;password=secret123;host=local
Event output
Event message: 
connection=db;user=admin;password=c8a776ac50189ec7ad12c9573865717cad8b37cba9af872059094fed920e70a642ac5d84e63a36da0c8fd6b94da9b0e6fdee07f12b42352afd1304155763d13b;host=local

Regex parser​

Parses a source attribute into one or more output attributes using a regular expression. See Regular expressions for details on the regex dialect and how the regex parser compares to the key-value parser.

The parser takes the following settings:

  • Source attribute: Log body or user attribute

  • Regular expression: Regex with one or more capture groups. The extraction mode determines how the parser uses the groups. To test regular expressions, preview the processor, or use a free regex parser such as Regex101.

  • Extraction mode: How Lumi creates user attributes from regex matches. See the following sections for more details on each mode and additional configuration settings.

    • Named capture groups: Creates an output attribute for each named capture group.
    • Target attributes: Creates an output attribute for each capture group, using names from a user-provided list.
    • Format string: Dynamic extraction method which creates output attributes based on user-provided format strings.
      Format strings refer to capture groups for user attribute names or values. For example, user: $2 or $1: $2. You can perform string processing on the matched content, such as to reorder groups or concatenate strings.

If a regex parser output shares a name with an existing user attribute, the parser overwrites the attribute, even if the regex matches an empty string or whitespace.

Named capture groups​

For the named capture groups mode, specify the following:

Regular expression: Must contain at least one named capture group in the pattern (?<NAME>REGEX). For example, (?<year>\d{4}) captures year: 2026. This mode ignores any unnamed capture groups.

Example

This example parses a log in Apache combined log format as represented in the tutorial data.

Processor configuration
Source attribute: Select the option to Extract from log body
Regular expression: 
(?<clientip>[^ ]*) (?<ident>[^ ]*) (?<user>[^ ]*) \[(?<req_time>[^\]]*)\] "(?<method>\S+)(?: +(?<uri>[^\"]*?)(?: +(?<version>\S+))?)?" (?<status>[^ ]*) (?<bytes>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<useragent>[^\"]*)")?
Event input
Event message: 
830:1e0e:525:e6a0:6479:cd69:c364:23c3 - - [24/Mar/2025:16:25:29 -0500] "POST /products/23394 HTTP/1.1" 200 1027 "https://techcrunch.com/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:110.0) Gecko/20100101 Firefox/110.0"
Event output
User attributes: 
bytes: 1027
clientip: 830:1e0e:525:e6a0:6479:cd69:c364:23c3
ident: -
method: POST
version: HTTP/1.1
referer: https://techcrunch.com/
status: 200
req_time: 24/Mar/2025:16:25:29 -0500
uri: /products/23394
user: -
useragent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:110.0) Gecko/20100101 Firefox/110.0

Target attributes​

For the target attributes mode, specify the following:

Regular expression: Must contain at least one capture group. The number of capture groups must match the number of output attribute names.

Output attributes: One or more output attribute names, separated by a comma. For example, host or host,user,action. Specify names in the same order as capture groups. Lumi doesn't use any capture group names specified in the regex pattern.

Example

This example parses a log in Apache combined log format as represented in the tutorial data.

Processor configuration
Source attribute: Select the option to Extract from log body
Regular expression: 
([^ ]*) ([^ ]*) ([^ ]*) \[([^\]]*)\] "(\S+)(?: +([^\"]*?)(?: +(\S+))?)?" ([^ ]*) ([^ ]*)(?: "([^\"]*)" "([^\"]*)")?
Output attributes: 
clientip, ident, user, req_time, method, uri, version, status, bytes, referer, useragent
Event input
Event message: 
830:1e0e:525:e6a0:6479:cd69:c364:23c3 - - [24/Mar/2025:16:25:29 -0500] "POST /products/23394 HTTP/1.1" 200 1027 "https://techcrunch.com/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:110.0) Gecko/20100101 Firefox/110.0"
Event output
User attributes: 
bytes: 1027
clientip: 830:1e0e:525:e6a0:6479:cd69:c364:23c3
ident: -
method: POST
version: HTTP/1.1
referer: https://techcrunch.com/
status: 200
req_time: 24/Mar/2025:16:25:29 -0500
uri: /products/23394
user: -
useragent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:110.0) Gecko/20100101 Firefox/110.0

Format string​

For the format string mode, specify the following:

Regular expression: Must contain at least one capture group.

Combine values from duplicate keys: Applies when there's multiple matches for the same attribute. Select this option to retain all matches as one string joined with a pipe character (|). Otherwise, only keep the first match.

Format strings: One or more pairs of format strings, consisting of Attribute format to define the attribute name and Value format to define the attribute value. Format strings reference capture groups using one-based indexing. Lumi doesn't use any capture group names specified in the regex pattern.

Consider the log body ERROR payment_service 503 and the regex (payment_service|auth_service|api_gateway) (\d{3}). The following table shows examples of format strings and the user attributes they create:

Use caseAttribute formatValue formatResult
Concatenate textcodehttp_$2code: http_503
Combine groupsresult$2_$1result: 503_payment_service
Key-value attribute$1$2payment_service: 503
Retain preexisting valuestatus_code_history$0,$2status_code_history: 200,503

For Value format, you can specify $0 to refer to any existing value of the output attribute. For example, if you already had the attribute status_code_history: 200, you can append a captured value to the attribute. If an existing value doesn't exist, $0 resolves to an empty string.

Example
Processor configuration
Source attribute: Select the option to Extract from log body
Regular expression: ([a-z_]+)=([a-z0-9-]+)
Format pairs: Attribute format $1, value format $2
Event input
Event message: env=prod region=us-east tier=web
Event output
User attributes: 
env: prod
region: us-east
tier: web

Status mapper​

Maps the value of a source attribute to the event status.

Lumi attempts to map status codes to human-readable values. For example, the HTTP status code 500 maps to Error. For more information, see the system attribute for status.

You can optionally include a fallback value that Lumi sets for the status when the source attribute doesn't exist or if it's unable to be interpreted.

Example
Processor configuration
Source attribute: http_code
Event input
Event metadata: http_code: 200
Event output
System attribute: status: ok

Note that status here is a system attribute, not a user attribute. If you want to remove the source attribute after the status mapping, use the attribute remover.

Timestamp mapper​

Maps the value of a source attribute to the event timestamp. Provide the name of the source attribute, timestamp format, and the time zone (optional).

Supported timestamp formats include ISO 8601, Common Log Format, and Unix epoch values. To automatically detect the format, select Auto. To define your own timestamp pattern, select Custom, and enter your pattern. Custom formats use Java DateTimeFormatter syntax. For details and examples, see Time formats.

When the timestamp is embedded in the event message, you can map it to the event timestamp as follows:

  1. Create a processor, such as the regex parser, to extract the timestamp as a new attribute.
  2. Use the newly extracted attribute in the timestamp mapper.
  3. Clean up the extracted attribute using the attribute remover.

For more details, see Manual timestamp mapping.

Example
Event input
Event message: 2025-08-05 15:45:00 INFO: Starting application...
Initial parsing
Grok parser to extract time: 2025-08-05 15:45:00
Timestamp processor
Source attribute: time
Time format: Custom: yyyy-MM-dd HH:mm:ss
Time zone ID: supply your time zone
Event output
Event timestamp: Aug 05, 03:45:00.000 PM
Example with Apache combined log format
Event input
Event message: 29.182.147.96 - - [24/Mar/2025:16:25:29 -0500] "POST /products/23394 ...
Initial parsing
Regex parser to extract time: 24/Mar/2025:16:25:29 -0500
Timestamp processor
Source attribute: time
Time format: CLF (Common Log Format)
Time zone ID: leave empty
Event output
Event timestamp, viewed from PDT time: Mar 24, 02:25:29.000 PM

In this example, the event message recorded the time as 4:25 PM CDT (denoted by the -0500 time zone specification). The user observed the event from the America/Los_Angeles time zone (PDT). As a result, the event displays the timestamp in Lumi as two hours prior.

Value mapper​

Maps a static value to an output user attribute.

For the static value, you can enter your own value or assign the Unix timestamp of event indexing. The Unix time represents seconds from Unix epoch—January 1, 1970, at 00:00:00 UTC. For more information about the event indexing timestamp, see Timestamp handling.

The processor creates a new attribute when it doesn't exist. If an attribute with the same name already exists, you can choose to override its value or leave it unchanged.

Example
Processor configuration
Static value: example.com
Event input
Event metadata: host: 23.192.228.84
Event output
User attribute: host: example.com

Limitations​

Lumi doesn't currently support extractions on time fields.

Learn more​

See the following topics for more information: