Skip to main content

Processors reference

AI summary
Explains how to configure processors in Imply Lumi pipelines to transform incoming events. Describes parsing, mapping, redaction, and lookup operations. Details configuration options and examples for each processor type.

About AI summaries.

Pipelines transform incoming events in Imply Lumi. A pipeline is made up of processors that perform discrete transformation tasks, such as creating or redacting a user attribute. This topic describes all supported processors, how to configure them, and examples of how to use them.

To learn how to create a pipeline and add processors, see Manage pipelines and processors.

Processor settings​

Use the following guidelines when configuring processors.

Source and output attributes​

When defining source and output attributes, keep the following in mind:

  • When the processor input calls for Source attribute, specify one of the following:
    • A field from incoming event metadata, such as sourcetype.
    • An attribute created by a previous processor in the same or different pipeline.
    • The option Extract from log body to read from the event message. You can select this option in the redaction processor as well as the grok, regex, and key-value parsers.
  • You can use the value _raw to refer to the log body as the source attribute for the attribute mapper or as a field in the expression processor.
  • Some processors don't have a Source attribute field; in the configuration, you refer to source attributes simply by name. For example, in the expression processor, conditional mapper, lookup mapper.
  • You can write to the event message using the message mapper.
  • You can't use system attributes as source or output attributes.
    Note that system attributes are allowed in the pipeline conditions.

Output attribute conflicts​

In some processors, you define the name of the user attribute that stores the processor result. When you do, select an option for When the output attribute already exists to control what happens if that name is already in use.

This option is only available in processors that have the Output attribute field as well as the key-value parser. The grok parser, regex parser, and redaction processor replace any existing user attributes by default.

For an existing output attribute, you can:

  • Preserve its original value: The default. Keeps the original attribute and skips processing for it. The exception is the conditional mapper, which defaults to replacement.

  • Replace it with the new value: Replaces the original attribute. Replacement happens even when the new value is an empty string ("") or whitespace.

    When the new value is null, the processor either skips replacement or replaces it with a null value. See the processor descriptions for details. Only the expression processor removes the attribute entirely if the new value is null.

  • Append to the existing value: Joins the new value to the existing one with the pipe character (|), such as val1|val2.

Remove source attribute​

When you map an attribute, the processor doesn't remove the source. To remove it, use the attribute remover.

Removing unused attributes can lead to better query performance and more efficient storage. It also simplifies your search experience and reduces complexity for any data maintenance tasks.

Regular expressions​

Processors can take regular expressions (regex) to parse logs, redact content, or replace text. Lumi uses Perl-compatible regular expressions, version 2 (PCRE2), which is compatible with Splunk® regular expressions. Be sure to use the PCRE2 flavor when testing regex in an external application such as Regex101.

Try out a processor​

For any processor, use the Try it out feature to preview the expected output for your test case. You can also simulate all processors in a pipeline. For details, see Simulate pipelines.

Attribute mapper​

Maps the value of a source attribute to an output user attribute.

You can specify _raw as the source attribute to refer to the log body. To test the processor, provide _log in the Sample attributes JSON.

You can select how to handle conflicts when the output attribute already exists. With the replace option:

  • When the source attribute doesn't exist, no replacement occurs.
  • When the source attribute exists but is null, the mapper replaces the output attribute with a null value.

To rename a user attribute, use the attribute mapper to create the new attribute, then add an attribute remover to remove the source.

Example
Processor configuration
Source attribute: status
Output attribute: http_status
Event input
Event metadata: {"status": 401}
Event output
User attribute: http_status: 401

Attribute remover​

Removes one or more source attributes. Specify multiple attributes as a comma-separated list.

Use this processor to drop unneeded fields to reduce storage size and improve query performance. You can also use the attribute remover to drop personally identifiable information, whether to remove it completely or to remove the source metadata after redaction.

Example
Processor configuration
Attributes to remove: userid
Event input
Event metadata: {"userid": "wilma"}
Event output
User attribute: none

Conditional mapper​

Maps a static value or source attribute to an output user attribute when a user-specified condition passes.

A conditional mapper can take one or more conditions. The processor evaluates them from highest to lowest priority and applies the mapping for the first condition that's met. If no conditions are satisfied, no mapping occurs.

A condition takes the following components:

  • Search expression in Lumi query syntax
  • Value to map, whether a static value or source attribute

In the search expression, you can use the comparison operators =, !=, <, <=, >, >=. The left-hand side of the comparison must refer to an attribute, and the right-hand side must be a string or numeric literal. If the attribute value is null, the result is false. Attribute names are case-sensitive; attribute values are compared case-insensitively.

The processor only creates a single output attribute. Create a separate processor for each output user attribute. You can select how to handle conflicts when the output attribute already exists. If the matched case's value is null, no replacement occurs.

Note the following similar processors:

  • A value mapper maps from a static value unconditionally.
  • An attribute mapper maps from a source attribute unconditionally.
  • A lookup mapper evaluates source attributes against a lookup table and creates one or more output attributes.
Example

Consider a static value replacement only for events that have a specific source type.

Processor configuration
Condition: sourcetype=access_combined
Mapper type: Value
Value / Attribute: redacted
Output attribute: user
Event input
Event metadata: {"sourcetype": "access_combined", "user": "wilma"}
Event output
User attributes: 
sourcetype: access_combined
user: redacted

This configuration only redacts user information when the event satisfies the processor condition for sourcetype and the pipeline condition.

Expression processor​

Computes an expression and maps the result to an output user attribute. Supports arithmetic, string processing, conditional branching, comparisons, and predicate tests.

You can select how to handle conflicts when the output attribute already exists.

When defining an expression, note the following:

  • Syntax: An expression refers to one or more source attributes and computes a result using functions and operators. Syntax follows Splunk® Search Processing Language (SPL) unless otherwise noted.

    • Case sensitivity: Function names aren't case-sensitive. The following syntax elements are case-sensitive: attribute names, string comparisons against an attribute value, string comparisons to string literals. For example, regex operations in REPLACE.

    • Reserved keywords: The following terms are reserved for processor use: _raw, function names, Boolean operators, TRUE, FALSE, and NULL in any case. If an attribute name conflicts with a reserved keyword, refer to the attribute in single quotes (''), such as 'match'. For an attribute literally named _raw, refer to its value using $['_raw'].

    • Nested JSON: Use JSONPath notation to refer to elements in a JSON attribute. For example, $.key refers to a root element named key. Refer to nested elements with dot or array notation. For example, $.path.expression or $.data[0]. Use bracket notation to refer to elements that conflict with a reserved keyword, such as $['Match'].

    • Variable definitions: You can define temporary variables for use in expressions. A variable definition follows the pattern let NAME = EXPRESSION;, where NAME is the variable name, and EXPRESSION is any valid expression for the processor. Supply the main expression after variable definitions. For example:

      let src_v6 = IF(MATCH(srcip, ":"), "ipv6", "ipv4");
      let dst_v6 = IF(MATCH(dstip, ":"), "ipv6", "ipv4");
      CASE(
        ISNOTNULL(srcip), src_v6,
        ISNOTNULL(dstip), dst_v6,
        "ipNotFound"
      )

      A variable name takes precedence over an attribute with the same name. To force the attribute value, refer to it using $.NAME, where NAME is the attribute name.

    • String escaping: You can refer to special characters in string literals using the following escape sequences. Unknown sequences return the literal backslash and character.

      Escape sequences
      SequenceResult
      \'Single quote
      \"Double quote
      \\Backslash
      \nNewline
      \tTab
      \rCarriage return
      \bBackspace
      \fForm feed
      \XLiteral \X

  • Type coercion: Expressions parse values as numbers when possible and use three-valued logic for logical operations.

    • Numeric coercion: Lumi coerces values to numbers in comparisons and with the addition (+) operator, otherwise falls back to lexicographic comparison and string concatenation. For example, "9" < "10" returns true, whereas "9a" < "10a" returns false. You can use ISNUM to detect whether an argument is a valid number.

    • Null coercion: Predicate expressions can yield true, false, or null values. Null values propagate through expressions and resolve to false in Boolean operations, IF, and CASE.

  • Output requirements: An expression must result in a numeric or string value. The output can be a multi-value string, where multiple values are separated by the pipe character (|) in a single string.

    • Boolean result: You can't create the processor when its expression yields a Boolean result, in which case Lumi returns a parse error. This happens if you use a predicate function or a Boolean operation as a top-level expression. To generate a string or numeric value, wrap the expression in IF or CASE.

    • Null result: If the expression yields a null result, the processor doesn't create a new attribute. If the output attribute already exists and you choose to replace its value, the processor removes the attribute from the event.

      A null result can occur when the expression references a nonexistent attribute, such as rounding a missing value, or when the expression is invalid, such as rounding a string value. To set a default result, wrap the entire expression in COALESCE.

      Null can also result from LIKE, MATCH, or REPLACE if you supply a complicated regex that exceeds resource limits. This is a safeguard to prevent pathological patterns from running indefinitely.

Example
Processor configuration
Expression: raw_bytes/1000 + ' ' + 'kB'
Output attribute: file_size
Event input
Event metadata: {"raw_bytes": 84352}
Event output
User attributes: 
file_size: 84.352 kB
raw_bytes: 84352

Supported functions​

You can use the following functions in the expression processor:

  • Arithmetic functions: Perform mathematical calculations on numeric values.

  • String functions: Manipulate and transform string values.

  • Conditional functions: Return a value based on a Boolean condition.
    A null condition is treated as false. You can nest conditions, such as CASE(<condition1>, IF(<condition2>, ...)).

  • Predicate functions: Evaluate a condition and return a Boolean or null result.
    Predicate functions can't be used as top-level expressions. All comparison operations are case-sensitive.

  • Null-handling functions: Produce or respond to null values.

  • Multi-value functions: Create, transform, and query multi-value fields.

See Expression functions reference for details on each function as well as supported operators.

ArithmeticBitwiseStringConditionalPredicateNull-handlingMulti-valueJSON
CEILBIT_ANDCOALESCECASEINNULLMVAPPENDJSON_EXTRACT
CEILINGBIT_NOTCONCATIFLIKENULLIFMVCOUNT
FLOORBIT_ORLENMATCHMVDEDUP
POWBIT_SHIFT_LEFTLENGTHCIDRMATCHMVFILTER
ROUNDBIT_SHIFT_RIGHTTRIMCONTAINSMVFIND
BIT_XORLTRIMSTARTS_WITHMVINDEX
RTRIMENDS_WITHMVJOIN
LOWERISNUMMVMAP
UPPERISNULLMVRANGE
REPLACEISNOTNULLMVSORT
SUBSTRMVZIP
SUBSTRINGMV_TO_JSON_ARRAY
TONUMBERSPLIT
TOSTRING
URLDECODE

Grok parser​

Parses a source attribute into one or more output attributes using a grok rule. You can use the log body as the source attribute.

  • Processor input: The grok parser takes the name of a source attribute and one or more newline-delimited grok rules. In a grok rule, you define capture expressions to match parts of the source value. Each expression can create an output attribute. If you supply multiple rules, the parser evaluates each one independently and captures matches for each.

  • Expression syntax: A capture expression takes the following form:

    %{PATTERN:OUTPUT:TYPE}

    • PATTERN: Required name of a preset grok pattern, such as TIMESTAMP_ISO8601. See Grok patterns for the available named patterns.

    • OUTPUT: Optional name for the output attribute.

      • If you use the same output name in multiple expressions, a pipe character (|) joins all matched values.
      • If a subsequent rule uses the same output name as a previous rule, the later rule defines the attribute value.
      • If the pattern already defines output names, you might not need one in your expression. For example, %{COMBINEDAPACHELOG} creates clientip, ident, auth, and more.
      • If no output name is defined, the parser discards the matched value.

    • TYPE: Optional data type identifier that casts the output type of the extracted value. Valid data types are string (default), int, long, float, double, boolean, and hex.

      The hex data type parses a base-16 input (hexadecimal string) and returns the base-10 numeric equivalent. For example, for an input string FF, the expression %{BASE16NUM:code:hex} stores code: 255.

  • Custom regex: In addition to using preset patterns, you can define a capture expression using custom regex in the form of named capture groups. For example, (?<port>[0-9]+). You can use a mix of patterns and custom regex in the same rule.

  • Comparison to regex parser: The regex parser also extracts structured data against a specified pattern. Grok expressions can be more human-readable because they can refer to named patterns, while the regex parser provides more extensible parsing capabilities, such as support for backreferences and different extraction modes.

  • Potential error states:

    • Lumi raises a compile error if you supply an invalid pattern name.
    • Aside from hex, if a data type isn't compatible with its output, the entire rule fails. Other rules are still valid.
    • If a hex output type is incompatible, the rule doesn't fail, and the parser stores the raw string for that match.

To test grok expressions, preview the processor or use a free online parser such as Grok Debugger.

Example
Processor configuration
Source attribute: Select the option to Extract from log body
Grok expression: %{TIMESTAMP_ISO8601:time} %{LOGLEVEL:status}: %{GREEDYDATA:message}
Event input
Event message: 2025-08-05 15:45:00 INFO: Starting application...
Event output
User attributes: 
time: 2025-08-05 15:45:00
status: INFO
message: Starting application...

For examples of how to map the extracted values to other event components, see the timestamp mapper and message mapper.

Example with Apache combined log format

This example parses a log in Apache combined log format as represented in the tutorial data.

Processor configuration
Source attribute: Select the option to Extract from log body
Grok expression: 
%{IP:clientip} %{DATA:ident} %{DATA:user} \[%{HTTPDATE:req_time}\] "%{WORD:method} %{DATA:uri} %{DATA:version}" %{NUMBER:status} %{NUMBER:bytes} "%{URI:referer}" "%{GREEDYDATA:useragent}"
Event input
Event message: 
830:1e0e:525:e6a0:6479:cd69:c364:23c3 - - [24/Mar/2025:16:25:29 -0500] "POST /products/23394 HTTP/1.1" 200 1027 "https://techcrunch.com/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:110.0) Gecko/20100101 Firefox/110.0"
Event output
User attributes: 
bytes: 1027
clientip: 830:1e0e:525:e6a0:6479:cd69:c364:23c3
ident: -
method: POST
version: HTTP/1.1
referer: https://techcrunch.com/
status: 200
req_time: 24/Mar/2025:16:25:29 -0500
uri: /products/23394
user: -
useragent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:110.0) Gecko/20100101 Firefox/110.0

Key-value parser​

Parses key-value pairs from a source attribute into one or more output attributes.

Each key-value pair parsed from the source attribute becomes its own output attribute. For example, the value index=main, source=/var/log/messages, sourcetype=access_combined creates user attributes for index, source, and sourcetype when parsed using the equality pattern.

The processor takes the following configuration:

  • Source attribute: Log body or user attribute.

  • Prefix: Optional value to prepend to the output attribute name. Lumi names output attributes using the format PREFIX.KEY, where PREFIX is your value and KEY is parsed from the source attribute. When the source is a user attribute, the UI pre-populates the attribute name. If unspecified, Lumi only uses the key for the output attribute name.

    The prefix can be helpful to trace the source of a parsed attribute, especially if you removed the source attribute. It can also prevent naming conflicts with other attributes, such as if your incoming event contains the JSON fields request.instanceId and response.instanceId.

  • Key-value pattern: Type of key-value format: CSV, equality, JSON, regex, XML.

  • When the output attribute already exists: Whether to preserve, replace, or append to any existing attributes.

    The following table describes how the key-value parser handles null values and empty strings in the replace mode:

    PatternParsed valueOutcome
    CSVNull valueN/A, never generated
    CSVEmpty stringOutput attribute is an empty string
    EqualityNull valueN/A, never generated
    EqualityEmpty stringDoesn't create output attribute (no replacement occurs)
    JSONNull valueOutput attribute exists as a null value
    JSONEmpty stringOutput attribute is an empty string
    RegexNull valueOutput attribute exists as a null value
    RegexEmpty stringDoesn't create output attribute (no replacement occurs)
    XMLNull valueOutput attribute exists as a null value
    XMLEmpty stringOutput attribute is an empty string

The following sections describe parsing behavior for each key-value pattern. Unless otherwise specified, the examples don't provide a prefix and replace existing output attributes.

CSV​

Example input: value, where the processor uses the comma delimiter, multiple attributes output, and key header

Parses delimiter-separated values, including CSV and TSV.

Additional configuration:

  • Delimiter: Select comma for CSV, tab for TSV, or custom to provide your own delimiter.
  • Output: Select whether to store parsed results as multiple attributes or a single one.
    • Multiple attributes: Creates a user attribute for each parsed field. For example, a,b,c yields three fields.
      Supply Headers for the respective keys (output attribute names) as a list separated by the same delimiter.
    • Single attribute: Creates a single user attribute that stores all parsed values in an array.
      Supply the name for Output attribute.
Example of key-value CSV parsing, multiple attributes
Processor configuration
Source attribute: Select the option to Extract from user attribute csv_attr
Key-value pattern: CSV
Delimiter: Custom ;
Output: Multiple attributes
Headers: key1;key2
Event input
Event metadata: csv_attr: value1;value2
Event output
User attributes: 
csv_attr: value1;value2
key1: value1
key2: value2
Example of key-value CSV parsing, single attribute
Processor configuration
Source attribute: Select the option to Extract from user attribute csv_attr
Key-value pattern: CSV
Delimiter: Custom ;
Output: Single attribute
Output attribute: key
Event input
Event metadata: csv_attr: value1;value2
Event output
User attributes: 
csv_attr: value1;value2
key: [value1, value2]

Equality​

Example input: key=value

Parses text joined by an equal sign and doesn't have space characters on either side of the sign. If an input contains duplicate keys, such as a=b a=c, the parser only takes the first match: a=b.

Logs can contain key-value pairs delimited by whitespace, tab (\t), comma (,), semicolon (;), pipe (|), and ampersand (&) characters. Key-value pairs can also be detected from a multi-line event separated by newlines (\n) or carriage returns (\r).

Take note of how delimiters affect parsing behavior:

Delimiter patternParser behaviorExample inputExample output
Mixed delimitersExtracts attributes even when you have different delimitersa=b | c=d & e=fa: b, c: d, e: f
Space after equalityDoesn't create an attribute when a space follows the equal signkey= valNo result
Space within valuesRetains space in value only for non-terminal pairskey1=Multiword value|key2=Another valuekey1: Multiword value and key2: Another
Delimiter within valueKeeps text up to the delimiter when a delimiter separates multiple values on a single keykey=val1,val2key: val1
Delimiter within enclosed valueCaptures text within enclosing characters ("", '', <>, {}, (), or []), ignoring delimiters withinkey=[val1;val2]key: [val1;val2]
Nested pairsExtracts key-value pairs nested within "", (), [], or {}. Enclosing pairs must surround the complete value, except parentheses can start within the value.key=[a=1;b=2]

key=Mozilla/5.0 (agent=Mac; ver=1)		key: [a=1;b=2], a: 1, and b: 2

key: Mozilla/5.0, agent: Mac, ver: 1
Missing closing characterEnds at delimiter or word break when no matching close is foundkey=a(b;c

key=[long value		key: a(b

key: [long
Example of key-value equality parsing
Processor configuration
Source attribute: Select the option to Extract from log body
Key-value pattern: Equality
Event input
Event message: {"message": "index=main, source=/var/log/messages, sourcetype=access_combined"}
Event output
User attributes: 
index: main
sourcetype: access_combined
source: /var/log/messages

JSON​

Example input: {"key": "value"}

Parses key-value pairs from JSON objects.

Select the checkbox Flatten into a single level to flatten the JSON structure, else preserve original nesting. For example, when parsing the input value {"key1": {"key2": "value"}}, you can store "key1": {"key2": "value"} (unchecked) or key1.key2: value (checked).

Example of key-value JSON parsing
Processor configuration
Source attribute: Select the option to Extract from user attribute request
Key-value pattern: JSON
Flatten into a single level: true
Subsequent processor
Attribute remover to remove request
Event input
Event metadata: {"request": {"customerId": "abc123", "device": {"id": "i714"}}}
Event output
User attributes: 
customerId: abc123
device.id: i714
Example of key-value JSON parsing with prefix
Processor configuration
Source attribute: Select the option to Extract from user attribute request
Prefix: request
Key-value pattern: JSON
Flatten into a single level: true
Subsequent processor
Attribute remover to remove request
Event input
Event metadata: {"request": {"customerId": "abc123", "device": {"id": "i714"}}}
Event output
User attributes: 
request.customerId: abc123
request.device.id: i714

Regex​

Example input: key_value with regex (\w*)_(\w*)

Parses text using a regular expression with two capture groups, where the first group is the key and the second group is the value. See Regular expressions for details on the regex dialect.

If a regex matches multiple values, the parser only stores the first one by default. To capture all matches, select the checkbox Combine values from duplicate keys. The output attribute stores all results joined by pipe (|). For example, consider the source attribute key:val1 key:val2 key:val3 and the regex (\w*):(\w*). The parser either stores key:val1 or key:val1|val2|val3.

Take note of the following behavior:

  • Trailing delimiter: If a captured value ends in a comma (,), semicolon (;), tab (\t), newline (\n), carriage return (\r), pipe (|), or ampersand (&) character, the parser trims that trailing character only when the log has more content after it.

  • Empty match: A capture group that matches an empty string doesn't produce an output attribute. A capture group that doesn't match anything still writes the output attribute as a null value.

Note that the regex parser can also parse text using regex. The regex parser generally provides more extensible parsing capabilities, such as support for backreferences and named capture groups. Use the key-value parser described in this section when you have a simple repeating key-value pattern, such as a=b c=d, and the keys are the output attribute names. You can also use the key-value parser to preserve existing output attributes.

Example of key-value regex parsing
Processor configuration
Source attribute: Select the option to Extract from log body
Key-value pattern: Regex
Regular expression: (\w+): ((?:.(?!\w+:))+)
Event input
Event message: 
Apr 23 08:45:11 192.168.56.1 Apr 23 08:45:10 HERMES/192.168.56.22 THOR: Info: MODULE: UserAccounts USER: neo PRIV: 0 LAST_LOGON: 04/22/2026 17:33:01 BADPWCOUNT: 2 NUM_LOGONS: 341 PASS_AGE: 12.00 days
Event output
User attributes: 
LAST_LOGON: 04/22/2026
MODULE: UserAccounts
PASS_AGE: 12.00 days
NUM_LOGONS: 341
PRIV: 0
USER: neo
BADPWCOUNT: 2

XML​

Example input: <root><key>value</key></root>

Parses text within an XML document's root element. Each input requires a single root element. The key-value parser doesn't extract the root name.

Select the checkbox Flatten into a single level to flatten the XML structure, else preserve original nesting.

An empty or self-closing XML element, such as <key></key>, produces an empty string. XML input only generates null when the element contains the XML attribute xsi:nil="true".

Example of key-value XML parsing
Processor configuration
Source attribute: Select the option to Extract from log body
Key-value pattern: XML

Event input
Event message:

<guestbook><guestcount>123</guestcount><guest rsvp="true">Wilma Rudolph</guest><venue><reception>courtyard</reception></venue></guestbook>

Event output with flattening
User attributes:

guest: Wilma Rudolph
guestcount: 123
venue.reception: courtyard
guest.rsvp: true

Event output retaining nested structure
User attributes:

guest: {
  rsvp: true
  Wilma Rudolph
}
guestcount: 123
venue: {
  reception: courtyard
}

Lookup mapper​

Looks up source attributes in a user-provided lookup table, and creates output attributes from the table values.

A lookup takes a search key (lookup ID) from source attributes, finds the corresponding row, then creates output attributes for the values in that row. Source attributes are metadata sent with the incoming event or attributes assigned by a previous processor. You can use one or more source attributes such as processId or firstName,lastName. If multiple rows satisfy the lookup condition, the processor uses the first matched row. If no rows match the condition, the processor doesn't create any output attributes.

A lookup table contains a delimiter-separated set of values. You can provide the column headers as part of the table itself or in the Headers field as comma-separated values. Denote the type of character separation in Delimiter. Be sure to match the delimiter to your lookup table. For example, the delimiter comma , is different from comma with a space , .

From the set of column headers, list source and output attributes for the lookup ID and output values:

  • Source attributes: Comma-separated list of incoming attributes to use as the lookup ID.
  • Output attributes: Comma-separated list of attributes to create from the lookup.

You can select how to handle conflicts when the output attribute already exists. If the result is a null value, no replacement occurs.

Consider the following lookup table, where you use product_id as the source attribute and category, description as the output attributes.

product_idcategorydescription
23394FurnitureLeather Sectional Sofa
32729ElectronicsRaspberry Pi 5
23002BooksMan's Search for Meaning
23394InstrumentsAnalog Theremin
78905JewelryArt Deco Diamond Bracelet

The following examples illustrate the lookup mapping behavior:

  • When an incoming event contains product_id: 23002, the processor creates category: Books and description: Man's Search for Meaning.
  • When an incoming event contains product_id: 23394, the processor creates category: Furniture and description: Leather Sectional Sofa.
  • When an incoming event contains product_id: 1234, the processor doesn't create any attributes.
Example

This example adds the description user attribute for events that store a specific product ID and category.

Processor configuration
Headers: Lookup CSV includes header line
Lookup CSV: 
product_id,category,description
23394,Furniture,Leather Sectional Sofa
32729,Electronics,Raspberry Pi 5
23002,Books,Man's Search for Meaning
23394,Instruments,Analog Theremin
28201,Jewelry,Art Deco Diamond Bracelet
Delimiter: ,
Source attributes: product_id,category
Output attribute: description
Event input
Event metadata: {"product_id": 23394, "category": "Instruments"}
Event output
User attributes: 
product_id: 23394
category: Instruments
description: Analog Theremin

Message mapper​

Maps the value of a source attribute to the event message.

You have the option to overwrite the event message with an empty string when the source attribute is missing or empty.

Example
Preceding processor
Grok parser to extract message: Starting application...
Processor configuration
Source attribute: message
Event input
Event message: 2025-08-05 15:45:00 INFO: Starting application...
Event output
Event message: Starting application...

Redaction processor​

Redacts a source attribute using a regular expression. You can use the log body as the source attribute. You don't specify an output attribute; the processor overwrites the source attribute with the redacted content.

Take note of the following for regex:

  • Must have at least one capture group.
  • Can match zero or more times in the source attribute. Each match is redacted.
  • Captures the entire value when the pattern is (.+). However, if you're redacting the entire value, consider the value mapper and select the replace mode.

In the processor configuration, you specify the redaction strategy, which determines how Lumi identifies and performs the replacement. The following sections describe these strategies in more detail.

String redaction​

Replaces the entirety of every regex match. Capture groups let you optionally retain content from the match.

To keep a capture group, backreference it in the replacement text using the syntax $N, where N is the one-based index of the group. For example, $2 references the second capture group. You can specify the capture groups in any order, for example, $3-$1-$2. When the replacement text doesn't reference a capture group, any specified capture group is ignored.

Consider the email address username@example.org that you want to redact to u***@example.org. The following configuration generates the redacted output:

  • Regex: (\w)\w*(@\w+\.\w+)
    \w matches word characters [a-zA-Z0-9_]. The first capture group matches the first character u, and the second capture group matches the email domain @example.org.
  • Replacement text: $1***$2
    The replacement text retains the two capture groups with asterisk characters in between.

See the following for additional examples.

Example of string redaction for a user identification number

This example redacts a Social Security number and retains the last four digits.

Processor configuration
Source attribute: Select the option to Extract from log body
Regular expression: (\d{3})-(\d{2})-(\d{4})
Strategy: String
Replacement text: xxx-xx-$3
Event input
Event message: 
2023-10-27 10:01:05 INFO UserID: 88421 - Username: jdoe - SSN: 999-00-1111 - Action: UpdateRecord
Event output
Event message: 
2023-10-27 10:01:05 INFO UserID: 88421 - Username: jdoe - SSN: xxx-xx-1111 - Action: UpdateRecord
Example of string redaction on multiple matches

When there are multiple regex matches against the source attribute, the processor operates on each match. Consider a log that contains two phone numbers.

Processor configuration
Source attribute: Select the option to Extract from log body
Regular expression: (phone=)"\d{3}-\d{3}-\d{4}"
Strategy: String
Replacement text: $1[REDACTED]
Event input
Event message: 
user=wilma, phone="800-555-0100", phone="800-555-0100"
Event output
Event message: 
user=wilma, phone=[REDACTED], phone=[REDACTED]

Hash redaction​

Replaces only the capture group for every regex match. The purpose of the capture group is to define the content to redact.

The processor replaces each capture group with its own cryptographic hash. Note the difference from the string strategy, where the entire regex is replaced and not just the capture groups. You can't backreference capture groups in the hash strategy.

In the processor configuration, select a hash algorithm such as MD5 or SHA-256. You can also supply a salt, such as a random string, to combine with the source attribute before hashing. Select whether to prepend or append the salt to the source attribute.

Example of hash redaction of a password

This example redacts the password from a log and replaces it with a hash.

Processor configuration
Source attribute: Select the option to Extract from log body
Regular expression: password=(.+);
Strategy: Hash
Algorithm: SHA-512
Salt: True
    Value: jv4w7m
    Position: Append
Event input
Event message: 
connection=db;user=admin;password=secret123;host=local
Event output
Event message: 
connection=db;user=admin;password=c8a776ac50189ec7ad12c9573865717cad8b37cba9af872059094fed920e70a642ac5d84e63a36da0c8fd6b94da9b0e6fdee07f12b42352afd1304155763d13b;host=local

Regex parser​

Parses a source attribute into one or more output attributes using a regular expression. See Regular expressions for details on the regex dialect. For other processors that can parse using regex, see the key-value parser and grok parser.

The parser takes the following settings:

  • Source attribute: Log body or user attribute

  • Regular expression: Regex with one or more capture groups. The extraction mode determines how the parser uses the groups. To test regular expressions, preview the processor, or use a free regex parser such as Regex101.

  • Extraction mode: How Lumi creates user attributes from regex matches. See the following sections for more details on each mode and additional configuration settings.

    • Named capture groups: Creates an output attribute for each named capture group.
    • Target attributes: Creates an output attribute for each capture group, using names from a user-provided list.
    • Format string: Dynamic extraction method which creates output attributes based on user-provided format strings.
      Format strings refer to capture groups for user attribute names or values. For example, user: $2 or $1: $2. You can perform string processing on the matched content, such as to reorder groups or concatenate strings.

If a regex parser output shares a name with an existing user attribute, the parser overwrites the attribute, even if the regex matches an empty string or whitespace.

Named capture groups​

For the named capture groups mode, specify the following:

Regular expression: Must contain at least one named capture group in the pattern (?<NAME>REGEX). For example, (?<year>\d{4}) captures year: 2026. This mode ignores any unnamed capture groups.

Example

This example parses a log in Apache combined log format as represented in the tutorial data.

Processor configuration
Source attribute: Select the option to Extract from log body
Regular expression: 
(?<clientip>[^ ]*) (?<ident>[^ ]*) (?<user>[^ ]*) \[(?<req_time>[^\]]*)\] "(?<method>\S+)(?: +(?<uri>[^\"]*?)(?: +(?<version>\S+))?)?" (?<status>[^ ]*) (?<bytes>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<useragent>[^\"]*)")?
Event input
Event message: 
830:1e0e:525:e6a0:6479:cd69:c364:23c3 - - [24/Mar/2025:16:25:29 -0500] "POST /products/23394 HTTP/1.1" 200 1027 "https://techcrunch.com/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:110.0) Gecko/20100101 Firefox/110.0"
Event output
User attributes: 
bytes: 1027
clientip: 830:1e0e:525:e6a0:6479:cd69:c364:23c3
ident: -
method: POST
version: HTTP/1.1
referer: https://techcrunch.com/
status: 200
req_time: 24/Mar/2025:16:25:29 -0500
uri: /products/23394
user: -
useragent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:110.0) Gecko/20100101 Firefox/110.0

Target attributes​

For the target attributes mode, specify the following:

Regular expression: Must contain at least one capture group. The number of capture groups must match the number of output attribute names.

Output attributes: One or more output attribute names, separated by a comma. For example, host or host,user,action. Specify names in the same order as capture groups. Lumi doesn't use any capture group names specified in the regex pattern.

Example

This example parses a log in Apache combined log format as represented in the tutorial data.

Processor configuration
Source attribute: Select the option to Extract from log body
Regular expression: 
([^ ]*) ([^ ]*) ([^ ]*) \[([^\]]*)\] "(\S+)(?: +([^\"]*?)(?: +(\S+))?)?" ([^ ]*) ([^ ]*)(?: "([^\"]*)" "([^\"]*)")?
Output attributes: 
clientip, ident, user, req_time, method, uri, version, status, bytes, referer, useragent
Event input
Event message: 
830:1e0e:525:e6a0:6479:cd69:c364:23c3 - - [24/Mar/2025:16:25:29 -0500] "POST /products/23394 HTTP/1.1" 200 1027 "https://techcrunch.com/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:110.0) Gecko/20100101 Firefox/110.0"
Event output
User attributes: 
bytes: 1027
clientip: 830:1e0e:525:e6a0:6479:cd69:c364:23c3
ident: -
method: POST
version: HTTP/1.1
referer: https://techcrunch.com/
status: 200
req_time: 24/Mar/2025:16:25:29 -0500
uri: /products/23394
user: -
useragent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:110.0) Gecko/20100101 Firefox/110.0

Format string​

For the format string mode, specify the following:

Regular expression: Must contain at least one capture group.

Combine values from duplicate keys: Applies when there's multiple matches for the same attribute. Select this option to retain all matches as one string joined with a pipe character (|). Otherwise, only keep the first match.

Format strings: One or more pairs of format strings, consisting of Attribute format to define the attribute name and Value format to define the attribute value. Format strings reference capture groups using one-based indexing. Lumi doesn't use any capture group names specified in the regex pattern.

Consider the log body ERROR payment_service 503 and the regex (payment_service|auth_service|api_gateway) (\d{3}). The following table shows examples of format strings and the user attributes they create:

Use caseAttribute formatValue formatResult
Concatenate textcodehttp_$2code: http_503
Combine groupsresult$2_$1result: 503_payment_service
Key-value attribute$1$2payment_service: 503
Retain preexisting valuestatus_code_history$0,$2status_code_history: 200,503

For Value format, you can specify $0 to refer to any existing value of the output attribute. The capture group resolves to an empty string if the attribute doesn’t exist. For example, if you already had the user attribute status_code_history: 199, you can use the value format string $0, $1 to generate a result similar to 200, 404.

Example
Processor configuration
Source attribute: Select the option to Extract from log body
Regular expression: ([a-z_]+)=([a-z0-9-]+)
Format pairs: Attribute format $1, value format $2
Event input
Event message: env=prod region=us-east tier=web
Event output
User attributes: 
env: prod
region: us-east
tier: web

Status mapper​

Maps the value of a source attribute to the event status.

Lumi attempts to map status codes to human-readable values. For example, the HTTP status code 500 maps to Error. For more information, see the system attribute for status.

You can optionally include a fallback value that Lumi sets for the status when the source attribute doesn't exist or if it's unable to be interpreted.

Example
Processor configuration
Source attribute: http_code
Event input
Event metadata: {"http_code": 200}
Event output
System attribute: status: ok

Note that status here is a system attribute, not a user attribute. If you want to remove the source attribute after the status mapping, use the attribute remover.

Timestamp mapper​

Maps the value of a source attribute to the event timestamp. Provide the name of the source attribute, timestamp format, and the time zone (optional).

Supported timestamp formats include ISO 8601, Common Log Format, and Unix epoch values. To automatically detect the format, select Auto. To define your own timestamp pattern, select Custom, and enter your pattern. Custom formats use Java DateTimeFormatter syntax. For details and examples, see Time formats.

When the timestamp is embedded in the log body, you can map it to the event timestamp as follows:

  1. Create a processor, such as the regex parser, to extract the timestamp as a new attribute.
  2. Use the newly extracted attribute in the timestamp mapper.
  3. Clean up the extracted attribute using the attribute remover.

For more details, see Assign event timestamps using a pipeline.

Example
Event input
Event message: 2025-08-05 15:45:00 INFO: Starting application...
Initial parsing
Grok parser to extract time: 2025-08-05 15:45:00
Timestamp processor
Source attribute: time
Time format: Custom: yyyy-MM-dd HH:mm:ss
Time zone ID: supply your time zone
Event output
Event timestamp: Aug 05, 03:45:00.000 PM
Example with Apache combined log format
Event input
Event message: 29.182.147.96 - - [24/Mar/2025:16:25:29 -0500] "POST /products/23394 ...
Initial parsing
Regex parser to extract time: 24/Mar/2025:16:25:29 -0500
Timestamp processor
Source attribute: time
Time format: CLF (Common Log Format)
Time zone ID: leave empty
Event output
Event timestamp, viewed from PDT time: Mar 24, 02:25:29.000 PM

In this example, the event message recorded the time as 4:25 PM CDT (denoted by the -0500 time zone specification). The user observed the event from the America/Los_Angeles time zone (PDT). As a result, the event displays the timestamp in Lumi as two hours prior.

Value mapper​

Maps a static input value to an output user attribute.

For the input value, you can enter your own value or assign the Unix timestamp of event indexing. The Unix time represents seconds from Unix epoch—January 1, 1970, at 00:00:00 UTC. For more information, see Store event indexing timestamp.

You can select how to handle conflicts when the output attribute already exists. An empty input generates an attribute with an empty string.

Example
Processor configuration
Static value: example.com
Event input
Event metadata: {"host": "23.192.228.84"}
Event output
User attribute: host: example.com

Limitations​

Lumi doesn't currently support extractions on time fields.

Learn more​

See the following topics for more information: