Processors reference

AI summary

Explains how processors in Imply Lumi transform events within pipelines. Covers available processor types for parsing, mapping, redacting, and removing event data. Describes configuration options and best practices for efficient data processing.

About AI summaries.

Pipelines transform incoming events in Imply Lumi. A pipeline is made up of processors that perform discrete transformation tasks, such as creating or redacting a user attribute. This topic describes all supported processors, how to configure them, and examples of how to use them.

To learn how to create a pipeline and add processors, see Manage pipelines and processors.

Processor settings

Use the following guidelines when configuring processors.

Source and output attributes

When defining source and output attributes, keep the following in mind:

You can use incoming event metadata as source attributes. For example, sourcetype.
An attribute created by one processor can be the source attribute for a later processor in the same or different pipeline.
Select processors let you use the event message (log body) as the source attribute. For example, the redaction processor.
You can write to the event message using the message mapper.
You can't use system attributes as source or output attributes.
Note that system attributes are allowed in the pipeline conditions.

Override output attributes

Some processors have the option to Override value when output attribute exists. This applies to situations in which you specify an output attribute with the same name as a previously existing user attribute or event metadata. When you choose to override, the processor replaces the original attribute value with the newly processed result.

For example, suppose you send an event with incoming metadata key: value1, and a processor computes the output attribute key: value2. You can choose whether to preserve value1 or override it to value2.

The override applies even when the new value is an empty string or one or more whitespace characters. No processing occurs when the new value is null, such as if you provide a nonexistent source attribute for the attribute mapper.

The grok and regex parsers override any existing user attributes by default. You can't disable override for these processors.

Remove mapped attributes

When you map an attribute, the processor doesn't remove the source. To remove it, use the attribute remover.

Removing unused attributes can lead to better query performance and more efficient storage. It also simplifies your search experience and reduces complexity for any data maintenance tasks.

Try out a processor

For any processor, use the Try it out feature to preview the expected output for your test case. You can also simulate all processors in a pipeline. For details, see Manage pipelines.

Arithmetic processor

warning

Deprecated. Use the expression processor instead, which covers all arithmetic operations and adds support for functions like ROUND() and string expressions.

Evaluates an arithmetic formula and outputs the result to an attribute.

You can reference existing attributes as variables in the formula. The formula supports the basic operators for addition (+), subtraction (-), multiplication (*), and division (/). Parentheses (()) control the order of operations.

In the arithmetic formula, surround operators with space characters. For example, val1 - val2 is a valid subtraction formula. Without the space characters, the processor interprets val1-val2 as a single attribute.

Configure the processor with the following settings:

Override value when output attribute exists: If an attribute with the same name already exists, you can select whether to override its value or leave it unchanged.
Round output decimal value: The number of decimal places allowed in the output value. For example, enter 2 to round a result of 0.888 to 0.89.
Replace invalid input values with zero: When you select this toggle, Lumi replaces any nonexistent attributes with zero. If you don't select this option, Lumi skips processing and doesn't evaluate the formula.

Example

Processor configuration: Arithmetic formula: (val1 + val2) / (val4 - val3); Output attribute: computed

Event input

Event metadata:

val1: 5
val2: 8
val3: 11
val4: 14

Event output: User attribute: computed: 4.333

Attribute mapper

Maps the value of a source attribute to an output user attribute.

The processor creates a new attribute when it doesn't exist. If an attribute with the same name already exists, you choose to override its value or leave it unchanged.

Example

Processor configuration: Source attribute: status; Output attribute: http_status

Event input: Event metadata: status: 401

Event output: User attribute: http_status: 401

Attribute remover

Removes one or more source attributes.

Use this processor to drop unneeded fields to reduce storage size and improve query performance. You can also use the attribute remover to drop personally identifiable information, whether to remove it completely or to remove the source metadata after redaction.

Example

Processor configuration: Attributes to remove: userid

Event input: Event metadata: userid: wilma

Event output: User attribute: none

Conditional mapper

Maps a static value or source attribute to an output user attribute when a user-specified condition passes.

A conditional mapper can take one or more conditions. The processor evaluates them from highest to lowest priority and applies the mapping for the first condition that's met. If no conditions are satisfied, no mapping occurs.

A condition takes the following components:

Search expression in Lumi query syntax
Value to map, whether a static value or source attribute

In the search expression, you can use the comparison operators =, !=, <, <=, >, >=. The left-hand side of the comparison must refer to an attribute, and the right-hand side must be a string or numeric literal. If the attribute value is null, the result is false. All string comparisons are case-sensitive.

The processor only creates a single output attribute. You can choose whether to override the attribute if it already exists. Create a separate processor for each output user attribute.

Note the following similar processors:

A value mapper maps from a static value unconditionally.
An attribute mapper maps from a source attribute unconditionally.
A lookup mapper evaluates source attributes against a lookup table and creates one or more output attributes.

Example

Consider a static value replacement only for events that have a specific source type.

Processor configuration: Condition: sourcetype=access_combined; Mapper type: Value; Value / Attribute: redacted; Output attribute: user

Event input

Event metadata:

sourcetype: access_combined
user: wilma

Event output

User attributes:

sourcetype: access_combined
user: redacted

This configuration only redacts user information when the event satisfies the processor condition for sourcetype and the pipeline condition.

Expression processor

Computes an arithmetic or string expression and maps the result to an output user attribute. Designate whether you want to override the output attribute if it already exists.

Invalid expressions don't generate an output attribute. For example, if you use a string attribute in an arithmetic formula.

Arithmetic processing

Perform arithmetic calculations such as val1 + val2. You can use operators for addition (+), subtraction (-), multiplication (*), division (/), and modulo (%). Parentheses (()) control the order of operations.

You can specify the following arithmetic functions:

ROUND(n, [precision]): Rounds a number n to the nearest integer.
Optionally specify precision to round to a specific number of decimal places.
FLOOR(n): Rounds a number down to the nearest integer.
CEIL(n): Rounds a number up to the nearest integer.

In case an attribute doesn't exist, you can use the COALESCE function to set a default value, such as 0 or 1.

String processing

Perform string operations such as concatenation: firstName + ' ' + lastName.

The processor converts numeric attributes to strings when they are combined with another attributes that can't be coerced into a number. For example, given the source attributes {"size": 2, "units": "GB"}, the expression size + ' ' + units yields the result 2 GB.

If the source attributes can be coerced to numbers even though they are string values, arithmetic processing applies. For example, with the source attributes {"foo": "2", "bar": "5"}, the expression foo + bar yields the result 7. To you intend to concatenate the values, use CONCAT(foo, bar) to yield the result 25.

You can use the following functions to perform operations such as trimming and text replacement.

CONCAT(a, b, ...): Joins two or more inputs into a single value.
COALESCE(a, b, ...): Returns the first non-null, non-empty value. Requires at least two arguments.
TRIM(s): Removes leading and trailing whitespace from a string.
LTRIM(s): Removes leading whitespace from a string.
RTRIM(s): Removes trailing whitespace from a string.
UPPER(s): Converts a string to uppercase.
LOWER(s): Converts a string to lowercase.
LENGTH(s): Returns the number of characters in a string.
SUBSTRING(s, start, length): Extracts part of a string from zero-based index start for the given length.
REPLACE(s, pattern, replacement): For a string s, finds regex pattern and substitutes with replacement.

For more information and examples, see Expression functions reference.

Example

Processor configuration: Expression: raw_bytes/1000 + ' ' + 'kB'; Output attribute: file_size

Event input: Event metadata: raw_bytes: 84352

Event output

User attributes:

file_size: 84.352 kB
raw_bytes: 84352

Grok parser

Parses a source attribute into one or more output attributes using a grok expression. You can use the event message as the source attribute.

In the parser configuration, provide the source attribute to parse and a grok expression. A grok expression is made up of one or more grok patterns in the following format:

%{PATTERN_NAME:OUTPUT}

PATTERN_NAME identifies a preset pattern, and OUTPUT is the label you assign to the output value that Lumi stores as a user attribute.

The grok parser extracts structured data when it matches the specified expression, similar to the regex parser. Grok expressions tend to be more human-readable than regex because they use preset templates for common patterns, such as TIMESTAMP_ISO8601. Unlike the regex parser, you don't supply the output attribute names in a separate field; you include them directly in the grok expression.

For a reference on the available patterns, see Grok patterns. Note that you can test your grok patterns using an online parser such as Grok Debugger before you add them to a processor.

Example

Processor configuration: Source attribute: Select the option to Extract from log body; Grok expression: %{TIMESTAMP_ISO8601:time} %{LOGLEVEL:status}: %{GREEDYDATA:message}

Event input: Event message: 2025-08-05 15:45:00 INFO: Starting application...

Event output

User attributes:

time: 2025-08-05 15:45:00
status: INFO
message: Starting application...

For examples of how to map the extracted values to other event components, see the timestamp mapper and message mapper.

Example with Apache combined log format

This example parses a log in Apache combined log format as represented in the tutorial data.

Processor configuration

Source attribute: Select the option to Extract from log body

Grok expression:

%{IP:clientip} %{DATA:ident} %{DATA:user} \[%{HTTPDATE:req_time}\] "%{WORD:method} %{DATA:uri} %{DATA:version}" %{NUMBER:status} %{NUMBER:bytes} "%{URI:referer}" "%{GREEDYDATA:useragent}"

Event input

Event message:

830:1e0e:525:e6a0:6479:cd69:c364:23c3 - - [24/Mar/2025:16:25:29 -0500] "POST /products/23394 HTTP/1.1" 200 1027 "https://techcrunch.com/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:110.0) Gecko/20100101 Firefox/110.0"

Event output

User attributes:

bytes: 1027
clientip: 830:1e0e:525:e6a0:6479:cd69:c364:23c3
ident: -
method: POST
version: HTTP/1.1
referer: https://techcrunch.com/
status: 200
req_time: 24/Mar/2025:16:25:29 -0500
uri: /products/23394
user: -
useragent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:110.0) Gecko/20100101 Firefox/110.0

Key-value parser

Parses key-value pairs from a source attribute into one or more output attributes.

Lumi creates a user attribute for each key-value pair. A single source attribute can generate multiple output attributes when the input value contains multiple instances of the pattern. For example, index=main, source=/var/log/messages, sourcetype=access_combined.

If you don't need to retain the original source value, you can use the attribute remover to remove it after key-value parsing. For additional details, see Remove mapped attributes.

The following table lists the supported key-value pattern types. Each example produces the user attribute key: value.

Pattern	Description	Processor configuration	Example input
CSV	Parses delimiter-separated values	Delimiter Output Headers or output attribute	`value` Headers: `key`
Equality	Parses text joined by an equal sign, without space characters	N/A	`key=value`
JSON	Parses text from JSON objects	Flatten or preserve nesting	`{"key": "value"}`
Regex	Parses text adhering to a regex with two capture groups. Also see regex parser.	Regular expression Combine values from duplicate keys	`key_value` Regex: `(\w)_(\w)`
XML	Parses text within an XML document's root element. Each input requires a single root element. The key-value parser doesn't extract the root name.	Flatten or preserve nesting	`<root><key>value</key></root>`

Output attribute names

Each parsed key-value pair creates an output user attribute. The name of the output attribute takes the format PREFIX.KEY, where PREFIX is an optional value you supply in Prefix, and KEY is the extracted key from the source attribute.

When your key-value pair comes from an existing source attribute, the Lumi UI populates Prefix with the name of the source attribute. You can remove or change this value.

Consider the example JSON input {"foo": "bar"}. The following table describes the output attribute names for various configurations:

Source attribute	Prefix	Output attribute
Extract from log body	--	`foo: bar`
Extract from log body	`parsed`	`parsed.foo: bar`
Extract from user attribute `fields`	--	`foo: bar`
Extract from user attribute `fields`	`fields`	`fields.foo: bar`

If there's already an output attribute with the same name, you can choose to preserve its original value or override it with the parsed value.

Duplicate keys

For the regex pattern, you can enable Combine values from duplicate keys to capture multiple matches for the same key. When this option is disabled, the processor stores only the first match. For example:

Input:
- Regex: (\w*):(\w*)
- Source value: key:val1 key:val2 key:val3
Output:
- Without combine: key:val1
- With combine: key:val1|val2|val3

For the equality pattern, if an input contains duplicate keys, such as a=b a=c a=d, the parser only takes the last match: a=d.

CSV format

The default settings process CSV data using a comma (,) delimiter. You can change the delimiter to a tab character or provide your own value in the Delimiter field.

You can control how Lumi stores the parsed values in the Output field. The default output behavior, Multiple attributes, creates a user attribute for each field. In Headers, supply the CSV headers, or other delimiter-separated headers, for the attribute names.

You can also store the parsed output as an array in a single output attribute. With this option, you don't provide any headers. In Output, select Single attribute, and supply a name for Output attribute.

Consider the input value1,value2 in each of the following configurations:

Multiple attributes
- Headers: key1,key2
- Result: key1: value1 and key2: value2
Single attributes
- Output attribute: parsed_csv
- Result: parsed_csv: [value1, value2]

Nested JSON

Use Flatten into a single level to control whether to preserve JSON nesting. If you don't select the flatten option, the processor preserves the original nesting of the source attribute value. For example:

Input:
- Source value: {"key1": {"key2": "value"}}
Output:
- Nested: "key1": {"key2": "value"}
- Flattened: key1.key2: value

Nested XML

Use Flatten into a single level to control whether to preserve XML nesting. Without flattening, the processor preserves the original nesting of the source attribute value. For example:

Input:
- Source value: <guestbook><guestcount>123</guestcount><guest rsvp="true">Wilma Rudolph</guest><venue><reception>courtyard</reception></venue></guestbook>

Output:

Nested:

venue: {reception=courtyard}
guestcount: 123
guest: {rsvp=true, =Wilma Rudolph}

Flattened:

venue.reception: courtyard
guestcount: 123
guest: Wilma Rudolph
guest.rsvp: true

Example

Processor configuration: Source attribute: Select the option to Extract from user attribute json_attr; Key-value pattern: JSON; Flatten into a single level: true

Subsequent processor: Attribute remover to remove json_attr

Event input: Event metadata: json_attr: {"jsonkey1": "value1", "outerkey": {"innerkey": "value2"}}

Event output

User attributes:

json_attr.jsonkey1: value1
json_attr.outerkey.innerkey: value2

Lookup mapper

Looks up source attributes in a user-provided lookup table, and creates output attributes from the table values.

A lookup table contains a delimiter-separated set of values. You can provide the column headers as part of the table itself or as comma-separated values in the Headers field. Denote the type of character separation in Delimiter. Be sure to match the delimiter to your lookup table. For example, the delimiter comma , is different from comma with a space , .

In Source attributes, enter the incoming attributes to use as lookup IDs from the table. Each source attribute should correspond to a table column.

In Output attributes, enter the attributes to create from the lookup. Each output attribute should correspond to a table column. Lumi assigns the attribute value from the first matched row. If your events already contain the output attributes, you can designate whether to override existing values.

Consider an example with the following lookup table:

`product_id`	`category`	`description`
23394	Furniture	Leather Sectional Sofa
32729	Electronics	Raspberry Pi 5
23002	Books	Man's Search for Meaning
23394	Instruments	Analog Theremin
78905	Jewelry	Art Deco Diamond Bracelet

You specify product_id as the source attribute and category, description as the output attributes. Based on the incoming event, the processor does the following:

When a source attribute contains product_id: 23002, the processor creates the output attributes category: Books and description: Man's Search for Meaning.
When a source attribute contains product_id: 1234, the processor doesn't create any attributes.

Example

This example adds the description user attribute for events that store a specific product ID and category.

Processor configuration

Headers: Lookup CSV includes header line

Lookup CSV:

product_id,category,description
23394,Furniture,Leather Sectional Sofa
32729,Electronics,Raspberry Pi 5
23002,Books,Man's Search for Meaning
23394,Instruments,Analog Theremin
28201,Jewelry,Art Deco Diamond Bracelet

Delimiter: ,

Source attributes: product_id,category

Output attribute: description

Event input

Event metadata:

product_id: 23394
category: Instruments

Event output

User attributes:

product_id: 23394
category: Instruments
description: Analog Theremin

Note that if you only select product_id as the source attribute, the resulting user attribute would be description: Leather Sectional Sofa, since it's the first matched row for product ID 23394.

Message mapper

Maps the value of a source attribute to the event message.

You have the option to overwrite the event message with an empty string when the source attribute is missing or empty.

Example

Preceding processor: Grok parser to extract message: Starting application...

Processor configuration: Source attribute: message

Event input: Event message: 2025-08-05 15:45:00 INFO: Starting application...

Event output: Event message: Starting application...

Redaction processor

Redacts a source attribute using a regular expression. You can use the event message as the source attribute.

The processor overrides the source attribute to store the redacted content. You can search on the redacted content such as user!=*redacted*.

Take note of the following for regex:

Must have at least one capture group.
Can match zero or more times in the source attribute. Redacts each match.
Captures the entire value when the pattern is (.+). However, if you're redacting the entire value, consider using the value mapper or conditional mapper with the override option.

The redaction strategy determines how Lumi identifies and replaces the redacted values. The processor uses regex capture groups differently depending on the strategy:

String: Replaces the entirety of every regex match. The purpose of the capture group is to optionally retain content from the match.
Hash: Replaces each capture group with a hash for every regex match. The purpose of the capture group is to define the redacted content.

The following sections describe the redaction strategies in more detail.

String redaction

With the string strategy for redaction, the regex describes the entire pattern to redact.

Regex capture groups define content you want to retain. To keep a capture group, backreference it in the replacement text using the syntax $N, where N is the one-based index of the group. For example, $2 references the second capture group. When the replacement text doesn't reference a capture group, the capture group is ignored. You can rearrange the order of the groups, such as $1$2 or $2$1.

Consider the email address username@example.org that you want to redact to u***@example.org. The following configuration generates the redacted output:

Regex: (\w)\w*(@\w+\.\w+)
This simplistic pattern only searches for alphanumeric and underscore characters. The regex contains a capture group for the first character and a capture group for the email domain.
Replacement text: $1***$2
The replacement text includes the asterisk redaction characters flanked by backreferences to the capture groups.

When there are multiple regex matches, the redaction and any backreference is applied for each match. For example, consider the following log that contains two phone numbers:

user="wilma", phone="800-555-0100", phone="800-555-0100", id="123-456-7890"

When you use the regex (phone=)"\d{3}-\d{3}-\d{4}" and replacement text $1"REMOVED", the processor keeps phone= and redacts the phone number for each match. The redacted line becomes:

user="wilma", phone="REMOVED", phone="REMOVED", id="123-456-7890"

Hash redaction

With the hash strategy for redaction, the regex describes the pattern to search for. The processor replaces each capture group with its own cryptographic hash. When there are multiple regex matches, the processor replaces each capture group of each match.

Note that this approach differs from the string strategy, where the entire regex is replaced and not just the capture groups. You can't backreference capture groups in the hash strategy.

In the processor configuration, you specify a hash algorithm such as MD5 or SHA-256. You also have the option to specify a salt, such as a random string, that gets combined with the source attribute before applying the hash function. You can either prepend or append the salt to the source attribute.

Example of partial string redaction

This example redacts a Social Security number and retains the last four digits.

Processor configuration: Source attribute: Select the option to Extract from log body; Regular expression: (\d{3})-(\d{2})-(\d{4}); Strategy: String; Replacement text: xxx-xx-$3

Event input

Event message:

2023-10-27 10:01:05 INFO UserID: 88421 - Username: jdoe - SSN: 999-00-1111 - Action: UpdateRecord

Event output

Event message:

2023-10-27 10:01:05 INFO UserID: 88421 - Username: jdoe - SSN: xxx-xx-1111 - Action: UpdateRecord

Example of string redaction with multiple matches

This example redacts multiple instances of medical diagnosis codes.

Processor configuration: Source attribute: Select the option to Extract from log body; Regular expression: ([A-Z][0-9]\w.\w+); Strategy: String; Replacement text: REDACTED

Event input

Event message:

{"patient_id":"P550","diagnosis_codes":["I11.0","I50.9","J44.9"]}

Event output

Event message:

{"patient_id":"P550","diagnosis_codes":["REDACTED","REDACTED","REDACTED"]}

Example of hash redaction of a password

This example redacts the password from a log and replaces it with a hash.

Processor configuration: Source attribute: Select the option to Extract from log body; Regular expression: password=(.+);; Strategy: Hash; Algorithm: SHA-512; Salt: True; Value: jv4w7m; Position: Append

Event input

Event message:

connection=db;user=admin;password=secret123;host=local

Event output

Event message:

connection=db;user=admin;password=c8a776ac50189ec7ad12c9573865717cad8b37cba9af872059094fed920e70a642ac5d84e63a36da0c8fd6b94da9b0e6fdee07f12b42352afd1304155763d13b;host=local

Regex parser

Parses a source attribute into one or more output attributes using a regular expression. You can use the event message as the source attribute.

In the parser configuration, provide the source attribute to parse, a regular expression with one or more capture groups, and a comma-separated list of output attributes. To test regular expressions, you can try out the processor or use a free regex parser such as Regex101.

The number of capture groups, denoted by (), determines the number of output attributes. If a capture group matches more than one result, the processor only takes the first match. Consider the following examples that parse the input string hello world:

The regex (\w+) matches on one or more word characters. Provide a single output attribute, such as greeting. The parser returns greeting: hello.
If your regex is (\w+) (\w+), provide two output attributes, such as greeting, addressee. In this case, the parser returns greeting: hello and addressee: world.

If you have an existing user attribute with the same name as one of the specified output attributes, the parser overwrites the previously existing user attribute. This behavior applies even when the match is an empty string or whitespace character.

In some cases, your input string contains both the attribute name and value. You can extract the attribute names directly from the source rather than provide a list of output attributes. To parse with regex to extract both attribute names and values, see the key-value parser and its regex pattern.

Example

Processor configuration: Source attribute: Select the option to Extract from log body; Regular expression: status: \[(\w*)\]; Output attributes: status

Event input: Event message: Deployment successful. System 1 status: [ok] System 2 status: [alert]

Event output: User attribute: status: ok

Example with Apache combined log format

This example parses a log in Apache combined log format as represented in the tutorial data.

Processor configuration

Source attribute: Select the option to Extract from log body

Regular expression:

([^ ]*) ([^ ]*) ([^ ]*) \[([^\]]*)\] "(\S+)(?: +([^\"]*?)(?: +(\S+))?)?" ([^ ]*) ([^ ]*)(?: "([^\"]*)" "([^\"]*)")?

Output attributes:

clientip, ident, user, req_time, method, uri, version, status, bytes, referer, useragent

Event input

Event message:

830:1e0e:525:e6a0:6479:cd69:c364:23c3 - - [24/Mar/2025:16:25:29 -0500] "POST /products/23394 HTTP/1.1" 200 1027 "https://techcrunch.com/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:110.0) Gecko/20100101 Firefox/110.0"

Event output

User attributes:

bytes: 1027
clientip: 830:1e0e:525:e6a0:6479:cd69:c364:23c3
ident: -
method: POST
version: HTTP/1.1
referer: https://techcrunch.com/
status: 200
req_time: 24/Mar/2025:16:25:29 -0500
uri: /products/23394
user: -
useragent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:110.0) Gecko/20100101 Firefox/110.0

Status mapper

Maps the value of a source attribute to the event status.

Lumi attempts to map status codes to human-readable values. For example, the HTTP status code 500 maps to Error. For more information, see the system attribute for status.

You can optionally include a fallback value that Lumi sets for the status when the source attribute doesn't exist or if it's unable to be interpreted.

Example

Processor configuration: Source attribute: http_code

Event input: Event metadata: http_code: 200

Event output: System attribute: status: ok

Note that status here is a system attribute, not a user attribute. If you want to remove the source attribute after the status mapping, use the attribute remover.

Timestamp mapper

Maps the value of a source attribute to the event timestamp. Provide the name of the source attribute, timestamp format, and the time zone (optional).

Supported timestamp formats include ISO 8601, Common Log Format, and Unix epoch values. To automatically detect the format, select Auto. To define your own timestamp pattern, select Custom, and enter your pattern. Custom formats use Java DateTimeFormatter syntax. For details and examples, see Time formats.

When the timestamp is embedded in the event message, you can map it to the event timestamp as follows:

Create a processor, such as the regex parser, to extract the timestamp as a new attribute.
Use the newly extracted attribute in the timestamp mapper.
Clean up the extracted attribute using the attribute remover.

For more details, see Manual timestamp mapping.

Example

Event input: Event message: 2025-08-05 15:45:00 INFO: Starting application...

Initial parsing: Grok parser to extract time: 2025-08-05 15:45:00

Timestamp processor: Source attribute: time; Time format: Custom: yyyy-MM-dd HH:mm:ss; Time zone ID: supply your time zone

Event output: Event timestamp: Aug 05, 03:45:00.000 PM

Example with Apache combined log format

Event input: Event message: 29.182.147.96 - - [24/Mar/2025:16:25:29 -0500] "POST /products/23394 ...

Initial parsing: Regex parser to extract time: 24/Mar/2025:16:25:29 -0500

Timestamp processor: Source attribute: time; Time format: CLF (Common Log Format); Time zone ID: leave empty

Event output: Event timestamp, viewed from PDT time: Mar 24, 02:25:29.000 PM

In this example, the event message recorded the time as 4:25 PM CDT (denoted by the -0500 time zone specification). The user observed the event from the America/Los_Angeles time zone (PDT). As a result, the event displays the timestamp in Lumi as two hours prior.

Value mapper

Maps a static value to an output user attribute.

For the static value, you can enter your own value or assign the Unix timestamp of event indexing. The Unix time represents seconds from Unix epoch—January 1, 1970, at 00:00:00 UTC. For more information about the event indexing timestamp, see Timestamp handling.

The processor creates a new attribute when it doesn't exist. If an attribute with the same name already exists, you can choose to override its value or leave it unchanged.

Example

Processor configuration: Static value: example.com

Event input: Event metadata: host: 23.192.228.84

Event output: User attribute: host: example.com

Limitations

Lumi doesn't currently support extractions on time fields.

Learn more

See the following topics for more information:

How to transform events with pipelines for a tutorial on pipelines.
Transform events using pipelines for an overview of pipelines and processors.
Manage pipelines and processors for how to create and manage pipelines and processors.

Processor settings​

Source and output attributes​

Override output attributes​

Remove mapped attributes​

Try out a processor​

Arithmetic processor​

Attribute mapper​

Attribute remover​

Conditional mapper​

Expression processor​

Arithmetic processing​

String processing​

Grok parser​

Key-value parser​

Output attribute names​

Duplicate keys​

CSV format​

Nested JSON​

Nested XML​

Lookup mapper​

Message mapper​

Redaction processor​

String redaction​

Hash redaction​

Regex parser​

Status mapper​

Timestamp mapper​

Value mapper​

Limitations​

Learn more​

Processor settings

Source and output attributes

Override output attributes

Remove mapped attributes

Try out a processor

Arithmetic processor

Attribute mapper

Attribute remover

Conditional mapper

Expression processor

Arithmetic processing

String processing

Grok parser

Key-value parser

Output attribute names

Duplicate keys

CSV format

Nested JSON

Nested XML

Lookup mapper

Message mapper

Redaction processor

String redaction

Hash redaction

Regex parser

Status mapper

Timestamp mapper

Value mapper

Limitations

Learn more