Skip to main content

Event splitting

AI summary
Explains how Imply Lumi splits batched data into individual events. Covers special handling for AWS CloudTrail and Windows Performance Monitor formats.

About AI summaries.

When you send events to Imply Lumi, you might stream them one at a time or group them in batches to make fewer calls. With batched events, Lumi needs to determine where one event ends and another one begins. Event splitting in Lumi is the process of parsing a single incoming payload into discrete events.

Imply Lumi performs event splitting after receiving events and before any pipeline transformations. Pipelines operate on events individually, so you can't configure them to split or merge events.

Event boundary detection​

To detect event boundaries, Lumi follows the standards for the data transmission protocol used to send the events, such as HEC, OTLP, or S2S. For file upload or S3 pull, Lumi creates an event for each JSON object, CSV row, or plain text line.

info

For the S2S protocol, when an event spans multiple lines, Lumi concatenates them into one. If you need to disable this, update the IAM key to clear the selection Should line merge.

Event types​

Lumi performs further event splitting and timestamp assignment for the specific log types described in this section.

AWS CloudTrail​

AWS CloudTrail records are formatted as a JSON object in which Records is an array that contains all the events. Lumi extracts each object in the array and creates an event for each. See Send AWS CloudTrail events to Lumi to learn how to send events.

Example batched input:

{
"Records": [
{
"eventVersion": "1.08",
"eventTime": "2023-01-02T02:51:00Z",
"awsRegion": "us-east-1",
...
},
{
"eventVersion": "1.08",
"eventTime": "2023-01-02T00:22:00Z",
"awsRegion": "us-east-1",
...
},
...
]
}

Example single event:

eventVersion: 1.08
eventTime: 2023-01-02T02:51:00Z
awsRegion: us-east-1

Lumi accepts and processes CloudTrail events differently by ingestion integration:

  • For file upload and S3 pull, Lumi splits a top-level Records array into one event per record.
  • Other integrations don't split CloudTrail records, so send one record per event.

After event splitting, Lumi processes the events using the AWS CloudTrail predefined pipeline unless you manually disable it. The pipeline requires the user attribute sourcetype=aws:cloudtrail.

Windows Performance Monitor​

Lumi splits an incoming Windows Performance Monitor (Perfmon) payload when it contains the following lines, in order:

  1. A timestamp
  2. Three header lines in key=value format, such as collection=CPU
  3. A line of column names that begins with instance
  4. One or more data rows

Any whitespace, such as tabs or spaces, separates columns and values. Lumi assigns the three header key-value pairs as user attributes on each event. Each data row becomes its own event, where the column names map to their corresponding row values. All events split from the same incoming event have the same timestamp as the incoming event.

If the incoming payload doesn't follow this structure, Lumi doesn't split it and ingests it as a single event. If the number of values in a data row doesn't match the number of column names, Lumi silently skips the row.

Example batched input:

06/23/2025 13:14:14.786 +0530
collection=CPU
category=CPU
object=Processor
instance %_Processor_Time %_User_Time %_Privileged_Time Interrupts/sec %_DPC_Time %_Interrupt_Time DPCs_Queued/sec DPC_Rate %_Idle_Time %_C1_Time %_C2_Time %_C3_Time C1_Transitions/sec C2_Transitions/sec C3_Transitions/sec
0 77.43091558329556 13.52817406291745 63.90274152037811 1487.2618390472518 2.614907735192046 11.94830572610538 148.93725107362184 0 20.71592864083621 20.71592864083621 0 0 1472.8836215093462 0 0
1 77.76318385342931 16.94372058183627 60.81946327159304 1301.4839270153648 1.4927305816420385 9.21685730294172 107.63910285471936 0 20.53827491106385 20.53827491106385 0 0 421.17293850641726 0 0

Example single event:

collection: CPU
category: CPU
object: Processor
instance: 0
%_Processor_Time: 77.43091558329556
%_User_Time: 13.52817406291745
...

Incoming events must have the associated sourcetype value to trigger the event splitting logic:

  • PerfmonMk:Processor
  • PerfmonMk:CPU
  • PerfmonMk:Memory
  • PerfmonMk:LogicalDisk
  • PerfmonMk:PhysicalDisk
  • PerfmonMk:Process
  • PerfmonMk:Network
  • PerfmonMk:Network_Interface
  • PerfmonMk:System
  • PerfmonMk:ProcessorInformation

The Splunk Add-on for Microsoft Windows assigns these source types when it collects Perfmon data in its default multikv mode. You can also assign sourcetype using your forwarding agent or Lumi IAM key. Don't use a pipeline since event splitting happens before pipeline transformation. Lumi doesn't split Perfmon data collected in single mode, which uses Perfmon:* source types.

info

Lumi performs Perfmon event splitting only when the entire set of logs arrives in a single payload. This applies when you forward events from a Splunk forwarder using S2S, which automatically concatenates multi-line events. To backfill using other ingestion integrations:

  • Concatenate all logs using \n line breaks to send through a single Splunk® HEC request
  • Use Splunk CSV format where _raw contains the entire set of logs wrapped in ", and _time contains the timestamp in ISO 8601 format. Upload the file in Lumi directly or backfill with S3 pull.

After event splitting, Lumi processes the events using the Windows event logs predefined pipeline unless you manually disable it.

Learn more​

For more information, see the following topics: