Event splitting
AI summary
About AI summaries.
When you send events to Imply Lumi, you might stream them one at a time or group them in batches to make fewer calls. With batched events, Lumi needs to determine where one event ends and another one begins. Event splitting in Lumi is the process of parsing a single incoming payload into discrete events.
Imply Lumi performs event splitting after receiving events and before any pipeline transformations. Pipelines operate on events individually, so you can't configure them to split or merge events.
Event boundary detection
To detect event boundaries, Lumi follows the standards for the data transmission protocol used to send the events, such as HEC, OTLP, or S2S. For file upload or S3 pull, Lumi creates an event for each JSON object, CSV row, or plain text line.
For the S2S protocol, when an event spans multiple lines, Lumi concatenates them into one. If you need to disable this, update the IAM key to clear the selection Should line merge.
Event types
Lumi performs further event splitting and timestamp assignment for the specific log types described in this section.
AWS CloudTrail
AWS CloudTrail records are formatted as a JSON object in which Records is an array that contains all the events.
Lumi extracts each object in the array and creates an event for each.
See Send AWS CloudTrail events to Lumi to learn how to send events.
Example batched input:
{
"Records": [
{
"eventVersion": "1.08",
"eventTime": "2023-01-02T02:51:00Z",
"awsRegion": "us-east-1",
...
},
{
"eventVersion": "1.08",
"eventTime": "2023-01-02T00:22:00Z",
"awsRegion": "us-east-1",
...
},
...
]
}
Example single event:
eventVersion: 1.08
eventTime: 2023-01-02T02:51:00Z
awsRegion: us-east-1
Lumi accepts and processes CloudTrail events differently by ingestion integration:
- For file upload and S3 pull, Lumi splits a top-level
Recordsarray into one event per record. - Other integrations don't split CloudTrail records, so send one record per event.
After event splitting, Lumi processes the events using the AWS CloudTrail predefined pipeline unless you manually disable it. The pipeline requires the user attribute sourcetype=aws:cloudtrail.
Windows Performance Monitor
Lumi splits an incoming Windows Performance Monitor (Perfmon) payload when it contains the following lines, in order:
- A timestamp
- Three header lines in
key=valueformat, such ascollection=CPU - A line of column names that begins with
instance - One or more data rows
Any whitespace, such as tabs or spaces, separates columns and values. Lumi assigns the three header key-value pairs as user attributes on each event. Each data row becomes its own event, where the column names map to their corresponding row values. All events split from the same incoming event have the same timestamp as the incoming event.
If the incoming payload doesn't follow this structure, Lumi doesn't split it and ingests it as a single event. If the number of values in a data row doesn't match the number of column names, Lumi silently skips the row.
Example batched input:
06/23/2025 13:14:14.786 +0530
collection=CPU
category=CPU
object=Processor
instance %_Processor_Time %_User_Time %_Privileged_Time Interrupts/sec %_DPC_Time %_Interrupt_Time DPCs_Queued/sec DPC_Rate %_Idle_Time %_C1_Time %_C2_Time %_C3_Time C1_Transitions/sec C2_Transitions/sec C3_Transitions/sec
0 77.43091558329556 13.52817406291745 63.90274152037811 1487.2618390472518 2.614907735192046 11.94830572610538 148.93725107362184 0 20.71592864083621 20.71592864083621 0 0 1472.8836215093462 0 0
1 77.76318385342931 16.94372058183627 60.81946327159304 1301.4839270153648 1.4927305816420385 9.21685730294172 107.63910285471936 0 20.53827491106385 20.53827491106385 0 0 421.17293850641726 0 0
Example single event:
collection: CPU
category: CPU
object: Processor
instance: 0
%_Processor_Time: 77.43091558329556
%_User_Time: 13.52817406291745
...
Incoming events must have the associated sourcetype value to trigger the event splitting logic:
PerfmonMk:ProcessorPerfmonMk:CPUPerfmonMk:MemoryPerfmonMk:LogicalDiskPerfmonMk:PhysicalDiskPerfmonMk:ProcessPerfmonMk:NetworkPerfmonMk:Network_InterfacePerfmonMk:SystemPerfmonMk:ProcessorInformation
The Splunk Add-on for Microsoft Windows assigns these source types when it collects Perfmon data in its default multikv mode.
You can also assign sourcetype using your forwarding agent or Lumi IAM key.
Don't use a pipeline since event splitting happens before pipeline transformation.
Lumi doesn't split Perfmon data collected in single mode, which uses Perfmon:* source types.
Lumi performs Perfmon event splitting only when the entire set of logs arrives in a single payload. This applies when you forward events from a Splunk forwarder using S2S, which automatically concatenates multi-line events. To backfill using other ingestion integrations:
- Concatenate all logs using
\nline breaks to send through a single Splunk® HEC request - Use Splunk CSV format where
_rawcontains the entire set of logs wrapped in", and_timecontains the timestamp in ISO 8601 format. Upload the file in Lumi directly or backfill with S3 pull.
After event splitting, Lumi processes the events using the Windows event logs predefined pipeline unless you manually disable it.
Learn more
For more information, see the following topics:
- Send events to Lumi for ways to send events to Lumi.
- Event formats for supported event formats.
- Timestamp detection to learn how Lumi detects and assigns event timestamps.
- Event model to learn how to assign user attributes.