Skip to main content

Event splitting reference

AI summary
Explains how Imply Lumi detects event boundaries when processing batched data from various sources. Covers standard protocol-based splitting and special handling for AWS CloudTrail and Windows Performance Monitor events.

About AI summaries.

When you send events to Imply Lumi, you might stream them one at a time or group them in batches to make fewer calls. With batched events, Lumi needs to determine where one event ends and another one begins. Event splitting in Lumi is the process of parsing a single incoming payload into discrete events.

Imply Lumi performs event splitting after receiving events and before any pipeline transformations. Pipelines operate on events individually, so you can't configure them to split or merge events.

Event boundary detection​

To detect event boundaries, Lumi follows the standards for the data transmission protocol used to send the events, such as HEC, OTLP, or S2S. For file upload or S3 pull, Lumi creates an event for each JSON object, CSV row, or plain text line.

info

For the S2S protocol, when an event spans multiple lines, Lumi concatenates them into one. If you need to disable this, update the IAM key to clear the selection Should line merge.

Incoming events can have attached metadata, such as to describe its source type or index. For the sourcetype values described in the following sections, Lumi performs further event processing.

AWS CloudTrail​

Criteria: sourcetype=aws:cloudtrail

AWS CloudTrail records are formatted as a JSON object in which Records is an array that contains all the events. Lumi extracts each object in the array and creates an event for each.

Example batched input:

{
  "Records": [
    {
      "eventVersion": "1.08", 
      "eventTime": "2023-01-02T02:51:00Z", 
      "awsRegion": "us-east-1",
      ...
    },
    {
      "eventVersion": "1.08", 
      "eventTime": "2023-01-02T00:22:00Z", 
      "awsRegion": "us-east-1", 
      ...
    },
    ...
  ]
}

Example single event:

eventVersion: 1.08
eventTime: 2023-01-02T02:51:00Z
awsRegion: us-east-1

Lumi then processes the events using the AWS CloudTrail predefined pipeline unless you manually disable it.

Windows Performance Monitor​

Criteria: when sourcetype is one of the following:

  • PerfmonMk:Processor
  • PerfmonMk:CPU
  • PerfmonMk:Memory
  • PerfmonMk:LogicalDisk
  • PerfmonMk:PhysicalDisk
  • PerfmonMk:Process
  • PerfmonMk:Network
  • PerfmonMk:Network_Interface
  • PerfmonMk:System
  • PerfmonMk:ProcessorInformation

Windows Performance Monitor (Perfmon) records are tab-separated values with several header lines. Lumi assigns the header key-value pairs as user attributes on each event. Each data row becomes its own event, where TSV column headers map to their corresponding row values. If the number of fields in a data row doesn't match the header count, Lumi silently skips over the line.

Example batched input:

06/23/2025 13:14:14.786 +0530
collection=CPU
category=CPU
object=Processor
instance	%_Processor_Time	%_User_Time	%_Privileged_Time	Interrupts/sec	%_DPC_Time	%_Interrupt_Time	DPCs_Queued/sec	DPC_Rate	%_Idle_Time	%_C1_Time	%_C2_Time	%_C3_Time	C1_Transitions/sec	C2_Transitions/sec	C3_Transitions/sec	
0	79.34612894430595	14.71359840693249	64.61188865652963	1590.3775442566346	2.833053109409361	12.885822207313547	155.45263533387762	0	18.82476423500179	18.82476423500179	0	0	1565.1705894677325	0	0	
1	79.98585061417258	17.82081794628469	61.96161316708216	1354.625259444422	1.5536097696761013	9.778602667961344	112.52478193468042	0	19.359221202868348	19.359221202868348	0	0

Example single event:

collection: CPU
category: CPU
object: Processor
instance: 0
%_Processor_Time: 79.34612894430595
%_User_Time: 14.71359840693249
...

Lumi then processes the events using the Windows event logs predefined pipeline unless you manually disable it.

info

Lumi performs Perfmon event splitting only when the entire set of logs arrives as a single event. This applies when you send it over S2S, which automatically concatenates multi-line events. If you send a plain text file with Perfmon logs, Lumi reads each line individually instead of applying Perfmon event splitting.

To backfill Perfmon logs, do one of the following:

  • Concatenate all logs using \n line breaks to send through a single Splunk HEC request
  • Use Splunk CSV format where _raw contains the entire set of logs wrapped in ", and _time contains the timestamp in ISO 8601 format. Upload the file in Lumi directly or backfill with S3 pull.

Learn more​

For more information, see the following topics: