Skip to main content

S2S event parsing reference

AI summary
Explains parsing settings for events from universal forwarders using the tcpout integration in Imply Lumi. Covers how Lumi handles event breaks and timestamps. Details configuration options when automatic timestamp detection fails.

About AI summaries.
info

This information only applies to tcpout configured on universal forwarders.

Imply Lumi supports a subset of Splunk® properties to configure parsing of events in the S2S protocol. This topic describes parsing settings for the tcpout integration. You assign them as S2S attributes on the IAM key. You can define conditional S2S attributes for when you have multiple event formats or source types associated with the same key.

Note that these settings don't apply to heavy forwarders or S2S over HTTP (httpout).

If your data isn't compatible—or can't be made compatible—with the available options, see other approaches to send events and explore how you can use pipelines to transform incoming events.

Event line breaking​

Lumi determines event breaks using the following heuristics:

  1. Detect and separate lines by newline characters \r or \n.
  2. Detect a datetime field as the start of a new event.

When an event spans multiple lines, Lumi merges them before storing the event. A datetime field at the start of a line identifies where the next event begins. To disable merging of multi-line events, clear the Should line merge checkbox. This option corresponds to SHOULD_LINEMERGE in a Splunk props.conf configuration file. For more information, see the Splunk documentation for event line breaking.

Event timestamps​

Lumi can automatically parse and extract timestamps from incoming events. For details about timestamp detection and assignment in Lumi, see Timestamp handling.

In cases where Lumi can't detect timestamps, you can configure timestamp parsing using the properties in this section. They correspond to properties of the same name in a Splunk props.conf file. For more information, see the Splunk documentation for timestamp recognition.

Time prefix​

String regular expression that matches the text pattern preceding the timestamp.

  • Default: empty string
  • Example: [\w\.:]*\s[\w-]*\s[\w-]*\s\[
  • Splunk configuration: TIME_PREFIX

Max timestamp lookahead​

Integer number that indicates the maximum character position to look for a timestamp. If you also set the time prefix, this position starts after the matched text.

  • Default: 128
  • Example: 20 means to search for the timestamp in the first 20 characters of the event message, following any time prefix.
  • Splunk configuration: MAX_TIMESTAMP_LOOKAHEAD

Time format​

String pattern in strptime format to extract timestamps. Supports Splunk-specific time variables. See the SPL reference for details.

  • Default: empty string
  • Example: %Y-%m-%d %H:%M:%S
  • Splunk configuration: TIME_FORMAT

Example​

Consider an event with the following message:

830:1e0e:525:e6a0:6479:cd69:c364:23c3 - - [24/Mar/2025:16:25:29 -0500] "POST /products/23394 HTTP/1.1" 200 1027 "https://techcrunch.com/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:110.0) Gecko/20100101 Firefox/110.0"

The timestamp, 24/Mar/2025:16:25:29 -0500, appears after the client IP address, any identity provided for the client, and any provided user ID.

The following settings control how Lumi parses this timestamp:

  1. Time prefix: [\w\.:]*\s[\w-]*\s[\w-]*\s\[
    Look for the timestamp following this regular expression pattern:

    • [\w\.:]* to match zero or more word characters, periods, or colons
    • \s to match a single space
    • [\w-]* to match zero or more word characters or hyphens
    • \s to match a single space
    • \[ to match a open square bracket

    Matches the string 830:1e0e:525:e6a0:6479:cd69:c364:23c3 - - [.

  2. Max timestamp lookahead: 20
    Look for the timestamp within the next 20 characters.
    Starts from 24, the characters immediately following the prefix.
    Matches the value 24/Mar/2025:16:25:29.

  3. Time format: %d/%b/%Y:%H:%M:%S
    Format the discovered timestamp according to this pattern.

Learn more​

For more information, see the following topics: