Skip to main content

External event formats reference

AI summary

An external data connection lets Imply Lumi query data that stays in your own Amazon S3 storage without ingesting it into Lumi. When you connect external data, you select the file type of your data on the Transformation page of the connection wizard. The file type determines how Lumi reads your data, which fields it extracts, and how you search the data.

This topic lists the file types you can use for external data, and the settings for each.

info

This topic applies to external data only. For the formats Lumi supports when you ingest events as managed data, see Event formats reference.

Supported file types​

All file types support keyword search. Field search depends on whether Lumi extracts fields from the file.

For all file types, Lumi sets the index attribute on each event to the connection's index attribute. This replaces any index field in your data.

File typeKeyword searchField searchNotes
CSVYesYes, all fieldsFields come from the header row or custom headers
HECYesHEC metadata only: host, source, sourcetypeThe event payload is searchable by keyword but isn't available as named fields
JSONYesYes, all fieldsFields come from the JSON keys
ParquetYesYes, all fieldsFields come from the file's schema
Plain textYesNoNo fields are extracted
Splunk DDSSYesIndex-time fields onlyFields include host, source, sourcetype, and any other fields indexed in Splunk

The following sections describe each file type and its settings.

CSV​

Lumi uses the first row or your custom headers as column names and extracts each column as a searchable field. The timestamp column becomes the event time. Lumi doesn't add a field for an empty value.

Configure the following:

  • Timestamp column: Column that holds the event time, for example timestamp.
  • Timestamp pattern: DateTimeFormatter pattern that matches the timestamp value and must include the time zone or offset. For example, yyyy-MM-dd'T'HH:mm:ssXXX matches 2026-10-01T14:55:36+01:00.
  • Delimiter: One or more characters that separate columns. Defaults to a comma (,).
  • Custom headers: Comma-separated list of column names to treat as headers. Leave blank to use the values in the first row as column names.
  • Rows to skip: Number of rows at the start of each file that Lumi skips before it reads the header row.
  • Field names: How field names map to event attributes. Select Keep as-is to preserve original names, or Strip leading underscores to remove a leading underscore from field names. For example, _id becomes id.

HEC​

HEC events carry their own timestamps and metadata, so no configuration is needed.

Lumi extracts the standard HEC metadata (host, source, sourcetype) as searchable fields. You can search the event payload by keyword but not by field name. This applies whether the payload is raw text or a JSON object.

Lumi replaces the index value from the original HEC event with the connection's index attribute. To search the connection, use the connection's index attribute, for example index=hec-json-logs.

JSON​

Lumi extracts each key in the JSON object as a searchable field.

Configure the following:

  • Format: JSON layout of your files. Select one of the following:
    • NDJSON: Newline-delimited JSON, with one object per line.
    • CloudTrail: AWS CloudTrail log files. Lumi reads the timestamps and field names automatically, so you don't need to configure the other options.
  • Timestamp path: Field that holds the event time, for example timestamp.
  • Timestamp pattern: DateTimeFormatter pattern that matches the timestamp value and must include the time zone or offset. For example, yyyy-MM-dd'T'HH:mm:ss.SSSX matches 2026-08-27T23:59:48.271Z.
  • Field names: How field names map to event attributes. Select Keep as-is to preserve original names, or Strip leading underscores to remove a leading underscore from field names. For example, _id becomes id.

Parquet​

Parquet files carry a schema, so Lumi extracts every column as a searchable field.

Configure the following:

  • Timestamp path: Column that holds the event time, for example timestamp.
  • Timestamp pattern: DateTimeFormatter pattern that matches the timestamp value and must include the time zone or offset. For example, yyyy-MM-dd'T'HH:mm:ssXXX matches 2026-10-01T14:55:36+01:00.
  • Field names: How field names map to event attributes. Select Keep as-is to preserve original names, or Strip leading underscores to remove a leading underscore from field names. For example, _id becomes id.

Plain text​

Plain text files have no inherent structure, so Lumi extracts no fields. Connections to plain text data support keyword search only, so you can search the raw text by keyword but not by field.

Configure the following:

  • Timestamp regex: Regular expression that locates the timestamp within each line. For example, \d{2}/[A-Za-z]{3}/\d{4}:\d{2}:\d{2}:\d{2} [+-]\d{4} finds 27/Aug/2026:23:32:19 +0000 in a line.
  • Date-time format: DateTimeFormatter pattern that parses the located timestamp and must include the time zone or offset. For example, dd/MMM/yyyy:HH:mm:ss Z parses 27/Aug/2026:23:32:19 +0000.

Splunk DDSS​

Events in Splunk® Dynamic Data Self Storage (DDSS) carry their own timestamps and metadata, so no configuration is needed.

Lumi extracts the index-time fields stored with each event as searchable fields. These include host, source, and sourcetype, plus any other fields indexed in Splunk, such as k8s.pod.name.

Learn more​

See the following topics for more information: