File formats

AI summary

Explains supported file and compression formats for batch ingestion in Imply Lumi, including CSV, JSON, and various compressed formats, useful for backfilling historical data or evaluating logs from stored files.

About AI summaries.

When you use Imply Lumi in your observability workflows, you typically have an incoming stream of events for the logs you want to capture and analyze. In some cases, you might need to ingest a batch of events stored in a file. This can be useful when performing a backfill to load historical events or for a quick evaluation of your data in Lumi.

Of the available ingestion integrations, you can use file upload and S3 pull to upload events as a batch. This topic lists the supported file and compression formats that you can use for these integrations.

File formats

The table in this section describes the file formats supported for batch ingestion. See the following sections for more details on additional file format requirements.

Format	File upload	S3 pull
Plain text	❌	✅
CSV	✅ *	✅
JSON	✅	✅

*CSV for file upload requires the fields _raw and _time. When you export data from Splunk® using the CSV format, the file includes these fields by default. For information about exporting data from Splunk, see Export data using Splunk Web.

Note the following restrictions for file formats. You can't ingest:

• A CSV file that doesn't contain a header line.
• A single JSON object that contains all the events.
• JSON files exported from Splunk.

JSON format details

With JSON, you can use the same format you send to Splunk HEC, where the JSON object contains event and time. For more information, see Format events for HTTP event collector in the Splunk documentation.

In addition, JSON files can have any of these structures:

• Separate objects: each object is considered a separate event.

  For example:

  {"time": "2025-11-14T22:46:11Z", "event": "example log 1"}{"time": "2025-11-14T23:46:11Z", "event": "example log 2"}

• Array of objects: each object in the array is an event.

  For example:

  [
    {"time": "2025-11-14T22:46:11Z", "event": "example log 1"},
    {"time": "2025-11-14T23:46:11Z", "event": "example log 2"}
  ]

• Newline-delimited objects: each line is considered an event.

  For example:

  {"time": "2025-11-14T22:46:11Z", "event": "example log 1"}
  {"time": "2025-11-14T23:46:11Z", "event": "example log 2"}

Compressed files

A compressed file format allows you to save storage space and streamline data management tasks. You can ingest a compressed file in one of the supported file formats.

Lumi supports the following compression formats:

• Brotli
• BZIP2
• DEFLATE
• GZIP
• LZMA
• LZ4
• Snappy
• XZ
• Z
• ZSTD

File format detection

To detect the format of a file, Lumi uses the following heuristic:

1. Check the file extension: Identify the file extension such as .csv or .csv.gz. When Lumi detects a known format extension, it performs the following checks:

   • A base file extension of .json or .ndjson indicates JSON format.
   • A base file extension of .csv indicates CSV format.

2. Inspect file contents: When Lumi can't determine the file format from the extension, it evaluates the file contents. Lumi reads the first 1024 bytes and checks for the following:

   • Contents starting with { or [ indicates JSON format.
   • Contents that contain commas in a consistent pattern on each line indicates CSV format.

3. If none of the preceding criteria are met, Lumi treats the file format as unknown and proceeds with line-based parsing.