Skip to main content

File formats

When you use Imply Lumi in your observability workflows, you typically have an incoming stream of events for the logs you want to capture and analyze. In some cases, you might need to ingest a batch of events stored in a file. This can be useful when performing a backfill to load historical events or for a quick evaluation of your data in Lumi.

Of the available ingestion integrations, you can use file upload and S3 pull to upload events as a batch. This topic lists the supported file and compression formats that you can use for these integrations.

File formats​

You can batch ingest events from the file formats listed in the following table:

FormatFile uploadS3 pull
Plain text❌✅
CSV✅ *✅
JSON✅✅

* CSV for file upload requires the fields _raw and _time. When you export data from Splunk® using the CSV format, the file includes these fields by default.

Lumi doesn't support the following file formats:

  • A single JSON object that contains all the events.
  • JSON files exported from Splunk.

For information about exporting data from Splunk, see Export data using Splunk Web.

JSON format details​

With JSON, you can use the same format you send to Splunk HEC, where the JSON object contains event and time. For more information, see Format events for HTTP event collector in the Splunk documentation.

In addition, JSON files can have any of these structures:

  • Separate objects: each object is considered a separate event.

    For example:

    {"time": "2025-11-14T22:46:11Z", "event": "example log 1"}{"time": "2025-11-14T23:46:11Z", "event": "example log 2"}

  • Array of objects: each object in the array is an event.

    For example:

    [
      {"time": "2025-11-14T22:46:11Z", "event": "example log 1"},
      {"time": "2025-11-14T23:46:11Z", "event": "example log 2"}
    ]

  • Newline-delimited objects: each line is considered an event.

    For example:

    {"time": "2025-11-14T22:46:11Z", "event": "example log 1"}
    {"time": "2025-11-14T23:46:11Z", "event": "example log 2"}

Compressed files​

A compressed file format allows you to save storage space and streamline data management tasks. You can ingest a compressed file in one of the supported file formats.

Lumi supports the following compression formats:

  • Brotli
  • BZIP2
  • DEFLATE
  • GZIP
  • LZMA
  • LZ4
  • Snappy
  • XZ
  • Z
  • ZSTD