Supported data and file formats

This topic is a reference for data and file format support in Imply Polaris.

Supported source data formats

Polaris supports the following data formats:

• Newline delimited JSON for batch file upload and push streaming.
• CSV files with a single header row for batch file upload.

Nested data, including newline-delimited JSON data, is not supported.

Supported time formats

Polaris supports the following time formats for all ingestion types. Example times are all for December 15th, 2022 at 1:00:00pm UTC.

• iso: An ISO 8601 or RFC 3339 timestamp. For example, 2021-12-15T14:00:23-8:00 or 2021-12-15 22:00:23Z.
• millisUtc: Milliseconds since midnight of January 1, 1970 (the UNIX epoch) in UTC. For example, 1639573200000.

Additional time format support

Batch ingestion using the tables API supports the following additional formats. Example times are all for December 15th, 2022 at 1:00:00pm UTC.

• posixUtc: A UNIX timestamp representing seconds since January 1, 1970 (the UNIX epoch) in UTC. For example, 1639573200.
• microUtc: Microseconds since midnight of January 1, 1970 (the UNIX epoch) in UTC. For example, 1639573200000000.
• nanoUtc: Nanoseconds since midnight of January 1, 1970 (the UNIX epoch) in UTC. For example, 1639573200000000000.

See Ingest data into a table by API for information on how to specify the time format for batch ingestion in the API.

Supported file compression formats

Polaris supports the following compression formats for uploaded files:

• bz2: Bzip2
• gz: Gzip
• xz: XZ
• sz: Snappy
• zst: Zstandard

ZIP files are not supported.

File size limit

Polaris supports individual files up to 10 GB. This limit refers to the size of the file transmitted by the browser or HTTP client.

You may upload a file that's larger than 10 GB on disk if your browser or client compresses the file less than 10 GB in transit.