Supported data and file formats
This topic is a reference for data and file format support in Imply Polaris.
Supported source data formats
The following table describes the data formats that Polaris supports for batch and streaming ingestion.
| Format | Batch ingestion | Streaming ingestion |
|---|---|---|
| Newline-delimited JSON | Yes | Yes |
| Delimiter-separated values | Yes | Yes |
| Apache Parquet | Yes | No |
| Apache ORC | Yes | No |
| Apache Avro | Yes (Avro OCF) | Yes (not supported for push streaming) |
| Protocol Buffers (Protobuf) | No | Yes (not supported for push streaming) |
Polaris supports nested data for all supported data formats.
For details on how to specify your input data schema for Avro and Protobuf formats, see Specify input schema by API.
Supported file compression formats
Polaris supports the following compression formats for uploaded files:
ZIP files and TAR files are not supported.
You can send gzipped data in push streaming ingestion with the HTTP header Content-Encoding: gzip.
File size limit
Polaris supports individual files up to 10 GB. This limit refers to the size of the file transmitted by the browser or HTTP client.
You may upload a file that's larger than 10 GB on disk if your browser or client compresses the file less than 10 GB in transit.
Learn more
For information on supported timestamp formats in Polaris, see Timestamp expressions.