Data overview

Like many database systems, Imply Polaris stores your data in tables. Behind the scenes, the tables are in the columnar format used by Apache Druid. Polaris lets you create up to 20 tables to power your OLAP use cases.

This topic provides a general overview of the data lifecycle for both the Polaris UI and the API. For more information on how to use the API, see Polaris API overview.

Supported input data formats

Prior to building tables and loading data, verify that your source data adheres to the supported data and file formats.

Create a schema

A schema defines the columns and other metadata about your tables. You can create a schema:

• Manually with the UI. See Create a schema.
• Using schema detection when you upload a file. Follow the Quickstart.
• Using the API. See Define table schemas by API.

Polaris offers the following schema optimizations:

• Rollup, a form of pre-aggregation that reduces the size of stored data and increases query performance.
• Partitioning, a way to organize your schema to increase query performance based upon columns you frequently use for filters.

Ingest data

Polaris supports the following data ingestion methods:

• Batch ingestion where you upload data files for processing.
• Push-based streaming where you use the Events API to push events to Polaris.

Polaris automatically scales the number of ingestion tasks within your project to maintain optimal performance.

Batch ingestion

Batch ingestion is a way to load data into Polaris based upon file uploads as follows:

1. Upload your source files to the Polaris file staging area through the UI or using the Tables API.
2. Ingest data into your table manually through the UI or using the Tables API.
3. While your data is loading, you can monitor the status of your ingestion job on the Ingestion Jobs tab in the UI or using the the Ingestion Jobs API.

Push-based streaming

You can push event data directly into Polaris using the Events API.

Work with existing tables and data

After you create a table, you can:

• Add batch data in the UI or using the API.
• Replace existing data for specific time intervals.
• Drop data in the UI or the API. When you drop data, you have the option to drop all data or the data for a time interval.
• Delete the table in the UI or using the API.

To access table operations within the Tables list, click the ... menu button at the far left of the table.

Known limitations

We are regularly updating Polaris to add features and fixes. If you run into an issue, check Known limitations.

Learn more

See the following topics for more information:

• Quickstart for a tutorial on how to upload data using batch ingestion.
• Load event data for information and examples on how to load event stream data.