Data partitioning

Partitioning is a method of organizing a large dataset into partitions to aid in data management and improve query performance in Imply Polaris.

By distributing data across multiple partitions, you decrease the amount of data that needs to be scanned at query time, which reduces the overall query response time.

For example, if you always filter your data by country, you can use the country dimension to partition your data. This improves the query performance because Polaris only needs to scan the rows related to the country filter.

Time partitioning

Polaris partitions datasets by timestamp based on the selected time partitioning granularity.

By default, time partitioning is set to day, which is sufficient for most applications. Depending on the use case and the size of your dataset, you may benefit from a finer or a coarser setting. For example:

• For highly aggregated datasets, where a single day contains less than one million rows, a coarser time partitioning may be appropriate.
• For datasets with finer granularity timestamps where queries often run on smaller intervals within a singe day, a finer time partitioning may be more suitable.

When using partitioning with rollup, partitioning time granularity must be coarser than or equal to the rollup granularity.

To change the time partitioning, in the Edit schema view, click Partitioning and choose from the available time partitioning settings. You can partition your data by hour, day, week, month, and year.

Generally, fine-tuning clustering and rollup is more impactful on performance than using time partitioning alone.

Clustering

In addition to partitioning by time, you can partition further using other columns. This is often referred to as clustering or secondary partitioning.

To achieve the best performance and the smallest overall memory footprint, we recommend choosing the columns you most frequently filter on. Doing so decreases access time and improves data locality, the practice of storing similar data together.

Sort order

When configuring clustering, select the column you filter on the most as your first dimension. This signals Polaris to sort the rows within each partition by that column, which often improves data compression.

Polaris always sorts the rows within a partition by timestamp first.

You can drag and drop the columns to change the order in with they appear for clustering.

The following screenshot shows a table before ingestion with time partitioning set to week and clustering configured on the continent, country, and language columns in that order.

Limitations

Time partitioning granularity must be coarser than or equal to the rollup granularity.