Skip to main content

Data lifecycle management

The lifecycle for data in Imply Polaris includes ingesting, querying, and deleting data. You can control the data lifecycle by configuring storage policies that determine the time to live (TTL) for data stored in Polaris. Setting a storage policy to remove obsolete data can decrease your storage costs and increase query performance.

This topic shows you how to set a storage policy on a table to automatically delete stale data.

Automatically delete data​

A retain-type storage policy, or retention policy, automatically deletes data with timestamps older than a specified time period. Polaris evaluates data that meet the criteria for automatic deletion every minute.

Data outside the retention period does not appear in queries or count towards your project size. Polaris schedules the data for permanent deletion 30 days after the retention period expires. Note that permanent deletion is a periodic, intermittent process. Once scheduled, permanent deletion may take up to an additional 30 days to complete.

Keep the following points in mind when setting or updating a storage policy for a table:

  • If you ingest data older than the retention period, Polaris ingests but immediately removes the data.
  • Polaris always retains data later than the retention period, that is, data with timestamps in the future.
  • When you update an existing retention period to a longer time period, Polaris does not automatically recover previously dropped or deleted data.
  • The table's partitioning setting limits the time granularity for the time period. For example, when the table has a partitioning granularity of day, you can't configure a retention period at the minute level.

Follow these steps to configure a storage policy for automatically deleting data:

  1. From the table view, click Manage > Edit table.
  2. Click Data retention.
  3. Select Specific and enter the time period for data retention. Polaris deletes data older than this period.
  4. Click Update.
  5. Click Save.

The following screenshot shows the Edit table view with a specific time period configured for data retention:

Retain storage policy

Use the API to create a table with a retention policy or to update an existing table to add a retention policy. For more information, see Set a storage policy by API.

Learn more​

See the following topics for more information: