Skip to main content

Restore or permanently delete data

After you soft delete data in Imply Polaris or data expires based on your retention period, you have a grace period that can be up to 30 days to:

  • Do nothing and let Polaris permanently delete the data after the grace period.
  • Proactively delete the data permanently before the grace period ends. Permanently deleting data reduces your data usage, helping you stay within your project size.
  • Restore the data if it is within the grace period. Note that there is some lag between recovering data and it becoming queryable again.
Data deletion

Imply reserves the right to delete data before the end of the 30 day grace period if needed.

This topic shows how to access deleted data in order to restore or permanently delete it.

Prerequisites​

To manage deleted data, you need the ManageTables and ManageIngestionJobs permissions. These permissions are assigned to the Organization Admin, Project Admin, and Data Manager groups by default. For information on permissions, see Permissions reference.

Restorable data​

For data to be eligible for recovery, it must have been soft deleted within the last 30 days or be within 30 days of the data exceeding the retention period.

Segment overlap​

You can't restore a segment whose interval overlaps with any interval already present in the table.

To restore the deleted interval, you must first delete the interval in the table that overlaps the same time interval.

Schema incompatibility​

Schema incompatibility occurs when there are changes in the declared schema, leading to inconsistencies between the declared schema of the restored data and the declared schema on the table.

When the schema of the deleted data isn't compatible with the schema declared on the table, Polaris still recovers the data. However, incompatible values of the recovered data become NULL at query time.

You can restore the nulled values by undeclaring the column that contains the incompatible values. This removes the type constraint since an undeclared column uses the data type that works for all data in the column. You can still declare the column again, using a compatible data type.

Restore data​

To recover data deleted from a table, do the following:

  1. Navigate to the table you want to recover data for.

  2. Select Manage > Manage deleted data.

  3. On the Manage deleted data page, identify the segments for the data you want to recover:

    UI page showing deleted segments that you can recover

    For more guidance on this step, see Identify data to recover.

  4. Select the data segments you want to recover. Then, select Recover.

    After you select Recover, a confirmation window appears.

  5. Verify that the intervals are correct and select Recover. A notification that confirms the recovery job appears. You can monitor the status from the table's page.

Permanently delete data​

To permanently delete data, follow the steps for recovering data, but select Permanently delete instead.

Identify data to recover​

The underlying data in Polaris is organized into segments. Segments are files that store data over units of time, the granularity of which is determined by the table's time partitioning. A given time interval may have zero, one, or more segments, depending on how much data has been ingested for that time interval.

When you delete data, Polaris removes all segments corresponding to your selected time range. To recover or permanently delete data, you need to identify the interval or intervals, that contain the relevant data. Use the following characteristics to identify the deleted segments to recover or permanently delete:

  • Time range of the data. Segments are designated by the start and end time of the __time column.
    For example, 2024-01-01, 00:00:00 to 2024-01-02, 00:00:00.
  • Date the data was ingested. For segments with the same time range, use the segment creation time for the date the data was ingested. This can help you differentiate segments from a previously deleted table compared to segments from a currently existing table with the same name.
  • Date the data was deleted. If you know when you deleted your data, such as from reviewing recent jobs, the deletion date can help you distinguish segments from within the same time range. This date also lets you know the remaining time on the grace period to recover the deleted data.

Learn more​

See the following topics for more information: