Skip to main content

Set a storage policy by API

info

Project-less regional API resources have been deprecated and will be removed by the end of September 2024.

You must include the project ID in the URL for all regional API calls in projects created after September 29, 2023. For example: https://ORGANIZATION_NAME.REGION.CLOUD_PROVIDER.api.imply.io/v1/projects/PROJECT_ID

Projects created before September 29, 2023 can continue to use project-less URLs until the end of September 2024. We strongly recommend updating your regional API calls to include the project ID prior to September 2024. See the API migration guide for more information.

You can set a storage policy on a table to configure how long to retain data for a given table. For information on setting a storage policy using the UI, see Data lifecycle management.

Data outside the retention period does not appear in queries or count towards your project size. Polaris schedules the data for permanent deletion 30 days after the retention period expires. Note that permanent deletion is a periodic, intermittent process. Once scheduled, permanent deletion may take up to an additional 30 days to complete.

Prerequisites​

You must have an API key with the ManageTables permission. In the examples below, the key value is stored in the variable named POLARIS_API_KEY. See API key authentication to obtain an API key and assign permissions. Visit Permissions reference for more information on permissions.

Create a table with a retention policy​

When creating or updating a table, you can set a storage policy on it for data lifecycle management. A retain-type storage policy, or retention policy, automatically deletes data with timestamps older than a specified time period. If not set, the default storage policy is to retain all data forever until explicitly deleted.

To create a table with a retention policy, include the storagePolicy property in the request payload, such as follows:

    "storagePolicy": {
        "retain": {
            "period": "P3M",
            "type": "period"
        }
    }

The retain-type policy only keeps data within the specified period and schedules deletion for data older than this period. Supply the period value as an ISO 8601 duration.

Sample request​

Send a POST request to the /v1/projects/PROJECT_ID/tables endpoint to create a new table.

For more information on creating tables, see Create a table by API.

curl --location --request POST "https://ORGANIZATION_NAME.REGION.CLOUD_PROVIDER.api.imply.io/v1/projects/PROJECT_ID/tables" \
--user ${POLARIS_API_KEY}: \
--header 'Content-Type: application/json' \
--data '{
    "name": "Koalas Retention",
    "type": "detail",
    "storagePolicy": {
        "retain": {
            "period": "P3M",
            "type": "period"
        }
    }
}'

Sample response​

The following example shows a successful response:

Click to view the response
{
    "schema": [
        {
            "name": "__time",
            "dataType": "timestamp"
        }
    ],
    "name": "Koalas Retention",
    "type": "detail",
    "version": 0,
    "availability": "available",
    "clusteringColumns": [],
    "compactionConfig": null,
    "createdBy": {
        "username": "api-key-pok_vipgj...bjjvyo",
        "userId": "a52cacf6-3ddc-48e5-8675-xxxxxxxxxxxx"
    },
    "createdByUser": {
        "username": "api-key-pok_vipgj...bjjvyo",
        "userId": "a52cacf6-3ddc-48e5-8675-xxxxxxxxxxxx"
    },
    "createdOnTimestamp": "2023-08-10T23:43:01.543495184Z",
    "createdTimestamp": "2023-08-10T23:43:01.543495184Z",
    "description": null,
    "id": "0189e1d5-05a6-7015-bb2c-de10182c7f03",
    "lastModifiedBy": {
        "username": "api-key-pok_vipgj...bjjvyo",
        "userId": "a52cacf6-3ddc-48e5-8675-xxxxxxxxxxxx"
    },
    "lastUpdateTimestamp": "2023-08-10T23:43:01.543499193Z",
    "modifiedByUser": {
        "username": "api-key-pok_vipgj...bjjvyo",
        "userId": "a52cacf6-3ddc-48e5-8675-xxxxxxxxxxxx"
    },
    "modifiedOnTimestamp": "2023-08-10T23:43:01.543499193Z",
    "partitioningGranularity": "day",
    "queryableSchema": [],
    "storagePolicy": {
        "retain": {
            "period": "P3M",
            "type": "period"
        }
    },
    "schemaMode": "flexible",
    "segmentCompactedBytes": 0,
    "segmentTotalBytes": 0,
    "totalDataSizeBytes": 0,
    "totalRows": 0
}

Update a table to add a retention policy​

You can add a storage policy to an existing table that already contains data. When you update the retention period on a table to a longer time period, Polaris does not recover previously deleted data.

To update the retention period back to the default behavior of retaining all data forever, use the following value for storagePolicy:

    "storagePolicy": {
        "retain": null
    }

When sending a PUT request to update a table, keep in mind the following differences from creating a table:

  • Supply the table name as a path parameter.
  • Include version in the request body.

For additional details on a table update request, see Create a table by API.

Sample request​

Send a PUT request to the /v1/projects/PROJECT_ID/tables/TABLE_NAME endpoint to update a table. See the Tables v1 API documentation for more information.

curl --location --request POST "https://ORGANIZATION_NAME.REGION.CLOUD_PROVIDER.api.imply.io/v1/projects/PROJECT_ID/tables/Koalas" \
--user ${POLARIS_API_KEY}: \
--header 'Content-Type: application/json' \
--data '{
    "name": "Koalas",
    "type": "aggregate",
    "version": 0,
    "storagePolicy": {
        "retain": {
            "period": "P1M",
            "type": "period"
        }
    }
}'

Sample response​

The following example shows a successful response:

Click to view the response
{
    "schema": [
        {
            "dataType": "timestamp",
            "type": "dimension",
            "name": "__time"
        }
    ],
    "timeResolution": "millisecond",
    "name": "Koalas",
    "type": "aggregate",
    "version": 1,
    "availability": "available",
    "clusteringColumns": [],
    "compactionConfig": null,
    "createdBy": {
        "username": "api-key-pok_vipgj...bjjvyo",
        "userId": "a52cacf6-3ddc-48e5-8675-xxxxxxxxxxxx"
    },
    "createdByUser": {
        "username": "api-key-pok_vipgj...bjjvyo",
        "userId": "a52cacf6-3ddc-48e5-8675-xxxxxxxxxxxx"
    },
    "createdOnTimestamp": "2023-08-10T23:27:51.549893Z",
    "createdTimestamp": "2023-08-10T23:27:51.549893Z",
    "description": null,
    "id": "0189e1c7-22fd-7ea3-addd-a1f06705afa0",
    "lastModifiedBy": {
        "username": "api-key-pok_vipgj...bjjvyo",
        "userId": "a52cacf6-3ddc-48e5-8675-xxxxxxxxxxxx"
    },
    "lastUpdateTimestamp": "2023-08-10T23:50:57.554676548Z",
    "modifiedByUser": {
        "username": "api-key-pok_vipgj...bjjvyo",
        "userId": "a52cacf6-3ddc-48e5-8675-xxxxxxxxxxxx"
    },
    "modifiedOnTimestamp": "2023-08-10T23:50:57.554676548Z",
    "partitioningGranularity": "day",
    "queryableSchema": [],
    "storagePolicy": {
        "retain": {
            "period": "P1M",
            "type": "period"
        }
    },
    "schemaMode": "flexible",
    "segmentCompactedBytes": 0,
    "segmentTotalBytes": 0,
    "totalDataSizeBytes": 0,
    "totalRows": 0
}

Learn more​

See the following topics for more information: