Skip to main content

View and manage ingestion jobs by API

info

Project-less regional API resources have been deprecated and will be removed by the end of September 2024.

You must include the project ID in the URL for all regional API calls in projects created after September 29, 2023. For example: https://ORGANIZATION_NAME.REGION.CLOUD_PROVIDER.api.imply.io/v1/projects/PROJECT_ID

Projects created before September 29, 2023 can continue to use project-less URLs until the end of September 2024. We strongly recommend updating your regional API calls to include the project ID prior to September 2024. See the API migration guide for more information.

After you create an ingestion job for Imply Polaris, you can monitor its progress, view job-related metrics, and cancel its execution. This topic walks you through the process of viewing job-related details and canceling an ingestion job.

Prerequisites​

This topic assumes that you have an API key with the ManageIngestionJobs permission. In the examples below, the key value is stored in the variable named POLARIS_API_KEY. For information about how to obtain an API key and assign permissions, see API key authentication. For more information on permissions, visit Permissions reference.

Monitor ingestion job progress​

To monitor the progress of your ingestion job, issue a GET request to the Jobs v1 API with the job ID in the path. For example, /v1/projects/PROJECT_ID/jobs/efb35e3e-406e-4127-ad2e-280fede4f431.

If you need to identify your job ID, send a GET request to the /v2/jobs endpoint to retrieve information about all jobs.

Sample request​

The following example shows how to monitor the progress of an example ingestion job with the job ID efb35e3e-406e-4127-ad2e-280fede4f431:

curl --location --request GET "https://ORGANIZATION_NAME.REGION.CLOUD_PROVIDER.api.imply.io/v1/projects/PROJECT_ID/jobs/efb35e3e-406e-4127-ad2e-280fede4f431" \
--user ${POLARIS_API_KEY}: \
--header "Content-Type: application/json" \
--data-raw ''

Sample response​

The following example shows a successful response for an ingestion job progress:

Click to view the response
{
    "type": "batch",
    "id": "efb35e3e-406e-4127-ad2e-280fede4f431",
    "target": {
        "type": "table",
        "tableName": "Koalas Subset"
    },
    "desiredExecutionStatus": "running",
    "createdBy": {
        "username": "service-account-docs-demo",
        "userId": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
    },
    "lastModifiedBy": {
        "username": "service-account-docs-demo",
        "userId": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
    },
    "executionStatus": "pending",
    "health": {
        "status": "ok"
    },
    "createdTimestamp": "2022-08-09T22:34:46.716017658Z",
    "lastUpdatedTimestamp": "2022-08-09T22:34:46.716017658Z",
    "source": {
        "type": "uploaded",
        "fileList": [
            "kttm-2019-08-19.json.gz",
            "kttm-2019-08-20.json.gz"
        ],
        "inputSchema": [
            {
                "dataType": "string",
                "name": "timestamp"
            },
            {
                "dataType": "string",
                "name": "city"
            },
            {
                "dataType": "string",
                "name": "session"
            },
            {
                "dataType": "long",
                "name": "session_length"
            }
        ],
        "formatSettings": {
            "format": "nd-json"
        }
    },
    "ingestionMode": "append",
    "mappings": [
        {
            "columnName": "__time",
            "expression": "TIME_PARSE(\"timestamp\")"
        },
        {
            "columnName": "city",
            "expression": "\"city\""
        },
        {
            "columnName": "session",
            "expression": "\"session\""
        },
        {
            "columnName": "max_session_length",
            "expression": "MAX(\"session_length\")",
            "isAggregation": true
        },
        {
            "columnName": "__count",
            "expression": "COUNT(*)",
            "isAggregation": true
        }
    ]
}

The executionStatus field shows the job's current execution status, and the health field describes the health of the job.

View metrics for a job​

Send a GET request to /v1/projects/PROJECT_ID/jobs/JOB_ID/metrics to view the metrics for a job. Replace JOB_ID with the ID of the job.

The following example shows metrics for an ingestion job in which two rows could not be parsed. By default, Polaris continues to ingest data when it encounters rows it cannot parse. You can specify a threshold for parsing exceptions for a job. The job fails if the number of parsing exceptions exceeds the threshold. Set this limit in maxParseExceptions when creating an ingestion job. You can view parse exception messages in the job logs.

You can also use the UI to monitor the health of your ingestion job. For more information, see Monitor streaming ingestion from a connection.

Sample request​

The following example shows how to view metrics for an ingestion job with the job ID db6a3110-d6c3-4a63-86b2-41d51a65ce11:

curl --location --request GET "https://ORGANIZATION_NAME.REGION.CLOUD_PROVIDER.api.imply.io/v1/projects/PROJECT_ID/jobs/db6a3110-d6c3-4a63-86b2-41d51a65ce11/metrics" \
--user ${POLARIS_API_KEY}:

Sample response​

The following example shows a successful response for metrics of an ingestion job:

{
    "totals": {
        "numRowsProcessed": 99999,
        "numRowsProcessedWithWarning": 0,
        "numRowsSkippedByFilter": 0,
        "numRowsSkippedByError": 2
    }
}

Cancel an ingestion job​

To cancel an ingestion job, issue a PUT request to the Jobs v1 API with the job ID in the path. In the request body, set the desiredExecutionStatus to canceled.

Sample request​

The following example shows how to cancel an ingestion job with the job ID efb35e3e-406e-4127-ad2e-280fede4f431:

curl --location --request PUT "https://ORGANIZATION_NAME.REGION.CLOUD_PROVIDER.api.imply.io/v1/projects/PROJECT_ID/jobs/efb35e3e-406e-4127-ad2e-280fede4f431" \
--user ${POLARIS_API_KEY}: \
--header "Content-Type: application/json" \
--data-raw '{
    "desiredExecutionStatus": "canceled"
}'

Sample response​

When you successfully cancel an ingestion job, the Jobs v1 API returns the 200 OK status code and the details of the canceled job.

Learn more​

See the following topics for more information: