View and manage ingestion jobs by API

After you create an ingestion job for Imply Polaris, you can monitor its progress, view job-related metrics, and cancel its execution. This topic walks you through the process of viewing job-related details and canceling an ingestion job.

Prerequisites

This topic assumes that you have an API key with the ManageIngestionJobs permission. In the examples below, the key value is stored in the variable named POLARIS_API_KEY. See API key authentication to obtain an API key and assign permissions. Visit Permissions reference for more information on permissions.

Monitor ingestion job progress

To monitor the progress of your ingestion job, issue a GET request to the Jobs v2 API with the job ID in the path. For example, /v2/jobs/efb35e3e-406e-4127-ad2e-280fede4f431.

Sample request

The following example shows how to monitor the progress of an example ingestion job with the job ID efb35e3e-406e-4127-ad2e-280fede4f431:

curl --location --request GET 'https://ORGANIZATION_NAME.REGION.CLOUD_PROVIDER.api.imply.io/v2/jobs/efb35e3e-406e-4127-ad2e-280fede4f431' \
--user ${POLARIS_API_KEY}: \
--header 'Content-Type: application/json' \
--data-raw ''
import os
import requests

url = "https://ORGANIZATION_NAME.REGION.CLOUD_PROVIDER.api.imply.io/v2/jobs/efb35e3e-406e-4127-ad2e-280fede4f431"

apikey = os.getenv("POLARIS_API_KEY")

payload = "\n"
headers = {
  'Authorization': f'Basic {apikey}',
  'Content-Type': 'application/json'
}

response = requests.request("GET", url, headers=headers, data=payload)

print(response.text)

Sample response

The following example shows a successful response for an ingestion job progress:

Click to view the response

{
    "type": "batch",
    "id": "efb35e3e-406e-4127-ad2e-280fede4f431",
    "target": {
        "type": "table",
        "tableName": "Koalas Subset"
    },
    "desiredExecutionStatus": "running",
    "createdBy": {
        "username": "service-account-docs-demo",
        "userId": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
    },
    "lastModifiedBy": {
        "username": "service-account-docs-demo",
        "userId": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
    },
    "executionStatus": "pending",
    "health": {
        "status": "ok"
    },
    "createdTimestamp": "2022-08-09T22:34:46.716017658Z",
    "lastUpdatedTimestamp": "2022-08-09T22:34:46.716017658Z",
    "source": {
        "type": "uploaded",
        "fileList": [
            "kttm-2019-08-19.json.gz",
            "kttm-2019-08-20.json.gz"
        ],
        "inputSchema": [
            {
                "dataType": "string",
                "name": "timestamp"
            },
            {
                "dataType": "string",
                "name": "city"
            },
            {
                "dataType": "string",
                "name": "session"
            },
            {
                "dataType": "long",
                "name": "session_length"
            }
        ],
        "formatSettings": {
            "format": "nd-json"
        }
    },
    "ingestionMode": "append",
    "mappings": [
        {
            "columnName": "__time",
            "expression": "TIME_PARSE(\"timestamp\")"
        },
        {
            "columnName": "city",
            "expression": "\"city\""
        },
        {
            "columnName": "session",
            "expression": "\"session\""
        },
        {
            "columnName": "max_session_length",
            "expression": "MAX(\"session_length\")"
        },
        {
            "columnName": "__count",
            "expression": "COUNT(*)"
        }
    ]
}

The executionStatus field shows the job's current execution status, and the health field describes the health of the job.

View metrics for a job

Send a GET request to /v2/jobs/JOB_ID/metrics to view the metrics for a job. Replace JOB_ID with the ID of the job.

Sample request

The following example shows how to view metrics for an ingestion job with the job ID db6a3110-d6c3-4a63-86b2-41d51a65ce11:

curl --location --request GET 'https://ORGANIZATION_NAME.REGION.CLOUD_PROVIDER.api.imply.io/v2/jobs/db6a3110-d6c3-4a63-86b2-41d51a65ce11/metrics' \
--user ${POLARIS_API_KEY}:
import os
import requests

url = "https://ORGANIZATION_NAME.REGION.CLOUD_PROVIDER.api.imply.io/v2/jobs/db6a3110-d6c3-4a63-86b2-41d51a65ce11/metrics"

apikey = os.getenv("POLARIS_API_KEY")

headers = {
  'Authorization': f'Basic {apikey}'
}

response = requests.get(url, headers=headers)

print(response.text)

Sample response

The following example shows a successful response for metrics of an ingestion job:

{
    "totals": {
        "numRowsProcessed": 99999,
        "numRowsProcessedWithWarning": 0,
        "numRowsSkippedByFilter": 0,
        "numRowsSkippedByError": 2
    }
}

The preceding example shows an ingestion job with two rows skipped due to parsing error. By default, Polaris continues to ingest data when it encounters rows it cannot parse. You can specify a threshold for parsing exceptions for a job. The job fails if the number of parsing exceptions exceeds the threshold. Set this limit in maxParseExceptions when creating an ingestion job.

You can view more details on the raised exceptions when requesting job logs.

Cancel an ingestion job

To cancel an ingestion job, issue a PUT request to the Jobs v2 API with the job ID in the path. In the request body, set the desiredExecutionStatus to canceled.

Sample request

The following example shows how to cancel an ingestion job with the job ID efb35e3e-406e-4127-ad2e-280fede4f431:

curl --location --request PUT 'https://ORGANIZATION_NAME.REGION.CLOUD_PROVIDER.api.imply.io/v2/jobs/efb35e3e-406e-4127-ad2e-280fede4f431' \
--user ${POLARIS_API_KEY}: \
--header 'Content-Type: application/json' \
--data-raw '{
    "desiredExecutionStatus": "canceled"
}'
import os
import requests
import json

url = "https://ORGANIZATION_NAME.REGION.CLOUD_PROVIDER.api.imply.io/v2/jobs/efb35e3e-406e-4127-ad2e-280fede4f431"

apikey = os.getenv("POLARIS_API_KEY")

payload = json.dumps({
  "desiredExecutionStatus": "canceled"
})
headers = {
  'Authorization': f'Basic {apikey}',
  'Content-Type': 'application/json'
}

response = requests.request("PUT", url, headers=headers, data=payload)

print(response.text)

Sample response

When you successfully cancel an ingestion job, the Jobs v2 API returns the 200 OK status code and the details of the canceled job.

Learn more

See the following topics for more information:

← Ingest using SQLIngest from files →