Ingestion status reference

You can view and monitor the status of an ingestion job in Imply Polaris by checking its overall execution status as well as the health of the job. Polaris displays the job health based on the status of row processing during ingestion.

This topic lists the health statuses that may be associated for an ingestion job.

Job health reference

The following table lists the row ingestion statuses that may be associated for an ingestion job.

Polaris UI label	Polaris API field	Druid field
Successful	numRowsProcessed	processed
Processed with warnings	numRowsProcessedWithWarnings	processedWithErrors
Filtered	numRowsSkippedByFilter	thrownAway
Unparseable	numRowsSkippedByError	unparseable

The row ingestion statuses are described as follows:

• Successful: Number of rows successfully ingested without parsing errors.
• Processed with warnings: Ingested rows that have one or more parsing errors. This typically occurs when Polaris can parse the input row but detects an invalid type, such as a string value ingested into a numeric column.
• Filtered: Number of rows skipped from a streaming ingestion job. Polaris skips rows when you specify an ingestion target interval or a SQL expression to filter the source data. Skipped rows include late arriving event data for streaming ingestion. This number does not include any header rows skipped in CSV data nor data ingested outside the system-wide time limits.

  The filtered row count only applies to streaming ingestion. Batch ingestion jobs do not track filtered rows.

• Unparseable: Number of rows discarded because they could not be parsed.

See the Druid documentation on Task reference for details on row processing fields exposed from Apache Druid.

View job health

Use either the UI or API to check the details of your ingestion job.

In the UI, Polaris shows the job status on the Job details page:

In the API, Polaris returns the row ingestion statuses in the totals field of a GET request for the job's metrics. For example, GET /v2/jobs/db6a3110-d6c3-4a63-86b2-41d51a65ce11/metrics returns the following:

{
    "totals": {
        "numRowsProcessed": 99999,
        "numRowsProcessedWithWarning": 0,
        "numRowsSkippedByFilter": 0,
        "numRowsSkippedByError": 2
    }
}

For how to view job metrics and limit the number of unparseable rows allowed per job, see View metrics for a job.

Learn more

See the following topics for more information:

• Create an ingestion job for creating and viewing ingestion jobs.
• Metrics reference for more metrics you can track for streaming ingestion.