Skip to main content

Manage jobs

After you create a job in Imply Polaris, you can monitor its progress and details or stop its execution. You can use past jobs as the basis for new jobs by editing and resubmitting the job spec directly in the UI. This topic shows how to view, manage, and recreate ingestion jobs in Polaris.

View jobs​

Access the Jobs page from the left sidebar in the UI to view all jobs for a project. There, you can monitor a job's progress and search for a specific job. You can also sort jobs by type, source of data being ingested, job status, or table name.

Polaris categorizes jobs into the following types:

Under certain conditions, Polaris cleans up the history for batch ingestion jobs, including SQL-based ingestion jobs. When a project exceeds 5000 finished batch ingestion jobs, Polaris deletes completed batch ingestion jobs older than 90 days. If there are fewer than 5000 finished batch ingestion jobs, Polaris retains all job history regardless of job age.

The following screenshot shows the Jobs page:

Jobs page

Job details​

To get more information about a specific job including its ingestion spec and its metrics, navigate to the Jobs page from the left navigation pane, and select the job to get to the Job details page of the job.

The Job details page takes you to the Overview tab by default. This view shows you the data source, target table, job status, and other metadata. For ingestion jobs, you can also view the status of the rows processed during ingestion. For reference on ingestion job statuses, see Ingestion status reference.

Polaris displays the job logs in the Logs pane. For streaming jobs, the logs contain parse exceptions for rows that cannot be parsed during ingestion. Parse exceptions may result due to incompatible data types.

The following screenshot shows an example Job details page:

Job details page for batch ingestion

To monitor a job using the API, see View and manage jobs by API.

Job spec​

Select the Job spec tab to view the JSON spec that describes the job. You can copy and modify the spec to use as the request body in your API requests. Refer to the Jobs API documentation for a description of the parameters.

Job spec page

Monitor streaming jobs​

On the Job details page, the Overview tab presents additional information specific to streaming jobs. Use these dashboards to monitor the status of your ingestion job.

You can also run a SQL query against the table to check that Polaris is ingesting new data in the connection source.

Polaris displays the following metrics for streaming jobs:

  • Latency: The lag from a message being produced to when it's consumed by Polaris.
    • Latency is displayed in terms of number of offsets for Kafka-based connections, including Kafka, Confluent Cloud, and push streaming sources.
    • Latency for Kinesis ingestion is displayed in milliseconds.
  • Throughput: The number of events processed per hour.

Job details page for streaming ingestion

Navigate to Monitoring > Streaming to view dashboards that monitor the overall health of your streaming jobs including:

  • Average Kafka and Kinesis lag
  • Events processed
  • Rows output
  • Events with processing exceptions
  • Events filtered out

Events filtered out shows the number of events that were empty, outside the message rejection period, or explicitly filtered out in the job spec.

Streaming metrics

Stop a job​

When you stop a batch ingestion job, Polaris does not ingest any data into your table. When you stop a streaming ingestion job, Polaris retains the data that's already ingested.

You can't stop jobs to delete data or drop a table.

To stop a job:

  1. Navigate to the Jobs page from the left navigation pane.
  2. Open the job you want to stop.
  3. Click Manage then Cancel job for batch ingestion jobs or Terminate for streaming ingestion jobs.
  4. Confirm your choice.

To stop an ingestion job using the API, see Stop a job.

For streaming jobs, you can also pause and resume ingestion. For details, see Configure streaming ingestion.

Recreate a past job​

After a job completes, you can use it as the basis to create a new job. For example, you may want to keep most of the details of the ingestion job spec but only change the file you ingest from or the table you ingest to. A recreated job is separate from the original job and has its own job ID.

To create a job based on a past job:

  1. Navigate to the Jobs page from the left navigation pane.
  2. Open the job you want to recreate.
  3. Select Manage > Recreate job. Polaris opens a code editor dialog and displays the JSON job spec associated with the completed job.
  4. Make any desired changes, and confirm the rest of the details.
  5. Click Recreate job to submit the new job.

Learn more​

See the following topics for more information: