Skip to main content

Manage ingestion jobs

After you create an ingestion job in Imply Polaris, you can monitor its progress and details or cancel its execution. You can use past ingestion jobs as the basis for new jobs by editing and resubmitting the job spec directly in the UI.

View ingestion jobs​

Access the Jobs page from the left sidebar in the UI to view all jobs for a project. There, you can monitor a job's progress and search for a specific job. You can also sort jobs by type, source of data being ingested, job status, or table name.

Under certain conditions, Polaris cleans up the history for batch ingestion jobs, including SQL-based ingestion jobs. When a project exceeds 5000 finished batch ingestion jobs, Polaris deletes completed batch ingestion jobs older than 90 days. If there are fewer than 5000 finished batch ingestion jobs, Polaris retains all job history regardless of job age.

The following screenshot shows the Jobs page:

Ingestion jobs page

Job details​

Select a job from the Jobs page to see more details about the job. The Job details page includes the source and target information with other metadata about the job.

Here, you can also view the status of the rows processed during ingestion. For reference on ingestion job statuses, see Ingestion status reference.

Polaris displays the job logs in the Logs pane on this page. The logs contain parse exceptions for rows that cannot be parsed. Parse exceptions may result due to incompatible data types. Note that only streaming jobs generate logs for parse exceptions.

The following screenshot shows an example Job details page:

Job details page for batch ingestion

Streaming ingestion jobs also display graphs for latency and throughput. Latency represents the lag from a message being produced to when it's consumed by Polaris. Latency is displayed in terms of number of offsets for Kafka-based connections, including Kafka, Confluent Cloud, and push streaming sources. Latency for Kinesis ingestion is displayed in milliseconds.

The throughput represents the number of events processed per hour.

Job details page for streaming ingestion

To monitor an ingestion job using the API, see Monitor ingestion job progress.

Job spec​

Select the Job spec tab to view the JSON spec that describes the job. You can copy and modify the spec to use as the request body in your API requests. Refer to the Jobs API documentation for a description of the parameters.

Job spec page

Cancel an ingestion job​

To cancel an ingestion job:

  1. Navigate to the Jobs page from the left navigation pane.
  2. Open the job you want to cancel.
  3. Click Stop job.
  4. Click Stop job to confirm your choice.

For canceled batch ingestion jobs, Polaris does not ingest any data into your table. A canceled streaming ingestion job may already have data ingested.

To cancel an ingestion job using the API, see Cancel an ingestion job.

Recreate from past ingestion job​

After a job completes, you can use it as the basis to create a new ingestion job. For example, you may want to keep most of the details of the ingestion job spec but only change the file you ingest from or the table you ingest to. A recreated job is separate from the original job and has its own job ID.

To create a job based on a past ingestion job:

  1. Navigate to the Jobs page from the left navigation pane.
  2. Open the job you want to recreate.
  3. Select Recreate job. Polaris opens a code editor dialog and displays the JSON job spec associated with the completed job.
  4. Make any desired changes, and confirm the rest of the details.
  5. Click Recreate job to submit the new job.

Learn more​

See the following topics for more information: