Skip to main content

Manage S3 jobs

AI summary
Explains how to monitor and manage S3 backfill jobs in Imply Lumi. Covers viewing job status, discovery and processing phases, and handling errors.

About AI summaries.

Use the S3 backfill integration to load events into Imply Lumi from batch files stored in AWS S3. For example, you might have a large set of files with historical data. After you create a backfill job, you can monitor its progress and metrics from the Jobs page. This topic covers how to use the Jobs page to view and manage your backfill jobs.

Jobs overview​

To view all jobs, click Jobs from the Lumi navigation menu. On this page, you can search and sort by the job creation date, status, and associated IAM key. To refresh the information on this page, click the refresh icon in the top right.

Jobs page

View job details​

From the Jobs page, you can click a job to see more details. The top of the job details pane shows when the job was created, the associated Lumi IAM key, and the unique identifier for the job.

The job details pane also shows details about the discovery and processing phases of a backfill job.

Job processing

Discovery​

A backfill job that starts running enters the discovery phase first, where Lumi identifies what objects to ingest and how many there are. Open the Discovery expander to view the following job details:

  • Objects discovered
  • Total size (bytes)
  • Created at
  • Last updated at

If a job is taking a long time in discovery, you might need to recreate the job with a more specific object filter or break it down into multiple smaller jobs.

Processing​

After a backfill job passes the discovery phase, it enters the processing phase. In this phase, Lumi begins ingesting events from the objects discovered. Open the Processing expander to view the following job details:

  • Objects processed
  • Objects failed
  • Events ingested
  • Processed size (bytes)
  • Started at
  • Last updated at

If a backfill job is canceled while in the discovery phase, the Processing section doesn't display when the job was started or last updated.

Spec​

To review the configuration you submitted for a backfill job, click the Spec tab. Ensure that you don't submit the same configuration multiple times as that can lead to duplicate events.

Job spec

Errors​

If an error occurs, Lumi displays it in the stage it was raised, Discovery or Processing. A backfill job halts when it enters an error state. Any ingested events remain ingested.

Backfill error

Note that permissions errors can also arise when you submit a backfill job, such as when Lumi detects an incorrect ARN on your IAM key or doesn't have the AWS permissions to access your objects. If this occurs, Lumi doesn't create the job.

Cancel a job​

You can cancel a job that's in a Running or Pending state. If you cancel a backfill job that was already running, any ingested events remain ingested.

To cancel a job, click the ellipsis and select Cancel job. Confirm the action in the dialog.

Backfill cancel

Learn more​

See the following topics for more information: