Skip to main content

Backfill events with S3 pull

AI summary
Explains how to use the S3 pull integration to ingest historical data from Amazon S3 into Imply Lumi. Covers creating and monitoring jobs through the Lumi UI.

About AI summaries.

Use the S3 pull integration to ingest historical data from an Amazon S3 bucket into Imply Lumi. The S3 pull integration is suitable for backfill ingestion, where you manually specify a predefined batch of objects to ingest.

This topic provides details to configure backfill ingestion using the S3 pull integration.

Prerequisites​

Before you continue, review Send events with S3 pull and ensure that you:

  • Met the prerequisites
  • Configured AWS access
  • Created a Lumi IAM key

How a backfill job works​

A backfill job follows these steps:

  1. You create a job to ingest data from S3.
  2. Lumi validates that it can access your objects.
    If you encounter permissions errors, see Configure AWS access.
  3. Lumi creates the backfill job in a Pending state and checks that it doesn't exceed more than 1,000,000 objects.
  4. Upon validation, Lumi transitions the job to Running, which includes the following stages:
    1. Discovery: Lumi identifies the objects to ingest.
    2. Processing: Lumi collects, processes, and retains the events.
  5. The job completes successfully, fails with an error, or is canceled by a user.

Before creating backfill jobs, note the following constraints and behavior:

  • Limit your job to a maximum of 1,000,000 objects. If you exceed this amount, Lumi doesn't proceed with ingestion. Create multiple smaller jobs, or refine your filter to reduce the size of your job. For details, see Reduce job scope.
  • It can take time for a job to begin, depending on the volume of discovered objects and any backlog of existing backfill requests.
  • Avoid creating backfill jobs with the same settings. This can lead to duplicate events.
  • Lumi assigns the user attribute filename and the system attribute correlationId to each event from a backfill job. Events from the same job have the same correlation ID. For more information, see S3 pull attributes.

Create a job​

Create a backfill job using the Lumi UI:

  1. From the Lumi navigation menu, click Integrations > S3 pull.

  2. In Select the job type, click Backfill.

  3. Select your AWS access method and your Lumi IAM key.

  4. In Create a new job, enter the following details. The job requires the bucket name and object filter; all other fields are optional.

    • Bucket name: Name of the S3 bucket containing the data.

    • Object filter: Glob pattern for object keys that defines which objects to include when ingesting data. The pattern must match the entire object key. See S3 object filters for examples.

      info

      As a best practice, use the most specific prefix that matches your objects. This can help speed up discovery time and avoid reaching the maximum object limit.

    • Region: AWS region of your S3 bucket. By default, it assumes the same region as the Lumi environment.

    • Modified after: Start date in ISO 8601 format. Only include objects that were created or modified after this date.

    • Modified before: End date in ISO 8601 format. Only include objects that were created or modified before this date.

    • Format override: Event format, such as CSV or JSON. By default, the job inherits the format selected on the IAM key. Otherwise, Lumi automatically detects the format.

    Backfill job

  5. Click Start job. Note that ingestion might not begin immediately.

  6. In Preview incoming data, view the events coming into Lumi. Lumi automatically refreshes the preview pane to display the latest events.

  7. Click Explore events to see more events associated with the IAM key. Adjust the time filter to choose the range of data displayed.

View job status​

Lumi displays the status of the job after you submit it. Click View and manage job for more details on the Jobs page. There, you can see the progress of discovery and processing for the job. You can also cancel the job and view past jobs. For more details, see View and manage jobs.

Backfill go to job

Learn more​

See the following topics for more information: