Skip to main content

Backfill events with S3 pull

You can use the S3 pull integration to ingest historical data from an Amazon S3 bucket into Imply Lumi. The process, often called backfill ingestion, involves manually specifying which objects to ingest. You provide an S3 bucket with optional prefix and suffix filters to limit what gets ingested. Lumi ingests events from the objects matching the specified filters.

This topic provides details on configuring backfill ingestion using the S3 pull integration. It assumes you have already reviewed the prerequisites, configured AWS access, and created a Lumi IAM key.

The following diagram shows how AWS services and Lumi interact to send events through backfill ingestion:

Lumi S3 pull backfill
info

The instructions in this topic reflect AWS configurations as of October 2025.

Create an ingestion job​

Create a new backfill ingestion job in Lumi:

  1. From the Lumi navigation menu, click Integrations > S3 pull.
  2. In Select type of job you're doing today, click Backfill.
  3. Select your IAM key.
  4. In Create a new job, enter the following details:
    • Bucket name: Name of the S3 bucket.
    • Object filter: Pattern that defines which files to include when ingesting data. For example: *.json.
    • Region (optional): Region of your S3 bucket. By default, it assumes the same region as the Lumi environment.
    • Modified before (optional): Only include objects that were modified before this date and time.
    • Modified after (optional): Only include objects that were modified after this date and time.
  5. Click Start job.
  6. In Preview incoming data, view the events coming into Lumi. Lumi automatically refreshes the preview pane to display the latest events.
  7. Click Explore events to see more events associated with the IAM key. Adjust the time filter to choose the range of data displayed.

Once events start flowing into Lumi, you can search them. See Search events with Lumi for details and information on supported search syntax.

Learn more​

See the following topics for more information: