Skip to main content

Backfill events with S3 pull

You can use the S3 pull integration to ingest historical data from an Amazon S3 bucket into Imply Lumi. The process, often called backfill ingestion, involves manually specifying which objects to ingest. You provide an S3 bucket with optional prefix and suffix filters to limit what gets ingested. Lumi ingests events from the objects matching the specified filters.

You can ingest S3 data through the Lumi UI or the API.

The following diagram shows how AWS services and Lumi interact to send events through backfill ingestion:

Lumi S3 pull backfill

This topic provides details to configure backfill ingestion using the S3 pull integration. It assumes you have already completed all the steps in Send events with S3 pull.

Create an ingestion job in the UI​

Create a backfill ingestion job using the Lumi UI:

  1. From the Lumi navigation menu, click Integrations > S3 pull.
  2. In Select type of job you're doing today, click Backfill.
  3. Select your IAM key.
  4. In Create a new job, enter the following details:
    • Bucket name: Name of the S3 bucket containing the data.
    • Object filter: Glob pattern for object keys that defines which objects to include when ingesting data. The pattern must match the entire object key. See S3 object filters for details.
    • Region (optional): Region of your S3 bucket. By default, it assumes the same region as the Lumi environment.
    • Modified after (optional): Start date in ISO 8601 format. Only include objects that were created or modified after this date.
    • Modified before (optional): End date in ISO 8601 format. Only include objects that were created or modified before this date.
  5. Click Start job.
  6. In Preview incoming data, view the events coming into Lumi. Lumi automatically refreshes the preview pane to display the latest events.
  7. Click Explore events to see more events associated with the IAM key. Adjust the time filter to choose the range of data displayed.

Create an ingestion job by API​

You can use the S3 pull API to ingest event data from an S3 bucket on demand.

To start an ingestion job, send a POST request to the /ingest endpoint. Replace IAM_KEY_TOKEN and REGION with your Lumi IAM key token and region, respectively.

curl --location 'https://IAM_KEY_TOKEN@s3-pull.REGION.api.lumi.imply.io/ingest' \
--header 'Content-Type: application/json' \
--data '{
    "bucket": BUCKET_NAME
}'

Include the S3 bucket name in the request body. All other body parameters are optional.

View body parameters

  • bucket

    • Type: string
    • S3 bucket name containing the data to backfill.

  • pattern

    • Type: string
    • Pattern that defines which files to include when ingesting data. By default, Lumi ingests all objects in the S3 bucket.

  • region

    • Type: string
    • Region of your S3 bucket. By default, Lumi assumes the same region as the Lumi environment.

  • modifiedAfter

    • Type: string
    • Start date in ISO 8601 format. Only include objects that were modified after this date and time. For example, 2025-01-01T00:00:00Z.

  • modifiedBefore

    • Type: string
    • End date in ISO 8601 format. Only include objects that were modified before this date and time. For example, 2025-01-31T23:59:59Z.

Sample request​

The following example shows how to ingest data from the example_logs.bz2 file in the example-bucket S3 bucket:

curl --location 'https://60252ae2-c123-4d56-b78f-910112bef518@s3-pull.us1.api.lumi.imply.io/ingest' \
--header 'Content-Type: application/json' \
--data '{
    "bucket": "example-bucket",
    "pattern": "logs/example_logs.bz2",
    "region":"us1",
    "modifiedAfter":"2025-01-01T00:00:00Z"
}'

A successful request returns an HTTP 200 OK message code and an empty response body.

Once events start flowing into Lumi, you can search them. See Search events with Lumi for details and information on supported search syntax.

S3 object filters​

You can define filters for S3 object keys to control which objects to ingest.

The following table lists the glob patterns supported by Lumi:

PatternDescriptionExample
**Matches zero or more path segmentslogs/**, **/*.json
*Matches any characters except path separatorslogs/2025-10-*.json
?Matches exactly one character except path separatorslogs/demo-logs-?.json
[abc]Matches any character in the setlogs/[aeu]*_logs.*
[a-z]Matches any character in the rangelogs/[a-z]*_logs.*
[!abc]Matches any character not in the set (negation)logs/[!aeu]*_logs.*
{a,b,c}Matches any of the alternatives (brace expansion)logs/*.{bz2,gz}

Learn more​

See the following topics for more information: