Skip to main content

Backfill events with S3 pull

AI summary
Explains how to use the S3 pull integration to ingest historical data from Amazon S3 into Imply Lumi. Covers creating backfill jobs through the UI or API. Describes job validation, object discovery, and filtering options to control ingestion.

About AI summaries.

Use the S3 pull integration to ingest historical data from an Amazon S3 bucket into Imply Lumi. The S3 pull integration is suitable for backfill ingestion, where you manually specify a predefined batch of objects to ingest.

You can ingest S3 data through the Lumi UI or the API.

This topic provides details to configure backfill ingestion using the S3 pull integration.

Prerequisites​

Before you continue, review Send events with S3 pull and ensure that you:

  • Met the prerequisites
  • Configured AWS access
  • Created a Lumi IAM key

How a backfill job works​

A backfill job follows these steps:

  1. You create a job to ingest data from S3.
  2. Lumi validates that it can access your objects.
    If you encounter permissions errors, see Configure AWS access.
  3. Lumi creates the backfill job in a Pending state and checks that it doesn't exceed more than 1,000,000 objects.
  4. Upon validation, Lumi transitions the job to Running, which includes the following stages:
    1. Discovery: Lumi identifies the objects to ingest.
    2. Processing: Lumi collects, processes, and retains the events.
  5. The job completes with one of the following statuses:
    • Succeeded: The ingestion completed successfully.
    • Failed: Lumi terminated the job due to an error during discovery or processing. See Errors.
    • Canceled: A user canceled the job.

Before creating backfill jobs, note the following constraints and behavior:

  • Limit your job to a maximum of 1,000,000 objects. If you exceed this amount, Lumi doesn't proceed with ingestion. Create multiple smaller jobs, or refine your filter to reduce the size of your job. For details, see Reduce job scope.
  • It can take time for a job to begin, depending on the volume of discovered objects and any backlog of existing backfill requests.
  • Avoid creating backfill jobs with the same settings. This can lead to duplicate events.
  • Lumi assigns the user attribute filename and the system attribute correlationId to each event from a backfill job. Events from the same job have the same correlation ID. For more information, see S3 pull attributes.

Create a job in the UI​

Create a backfill job using the Lumi UI:

  1. From the Lumi navigation menu, click Integrations > S3 pull.

  2. In Select the job type, click Backfill.

  3. Select your AWS access method and your Lumi IAM key.

  4. In Create a new job, enter the following details. The job requires the bucket name and object filter; all other fields are optional.

    • Bucket name: Name of the S3 bucket containing the data.

    • Object filter: Glob pattern for object keys that defines which objects to include when ingesting data. The pattern must match the entire object key. See S3 object filters for examples.

      info

      As a best practice, use the most specific prefix that matches your objects. This can help speed up discovery time and avoid reaching the maximum object limit.

    • Region: AWS region of your S3 bucket. By default, it assumes the same region as the Lumi environment.

    • Modified after: Start date in ISO 8601 format. Only include objects that were created or modified after this date.

    • Modified before: End date in ISO 8601 format. Only include objects that were created or modified before this date.

    • Format: Format of the events in the objects. By default, the job inherits the event format from the IAM key. If the IAM key doesn't specify one, Lumi automatically detects the event format.

    Backfill job

  5. Click Start job. Note that ingestion might not begin immediately.

  6. In Preview incoming data, view the events coming into Lumi. Lumi automatically refreshes the preview pane to display the latest events.

  7. Click Explore events to see more events associated with the IAM key. Adjust the time filter to choose the range of data displayed.

Create a job by API​

You can use the S3 pull API to ingest event data from an S3 bucket on demand.

To start a backfill job, send a POST request to the /ingest endpoint. Replace S3_PULL_BACKFILL_ENDPOINT with your Lumi endpoint URL. You can find this URL in the Authentication and access pane of the S3 pull backfill integration in the UI.

curl --location 'S3_PULL_BACKFILL_ENDPOINT' \
--header 'Content-Type: application/json' \
--data '{
    "bucket": "BUCKET_NAME"
}'

Include the S3 bucket name in the request body. All other body parameters are optional.

View request body parameters

  • bucket

    • Type: string
    • S3 bucket name containing the data to backfill.

  • pattern

    • Type: string
    • Pattern that defines which objects to include when ingesting data. By default, Lumi ingests all objects in the S3 bucket. See S3 object filters for details.

  • region

    • Type: string
    • AWS region of your S3 bucket. For example, us-east-1. By default, Lumi assumes the same region as the Lumi environment.

  • modifiedAfter

    • Type: string
    • Start date in ISO 8601 format. Only include objects that were modified after this date and time. For example, 2025-01-01T00:00:00Z.

  • modifiedBefore

    • Type: string
    • End date in ISO 8601 format. Only include objects that were modified before this date and time. For example, 2025-01-31T23:59:59Z.

  • format

    • Type: string enum with options csv, json, splunk_csv, splunk_hec, plain
    • Format of events in the S3 objects. By default, Lumi uses the format on the IAM key else auto-detects the format.

  • formatOptions

    • Type: object

    • Options for custom CSV format. Only applies when format is csv. Schema:

      "formatOptions": {
        "type": "csv",
        "delimiter": <string field delimiter>,
        "headers": <array of header names>,
        "skipHeaderRows": <integer number of header rows to discard>
      }

      For example:

      "formatOptions": {
        "type": "csv",
        "delimiter": "\t",
        "headers": ["timestamp", "message", "host"],
        "skipHeaderRows": 1
      }

Sample request​

The following example shows how to ingest data from the example_logs.bz2 file in the example-bucket S3 bucket:

curl --location 'https://60252ae2-c123-4d56-b78f-910112bef518@s3-pull.us1.api.lumi.imply.io/ingest' \
--header 'Content-Type: application/json' \
--data '{
    "bucket": "example-bucket",
    "pattern": "logs/example_logs.bz2",
    "region": "us-east-1",
    "modifiedAfter": "2025-01-01T00:00:00Z"
}'

A successful request returns an HTTP 200 OK message code and the task ID (taskId) in the response body.

View job status​

If you use the Lumi UI to submit a job, Lumi displays the status of the job. Click View and manage job for more details on the Jobs page. There, you can see the progress of discovery and processing for the job. You can also cancel the job and view past jobs. For more details, see View and manage jobs.

Backfill go to job

Learn more​

See the following topics for more information: