Skip to main content

Set up recurring ingestion with S3 pull

You can use the S3 pull integration to configure recurring ingestion from Amazon S3 to Imply Lumi. With recurring ingestion, Lumi listens for event notifications from an S3 bucket and automatically retrieves and ingests events from new objects as they are created.

This topic provides details on configuring recurring ingestion using the S3 pull integration. It assumes you have already reviewed the prerequisites, configured AWS access, and created a Lumi IAM key.

info

The instructions in this topic reflect AWS configurations as of October 2025.

Workflow​

The following workflow describes how recurring ingestion works between Amazon S3 and Lumi:

  1. Complete the steps in Send events with S3 pull.
  2. Create an Amazon Simple Notification Service (Amazon SNS) topic and subscribe the Lumi endpoint to it.
  3. Configure an S3 event notification to publish messages about new objects to the SNS topic.
  4. When a new object is added to the S3 bucket, Amazon S3 generates an event notification and publishes it to the SNS topic.
  5. Amazon SNS delivers the notification to the Lumi endpoint.
  6. Lumi processes the notification and retrieves the event data from Amazon S3 for ingestion.

The following diagram shows how AWS services and Lumi interact to send events through recurring ingestion:

S3 pull recurring integration diagram

Create an SNS topic and subscription​

An SNS topic acts as a communication channel for publishing messages that you can then broadcast to different subscribers.

Obtain the Lumi endpoint URL:

  1. From the Lumi navigation menu, click Integrations > S3 pull.
  2. In Select type of job you're doing today, select Recurring.
  3. Select your IAM key.
  4. The page populates with the information required to create an SNS subscription. Save the endpoint URL. Lumi endpoint URL

Create an SNS topic in AWS and subscribe the Lumi endpoint to it:

  1. In the AWS console, go to the SNS dashboard.
  2. Click Topics in the left sidebar, then click Create topic.
  3. Select Standard for Type.
  4. Enter a name for the topic.
  5. Click Create topic. Save the ARN of the topic. You'll need it in a later step.
  6. Click Subscriptions in the left sidebar, then click Create subscription.
  7. Enter the following details:
    • Topic ARN: Select the SNS topic you created.
    • Protocol: HTTPS
    • Endpoint: Lumi endpoint URL resembling https://IAM_KEY_TOKEN@s3-pull.us1.api.lumi.imply.io/notifications.
  8. Click Create subscription.

Create an event notification​

Create an event notification for your S3 bucket to trigger a notification when a new object is created:

  1. In the AWS console, go to Amazon S3.
  2. Select your S3 bucket.
  3. Go to the Properties tab.
  4. In Event notifications, click Create event notification.
  5. In General configuration, enter a name for the event.
  6. Optionally, specify a prefix and a suffix to only send notifications for desired objects:
  7. In Event types, select All object create events (s3:ObjectCreated:*).
  8. In Destination, select SNS topic.
  9. Select the SNS topic you created previously. AWS S3 event notification destination
  10. Click Save changes.

Check Lumi for events​

After you add objects to the S3 bucket, you can preview the incoming data in Lumi:

  1. From the Lumi navigation menu, click Integrations > S3 pull.
  2. In Select type of job you're doing today, select Recurring.
  3. In Select or create an IAM key, select your key.
  4. In Preview incoming data, view the events coming into Lumi. Lumi automatically refreshes the preview pane to display the latest events.
  5. Click Explore events to see more events associated with the IAM key. Adjust the time filter to choose the range of data displayed.

Once events start flowing into Lumi, you can search them. See Search events with Lumi for details and information on supported search syntax.

Learn more​

See the following topics for more information: