Send events with S3 pull

You can load data into Imply Lumi from an Amazon Simple Storage Service (Amazon S3) bucket using the S3 pull integration. With S3 pull, Lumi listens for event notifications from an Amazon S3 bucket and automatically ingests events from new objects as they are created.

This topic provides details on sending events from an Amazon S3 bucket to Lumi. For other approaches to sending events, see Send events to Lumi.

info

The instructions in this topic reflect Amazon Web Services (AWS) configurations as of September 2025.

General setup

The general setup for ingesting objects from Amazon S3 to Lumi is as follows:

1. Set up the necessary AWS permissions and identity, including a permissions policy and an Identity and Access Management (IAM) role.
2. Configure a Lumi IAM key using the Amazon Resource Name (ARN) of the IAM role you created. As part of this process, Lumi provides an endpoint to receive notifications about new objects.
3. Create an Amazon Simple Notification Service (Amazon SNS) topic and subscribe the Lumi endpoint to it.
4. Configure an S3 event notification to publish messages about new objects to the SNS topic.

When a new object is added to the S3 bucket, Amazon S3 generates an event notification and publishes it to the SNS topic. Amazon SNS then delivers the notification to the Lumi endpoint. Lumi processes the notification and retrieves the event data from Amazon S3.

The following diagram illustrates how AWS services work together to send events from an S3 bucket to Lumi:

Prerequisites

To send events from an S3 bucket to Lumi, you need the following:

• Access to Lumi with the Data manager role or higher. For information on roles and permissions, see Manage roles.
• Amazon IAM permissions to create roles, create policies, and attach policies to roles. For more information, see Allow users and groups to create and modify roles in the AWS documentation.
• Amazon S3 bucket, including the ARN of the bucket. For information on how to find the ARN of an S3 bucket, see Find ARNs in the AWS documentation.

Configure AWS IAM access for Lumi

This section describes how to configure AWS IAM access for Lumi. First, you create a permissions policy to grant access to the S3 bucket. Then, you create an IAM role and a trust policy to allow Lumi to assume that role.

Create an AWS permissions policy

Create a permissions policy that grants access to the S3 bucket:

1. In the AWS console, go to the IAM dashboard.

2. Click Policies in the left sidebar, then click Create policy.

3. In the Policy editor section, click JSON.

4. Replace the contents in the policy editor with the following policy. Replace both instances of S3 ARN with the ARN of your S3 bucket.

   {
     "Version": "2012-10-17",
     "Statement": [
       {
         "Effect": "Allow",
         "Action": [
           "s3:ListBucket"
         ],
         "Resource": [
           "S3 ARN"
         ]
       },
       {
         "Effect": "Allow",
         "Action": [
           "s3:GetObject"
         ],
         "Resource": [
           "S3 ARN/*"
         ]
       }
     ]
   }

   The policy editor should resemble the following:

   

5. Optionally, click Actions and select Optimize for readability to format the JSON.

6. Click Next.

7. Enter a name for the policy, then click Create policy.

Create an AWS IAM role

Lumi provides a trust policy template that allows Lumi to assume an IAM role in your AWS account.

Start by obtaining the trust policy from Lumi:

1. From the Lumi navigation menu, click Integrations > S3 pull.
2. Under Configure AWS access: Create an AWS IAM role, copy the trust policy. You'll use it to create an IAM role in AWS.

Create an IAM role with the trust policy:

1. In the AWS console, go to the IAM dashboard.

2. Click Roles in the left sidebar, then click Create role.

3. In the Trusted entity type section, select Custom trust policy.

4. Replace the contents in the policy editor with the trust policy you copied from Lumi.

   The policy editor should resemble the following:

   

5. Click Next.

6. Search for and select the permissions policy you created in the previous step.

7. Click Next.

8. Enter a name for the role, then click Create role.

9. Click the role to view its details. Save the ARN of the role. You'll need it in a later step.

Configure event forwarding

This section explains how to configure event forwarding from Amazon S3 to Lumi.

First, you configure an IAM key in Lumi using your AWS IAM role ARN. You then create an SNS topic and subscribe Lumi to it. Finally, you configure your S3 bucket to publish event notifications to the SNS topic.

Configure an IAM key

Configure a Lumi IAM key using your AWS IAM role ARN:

1. From the Lumi navigation menu, click Integrations > S3 pull.
2. Select or create an IAM key.
3. For a new key, enter a name, global attributes, and the ARN of the AWS IAM role you created in the previous step. For an existing key, update it with the ARN of the AWS IAM role.
4. Save your changes.
5. The page populates with the information required to create a new subscription. Save the endpoint URL. You'll need it in a later step.

Create an SNS topic and subscription

An SNS topic acts as a communication channel for publishing messages that you can then broadcast to different subscribers.

Create an SNS topic in AWS and subscribe the Lumi endpoint to it:

1. In the AWS console, go to the SNS dashboard.
2. Click Topics in the left sidebar, then click Create topic.
3. Select Standard for Type.
4. Enter a name for the topic.
5. Click Create topic. Save the ARN of the topic. You'll need it in a later step.
6. Click Subscriptions in the left sidebar, then click Create subscription.
7. Enter the following details:
   • Topic ARN: Select the SNS topic you created.
   • Protocol: HTTPS
   • Endpoint: Lumi endpoint URL provided in the previous step.
8. Click Create subscription.

Create an event notification

Create an event notification for your S3 bucket to trigger a notification when a new object is created:

1. In the AWS console, go to Amazon S3.
2. Select your S3 bucket.
3. Go to the Properties tab.
4. In the Event notifications section, click Create event notification.
5. Enter a name for the event.
6. In the Event types section, select All object create events (s3:ObjectCreated:*).
7. In the Destination section, select SNS topic.
8. Select the SNS topic you created previously.
9. Click Save changes.

Check Lumi for events

After you configure event forwarding and add objects to the S3 bucket, you can preview the events in Lumi:

1. From the Lumi navigation menu, click Integrations > S3 pull.
2. In Select or create an IAM key, select your key.
3. In Preview incoming data, view the events coming into Lumi. Lumi automatically refreshes the preview pane to display the latest events.
4. Click Explore events to see more events associated with the IAM key. Adjust the time filter to choose the range of data displayed.

Once events start flowing into Lumi, you can search them. See Search events with Lumi for details and information on supported search syntax.

Learn more

See the following topics for more information:

• Send events to Lumi for other options to send events.
• IAM keys reference for details on IAM keys.
• Create pipelines to transform events for information on how to transform events in Lumi.