Send events with S3 pull

You can load data into Imply Lumi from an Amazon Simple Storage Service (Amazon S3) bucket using the S3 pull integration. With S3 pull, Lumi automatically ingests new objects from your S3 bucket, treating each line in an object as a separate event.

This topic provides details on loading data into Lumi from Amazon S3. For other approaches to sending events, see Send events to Lumi.

info

The instructions in this topic reflect Amazon Web Services (AWS) configurations as of September 2025.

General setup

The general setup for ingesting objects from Amazon S3 into Lumi is as follows:

1. Set up the necessary AWS permissions and identity, including a permissions policy and an Identity and Access Management (IAM) role.
2. Configure a Lumi IAM key using the ARN of the IAM role you created. As part of this process, Lumi generates an endpoint to receive notifications about new objects.
3. Create an Amazon Simple Notification Service (Amazon SNS) topic and subscribe the Lumi endpoint to it.
4. Configure an S3 event notification to publish messages about new objects to the SNS topic.

When a new object is added to the S3 bucket, Amazon S3 generates an event notification and publishes it to the SNS topic. Amazon SNS then delivers the notification to the Lumi endpoint. Lumi processes the notification and retrieves the event data from Amazon S3.

The following diagram illustrates how AWS services work together to send events from an S3 bucket into Lumi:

Prerequisites

To send events from an S3 bucket into Lumi, you need the following:

• Access to Lumi with the Data manager role or higher. For information on roles and permissions, see Manage roles.
• Amazon Resource Name (ARN) of the Lumi IAM role. To obtain the Lumi ARN, contact Lumi customer support.
• Your tenant ID in Lumi. To obtain the tenant ID, contact Lumi customer support.
• Lumi IAM key. See Create an IAM key for details.
• Amazon IAM permissions to create roles, create policies, and attach policies to roles. For more information, see Allow users and groups to create and modify roles in the AWS documentation.
• Amazon S3 bucket, including the ARN of the bucket. For information on how to find the ARN of an S3 bucket, see Find ARNs in the AWS documentation.

Configure AWS IAM access for Lumi

This section describes how to configure AWS IAM access for Lumi. First, you create a permissions policy to grant access to the S3 bucket. Then, you create an IAM role and a trust policy to allow Lumi to assume that role.

Create an AWS permissions policy

Create a permissions policy that grants access to the S3 bucket:

1. In the AWS console, go to the IAM dashboard.

2. Click Policies in the left sidebar, then click Create policy.

3. In the Policy editor section, click JSON.

4. Replace the contents in the policy editor with the following policy. Replace both instances of S3 ARN with the ARN of your S3 bucket.

   {
      "Version": "2012-10-17",
      "Statement": [
         {
            "Effect": "Allow",
            "Action": [
               "s3:ListBucket"
            ],
            "Resource": [
               "S3 ARN"
            ]
         },
         {
            "Effect": "Allow",
            "Action": [
               "s3:GetObject"
            ],
            "Resource": [
               "S3 ARN/*"
            ]
         }
      ]
   }
   

   The policy editor should resemble the following:

   

5. Optionally, click Actions and select Optimize for readability to format the JSON.

6. Click Next.

7. Enter a name for the policy, then click Create policy.

Create an AWS IAM role

Create an IAM role with a trust policy that lets Lumi assume that role:

1. In the AWS console, go to the IAM dashboard.

2. Click Roles in the left sidebar, then click Create role.

3. In the Trusted entity type section, select Custom trust policy.

4. Replace the contents in the policy editor with the following policy.
   Replace LUMI ARN with the provided Lumi ARN.
   Replace TENANT ID with your tenant ID in Lumi.

   {
      "Version": "2012-10-17",
      "Statement": [
         {
            "Sid": "Statement1",
            "Effect": "Allow",
            "Principal": {
               "AWS": "LUMI ARN"
            },
            "Action": "sts:AssumeRole",
            "Condition": {
            "StringEquals": {
               "sts:ExternalId": "TENANT ID"
               }
            }
         }
      ]
   }

   The policy editor should resemble the following:

   

5. Click Next.

6. Search for and select the permissions policy you created in the previous step.

7. Click Next.

8. Enter a name for the role, then click Create role.

9. Click the role to view its details. Save the ARN of the role. You'll need it in a later step.

Configure event forwarding

This section explains how to configure event forwarding from Amazon S3 to Lumi.

First, you configure an IAM key in Lumi using your AWS role ARN. You then create an SNS topic and subscribe Lumi to it. Finally, you configure your S3 bucket to publish event notifications to the SNS topic.

Configure an IAM key

Configure an IAM key to generate an S3 pull endpoint in Lumi:

1. From the Lumi navigation menu, click Integrations > S3 pull.
2. Select or create an IAM key.
3. For a new key, enter a name, global attributes, and the ARN of the AWS IAM role you created in the previous step. For an existing key, update it with the ARN of the AWS IAM role.
4. Save your changes.
5. The page populates with the information required to create a new subscription. Save the endpoint URL. You'll need it in a later step.

Create an SNS topic

An SNS topic acts as a communication channel for publishing messages that you can then broadcast to different subscribers.

Create an SNS topic:

1. In the AWS console, go to the SNS dashboard.
2. Click Topics in the left sidebar, then click Create topic.
3. Select Standard for Type.
4. Enter a name for the topic.
5. Click Create topic. Save the ARN of the topic. You'll need it in a later step.

Subscribe to the SNS topic

Create a subscription to link the SNS topic to the S3 pull endpoint in Lumi:

1. In the AWS console, go to the SNS dashboard.
2. Click Subscriptions in the left sidebar, then click Create subscription.
3. Enter the following details:
   • Topic ARN: Select the SNS topic you created previously.
   • Protocol: HTTPS
   • Endpoint: Lumi S3 pull endpoint
4. Click Create subscription.

Create an event notification

Create an event notification for your S3 bucket to trigger an event when a new object is created:

1. In the AWS console, go to Amazon S3.
2. Select your S3 bucket.
3. Go to the Properties tab.
4. In the Event notifications section, click Create event notification.
5. Enter a name for the event.
6. In the Event types section, select All object create events (s3:ObjectCreated:*).
7. In the Destination section, select SNS topic.
8. Select the SNS topic you created in the previous step.
9. Click Save changes.

Check Lumi for events

After you configure event forwarding and add objects to the S3 bucket, you can preview the events in Lumi:

1. From the Lumi navigation menu, click Integrations > S3 pull.
2. In Select or create an IAM key, select your key.
3. In Preview incoming data, view the events coming into Lumi. Lumi automatically refreshes the preview pane to display the latest events.
4. Click Explore events to see more events associated with the IAM key. Adjust the time filter to choose the range of data displayed.

Once events start flowing into Lumi, you can search them. See Search events with Lumi for details and information on supported search syntax.

Learn more

See the following topics for more information:

• Send events to Lumi for other options to send events.
• IAM keys reference for details on IAM keys.
• Transform events with pipelines for information on how to transform events in Lumi.