How to send events with S3 pull

In this tutorial, you learn how to send events from an Amazon Simple Storage Service (Amazon S3) bucket to Imply Lumi using S3 pull.

The steps show you how to:

• Set up the necessary AWS permissions and identity.
• Create an Amazon Simple Notification Service (Amazon SNS) topic and subscription.
• Configure an S3 event notification to publish messages to the SNS topic.
• Upload a sample log file to the S3 bucket.
• Preview the data and view events in Lumi.

To complete the steps, you use sample web traffic data from a fictional online store. For details about the dataset and its format, see the tutorial data overview.

The following diagram summarizes the end-to-end process of sending events to Lumi using S3 pull. Yellow shaded boxes represent steps taken within Lumi, and blue shaded boxes represent steps taken outside Lumi. Click any box in the diagram to jump to that step.

Prerequisites

To complete the tutorial, you need the following:

• Access to Lumi with the Data manager role or higher. For information on roles and permissions, see Manage roles.
• Amazon IAM permissions to create roles, create policies, and attach policies to roles. For more information, see Allow users and groups to create and modify roles in the AWS documentation.
• Amazon S3 bucket, including the ARN of the bucket. For information on how to find the ARN of an S3 bucket, see Find ARNs in the AWS documentation.

1. Create a permissions policy

In this section, you create a permissions policy that grants access to a specific S3 bucket.

1. In the AWS console, go to the IAM dashboard.

2. Click Policies in the left sidebar, then click Create policy.

3. In the Policy editor section, click JSON.

4. Replace the contents in the policy editor with the following policy. Replace both instances of S3 ARN with the ARN of your S3 bucket.

   {
     "Version": "2012-10-17",
     "Statement": [
       {
         "Effect": "Allow",
         "Action": [
           "s3:ListBucket"
         ],
         "Resource": [
           "S3 ARN"
         ]
       },
       {
         "Effect": "Allow",
         "Action": [
           "s3:GetObject"
         ],
         "Resource": [
           "S3 ARN/*"
         ]
       }
     ]
   }

   The policy editor should resemble the following:

   

5. Optionally, click Actions and select Optimize for readability to format the JSON.

6. Click Next.

7. Enter a name for the policy, then click Create policy.

2. Create an IAM role

In this section, you create an AWS IAM role and add a trust policy that allows Lumi to assume that role.

Start by obtaining a trust policy template from Lumi:

1. From the Lumi navigation menu, click Integrations > S3 pull.
2. Under Configure AWS access: Create an AWS IAM role, copy the trust policy. You'll use it to create an IAM role in AWS.

Create an IAM role with the trust policy:

1. In the AWS console, go to the IAM dashboard.

2. Click Roles in the left sidebar, then click Create role.

3. In the Trusted entity type section, select Custom trust policy.

4. Replace the contents in the policy editor with the trust policy you copied from Lumi.

   The policy editor should resemble the following:

   

5. Click Next.

6. Search for and select the permissions policy you created in the previous step.

7. Click Next.

8. Enter a name for the role, then click Create role.

9. Click the role to view its details. Save the ARN of the role. You'll need it in a later step.

3. Create an IAM key

In this section, you configure an IAM key in Lumi using your AWS IAM role ARN. As part of this process, Lumi provides an endpoint that can receive notifications about new objects.

1. From the Lumi navigation menu, click Integrations > S3 pull.
2. In the Select or create an IAM key pane, click Select or create key > Create key.
3. Enter a name, global attributes, and the ARN of the AWS IAM role you created in the previous step.
4. Save your changes.
5. The page populates with the information required to create a new subscription. Save the endpoint URL. You'll need it in a later step.

4. Create an SNS topic and subscription

SNS topics act as communication channels for publishing messages that you can then broadcast to different subscribers. In this section, you create an SNS topic in AWS and subscribe the Lumi endpoint to it.

1. In the AWS console, go to the SNS dashboard.
2. Click Topics in the left sidebar, then click Create topic.
3. Select Standard for Type.
4. Enter a name for the topic.
5. Click Create topic. Save the ARN of the topic. You'll need it in a later step.
6. Click Subscriptions in the left sidebar, then click Create subscription.
7. Enter the following details:
   • Topic ARN: Select the SNS topic you created.
   • Protocol: HTTPS
   • Endpoint: Lumi endpoint URL provided in the previous step.
8. Click Create subscription.

5. Create an event notification

In this section, you create an event notification for your S3 bucket to trigger a notification when a new object is created.

1. In the AWS console, go to Amazon S3.

2. Select your S3 bucket.

3. Go to the Properties tab.

4. In the Event notifications section, click Create event notification.

5. Enter a name for the event.

6. In the Event types section, select All object create events (s3:ObjectCreated:*).

7. In the Destination section, select SNS topic.

8. Select the SNS topic you created previously.

   

9. Click Save changes.

6. Send data

In this section, you upload a sample log file to the S3 bucket to test event routing to Lumi.

1. Download and save the example data file site_visitors.log.
2. In the AWS console, go to Amazon S3.
3. Select your S3 bucket.
4. Click Upload > Add files, then choose the site_visitors.log file you downloaded.
5. Click Upload.
6. Successful upload returns the message: "Upload succeeded."

7. Preview data

In this section, you preview the data you sent to Lumi and view the events in the explore view.

1. Return to the S3 pull integration page in Lumi. In the Preview incoming data section, view the newly added events.

2. Select Explore events. Lumi takes you to the explore view and applies search filters for your IAM key and the integration. You may need to adjust the time range selector and refresh the page to see the events.

   

3. Select an event to view its attributes. For details on the attributes, see Event model.

   

For information on searching events and filtering on event attributes, see Search events with Lumi.

Learn more

For more information, see the following topics:

• Send events with S3 pull for details on the integration.
• Send events to Lumi for other options to send events.
• How to search events with Lumi to walk through a set of example queries using Lumi query syntax.