Skip to main content

Send events with S3 pull

You can send events from an Amazon Simple Storage Service (Amazon S3) bucket to Imply Lumi using the S3 pull integration.

The S3 pull integration supports the following types of ingestion jobs:

  • Recurring: Lumi continuously listens for event notifications from AWS. When new objects arrive, Lumi automatically ingests the data. For details, see Recurring ingestion.
  • Backfill: You manually specify which objects to ingest. Lumi ingests the historical data once. For details, see Backfill ingestion.

To send events from an S3 bucket to Lumi, you need to configure AWS access and create a Lumi IAM key. Then, you can create either a recurring ingestion job for real-time data or a backfill ingestion job for historical data.

This topic covers the initial setup. For steps specific to recurring or backfill ingestion, see Recurring ingestion and Backfill ingestion respectively.

info

The instructions in this topic reflect AWS configurations as of October 2025.

Prerequisites​

To send events from an S3 bucket to Lumi, you need the following:

  • Access to Lumi with the Data manager role or higher. For information on roles and permissions, see Manage roles.
  • Amazon IAM permissions to create roles, create policies, and attach policies to roles. For more information, see Allow users and groups to create and modify roles in the AWS documentation.
  • Amazon S3 bucket, including the ARN of the bucket. For information on how to find the ARN of an S3 bucket, see Find ARNs in the AWS documentation.

Supported data formats​

The S3 pull integration supports the following object formats:

  • JSON
  • CSV
  • Plain text
  • Splunk® HEC
  • Splunk® CSV

The integration also supports ingesting files from compression formats.

Configure AWS access​

Before you can load data from an S3 bucket into Lumi, you need to configure AWS access. This involves setting up the necessary AWS permissions and identity, including a permissions policy and an IAM role.

Create an AWS permissions policy​

Create a permissions policy that grants access to the S3 bucket:

  1. In the AWS console, go to the IAM dashboard.

  2. Click Policies in the left sidebar, then click Create policy.

  3. In the Policy editor section, click JSON.

  4. Replace the contents in the policy editor with the following policy. Replace both instances of S3 ARN with the ARN of your S3 bucket.

    {
      "Version": "2012-10-17",
      "Statement": [
        {
          "Effect": "Allow",
          "Action": [
            "s3:ListBucket"
          ],
          "Resource": [
            "S3 ARN"
          ]
        },
        {
          "Effect": "Allow",
          "Action": [
            "s3:GetObject"
          ],
          "Resource": [
            "S3 ARN/*"
          ]
        }
      ]
    }

    The policy editor should resemble the following:

    AWS IAM policy editor

  5. Optionally, click Actions and select Optimize for readability to format the JSON.

  6. Click Next.

  7. Enter a name for the policy, then click Create policy.

Handling objects encrypted with SSE-KMS​

If your S3 bucket uses server-side encryption with AWS Key Management Service (AWS KMS) keys (SSE-KMS), whether with customer managed keys or AWS managed keys, the IAM role you use for Lumi must include the kms:Decrypt permission. For more information, see the AWS documentation on SSE-KMS encryption.

To grant this permission, add the following statement to the permissions policy:

{
    "Effect": "Allow",
    "Action": "kms:Decrypt",
    "Resource": "arn:aws:kms:REGION:ACCOUNT_ID:key/KMS_KEY_ID" 
}

Replace REGION, ACCOUNT_ID, and KMS_KEY_ID with your AWS region, account ID, and the ARN of the specific KMS key used to encrypt the data.

Create an AWS IAM role​

Lumi provides a trust policy template that allows Lumi to assume an IAM role in your AWS account.

Start by obtaining the trust policy from Lumi:

  1. From the Lumi navigation menu, click Integrations > S3 pull.
  2. In the Configure AWS access section, click Show instructions.
  3. Scroll down to the Create an AWS IAM role step and copy the trust policy. You'll use it to create an IAM role in AWS.

Create an IAM role with the trust policy:

  1. In the AWS console, go to the IAM dashboard.

  2. Click Roles in the left sidebar, then click Create role.

  3. In the Trusted entity type section, select Custom trust policy.

  4. Replace the contents in the policy editor with the trust policy you copied from Lumi.

    The policy editor should resemble the following:

    AWS custom trust policy

  5. Click Next.

  6. Search for and select the permissions policy you created in the previous step.

  7. Click Next.

  8. Enter a name for the role, then click Create role.

  9. Click the role to view its details. Save the ARN of the role. You'll need it in a later step.

Configure a Lumi IAM key​

Configure a Lumi IAM key using the Amazon Resource Name (ARN) of the IAM role you created:

  1. From the Lumi navigation menu, click Integrations > S3 pull.
  2. In Select or create an IAM key, enter the following details:
    • For a new key, enter a name, optional description, global attributes, and the ARN of your AWS IAM role.
    • For an existing key, enter the ARN of your AWS IAM role.
  3. Save your changes.
tip

As part of this process, Lumi provides an endpoint to receive notifications about new objects.

Once you configure access to AWS and Lumi, you can create a recurring or backfill ingestion job to start sending events to Lumi.

Learn more​

See the following topics for more information: