Ingest from Amazon S3

You can create a connection to Amazon S3, an object storage service provided by Amazon Web Services, to ingest data into Imply Polaris.

Create a unique connection for each S3 bucket from which you want to ingest data.

S3 connection information

A Polaris connection to Amazon S3 takes the following information:

• Information about the S3 bucket to ingest from.

  • Bucket name: The name of the S3 bucket that contains the data to ingest.
  • Prefix (optional): You can limit access to designated files in the S3 bucket by specifying a prefix. The connection will be limited to the set of files matching this prefix. For example, logs/20221014T00:00:00.
  • AWS S3 endpoint: The endpoint of the S3 service, such as s3.us-east-1.amazonaws.com.

• Authorization to access the S3 bucket. For more information, see Secure connections to AWS and the AWS documentation on Managing access to resources.

  • ARN of IAM role: The Amazon Resource Name (ARN) of the AWS assumed role to use for access. For example, arn:aws:iam:::123456789012:role/s3-access-role.

  • Trust policy attached to the IAM role: Authorizing access to your S3 data from Polaris requires a trust policy added to your IAM role to allow Polaris to assume the role. For more information, see Trust policy.

  • Permissions policy attached to the IAM role: In order to grant Polaris access to view and ingest data from your S3 buckets, attach to the IAM role a permissions policy that lists your S3 resources and includes the following actions:

    • s3:GetObject to retrieve objects from the S3 bucket.
    • s3:ListBucket (optional) to list the objects in the S3 bucket. This permission is not required to ingest from S3; however, Imply strongly recommends you include the permission because it makes viewing and selecting objects to ingest more straightforward. Note that s3:ListBucket is the name of the permission that allows a user to list the objects in a bucket. ListObjectsV2 is the name of the API call that lists the objects in a bucket.

Example IAM policy

The following example shows an IAM policy that can be attached to your IAM role. The policy grants the role the listed permissions for Polaris to view and obtain data from your S3 bucket. Replace S3 ARN with the ARN for your S3 resource—for example, arn:aws:s3:::bucket_name.

{
  "Version": "2012-10-17",
  "Statement": [
      {
          "Effect": "Allow",
          "Action": [
              "s3:ListBucket"
          ],
          "Resource": [
              "S3 ARN"
          ]
      },
      {
          "Effect": "Allow",
          "Action": [
              "s3:GetObject"
          ],
          "Resource": [
              "S3 ARN/*"
          ]
      }
  ]
}

The following screenshot shows an example connection created in the UI:

Learn more

To learn how to ingest data from Amazon S3 using the Polaris API, see Ingest data from Amazon S3 by API.