Connect to Amazon Kinesis

info

Polaris only supports ingesting from Amazon Kinesis when your cloud provider is AWS.

To ingest data from Amazon Kinesis into Imply Polaris, create an Amazon Kinesis connection and use it as the source of an ingestion job. Create a unique connection for each Kinesis stream from which you want to ingest data.

Polaris authenticates with Kinesis using IAM role assumption. Before setting up a Kinesis connection, familiarize yourself with IAM role assumption in Polaris.

This topic provides reference information to create a Kinesis connection.

tip

For an end-to-end guide on Kinesis ingestion in Polaris, see Guide for Kinesis ingestion.

Create a connection

Create a Kinesis connection as follows:

1. Click Sources from the left navigation menu.
2. Click Create source and select Amazon Kinesis.
3. Enter the connection information.
4. Click Test connection to confirm that the connection is successful.
5. Click Create connection to create the connection.

The following screenshot shows an example connection created in the UI. For more information, see Create a connection.

info

To successfully test the connection to Kinesis, ensure your Kinesis stream contains data.

Connection information

Follow the steps in Create a connection to create the connection. The connection requires the following information from Kinesis:

• Information about the Kinesis data stream to ingest from.

  • Stream name: The name of the Kinesis stream that contains the data to ingest.
  • AWS endpoint: The endpoint of the Kinesis stream, such as kinesis.us-east-1.amazonaws.com. There is no limitation on a region for the Kinesis data stream. To find your AWS endpoint, refer to the AWS service endpoints documentation.

• Authorization to access the Kinesis data streams. For more information, see Secure connections to AWS and the AWS documentation on Managing access to resources.

  • ARN of IAM role: The Amazon Resource Name (ARN) of your AWS role Imply will assume. For example, arn:aws:iam:::123456789012:role/kinesis-access-role.

  • Trust policy attached to the IAM role: Authorizing access to your Kinesis data from Polaris requires both the correct permissions defined in the IAM policy attached to your IAM role as well as a trust policy added to your IAM role to allow Polaris to assume the role. For an example, see Trust policy.

  • IAM policy with access permissions: In order to grant Polaris access to view and ingest data from your Kinesis streams, attach to the IAM role a permissions policy that lists your Kinesis stream resources and includes the following actions:

    • kinesis:ListStreams to view your Kinesis data streams.
    • kinesis:DescribeStream to obtain information about your Kinesis data streams.
    • kinesis:ListShards to list the shards in a stream.
    • kinesis:GetShardIterator to get a shard iterator. A shard iterator specifies the position from which to start reading data records from a shard in a stream.
    • kinesis:GetRecords to get data records from a Kinesis data stream's shard.

Example IAM permissions policy

The following example shows an IAM policy that can be attached to your IAM role. The policy grants the role the listed permissions for Polaris to view and obtain data from your Kinesis data stream. Replace KINESIS ARN with the ARN for your Kinesis data stream—for example, arn:aws:kinesis:*:111122223333:stream/my-stream.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "kinesis:ListStreams",
        "kinesis:DescribeStream"
        ],
      "Resource": ["*"]
    },
    {
      "Effect": "Allow",
      "Action": [
        "kinesis:ListShards",
        "kinesis:GetShardIterator",
        "kinesis:GetRecords"
      ],
      "Resource": ["KINESIS ARN"]
    }
  ]
}

Learn more

To learn how to ingest data from Amazon Kinesis using the Polaris API, see Ingest data from Amazon Kinesis by API.