Skip to main content

Connect to Amazon Kinesis

info

Polaris only supports ingesting from Amazon Kinesis when your cloud provider is AWS.

To ingest data from Amazon Kinesis into Imply Polaris, create an Amazon Kinesis connection and use it as the source of an ingestion job. Create a unique connection for each Kinesis stream from which you want to ingest data.

Polaris authenticates with Kinesis using IAM role assumption. Before setting up a Kinesis connection, familiarize yourself with IAM role assumption in Polaris.

This topic provides reference information to create a Kinesis connection.

tip

For an end-to-end guide on Kinesis ingestion in Polaris, see Guide for Kinesis ingestion.

Create a connection​

Create a Kinesis connection as follows:

  1. Click Sources from the left navigation menu.
  2. Click Create source and select Amazon Kinesis.
  3. Enter the connection information.
  4. Click Test connection to confirm that the connection is successful.
  5. Click Create connection to create the connection.

The following screenshot shows an example connection created in the UI. For more information, see Create a connection.

Kinesis connection UI

info

To successfully test the connection to Kinesis, ensure your Kinesis stream contains data.

Connection information​

Follow the steps in Create a connection to create the connection. The connection requires the following information from Kinesis:

  • Stream name: The name of the Kinesis stream that contains the data to ingest.

  • AWS endpoint: The endpoint of the Kinesis stream, such as kinesis.us-east-1.amazonaws.com. There is no limitation on a region for the Kinesis data stream. To find your AWS endpoint, refer to the AWS service endpoints documentation.

Authentication​

Authenticate using IAM role assumption to grant Polaris the permissions to access your data from Kinesis.

You should be familiar with IAM role assumption through Polaris, see Secure connections to AWS. Also see the AWS documentation on Managing access to resources.

In the Kinesis connection, supply the following:

  • ARN of IAM role: The Amazon Resource Name (ARN) of your AWS role Imply will assume. For example, arn:aws:iam:::123456789012:role/kinesis-access-role.

  • Trust policy attached to the IAM role: Authorizing access to your Kinesis data from Polaris requires a trust policy added to your IAM role. The trust policy allows Polaris to assume the role. For an example, see Trust policy.

  • IAM policy with access permissions: Attach to a permissions policy to the IAM role. The permissions policy should list your Kinesis stream resources as well as actions that grant Polaris access to your data. See an example in the following section. The following actions are required:

Example IAM permissions policy​

The following example shows an IAM permissions policy that you attach to your IAM role. The policy grants the role the listed permissions for Polaris to view and obtain data from your Kinesis data stream. Replace KINESIS ARN with the ARN for your Kinesis data stream—for example, arn:aws:kinesis:*:111122223333:stream/my-stream.

Show the policy
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "kinesis:ListStreams",
        "kinesis:DescribeStream"
        ],
      "Resource": ["*"]
    },
    {
      "Effect": "Allow",
      "Action": [
        "kinesis:ListShards",
        "kinesis:GetShardIterator",
        "kinesis:GetRecords"
      ],
      "Resource": ["KINESIS ARN"]
    }
  ]
}

Learn more​

To learn how to ingest data from Amazon Kinesis using the Polaris API, see Ingest data from Amazon Kinesis by API.

To include Kinesis metadata with the ingestion job, see Ingest Kinesis metadata.