Ingest from Amazon Kinesis

You can create a connection to Amazon Kinesis, a real-time data processing platform provided by Amazon Web Services, to ingest data into Imply Polaris.

Create a unique connection for each Kinesis stream from which you want to ingest data.

Kinesis connection information

A Polaris connection to Amazon Kinesis requires the following:

• Information about the Kinesis data stream to ingest from.

  • Stream name: The name of the Kinesis stream that contains the data to ingest.
  • AWS endpoint: The endpoint of the Kinesis stream, such as kinesis.us-east-1.amazonaws.com. There is no limitation on a region for the Kinesis data stream.

• Authorization to access the Kinesis data streams. For more information, see Secure connections to AWS and the AWS documentation on Managing access to resources.

  • ARN of IAM role: The Amazon Resource Name (ARN) of the AWS assumed role to use for access. For example, arn:aws:iam:::123456789012:role/kinesis-access-role.

  • Trust policy attached to the IAM role: Authorizing access to your Kinesis data from Polaris requires both the correct permissions defined in the IAM policy attached to your IAM role as well as a trust policy added to your IAM role to allow Polaris to assume the role. For more information, see Trust policy.

  • IAM policy with access permissions: In order to grant Polaris access to view and ingest data from your Kinesis streams, attach to the IAM role a permissions policy that lists your Kinesis stream resources and includes the following actions:

    • kinesis:ListStreams to view your Kinesis data streams.
    • kinesis:DescribeStream to obtain information about your Kinesis data streams.
    • kinesis:ListShards to list the shards in a stream.
    • kinesis:GetShardIterator to get a shard iterator. A shard iterator specifies the position from which to start reading data records from a shard in a stream.
    • kinesis:GetRecords to get data records from a Kinesis data stream's shard.

Example IAM policy

The following example shows an IAM policy that can be attached to your IAM role. The policy grants the role the listed permissions for Polaris to view and obtain data from your Kinesis data stream. Replace KINESIS ARN with the ARN for your Kinesis data stream—for example, arn:aws:kinesis:*:111122223333:stream/my-stream.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "kinesis:ListStreams",
        "kinesis:DescribeStream"
        ],
      "Resource": ["*"]
    },
    {
      "Effect": "Allow",
      "Action": [
        "kinesis:ListShards",
        "kinesis:GetShardIterator",
        "kinesis:GetRecords"
      ],
      "Resource": ["KINESIS ARN"]
    }
  ]
}

Example Kinesis connection

The following screenshot shows an example connection created in the UI. For more information on creating connections in the UI, see Create a connection.

To successfully test the connection to Kinesis, ensure your Kinesis stream contains data.

Learn more

To learn how to ingest data from Amazon Kinesis using the Polaris API, see Ingest data from Amazon Kinesis by API.