Skip to main content

Connect to Amazon MSK

To ingest data from Amazon Managed Streaming for Apache Kafka (MSK) into Imply Polaris, create a Kafka connection and use it as the source of an ingestion job. This topic provides reference information specific to MSK to create a Kafka connection.

For details on creating the Kafka connection, see Connect to Apache Kafka.

Prerequisites​

Polaris authenticates with Amazon MSK using IAM role assumption or SASL/SCRAM.

For IAM role assumption, the following prerequisites apply:

  • Your Polaris cluster is hosted on AWS.
  • Your MSK cluster is set up to use IAM access control. For more details, see the AWS documentation.
  • You are familiar with IAM role assumption in Polaris. See Secure connections to AWS.

For SASL/SCRAM, ensure that you set up SASL/SCRAM authentication for your Amazon MSK cluster. For more details, see the AWS documentation.

Connection information​

The details required for a Kafka connection for MSK are described in Connect to Apache Kafka.

For details on where to find the bootstrap servers in Amazon MSK, see Getting the bootstrap brokers using the AWS Management Console.

Authentication​

For Polaris to access your MSK data using IAM role assumption, you need the following:

  • ARN of IAM role: The Amazon Resource Name (ARN) of your AWS role that Imply will assume. For example, arn:aws:iam:::123456789012:role/msk-access-role.

  • Trust policy attached to the IAM role: Attach a trust policy that lists Imply's IAM role ARN so that Imply has authorization to assume your role. For an example, see Trust policy.

  • Permissions policy attached to the IAM role: Attach a permissions (authorization) policy that lists your MSK resource and includes the following actions. For details on each of the actions, see the AWS documentation on IAM access control for Amazon MSK.

    • kafka-cluster:Connect
    • kafka-cluster:DescribeTopic
    • kafka-cluster:DescribeGroup
    • kafka-cluster:AlterGroup
    • kafka-cluster:ReadData

Example IAM permissions policy​

The following example shows an IAM permissions policy that you attach to your AWS IAM role. The policy grants permissions to read data from the specified topic in your MSK cluster. Imply inherits these permissions when assuming your role. In Resource, list the appropriate ARNs for your MSK cluster, topic, or group.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "kafka-cluster:Connect",
            ],
            "Resource": [
                "arn:aws:kafka:us-east-1:0123456789012:cluster/MyTestCluster/abcd1234-0123-abcd-5678-1234abcd-1"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "kafka-cluster:DescribeTopic",
                "kafka-cluster:ReadData"
            ],
            "Resource": [
                "arn:aws:kafka:us-east-1:0123456789012:topic/MyTestCluster/*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "kafka-cluster:AlterGroup",
                "kafka-cluster:DescribeGroup"
            ],
            "Resource": [
                "arn:aws:kafka:us-east-1:0123456789012:group/MyTestCluster/*"
            ]
        }
    ]
}