Skip to main content

AWS PrivateLink for consuming data from an event stream

AWS PrivateLink is an AWS networking technology that enables private communication pathways between virtual private clouds (VPCs) and AWS or third-party services. You can use AWS PrivateLink to establish private connectivity between your Apache Kafka cluster and Imply Polaris within the AWS network. When you connect to Polaris through AWS PrivateLink, the network traffic between your AWS VPC and Polaris doesn't traverse the public internet.

This topic describes how to set up a secure connection between AWS and Polaris using AWS PrivateLink to consume data from an event stream in a Kafka cluster hosted on AWS. Don't use the methods in this topic to query and publish data from an event stream.

The images and instructions in this topic reflect the AWS service configurations as of October 2024. They may not represent the current state of the product.

info

All traffic that flows through AWS PrivateLink incurs interface endpoint expenses. For information on interface endpoint pricing, see AWS PrivateLink pricing.

Prerequisites​

Before you proceed, ensure that you have the 12-digit AWS account ID associated with Polaris. To obtain the AWS account ID, contact Polaris support. You use the ID to allow Polaris principal access to your AWS account. For more information on AWS account principals, see AWS JSON policy elements: Principal.

You can establish a secure connection between your AWS VPC and Polaris using either a single or multiple AWS PrivateLink endpoint services. Both methods require you to deploy additional AWS infrastructure, grant Polaris principal access to the endpoint service, and accept connection requests from Polaris. For the single AWS PrivateLink endpoint service option, you must also update your Kafka broker configurations to modify listeners and advertised listeners.

When deciding which method to use, consider the following:

  • Cost: The single endpoint service is more cost-effective, as it uses less AWS infrastructure.
  • Configuration modifications: With the single endpoint service, you must update the advertised listeners on the Kafka brokers when you add a new broker. The multi-endpoint service approach doesn't require you to modify the advertised listeners. However, every time you add a new Kafka broker, you need to inform Polaris support for Polaris to subscribe to the new endpoint service.

These solutions aren't managed by Amazon. For an Amazon-managed solution, see Amazon multi-VPC private connectivity.

The following diagram shows the architecture for the single AWS PrivateLink endpoint service setup:

Single AWS PrivateLink endpoint service connection diagram

Follow these steps to establish a private connection between Polaris and a Kafka cluster through a single AWS PrivateLink endpoint service:

  1. Create a Network Load Balancer (NLB) in your AWS VPC. Make sure to select this load balancer when you create the endpoint service.

  2. If your NLB has a security group, either allow inbound traffic from Polaris IP addresses or disable the evaluation of inbound security group rules for AWS PrivateLink traffic sent to the NLB. For more information, see Update the security settings in the AWS documentation.

  3. Follow the instructions in Create an endpoint service to create an AWS endpoint service using the NLB.

  4. Grant permission to the Polaris principal to connect to your endpoint service:

    • Go to the Allow principles tab for your endpoint service.
    • Click Allow principles.
    • For Principles to add, enter the ARN of the Polaris principle following the format arn:aws:iam::polaris-account-id:root.
    • Click Allow principles. For more information, see Manage permissions in the AWS documentation.

  5. Contact Polaris support to complete the setup. Polaris requires the following information to subscribe to the service:

    • Endpoint service name: Service name, for example com.amazonaws.vpce.us-east-1.vpce-svc-01abc234de56fg7hi.
    • Availability zone ID: Availability zone (AZ) ID for your endpoint service. Note that an AZ ID is different from an AZ name. For example, use1-az4 is an AZ ID for an AZ in the us-east-1a region: AWS availability zone For more information, see Availability Zone IDs for your AWS resources in the AWS documentation.
    • Broker domain: Fully qualified domain name (FQDN) associated with the endpoint service, for example *.something.us-west-2.amazonaws.com.

  6. After Polaris support completes the setup, you should see a new connection request under the Endpoint connections tab of your endpoint service. Approve the connection request from Polaris:

    • Go to the Endpoint connections tab for your endpoint service.
    • Click Actions > Accept endpoint connection request. For more information, see Accept or reject connection requests in the AWS documentation.

    The following screen capture shows endpoint connection requests for an AWS endpoint service: AWS private endpoint connection

The following diagram shows the architecture for the multi AWS PrivateLink endpoint service setup:

Multi AWS PrivateLink endpoint service connection diagram

To establish a private connection between Polaris and a Kafka cluster using multiple AWS PrivateLink endpoint services and NLBs, follow the instructions in the Single AWS PrivateLink endpoint service section with these key adjustments:

  1. In your AWS VPC, create a separate NLB for every Kafka broker in the cluster.
  2. If your NLBs use security groups, either allow inbound traffic from Polaris IP addresses or disable inbound security group evaluation as described in the AWS security settings documentation.
  3. Create an AWS PrivateLink endpoint service for each NLB.
  4. For each endpoint service, grant permissions to the Polaris principal. For more information, see the AWS permissions documentation.
  5. Provide Polaris with the required information for each endpoint service. See step 5 in the Single AWS PrivateLink endpoint service section for details.
  6. After Polaris support finishes the setup, you will receive connection requests for each endpoint service. Approve the requests to complete the configuration.

Learn more​

See the following topics for more information: