Skip to main content

Azure Private Link for consuming data from an event stream

Azure Private Link is a networking service in Microsoft Azure that allows you to establish a unidirectional secure connection from your Azure virtual network (VNet) to Imply Polaris over a private endpoint. When you connect to Polaris through Azure Private Link, traffic between your Azure VNet and Polaris travels the Microsoft backbone network and doesn't traverse the public internet.

You can use Azure Private Link to establish a secure connection between Polaris and Apache Kafka clusters running within your Azure account.

This topic describes various ways you can set up a secure connection between Azure and Polaris using Azure Private Link to consume data directly from an event stream. Don't use the methods in this topic to query and publish data from an event stream.

info

All traffic that flows through Azure Private Link incurs interface endpoint expenses. For information on interface endpoint pricing, see Azure Private Link pricing.

General workflow​

The following is the general workflow to establish a private connection between Azure and Polaris VNets for consuming event data:

  1. Create a Private Link service for your Azure subscription. For detailed instructions, refer to the official Azure documentation.

  2. Provide the Private Link service configuration information to Polaris support. Polaris must subscribe to your service to receive events. The required information varies based on the Kafka installation method. See the following sections for more details on each method.

    Show connection requirements
    Azure Event Hubs
    • Resource ID: Resource ID for the Azure Event Hubs namespace.
    • Host name: Host name of the Azure Event Hubs namespace.
    Multi-availability zone cluster
    • Alias: Service alias.
    • Zone: Availability zone for each Private Link service.
    • Host name: DNS domain name.
    Single availability zone cluster
    • Alias: Service alias.
    • Host name: Host name of the Kafka installation.

  3. Approve the connection request.

Connect to Azure Event Hubs​

Follow the instructions in Allow access to Azure Event Hubs namespaces via private endpoints to configure private access for your Azure Event Hubs namespace. Once you create the private endpoint, contact Polaris support to complete the configuration on the Polaris side.

Polaris requires the following information to subscribe to the service:

  • Resource ID: Resource ID for the Azure Event Hubs namespace. For example: /subscriptions/1237f700-xxxx-0000-xxxx-abcdefgh1234/resourceGroups/polaris-demo-eastus/providers/Microsoft.EventHub/namespaces/private-link-test. Polaris uses the resource ID to subscribe to the Azure Private Link service of the Azure Event Hubs namespace. To obtain the resource ID, go to the Azure Event Hubs namespace page, click Overview > Essentials > JSON View.
  • Host name: Host name of your Azure Event Hubs namespace. For example: private-link-test.servicebus.windows.net. To obtain the host name, go to the Azure Event Hubs namespace page, click Overview > Essentials.

Wait for Polaris to complete the deployment, then manually approve the connection request as the Azure Private Link resource owner.

Connect to a multi-availability zone Kafka cluster​

The following is the general workflow to establish a private connection between Polaris and a multi-availability zone Kafka cluster hosted on Azure:

  1. Set up a Standard Load Balancer for each broker in your Kafka cluster.
  2. Create three Private Link services, one service for each of the zonal service aliases.
  3. Contact Polaris support to complete the setup. Polaris requires the following information to subscribe to each service:
    • Alias: Service alias. For example: s-d1vba-privatelink-1.123abc12-xxxx-0000-xxxx-abc123efg456.eastus.azure.privatelinkservice. Provide a zonal alias for each Private Link service—a total of three aliases, one for each service.
    • Zone: Availability zone of the Kafka cluster. Provide an availability zone for each Private Link service—a total of three availability zones, one for each service.
    • Host name: DNS domain name.
  4. Polaris sets up a private DNS zone for the Polaris VNet, where a DNS A record is added to point each Kafka broker's DNS to the Azure Private Link service dedicated to that broker.
  5. Approve the connection request.

The following diagram shows the Azure Private Link connection between a multi-availability zone Kafka cluster and Polaris:

Azure multi-zonal cluster connection diagram

Connect to a single availability zone Kafka cluster​

The following is the general workflow to establish a private connection between Polaris and a single availability zone Kafka cluster hosted on Azure:

  1. Set up a single Standard Load Balancer as follows:
    • Assign a unique listener to each broker in the Kafka cluster.
    • Ensure that the target of the listener is the broker assigned to it.
  2. Create a Private Link service for the Standard Load Balancer.
  3. Update the advertised.listeners on your brokers to add a new entry which advertises a host name and ports that the Standard Load Balancer is configured to listen on. For example, for the host name my-kafka.cluster, the advertised listeners should be updated to my-kafka.cluster:9001, my-kafka.cluster:9002, my-kafka.cluster:9003, etc. The listeners should include all ports.
  4. Contact Polaris support to complete the setup. Polaris requires the following information to subscribe to the service:
    • Alias: Service alias.
    • Host name: Host name of your Kafka installation.
  5. Polaris sets up a private DNS zone for the Polaris VNet, where a DNS A record is added to point the Kafka broker's DNS to the Azure Private Link service.
  6. Approve the connection request.

The following diagram shows the Azure Private Link connection between a single availability zone Kafka cluster and Polaris:

Azure single zone cluster connection diagram

Learn more​

See the following topics for more information: