Skip to main content

Azure Private Link for consuming data from an event stream

Azure Private Link is a networking service in Microsoft Azure that allows you to establish a unidirectional secure connection from your Azure virtual network (VNet) to Imply Polaris over a private endpoint. When you connect to Polaris through Azure Private Link, traffic between your Azure VNet and Polaris travels the Microsoft backbone network and doesn't traverse the public internet.

You can use Azure Private Link to establish a secure connection between Polaris and Apache Kafka clusters running within your Azure account.

This topic describes various ways to set up a secure connection between Azure and Polaris using Azure Private Link to consume data directly from an event stream. Don't use the methods in this topic to query or publish data from an event stream.

The images and instructions in this topic reflect the Azure service configurations as of September 2024. They may not represent the current state of the product.

info

All traffic that flows through Azure Private Link incurs interface endpoint expenses. For information on interface endpoint pricing, see Azure Private Link pricing.

Connect to Azure Event Hubs​

The following is the general workflow to establish a private connection between Polaris and Azure Event Hubs:

  1. Follow the instructions in Allow access to Azure Event Hubs namespaces via private endpoints to configure private access for your Azure Event Hubs namespace.

  2. Contact Polaris support to complete the setup. Polaris requires the following information to subscribe to the service:

    • Resource ID: Resource ID for the Azure Event Hubs namespace. For example: /subscriptions/1237f700-xxxx-0000-xxxx-abcdefgh1234/resourceGroups/polaris-demo-eastus/providers/Microsoft.EventHub/namespaces/private-link-test. Polaris uses the resource ID to subscribe to the Azure Private Link service of the Azure Event Hubs namespace. To obtain the resource ID, go to the Azure Event Hubs namespace page, click Overview > Essentials > JSON View.
    • Host name: Host name of your Azure Event Hubs namespace. For example: private-link-test.servicebus.windows.net. To obtain the host name, go to the Azure Event Hubs namespace page, click Overview > Essentials.

  3. Approve the private connection request from Polaris:

    • In the Azure portal, navigate to Event Hubs.
    • Select the namespace you want to manage.
    • Go to Settings > Networking.
    • In the Private endpoint connections tab, select the private endpoint you want to approve and click Approve.
      For more information, see Azure documentation on how to approve a private endpoint connection.

    The following screen capture shows a pending request to establish a private connection to an Azure Event Hubs namespace: Azure private endpoint connection

Connect to a multi-availability zone Kafka cluster​

The following is the general workflow to establish a private connection between Polaris and a multi-availability zone Kafka cluster hosted on Azure:

  1. Set up a Standard Load Balancer for each broker in your Kafka cluster.
  2. Create three Private Link services, one service for each of the zonal service aliases. For detailed instructions, refer to the official Azure documentation.
  3. Contact Polaris support to complete the setup. Polaris requires the following information to subscribe to each service:
    • Alias: Service alias. For example: s-d1vba-privatelink-1.123abc12-xxxx-0000-xxxx-abc123efg456.eastus.azure.privatelinkservice. Provide a zonal alias for each Private Link service—a total of three aliases, one for each service.
    • Zone: Availability zone of the Kafka cluster. Provide an availability zone for each Private Link service—a total of three availability zones, one for each service.
    • Host name: DNS domain name.
  4. Polaris sets up a private DNS zone for the Polaris VNet, where a DNS A record is added to point each Kafka broker's DNS to the Azure Private Link service dedicated to that broker.
  5. Approve the private connection request from Polaris:
    • In the Azure portal, navigate to Private link services.
    • Select the namespace you want to manage.
    • Go to Settings > Private endpoint connections.
    • Select the private endpoint you want to approve and click Approve.

The following diagram shows the Azure Private Link connection between a multi-availability zone Kafka cluster and Polaris:

Azure multi-zonal cluster connection diagram

Connect to a single availability zone Kafka cluster​

The following is the general workflow to establish a private connection between Polaris and a single availability zone Kafka cluster hosted on Azure:

  1. Set up a single Standard Load Balancer as follows:
    • Assign a unique listener to each broker in the Kafka cluster.
    • Ensure that the target of the listener is the broker assigned to it.
  2. Create a Private Link service for the Standard Load Balancer. For detailed instructions, refer to the official Azure documentation.
  3. Update the advertised.listeners on your brokers to add a new entry which advertises a host name and ports that the Standard Load Balancer is configured to listen on. For example, for the host name my-kafka.cluster, the advertised listeners should be updated to my-kafka.cluster:9001, my-kafka.cluster:9002, my-kafka.cluster:9003, etc. The listeners should include all ports.
  4. Contact Polaris support to complete the setup. Polaris requires the following information to subscribe to the service:
    • Alias: Service alias.
    • Host name: Host name of your Kafka installation.
  5. Polaris sets up a private DNS zone for the Polaris VNet, where a DNS A record is added to point the Kafka broker's DNS to the Azure Private Link service.
  6. Approve the private connection request from Polaris:
    • In the Azure portal, navigate to Private link services.
    • Select the namespace you want to manage.
    • Go to Settings > Private endpoint connections.
    • Select the private endpoint you want to approve and click Approve.

The following diagram shows the Azure Private Link connection between a single availability zone Kafka cluster and Polaris:

Azure single zone cluster connection diagram

Learn more​

See the following topics for more information: