Ingest from Kafka and and Amazon MSK

You can create a connection to Kafka or Amazon MSK (MSK) to ingest data into Imply Polaris. MSK is a fully managed, cloud-native service for Apache Kafka.

Ingestion from Kafka or MSK to Polaris uses exactly once semantics.

Data in transit between Polaris and Kafka is encrypted with TLS. Polaris authenticates with your Kafka cluster using Simple Authentication Security Layer (SASL) authentication.

Kafka or MSK connection information

A Polaris connection to Kafka or MSK requires the following:

  • Topic name: The name of the Kafka topic containing your event data.

  • Bootstrap servers: A list of one or more host and port pairs representing the addresses of brokers in the Kafka cluster. This list should be in the form host1:port1,host2:port2,... For details on where to find the bootstrap server in Amazon MSK, see Getting the bootstrap brokers using the AWS Management Console.

  • Apache Kafka username and password for Polaris to use to make the connection. Polaris supports SASL/PLAIN and SASL/SCRAM authentication. For SASL SCRAM connections, you must also provide the SCRAM mechanism, either SCRAM-SHA-256 or SCRAM-SHA-512

    Polaris does not currently support IAM authentication for Amazon MSK.

Example MSK connection

The following screenshot shows an example connection created in the UI. For more information on creating connections in the UI, see Create a connection.

Polaris sources

When you have your table, connection, and ingestion job set up for Kafka or MSK, Polaris automatically ingests data from the Kafka topic as data enters the topic defined in the connection.

Learn more

To learn how to ingest data from Kafka or MSK using the Polaris API, see Ingest data from Kafka and MSK by API.

← Ingest from S3Ingest from Kinesis →