Skip to main content

Connect to Confluent Schema Registry

When you ingest streaming data in Avro or Protobuf format, you must provide additional schema information for Polaris to parse the input data. You can provide this schema "inline" directly in the job request, or through a connection to Confluent Schema Registry.

Prerequisites​

Before you proceed, familiarize yourself with creating a connection and starting an ingestion job.

Confluent Schema Registry connection information​

Follow the steps in Create a connection to create the connection. The connection requires the following information from Confluent:

  • URLs of the Schema Registry: One or more Schema Registry endpoints. For example, https://psrc-xxxxx.us-east-1.aws.confluent.cloud.

  • Confluent Schema Registry API key: Create an API key for Confluent Cloud Schema Registry, and save the key and secret to supply in the connection. For more information, refer to the Confluent documentation.

Use the Schema Registry​

This section guides you through using a Schema Registry connection to ingest Avro or Protobuf data from a Confluent Cloud or Kafka connection. Before following along, ensure you have a Confluent Cloud or Kafka connection. To preview your data in the UI (recommended), add data to the stream before continuing.

See Specify data schema by API for a diagram of an example workflow for working with tables, connections, and ingestion jobs.

  1. From a table view, click Insert data in the top navigation panel.

  2. Select Confluent Cloud or Kafka as the source, and select your connection. Click Next.

  3. In the Input format menu, select your data format, either Avro or Protobuf. Polaris displays options to determine your schema using a local file or from a registry.

  4. Below the second option, select Connect to registry > New connection.

  5. Provide a connection name, and enter the Schema Registry URL and API key and secret. Then click Create connection.

    Schema registry example

  6. Polaris interprets the schema and samples data from the Kafka topic defined in the Confluent Cloud connection. You can edit data types and add additional input fields. Click Continue.

  7. Map your input fields to table columns. You can also configure transforms, add a filter to limit what data is ingested, and select the offset for where in the stream Polaris starts ingesting data.

  8. Click Start ingestion.

Learn more​

See the following topics for more information: