Connect to Confluent Schema Registry

When you ingest streaming data in Avro or Protobuf format, you must provide additional schema information for Polaris to parse the input data. You can provide this schema "inline" directly in the job request, or through a connection to Confluent Schema Registry.

For details on how to use a Confluent Schema Registry connection for parsing input data, see Specify data schema by API.

Confluent Schema Registry connection information

A Polaris connection to Confluent Schema Registry requires the following information:

  • URLs of the Schema Registry: One or more Schema Registry endpoints. For example, https://psrc-xxxxx.us-east-1.aws.confluent.cloud.

  • Confluent Schema Registry API key: You provide the API key and API secret in the username and password fields for authenticating to Confluent Schema Registry from Polaris. For information on creating API keys in Confluent Schema Registry, see Quick Start for Schema Management on Confluent Cloud.

    Confluent Schema Registry supports multiple authentication modes, including HTTP Basic authentication or SASL authentication. For more information, see the Confluent docs on Schema Registry Security Overview.

Schema registry example

The following screenshot shows an example of using a Schema Registry connection when ingesting Protobuf data from a Confluent Cloud connection. To follow along with this example, create a table and an authenticated Confluent Cloud connection. See Specify data schema by API for a diagram of an example workflow for working with tables, connections, and ingestion jobs.

  1. From the table view, click Insert data in the top navigation panel.

  2. Select the Confluent Cloud tile, and select the connection to ingest from. Click Next.

  3. In the Input format menu, select Protobuf. Polaris displays options to determine your schema using a local file or from a registry. Below the second option, select Connect to registry. Schema registry example

  4. Polaris shows the available connections to schema registries. To create a new connection, select New connection. Schema registry example

  5. Complete the New connection dialog, and click Create connection. Schema registry example

  6. Polaris interprets the schema and samples data from the Kafka topic defined in the Confluent Cloud connection. You can edit data types and add additional input fields. Click Continue. Schema registry example

  7. Map your input fields to table columns. You can also configure transforms, add a filter to limit what data is ingested, and select the offset for when in the stream Polaris starts ingesting data. When you are finished, click Start ingestion. Schema registry example

Learn more

See the following topics for more information:

← Push event dataOverview →