• Developer guide
  • API reference

›Ingestion sources

Getting started

  • Introduction to Imply Polaris
  • Quickstart
  • Execute a POC
  • Create a dashboard
  • Navigate the console
  • Key concepts

Tables and data

  • Overview
  • Introduction to tables
  • Table schema
  • Create an ingestion job
  • Timestamp expressions
  • Data partitioning
  • Introduction to rollup
  • Approximation algorithms
  • Replace data

Ingestion sources

  • Ingestion sources overview
  • Supported data formats
  • Create a connection
  • Ingest from files
  • Ingest data from a table
  • Ingest from S3
  • Ingest from Kafka and MSK
  • Ingest from Kinesis
  • Ingest from Confluent Cloud
  • Kafka Connector for Imply Polaris
  • Push event data
  • Connect to Confluent Schema Registry

Analytics

  • Overview
  • Manage data cubes
  • Visualize data
  • Data cube dimensions
  • Data cube measures
  • Dashboards
  • Visualizations reference
  • Set up alerts
  • Set up reports
  • Embed visualizations
  • Query data

Monitoring

  • Overview

Management

  • Overview
  • Pause and resume a project

Billing

  • Overview
  • Polaris plans
  • Estimate project costs

Usage

  • Overview

Security

    Polaris access

    • Overview
    • Invite users to your organization
    • Manage users
    • Permissions reference
    • Manage user groups
    • Enable SSO
    • SSO settings reference
    • Map IdP groups

    Secure networking

    • Connect to AWS
    • Create AWS PrivateLink connection

Developer guide

  • Overview
  • Authentication

    • Overview
    • Authenticate with API keys
    • Authenticate with OAuth
  • Manage users and groups
  • Migrate deprecated resources
  • Create a table
  • Define a schema
  • Upload files
  • Create an ingestion job
  • Ingestion sources

    • Ingest from files
    • Ingest from a table
    • Get ARN for AWS access
    • Ingest from Amazon S3
    • Ingest from Kafka and MSK
    • Ingest from Amazon Kinesis
    • Ingest from Confluent Cloud
    • Push event data
    • Kafka Connector for Imply Polaris
    • Kafka Connector reference
  • Filter data to ingest
  • Ingest nested data
  • Ingest and query sketches
  • Specify data schema
  • Query data
  • Update a project
  • Link to BI tools
  • Connect over JDBC
  • Query parameters reference
  • API documentation

    • OpenAPI reference
    • Query API

Product info

  • Release notes
  • Known limitations
  • Druid extensions

Connect to Confluent Schema Registry

When you ingest streaming data in Avro or Protobuf format, you must provide additional schema information for Polaris to parse the input data. You can provide this schema "inline" directly in the job request, or through a connection to Confluent Schema Registry.

For details on how to use a Confluent Schema Registry connection for parsing input data, see Specify data schema by API.

Confluent Schema Registry connection information

A Polaris connection to Confluent Schema Registry requires the following information:

  • URLs of the Schema Registry: One or more Schema Registry endpoints. For example, https://psrc-xxxxx.us-east-1.aws.confluent.cloud.

  • Confluent Schema Registry API key: You provide the API key and API secret in the username and password fields for authenticating to Confluent Schema Registry from Polaris. For information on creating API keys in Confluent Schema Registry, see Quick Start for Schema Management on Confluent Cloud.

    Confluent Schema Registry supports multiple authentication modes, including HTTP Basic authentication or SASL authentication. For more information, see the Confluent docs on Schema Registry Security Overview.

Schema registry example

The following screenshot shows an example of using a Schema Registry connection when ingesting Protobuf data from a Confluent Cloud connection. To follow along with this example, create a table and an authenticated Confluent Cloud connection. See Specify data schema by API for a diagram of an example workflow for working with tables, connections, and ingestion jobs.

  1. From the table view, click Insert data in the top navigation panel.

  2. Select the Confluent Cloud tile, and select the connection to ingest from. Click Next.

  3. In the Input format menu, select Protobuf. Polaris displays options to determine your schema using a local file or from a registry. Below the second option, select Connect to registry. Schema registry example

  4. Polaris shows the available connections to schema registries. To create a new connection, select New connection. Schema registry example

  5. Complete the New connection dialog, and click Create connection. Schema registry example

  6. Polaris interprets the schema and samples data from the Kafka topic defined in the Confluent Cloud connection. You can edit data types and add additional input fields. Click Continue. Schema registry example

  7. Map your input fields to table columns. You can also configure transforms, add a filter to limit what data is ingested, and select the offset for when in the stream Polaris starts ingesting data. When you are finished, click Start ingestion. Schema registry example

Learn more

See the following topics for more information:

  • Specify data schema by API
  • Supported data and file formats
← Push event dataOverview →
  • Confluent Schema Registry connection information
  • Schema registry example
  • Learn more
Key links
Try ImplyApache Druid siteImply GitHub
Get help
Stack OverflowSupportContact us
Learn more
BlogApache Druid docs
Copyright © 2023 Imply Data, Inc