A connection in Imply Polaris defines a source of data to ingest into a table. You can create multiple jobs from a single connection. This allows you to ingest data from the connection into multiple tables. If a table already has an ingestion job sourced from a streaming connection, you cannot create another job to stream data into that table.
Polaris supports the following connection types:
- Amazon S3
- Apache Kafka and Amazon MSK
- Amazon Kinesis
- Confluent Cloud
- Push streaming
- Confluent Schema Registry
Connections are not required for ingesting from Polaris tables or from files uploaded to Polaris. For more information on ingestion sources, see Ingestion sources overview.
Connections to certain sources of data require credentials for Polaris to access the data. The connection type determines what credentials are required. Polaris never displays existing credentials.
This topic describes how to create and manage connections in Polaris and how to ingest data into Polaris from the connection.
The following permissions are required for ingestion jobs involving connections:
ManageTablesto create and edit tables
ManageConnectionsto create and edit connections
ManageIngestionJobsto create ingestion jobs
For more information on permissions, see Permissions reference.
Create a connection
To create a new connection:
Click Sources from the left navigation menu. You can also add connections when you create a new table or add data to an existing table.
The following screenshot shows the sources view:
Click Create source and select the desired source. Polaris displays a New connection dialog with form fields specific to the connection.
Provide the following information for the connection:
- Connection name. The name of the connection in Polaris. The connection name must be unique within your Polaris organization and cannot be changed. A valid connection name is limited to 64 characters. Valid characters include alphanumeric characters, periods, hyphens, and underscores.
- Description. An optional description of the connection.
- Other details specific to the connection type. Visit the reference topic for each connection source for more information. For an overview of available ingestion sources, see Ingestion sources overview.
Verify the connection is successful before creating the connection. Select Test connection from the New connection dialog.
The following screenshot shows an example of a successful connection:
Polaris displays a warning if it is unable to connect to the data source; however, you can still save your connection information.
Your connections are displayed by source type on the Sources page. Click the ellipsis icon next to any connection to edit, test, or delete the connection.
Only users who have permissions to create connections can update and remove connection credentials and delete connections.
You can modify or delete an inactive connection. If the connection has an active ingestion job, cancel the job before you update or delete the connection.
To delete a connection:
- Navigate to Sources from the left pane.
- Click the ellipsis icon next to the connection and select Delete.
- Confirm the request.
Monitor streaming ingestion from a connection
Navigate to Jobs from the left pane to monitor the status of your ingestion job.
To view specific errors related to event ingestion, go to Tables. Click the menu icon on the right side of the row for your table and select View jobs.
You can also run a SQL query against the table to check that new data in the connection source is being ingested.
Go to Streaming to view dashboards that monitor the overall health of your event stream including:
- Ingest latency
- The number of Events processed
- Rejections because Events arrived too late
- Unparseable events
- Rows output
See the following topics for more information: