Skip to main content

Kafka Connector configuration reference

This topic contains a list of the configuration options for the Kafka Connector for Imply Polaris (Kafka connector). See Kafka Connector for Imply Polaris for instructions on how to use the HTTP sink connector.

You can download sample configuration file to help you get started with a standalone configuration.

Kafka consumer client configurations​

topics​

A list of one or more comma-separated Kafka topics to consume. For example: example-topic-1, example-topic-2. If you configure more than one topic, set the http.url to the Polaris Events base URL and use topic.endpoint.mapping to map topics to individual Polaris stream connections.

  • Required: Yes
  • Type: String
  • Default: None

tasks.max​

The maximum number of connector tasks. For production scenarios, increase the maximum number of connector tasks to the number of CPU cores available to the Kafka connector. For example, if you deploy the connector to 3 servers and each server has 8 CPU cores set this value to 24 (3x8) unless it exceeds the number of partitions.

Each task is assigned to a thread. Each task is capable of handling multiple Kafka partitions, but a single partition must be handled by only one task.

  • Required: Yes
  • Type: Integer
  • Default: 1

Polaris connection configurations​

http.url​

A Polaris URL.

For a single topic, the Polaris streaming connection URL. For example, https://example.us-east-1.aws.api.imply.io/v1/projects/PROJECT_ID/events/kafka-connetor.

To map one or more topics to a single Polaris streaming connector, use the base URL for the Events API for http.url, for example https://example.us-east-1.aws.api.imply.io/v1/projects/PROJECT_ID/events/. Specify the mapping of Kafka topic to Polaris endpoint in topic.endpoint.mapping.

  • Required: Yes
  • Type: String (URL)
  • Default None

topic.endpoint.mapping​

Required only for use with multiple mappings. A comma-separated list of Kafka topics mapped to Polaris connections. For example: stream-1:connector-1, stream-2:connector-2, stream-3: connector-2. You cannot map a single stream to multiple endpoints.

  • Required: Only for more than one stream-to endpoint mapping
  • Type: String (URL)
  • Default None

http.proxy.host​

Host or IP address of for your proxy server.

  • Required: No
  • Type: String
  • Default None

http.proxy.port​

The port for your proxy server.

  • Required: No
  • Type: Integer
  • Default None

http.proxy.user​

Username to authenticate to your proxy server.

  • Required: No
  • Type: String
  • Default None

http.proxy.password​

Password for your proxy server user.

  • Required: No
  • Type: String
  • Default None

HTTP sink connector authorization​

http.authorization.type​

For API key authentication set to static. Do not set this property for OAuth authentication.

  • Required: Only for API key authentication
  • Type: String
  • Default None. Always use static

http.headers.authorization​

For API key authentication only, configure the basic authentication header used for API requests from the connector: Basic API_KEY. Substitute the value of your API key for API_KEY.

  • Required: Only for API key authentication
  • Type: String (HTTP authentication header value)
  • Default None

oauth2.access.token.url​

OAuth URL for Polaris. For example, https://id.imply.io/auth/realms/example/protocol/openid-connect/token. Replace example with your own organization ID.

  • Required: Only for OAuth authentication
  • Type: String (URL)
  • Default None

oauth2.client.id​

Your Polaris API client ID. See Authenticate with OAuth.

  • Required: Only for OAuth authentication
  • Type: String (API client)
  • Default None

oauth2.client.secret​

Your Polaris API client secret. See Authenticate with OAuth.

  • Required: Only for OAuth authentication
  • Type: String (API secret)
  • Default None

oauth2.client.authorization.mode​

URL for OAuth API clients.

  • Required: Only for OAuth authentication
  • Type: String (API secret)
  • Default: URL

Kafka bootstrap configuration​

reporter.bootstrap.servers​

The host and port where the Kafka bootstrap server runs.

  • Required: Yes
  • Type: String (host:port)
  • Default: localhost:9092