Set up Clarity

There are two ways you can use Clarity with an on-prem Imply deployment. The recommended way is to use SaaS Clarity (http://clarity.imply.io/) with on-prem Imply. This approach offers the simplest set up and maintenance. It's also possible to set up an on-prem Clarity instance as well. Both approaches are described below.

Imply Private with SaaS Clarity

To set up Imply Private with SaaS Clarity, enable the Clarity emitter extension, as follows:

In the Imply configuration file, common.runtime.properties, make the following two changes:

Add "clarity-emitter" to druid.extensions.loadList.

Add the following emitter configs at the end of the file. If you have existing emitter configs, remove those first.

# Enable JVM monitoring.
druid.monitoring.monitors=["org.apache.druid.java.util.metrics.JvmMonitor"]

# Enable Clarity emitter.
druid.emitter=clarity

# API details provided by Imply.
druid.emitter.clarity.recipientBaseUrl=https://cc.imply.io/d/<orgname>
druid.emitter.clarity.basicAuthentication=<orgname>:<apikey>

# Cluster name; should be different for each cluster.
druid.emitter.clarity.clusterName=<my-cluster-name>

Restart all nodes in the cluster, which enables the metrics.

If druid.extensions.loadList has been independently specified for a service, then you need to make this configuration in each respective service's druid.extensions.loadList as well, followed by a service restart. For example if the broker service configuration includes druid.extensions.loadList, clarity-emitter needs to be added to druid/broker/runtime.properties.

Optional Clarity emitter configurable properties

You can configure Clarity behavior and settings using the following properties common.runtime.properties. Prepend all properties with druid.emitter.clarity. followed by the field name. For example, druid.emitter.clarity.recipientBaseUrl.

Field	Type	Description	Default	Required
recipientBaseUrl	String	HTTP endpoint events will be posted to, e.g. http://:/d/	[required]	yes
basicAuthentication	String	Basic auth credentials, typically :	null	no
clusterName	String	Cluster name used to tag events	null	no
anonymous	Boolean	Should hostnames be scrubbed from events?	FALSE	no
maxBufferSize	Integer	Maximum size of event buffer	min(250MB, 10% of heap)	no
maxBatchSize	Integer	Maximum size of HTTP event payload	5MB	no
flushCount	Integer	Number of events before a flush is triggered	500	no
flushBufferPercentFull	Integer	Percentage of buffer fill that will trigger a flush (byte-based)	25	no
flushMillis	Integer	Period between flushes if not triggered by flushCount or flushBufferPercentFull	60s	no
flushTimeOut	Integer	Flush timeout	Long.MAX_VALUE	no
timeOut	ISO8601 Period	HTTP client response timeout	PT1M	no
batchingStrategy	String [ARRAY, NEWLINES]	How events are batched together in the payload	ARRAY	no
compression	String [NONE, LZ4, GZIP]	Compression algorithm used	LZ4	no
lz4BufferSize	Integer	Block size for the LZ4 compressor in bytes	65536	no
samplingRate	Integer	For sampled metrics, what percentage of metrics will be emitted	100	no
sampledMetrics	List	Which event types are sampled	["query/wait/time", "query/segment/time", "query/segmentAndCache/time"]	no
sampledNodeTypes	List	Which node types are sampled	["druid/historical", "druid/peon", "druid/realtime"]	no

SSL options

All keys are prefixed by druid.emitter.clarity.ssl, for example, druid.emitter.clarity.ssl.trustStorePath.

If trustStorePath is not specified, a custom SSL context will not be created, and the default SSL context will be used instead.

Property	Description	Default	Required
protocol	SSL protocol to use.	TLSv1.2	no
trustStoreType	The type of the key store where trusted root certificates are stored.	`java.security.KeyStore.getDefaultType()`	no
trustStorePath	The file path or URL of the TLS/SSL Key store where trusted root certificates are stored.	none	no
trustStoreAlgorithm	Algorithm to be used by TrustManager to validate certificate chains	`javax.net.ssl.TrustManagerFactory.getDefaultAlgorithm()`	no
trustStorePassword	The Password Provider or String password for the Trust Store.	none	yes, if trustStorePath is specified.

On-prem Clarity

Under the covers, Clarity uses Druid to store metrics. For a production on-prem installation, you should install a separate Druid cluster (the collection cluster) to receive performance data from the monitored Druid cluster (the monitored cluster).

In evaluation settings, it's possible to have a single cluster acting as both the monitored and collection cluster. However, in a production setting, this is strongly discouraged; the monitoring cluster should run independently from the cluster being monitored to ensure that monitoring functions, such as alerting, continue working if the availability or performance of the the production cluster is degraded. It also prevents Clarity operations from impacting production cluster performance.

Similarly, running Pivot from the secondary Imply instance provides the equivalent advantage.

Enabling Clarity on-prem involves these steps:

Set up a metrics collection cluster.
Configure your monitored cluster to emit metrics to Kafka.
Configure the Kafka topic to which the monitored cluster emits metrics as a data source on your metrics collection cluster.
Enable the embedded Clarity UI in your Pivot configuration.

Clarity architecture

Step 1: Set up a metrics collection cluster

You can skip this step if you plan to use the same cluster for metrics emitting and metrics collection. However, note that in production, you should use separate clusters.

To minimize sizing requirements of the metrics cluster, use load and drop rules to set a retention window on your data.

Most metrics are query telemetry events, which are emitted once per query per segment. Since it's common for clusters to have thousands of segments or more, there can be quite a lot of these events! If you have high query concurrency and wish to limit the amount of telemetry emitted, use the druid.emitter.clarity.samplingRate property documented in the table below. This property should be set on the metrics emitting cluster, not the metrics collection cluster.

For a large metrics cluster, you will need to increase the size of taskCount in your Kafka supervisor spec. This is the amount of parallelism used to process metrics. You may need to increase the number of data servers as well. Ensure that the druid-histogram extension is in the druid.extensions.loadList in the druid/_common/common.runtime.properties config file. This extension is used for computing 98th percentile latency metrics.

Step 2: Enable the metric emitter on the monitored cluster

For every cluster that you want to monitor, configure the Clarity emitter by following these steps:

Ensure that the clarity-emitter-kafka extension is in the druid.extensions.loadList in druid/_common/common.runtime.properties file for the emitting cluster.

Remove or comment out existing druid.emitter and druid.emitter.* configs in druid/_common/common.runtime.properties and replace them with the following:

druid.emitter=clarity-kafka
druid.emitter.clarity.topic=druid-metrics
druid.emitter.clarity.producer.bootstrap.servers=kafka1.example.com:9092
druid.emitter.clarity.clusterName=clarity-collection-cluster

Replace kafka1.example.com:9092 with a comma-delimited list of Kafka brokers in your environment.
The "clarity-collection-cluster" string can be anything you want, but it is intended to be used to help Clarity users tell different clusters apart in the Clarity UI.

The Clarity emitter will write to a "druid-metrics" topic. Start up Druid and verify that the druid-metrics topic exists as a datasource in the collection cluster.

druid-metrics datasource

Step 3: Configure Kafka ingestion on your metrics collection cluster

Ensure that the Druid Kafka indexing service extension is loaded on the metrics collection cluster. See extensions for information on loading Druid extension.

Download the Clarity Kafka supervisor spec from https://static.imply.io/support/clarity-kafka-supervisor.json. Apply the spec by running the following command from the directory to which you downloaded the spec:

curl -XPOST -H'Content-Type: application/json' -d@clarity-kafka-supervisor.json http://<overlord_address>:8090/druid/indexer/v1/supervisor

Replace overlord_address with the IP address of the machine running the overlord process in your Imply cluster. This is typically the Master server in the Druid cluster.

Step 4: Configure Clarity-specific settings

Keep in mind that Clarity maintains a connection to the Druid collection cluster that is apart from Pivot's own connection to Druid. Accordingly, you need to configure the connection separately.

Add the following minimum configuration settings to your Pivot configuration file. You can find the Pivot configuration file in conf/pivot/config.yaml and conf-quickstart/pivot/config.yaml (for a quickstart instance) in your Imply installation home.

# Specify the metrics cluster to connect to
metricsCluster:
 host: localhost:8082 # Enter the IP of your metrics collecting broker node here

# Enter the name of your clarity data source
metricsDataSource: druid-metrics

# Instead of relying on auto-detection you can explicitly specify which clusters should be available from the cluster dropdown
clientClusters: ["default"]

# If your metrics data source does not have a histogram (approxHistogram) metric column then take it out of the UI by suppressing it
#suppressQuantiles: true

# If your metrics data source does have a histogram you can specify a tuning config here
#quantileTuning: "resolution=40

As noted in the comment, replace localhost in the metricsCluster configuration with the IP address of the metrics collection cluster.

You need to provide at least one cluster name in clientClusters parameter or Pivot may fail to start up. The name given should match the one used in druid.emitter.clarity.clusterName in the emitting cluster's common.runtime.properties configuration file.

Depending on the configuration of the Druid collecting cluster, you may need additional settings. For example, if authentication is enabled in Druid, add the defaultDbAuthToken property with the auth type, you need to add a username and password to the metricsCluster configuration, as follows:

metricsCluster:
 host: <broker_host>:<broker_port>
 ...
 defaultDbAuthToken:
    type: 'basic-auth'
    username: <auth_user_name>
    password: <auth_password>

If TLS is enabled, add the protocol property and provide the certificate information to the metricsCluster configuration:

metricsCluster:
  host: <MetricClusterBrokerHost>:<BrokerPort>
  protocol: tls
  ca: <certificate>

For a self-signed certificate, you can use tls-loose as the protocol:

metricsCluster:
  host: <MetricClusterBrokerHost>:<BrokerPort>
  protocol: tls-loose

Likewise, you can use any connection parameter available for connecting Pivot to Druid in the metricsCluster configuration for connecting Clarity to the metrics collection cluster as well. See metricsCluster settings for more information on those settings.

Access Clarity

If Pivot is running, you need to restart it to have the configuration change take effect. After restarting Pivot, you can open Clarity at the following address:

http://<pivot_address>:9095/clarity

Users in Pivot need to have the AccessClarity permission to be able to access the Clarity UI. Of the built-in roles, only "Super Admins" have this permission, so you'll need to allocate this permission to the users and roles as appropriate for your system.

metricsCluster connection optional parameters

Step 4 above describes the basic connection settings to connect Clarity to the metrics collection cluster.

The following connection settings are optional, or required only as necessitated by your metrics collection Druid configuration.

The settings are equivalent to those in the Pivot configuration. However, those settings are separate from Clarity's.

Field	Description
`timeout`	The timeout for the metric queries. Default is 40000.
`protocol`	The connection protocol, one of `plain` (the default), `tls-loose` or `tls`. If `tls` specify the `ca`, `cert`, `key` and `passphrase`.
`ca`	If connecting via TLS, a trusted certificate of the certificate authority if using self-signed certificates. Should be PEM formatted text.
`cert`	If connecting via TLS, the client side certificate to present. Should be PEM-formatted text.
`key`	If connecting via TLS, the private key file name. The key should be PEM-formatted text.
`passphrase`	If connecting via TLS, a passphrase for the private key, if needed.
`defaultDbAuthToken`	If Druid authentication is enabled, the default token that will be used to authenticate against this connection.
`socksHost`	If Clarity needs to connect to Druid via a SOCKS5 proxy, the hostname of the proxy host.
`socksUsername`	The user for the Socks proxy, if needed.
`socksPassword`	The password for proxy authentication, if needed.