The following document describes how to use Kubernetes and the Helm chart to deploy a highly available, production-ready Imply cluster that is managed by the Imply Manager. You should generally reference this guide after you have followed the Kubernetes Quickstart to deploy a cluster to a scalable, distributed Kubernetes cluster (for example: Rancher, GKE, AKS, or EKS).
The most straightforward way to modify the Helm chart is to pull the values.yaml configuration file to your local machine and open it up in your favorite text editor. To fetch the configuration file for the latest version of the chart, run:
helm show values imply/imply > values.yamlOnce you are finished with your modifications, you can update the deployed release by running:
helm upgrade {releaseName} imply/imply -f values.yamlThe druid section of the values will allow us to set defaults for new clusters. The first time we deploy our chart these values are used in the initial cluster creation. If any of the default values need to be modified after the initial cluster creation happens this should be done within the Manager UI.
More information about the specific values within the druid section can be found in the documentation below.
The default deployment deploys one of each master, query, and data nodes. To increase the number of nodes running in the cluster we can modify their replicaCount in values.yaml.
For example, to modify the master replica count to be 3 we would change the following section:
master:
replicaCount: 1to:
master:
replicaCount: 3Adding additional data tiers can be completed much the same way as scaling. By default one data node is created in the default tier (dataTier1). To
create a second data tier change the replicaCount for the dataTier2 to the desired count. All configuration options available from the default tier
are available to both dataTier2 and dataTier3.
More information on Druid data tiers can be found here.
When adding an additional cluster the first step is getting the name of the original cluster deployment. Using the command helm list a list of the currently deployed releases.
Now we will create a values file for this cluster by running:
helm show values imply/imply > cluster-two.yamlOpen cluster-two.yaml with your favorite editor.
Update the deployments section to disable everything but the agents deployment as our second cluster will reuse those resources. For example:
...
deployments:
manager: false
agents: true
zookeeper: false
mysql: false
minio: false
...Next we will specify managerHost to point to the manager service defined in the existing deployment and set a name for the new cluster, e.g.:
...
agents:
managerHost: wanton-toucan-imply-manager-int
clusterName: cluster-two
...In this example wanton-toucan is the release name of the deployment we found using helm list.
First make any changes to scaling Now the second cluster can be deployed using the following command:
helm install -f cluster-two.yaml imply/imply
Any changes to the druid settings in the second cluster will not change the defaults. These values should be modified in the manager. Only changes to the master/query/dataTierX will take effect.
If using an external database we can disable the default MySQL deployment. To disable, edit the values.yaml and update the deployments section:
deployments:
manager: true
agents: true
zookeeper: true
mysql: false # this line should be updated to false to disable
minio: trueNow we will need to specify the connection details for the external database. Ensure that this database will be reachable from the pods in the deployment. Please see the sections below on configuring MySQL or PostgreSQL.
If your cluster is already running the data will not be migrated from the default MySQL deployment to your external database. This will have to be completed manually if required.
The Imply Manager application and the Druid clusters require a MySQL database, version 5.7.21 or later.
To use an external MySQL database we will update the metadataStore section for both the manager and druid. For example:
manager OR druid:
...
metadataStore:
type: mysql
host: mysql.example.com
port: 3306
user: root
password: imply
database: imply-manager # for the manager section only
...The database name should not contain any spaces or dashes.
To use an external MySQL database we will update the metadataStore section for both the manager and druid. For example:
manager OR druid:
...
metadataStore:
type: postgresql
host: postgresql.example.com
port: 5432
user: root
password: imply
database: imply-manager # for the manager section only
...The database name should not contain any spaces or dashes.
To enable TLS validation we add the TLS certificate to the metadataStore section. For example:
manager OR druid:
metadataStore:
...
tlsCert: |
-----BEGIN CERTIFICATE-----
...
-----END CERTIFICATE-----
...A Zookeeper deployment is included in the Helm chart. It is configured by default in a quick-start configuration that is not recommended in production. For production deployments it should either be configured as highly available or to use an existing highly available Zookeeper ensemble.
Check the Zookeeper Documentation for more information on clustered Zookeeper ensembles.
To increase the number of Zookeeper nodes we can adjust the replicaCount in the zookeeper section of values.yaml. For example:
...
zookeeper:
replicaCount: 1 # Change this value
...Zookeeper works best with an odd number of nodes and more nodes allows for more fault tolerance. Setting the replicaCount to 3 or a higher odd number will enable a highly available Zookeeper ensemble.
If using an external Zookeeper ensemble we can disable the default Zookeeper deployment. To disable, edit the values.yaml and update the deployments section:
deployments:
manager: true
agents: true
zookeeper: false # this line should be updated to false to disable
mysql: true
minio: trueNow we will need to specify the connection details for the external database. Ensure that this database will be reachable from the pods in the deployment. We can now update the new cluster defaults to use the external Zookeeper ensemble:
druid:
...
zk:
connectString: "zookeeper.example.com:2181"If your cluster is already created the Manager UI should be used to point the cluster to the external Zookeeper ensemble.
MinIO is not intended for production usage and is provided for a quick-start configuration only. For more information on the types of deep storage available in Druid as well as how it should be selected review the Druid Deep storage documentation.
When using external deep storage we can disable the default MinIO deployment. To disable, edit the values.yaml and update the deployments section:
deployments:
manager: true
agents: true
zookeeper: true
mysql: true
minio: false # this line should be updated to false to disableWhen using a named deep storage type below the required settings as well as authentication information and index logs will be set automatically. The druid extension will also be added to the classpath automatically. If any of these settings, or additional settings need to be changed they can be added to the commonRuntimeProperties in the druid section of values.yaml. If no additional settings are require the MinIO overrides should be removed.
The section:
druid:
...
commonRuntimeProperties:
- "# MinIO configurations"
- "druid.s3.endpoint.url=http://{{ .Release.Name }}-minio:9000"
- "druid.s3.enablePathStyleAccess=true"should be updated to:
druid:
...
commonRuntimeProperties: [] # for no settings, or just remove the s3 overrides for MinIOIf your cluster is already created the Manager UI should be used to point the cluster to the external deep storage implementation.
New cluster defaults for S3 can be set under deepStorage in the druid section of values.yaml. For example:
druid:
...
deepStorage:
type: s3
path: "s3://my-bucket-name/" # the bucket to use
user: AKIAIOSFODNN7EXAMPLE # the AWS Access Key ID
password: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY # the AWS Secret Access Key
...The user and password fields can be omitted if the nodes the pods in this deployment are running on have instance roles which allow them access to the bucket.
More configuration options can be found in the Druid S3 extension documentation.
New cluster defaults for HDFS can be set under deepStorage in the druid section of values.yaml. For example:
druid:
...
deepStorage:
type: hdfs
path: "hfds://hdfs.example.com:9000/druid"
...All HDFS configuration files should also be placed on the Druid classpath. Review the documentation on Adding custom user files for how to add them.
More configuration options can be found in the Druid HDFS extension documentation.
Using an existing Volume or to make a new claim of a PersistentVolume to use for deep storage allows the use of any supported volume types to be used as deep storage. The type used needs to support the ReadWriteMany access mode as well. Note that different supported types will have different levels of data durability and this should be taken into account.
To make a new claim enable the volumeClaim as shown below:
volumeClaim:
enabled: true # enable to volume claim
mountPath: "/mnt/deep-storage"
storageClassName: # leave blank to use the default StorageClass or specify
resources: {}
selector: {}For the resources and selector sections see the documentation for the StorageClass being used.
To use an existing PersistentVolume we will need to mount it to each of:
replicaCount greater than zero)Update the following sections to mount the PersistentVolume:
master/query/dataTierX:
...
extraVolumes: []
extraVolumeMounts: []
...to:
master/query/dataTierX:
...
extraVolumes:
- name: deep-storage
nfs:
server: 10.108.211.244
path: /segments
extraVolumeMounts:
- mountPath: "/mnt/deep-storage"
name: deep-storage
...The above example shows a NFS volume being mounted to the /mnt/deep-storage path. This is the same path that is used automatically by the volumeClaim if used.
To update the defaults for new clusters to use the mount we will update deepStorage in the druid section of values.yaml:
druid:
...
deepStorage:
type: local
path: "/mnt/deep-storage"
...To use other deep storage types supported by druid extensions requires overriding the druid properties. deepStorage in the druid section of values.yaml can be left with the values currently in it and then commonRuntimeProperties will contain the overrides. For example:
druid:
...
commonRuntimeProperties:
- "# GCS configurations"
- druid.storage.type=google
- druid.google.bucket=imply-storage-bucket
- druid.google.prefix=segments
- druid.indexer.logs.type=google
- druid.indexer.logs.bucket=imply-storage-bucket
- druid.indexer.logs.prefix=logs
...The above example shows an example configuration that would use GCS for deep storage and Druid index logs given that the pods have instance roles that allow them to access the GCS buckets. When using extensions that are not yet implemented in the Manager UI you will also need to ensure the extension is loaded as well. For many of the Druid extensions this is as easy a checking a box in the Manager UI for others please refer to Adding custom user files to add the extension to the Druid classpath.
By default the cluster is not accessible outside of Kubernetes. To connect to the Manager UI, Pivot or the Druid Console you will need to run a kubectl port-forward command to access it. If you want to have users access these components without using or giving them kubectl access to Kubernetes you can expose them as a Service.
Different types of services are available and you should review the Kubernetes documentation of them. In the examples below we will use a LoadBalancer service. This service type may not be available or supported by default depending on your Kubernetes configuration.
To enable the LoadBalancer service for the Manager UI update the manager.service entry in values.yaml:
manager:
...
service:
enabled: true # set to true to enable
type: LoadBalancer
port: "{{ ternary 80 443 (empty .Values.security.tls) }}" # will use port 80 if tls is not enabled, or 443 if enabled or any port specified
# nodePort:
# loadBalancerIP: set this to request a specific IP, refer to your Kubernetes provider documentation for more information
protocol: TCP # TCP or HTTP, refer to your Kubernetes provider documentation for more information
annotations: {} # provider specific annotation to customize the service
...To enable the LoadBalancer service for the Manager UI update the manager.service entry in values.yaml:
manager:
...
service:
type: LoadBalancer
routerPort: 8888 # Leave blank to not expose the router through the Service
pivotPort: 9095 # Leave blank to not expose Pivot through the Service
# routerNodePort:
# pivotNodePort:
# loadBalancerIP:
protocol: TCP
annotations: {}
...See the TLS Docs for information on how to generate certificates and general TLS information.
With Kubernetes we can use Kubernetes secrets to store our TLS information. To create the secret containing the certificate and key using the following command:
kubectl create secret tls imply-ca --key path/to/ca.key --cert path/to/ca.crtNow we will modify the values.yaml to enable TLS. Uncomment the section:
security:
...
# tls:
# secretName: imply-caYou can verify that TLS was successfully enabled by running:
kubectl logs -lapp.kubernetes.io/name=imply-manager --tail=10000And verify that TLS: Enabled shows in the logs.
See the Authentication Docs for information on enabling authentication will affect your cluster.
With Kubernetes we can use Kubernetes secrets. To create the secret containing the authentication token using the following command:
kubectl create secret generic imply-auth-token --from-literal="auth-token=supersecret"Passing the authentication token in this way to agent will leave the token in your history and is meant as an example only. Please consult the Kubernetes secrets documentation for secure ways to create the secret.
Now we will modify the values.yaml to enable authentication. Uncomment the section:
security:
...
# auth:
# secretName: imply-auth-tokenYou can verify that authentication was successfully enabled by running:
kubectl logs -lapp.kubernetes.io/name=imply-manager --tail=10000And verify that Authentication: Enabled shows in the logs.
Custom user files can be pushed by the manager to the nodes of the Imply cluster by specifying them in the Setup / Advanced Config section of the manager. These files can be written to the following locations:
/opt/imply/user)Additionally, these files can be processed as follows:
tar -x)chmod +x)There are two ways to make these files available to the manager:
manager:/// schemeTo make the files available to the manager locally, they will need to be placed in the /mnt/var/user directory inside the manager container.
Once the files are available in that directory, reference them using the manager:/// scheme. As an example:
Imply Manager will automatically attempt to create the required databases if they do not exist. There is one database for the manager application and one database for each Imply cluster. In the case that the provided user does not have authorization to create these databases, you should create them manually prior to running the application. Both Druid and the Imply Manager expect a database with a 'utf8mb4' default character set. An example statement to create the manager database:
CREATE SCHEMA `imply_manager` DEFAULT CHARACTER SET utf8mb4;