Install Imply Private on Kubernetes

Deploying Imply over Kubernetes gives you a distributed, production-ready Imply cluster that takes advantage of the deployment, scaling, and redundancy features of the Kubernetes orchestration platform.

Imply provides a Helm chart that offers a configuration-driven approach for defining, deploying, and scaling the Imply cluster on Kubernetes.

Imply works with several Kubernetes engines, including Rancher, Google Kubernetes Engine (GKE), Azure Kubernetes Services (AKS), and Amazon EKS.

Getting started

There are several ways to get started with Imply and Kubernetes. If you are new to both, and just want to kick the tires, the Imply quickstart takes you through the steps for installing a single machine Kubernetes cluster using Minikube. This is suitable for learning use cases only. For a production-ready cluster, Imply recommends a hosted Kubernetes engine, and has instructions for deploying Imply with following providers:

After deploying Imply with Kubernetes, you can find information about scaling and managing your cluster below.

Google Kubernetes Engine 1.13.11
Amazon Elastic Kubernetes Service 1.13.10
Azure Kubernetes Service 1.13.12
Rancher 2.0.12

Newer versions of Kubernetes are supported, but may not have been specifically validated.

Additionally, using the Imply Helm chart requires:

Helm 2.16.0 or 3.2.0, or later

We strongly recommend deploying Imply Manager on a Linux host to avoid virtualization overhead. Running the manager on an OSX host is suitable for a quickstart environment but is not recommended for production deployments. Windows hosts are not supported.

Adapting the Imply Manager Helm Chart

The Imply quickstart describes how to install a single-machine Imply cluster with Minukube. The Imply Helm chart introduced in the quickstart can serve as a good starting point for creating a Helm chart suitable for your own environment.

If you have not followed the steps for installing Imply on Minikube in the Imply quickstart, you will need to set up the Imply Helm repository before proceeding. Run the following commands to add the Imply repository:

helm repo add imply https://static.imply.io/onprem/helm

helm repo update

Fetch the configuration file for the latest version of the chart by running:

helm show values imply/imply > values.yaml

After modifying the chart, you can update the deployed release by running:

helm upgrade {releaseName} imply/imply -f values.yaml

The druid section of the values configuration file allows you to set default settings for new clusters. The first time you deploy a chart, these values are used in the initial cluster creation.

By default, when you run the helm upgrade command, changes in the druid section are not applied to existing clusters; they apply as default values when creating new clusters. The configuration of existing clusters can be configured in the Manager UI.

Alternatively, you can use the update flag (experimental) to have any modified values in the druid configuration section applied to existing clusters. Note that druid configuration updates to existing deployments works for single-cluster deployments only.

The update flag takes these values:

disabled – The default, directs the changes represented in the modified values in the druid section to be applied to new clusters only.
hard – Updates an existing cluster through a hard upgrade; that is, nodes are shut down, resulting in possible service interruption.
rolling – Updates an existing cluster through a rolling upgrade.

See related comments in values.yaml for more information.

Scaling out the cluster

Adding additional master/query/data nodes

The default deployment deploys one of each master, query, and data nodes. To increase the number of nodes running in the cluster we can modify their replicaCount in values.yaml.

For example, to modify the master replica count to be 3 we would change the following section:

master:
  replicaCount: 1

to:

master:
  replicaCount: 3

Increasing the disk available on the data nodes

To increase the disk available to data nodes we can either increase the size of the persistent volume we are requesting or add additional volume requests and span the Druid segment cache across them. Additional volumes can take advantage of multiple volumes on a node.

It is recommended to carefully consider the size allocated to data nodes before going to production as increasing the size of a volume is not supported by all storageClass implementations.

Increasing the size of volume claims for StatefulSet is currently out of the scope of this documentation. Depending on the storageClass used this could be hard to change. See https://github.com/kubernetes/kubernetes/issues/68737 for more information.

Increasing requested volume size

To make a larger initial request of volume size modify the following section in values.yaml:

dataTier1:
  ...
  segmentCacheVolume:
    storageClassName:
    resources:
      requests:
        storage: 40Gi
  ...

The storage size should be adjusted to the desired size.

Configure Druid with this setting by updating the historical runtime properties in values.yaml:

druid:
  historicalRuntimeProperties:
    - 'druid.segmentCache.locations=[{"path": "/mnt/var", "maxSize": 40000000000}]'
    - 'druid.server.maxSize=40000000000'

Keep in mind that this size is the amount of storage used for segment cache only. There should be about 25 GB of additional space available for other internal processes such as temporary storage space for ingestion, heap dumps, and log files.

See the Druid configuration documentation for more information.

Adding another volume claim

To add another volume claim we should add the claim information to the extraVolumeClaimTemplates section in values.yaml:

dataTier1:
  ...
  extraVolumeClaimTemplates:
    - metadata:
        name: var2
      spec:
        accessModes:
          -ReadWriteOnce
        resources:
          requests:
            storage: 20Gi
  ...

Once we have added the additional volume claim we want to make sure that the data pods mount this volume. Update the extraVolumeMounts section in values.yaml:

dataTier1:
  ...
  extraVolumeMounts:
    - mountPath: "/mnt/var2"
      name: var2
  ...

The name in the extraVolumeMounts section should match the name from the extraVolumeClaimTemplates section and should not be var or tmp as those volume names are used by the default segmentCacheVolume and tmpVolume. To have Druid use this new volume we will have to make sure it knows about it by updating the historical runtime properties in values.yaml:

druid:
  historicalRuntimeProperties:
    - 'druid.segmentCache.locations=[{"path": "/mnt/var", "maxSize": 40000000000}, {"path": "/mnt/var2", "maxSize": 20000000000}]'
    - 'druid.server.maxSize=60000000000'

This adds /mnt/var2 as another available location to cache segments and sets druid.server.maxSize to the combined size of the two locations. See the Druid configuration documentation for more information.

Adding data tiers

Adding additional data tiers can be completed much the same way as scaling. By default one data node is created in the default tier (dataTier1). To create a second data tier change the replicaCount for the dataTier2 to the desired count. All configuration options available from the default tier are available to both dataTier2 and dataTier3.

More information on Druid data tiers can be found in the Druid multitenancy documentation.

Adding clusters

When adding an additional cluster the first step is getting the name of the original cluster deployment. Using the command helm list a list of the currently deployed releases.

Now we will create a values file for this cluster by running:

helm show values imply/imply > cluster-two.yaml

Open cluster-two.yaml with your favorite editor.

Update the deployments section to disable everything but the agents deployment, as our second cluster will reuse those resources, for example:

...
deployments:
  manager: false
  agents: true

  zookeeper: false
  mysql: false
  minio: false
...

Next we will set the managerHost value to point to the Manager service defined in the existing deployment and set a name for the new cluster, for example:

...
agents:
  managerHost: wanton-toucan-imply-manager-int
  clusterName: cluster-two
...

In this example wanton-toucan is the release name of the deployment we found using helm list.

First, make sure that your Kubernetes environment has enough capacity to accommodate the second Druid Cluster, as defined by the cluster-two.yaml settings. Then deploy the second cluster by running:

helm install -f cluster-two.yaml cluster-two imply/imply

Any changes to the druid settings in the second cluster will not change the defaults. These values should be modified in the Manager. Only changes to the master/query/dataTierX will take effect.

Using an external metadata storage DB

To use an external database for metadata storage, disable the default database and configure the connection to the external database.

Disable the default database by editing values.yaml and updating the deployments section as follows:

deployments:
  manager: true
  agents: true

  zookeeper: true
  mysql: false # this line should be updated to false to disable
  minio: true

Now we will need to specify the connection details for the external database. Ensure that this database will be reachable from the pods in the deployment. See the following sections on configuring MySQL or PostgreSQL.

If your cluster is already running the data will not be migrated from the default MySQL deployment to your external database. This will have to be completed manually if required.

Using an external MySQL DB

The Imply Manager application and the Druid clusters require a MySQL database, version 5.7.21 or later.

To use an external MySQL database, update the metadataStore section for both manager and druid, for example:

manager OR druid:
  ...
  metadataStore:
    type: mysql
    host: mysql.example.com
    port: 3306
    user: root
    password: imply
    database: imply-manager # for the manager section only
  ...

The database name should not contain any spaces or dashes.

Using an external PostgreSQL DB

To use an external MySQL database, update the metadataStore section for both manager and druid, for example:

manager OR druid:
  ...
  metadataStore:
    type: postgresql
    host: postgresql.example.com
    port: 5432
    user: root
    password: imply
    database: imply-manager # for the manager section only
  ...

The database name should not contain any spaces or dashes.

Enabling TLS

To enable TLS validation, add the TLS certificate to the metadataStore section, for example:

manager OR druid:
  metadataStore:
    ...
    tlsCert: |
      -----BEGIN CERTIFICATE-----
      ...
      -----END CERTIFICATE-----
    ...

Zookeeper

A Zookeeper deployment is included in the Helm chart. It is configured by default in a quickstart configuration that is not recommended in production. For production deployments it should either be configured as highly available or to use an existing highly available Zookeeper ensemble.

Check the Zookeeper Documentation for more information on clustered Zookeeper ensembles.

Configuring a highly available ensemble

To increase the number of Zookeeper nodes we can adjust the replicaCount in the zookeeper section of values.yaml. For example:

...
zookeeper:
  replicaCount: 1 # Change this value
...

Zookeeper works best with an odd number of nodes and more nodes allows for more fault tolerance. Setting the replicaCount to 3 or higher odd number will enable a highly available Zookeeper ensemble.

Using an external Zookeeper ensemble

If using an external Zookeeper ensemble we can disable the default Zookeeper deployment. To disable, edit the values.yaml and update the deployments section:

deployments:
  manager: true
  agents: true

  zookeeper: false # this line should be updated to false to disable
  mysql: true
  minio: true

Now we will need to specify the connection details for the external database. Ensure that this database will be reachable from the pods in the deployment. We can now update the new cluster defaults to use the external Zookeeper ensemble:

druid:
  ...
  zk:
    connectString: "zookeeper.example.com:2181"

If your cluster is already created the Manager UI should be used to point the cluster to the external Zookeeper ensemble.

Deep storage

MinIO is not intended for production use and is provided for a quickstart configuration only. For information on the deep storage types supported by Druid, as well as how to choose one, see the Druid Deep storage documentation.

To use an external deep storage system, disable the default storage, MinIO, and configure the connection to the external system.

Disable the default by editing values.yaml and updating the deployments section as follows:

deployments:
  manager: true
  agents: true

  zookeeper: true
  mysql: true
  minio: false # this line should be updated to false to disable

When using a named deep storage type, the required settings as well as authentication information and index logs are set automatically. The Druid extension is added to the classpath automatically as well. If these settings or any others need to be changed, add them to commonRuntimeProperties in the druid section of values.yaml. If no additional settings are required, remove the MinIO overrides, as follows.

Update this section:

druid:
  ...
  commonRuntimeProperties:
    - "# MinIO configurations"
    - "druid.s3.endpoint.url=http://{{ .Release.Name }}-minio:9000"
    - "druid.s3.enablePathStyleAccess=true"

To the following:

druid:
  ...
  commonRuntimeProperties: [] # for no settings, or just remove the s3 overrides for MinIO

If your cluster is already created, use the Manager UI to configure the external deep storage connection.

To configure the deep storage connection, follow the instructions for the type of system you want to use:

Amazon S3
Microsoft Azure
Google Cloud Storage
HDFS
NFS and K8S Supported Volumes

S3

To use Amazon S3 as the deep storage mechanism, set the deepStorage configuration in the druid section of values.yaml as follows:

druid:
  ...
  deepStorage:
    type: s3
    path: "s3://my-bucket-name/" # the bucket to use
    user: AKIAIOSFODNN7EXAMPLE # the AWS Access Key ID
    password: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY # the AWS Secret Access Key
  ...

The user and password fields can be omitted if the nodes the pods in this deployment are running on have instance roles which allow them access to the bucket.

More configuration options can be found in the Druid S3 extension documentation.

Azure

To use Microsoft Azure for deep storage, configure the deepStorage type in the druid section of values.yaml as follows:

druid:
  ...
 deepStorage:
    type: azure
    path: <container/optional-prefix>
    user: <azure-service-account-name>
    password: <azure-account-shared-key>
  ...

The path you specify will be prepended by azure://, so a path value of druid would result in the full path of azure://druid. Accordingly, the Druid segment location will be azure://druid/<partial_cluster-id>/segment

For the Azure shared account key to use, see Authorize with Shared Key in the Azure documentation.

For more configuration options, see the Druid Azure extension documentation.

GCS

To use Google Cloud Storage for deep storage, configure the deepStorage type in the druid section of values.yaml as follows:

druid:
 ...
 deepStorage:
    type: google
    path: <bucket/path>
    user: 
    password:
      {
        "type": "service_account",
        "project_id": "project-id",
        "private_key_id": "key-id",
        ...
      }
 ...

The path you specify will be prepended by gs://, so a path value of druid would result in a full path gs://druid. Accordingly, the Druid segment location will be gs://druid/<partial_cluster-id>/segment.

The user value can be left empty, as shown here, since it isn't relevant to the GCS connection configuration.

A Google service account key is only required if you are running outside of GCS or the service account attached to the VM does not have read/write access to the bucket.

For the Google service account key to use, see Creating and managing service account keys in the Google Cloud documentation.

For more configuration options, see the Druid Google Cloud Storage extension documentation.

HDFS

New cluster defaults for HDFS can be set under deepStorage in the druid section of values.yaml. For example:

druid:
  ...
  deepStorage:
    type: hdfs
    path: "hfds://hdfs.example.com:9000/druid"
  ...

All HDFS configuration files should also be placed on the Druid classpath. Review the documentation on Adding custom user files for how to add them.

For more configuration options, see the Druid HDFS extension documentation.

NFS and K8S Supported Volumes

Using an existing Volume or making a new claim of a PersistentVolume to use for deep storage allows the use of any supported volume types to be used as deep storage. The type used needs to support the ReadWriteMany access mode as well. Note that different supported types will have different levels of data durability and this should be taken into account.

To make a new claim, enable the volumeClaim as shown below:

volumeClaim:
  enabled: true # enable to volume claim
  mountPath: "/mnt/deep-storage"
  storageClassName: # leave blank to use the default StorageClass or specify
  resources: {}
  selector: {}

For the resources and selector sections, see the documentation for the StorageClass being used.

To use an existing PersistentVolume, we will need to mount it to each of:

master
query
dataTierX (only required for dataTiers with replicaCount greater than zero)

Update the following sections to mount the PersistentVolume:

master/query/dataTierX:
  ...
  extraVolumes: []
  extraVolumeMounts: []
  ...

to:

master/query/dataTierX:
  ...
  extraVolumes:
    - name: deep-storage
      nfs: 
        server: 10.108.211.244
        path: /segments
  extraVolumeMounts:
    - mountPath: "/mnt/deep-storage"
      name: deep-storage 
  ...

The above example shows a NFS volume being mounted to the /mnt/deep-storage path. This is the same path that is used automatically by the volumeClaim if used.

To update the defaults for new clusters to use the mount, update deepStorage in the druid section of values.yaml:

druid:
  ...
  deepStorage:
    type: local
    path: "/mnt/deep-storage"
  ...

Accessing the cluster via Services

By default the cluster is not accessible outside of Kubernetes. To connect to the Manager UI, Pivot, or the Druid Console you will need to run a kubectl port-forward command to access it.

If you want to have users access these components without using or giving them kubectl access to Kubernetes, you can expose them as a Service.

Different types of services are available; you should review the Kubernetes documentation to learn more. In the following examples, we will use a LoadBalancer service. This service type may not be available or supported by default, depending on your Kubernetes configuration.

Manager

To enable the LoadBalancer service for the Manager UI, update the manager.service entry in values.yaml:

manager:
  ...
  service:
    enabled: true # set to true to enable
    type: LoadBalancer
    port: "{{ ternary 80 443 (empty .Values.security.tls) }}" # will use port 80 if tls is not enabled, or 443 if enabled or any port specified
    # nodePort:
    # loadBalancerIP: set this to request a specific IP, refer to your Kubernetes provider documentation for more information
    protocol: TCP # TCP or HTTP, refer to your Kubernetes provider documentation for more information
    annotations: {} # provider specific annotation to customize the service
  ...

Query nodes

To enable the LoadBalancer service for the cluster's query nodes, update the query.service entry in values.yaml:

query:
  ...
  service:
    type: LoadBalancer
    routerPort: 8888  # Leave blank to not expose the router through the Service
    pivotPort: 9095   # Leave blank to not expose Pivot through the Service
    # routerNodePort:
    # pivotNodePort:
    # loadBalancerIP:
    protocol: TCP
    annotations: {}
  ...

Security

TLS

See the TLS Docs for information on how to generate certificates and general TLS information.

With Kubernetes we can use Kubernetes secrets to store our TLS information.

Create the secret containing the certificate and key using the following command:

kubectl create secret tls imply-ca --key path/to/ca.key --cert path/to/ca.crt

Now modify the values.yaml to enable TLS by uncommenting the section:

security:
  ...
  # tls:
  #   secretName: imply-ca

You can verify that TLS was successfully enabled by running:

kubectl logs -lapp.kubernetes.io/name=imply-manager --tail=10000

And verify that TLS: Enabled appears in the logs.

Authentication

See the Authentication Docs for information on how enabling authentication will affect your cluster.

With Kubernetes we can use Kubernetes secrets.

Create the secret containing the authentication token using the following command:

kubectl create secret generic imply-auth-token --from-literal="auth-token=supersecret"

Passing the authentication token in this way to the agent will leave the token in your history and is meant as an example only. Consult the Kubernetes secrets documentation for secure ways to create the secret.

Now we will modify the values.yaml to enable authentication. Uncomment the section:

security:
  ...
  # auth:
  #   secretName: imply-auth-token

You can verify that authentication was successfully enabled by running:

kubectl logs -lapp.kubernetes.io/name=imply-manager --tail=10000

And verify that Authentication: Enabled shows in the logs.

Adding custom user files

Custom user files can be pushed by the Manager to the nodes of the Imply cluster by specifying them in the Setup / Advanced Config section of the manager. These files can be written to the following locations:

User files (/opt/imply/user)
Druid's classpath
as a Hadoop dependency

Additionally, these files can be processed as follows:

None
Requires unpacking (extracts using tar -x)
Is executable (sets chmod +x)

There are two ways to make these files available to the Manager:

Make them available via HTTP(S) and provide the URL
Make them available to the Manager locally and reference them using the manager:/// scheme

To make the files available to the Manager locally, they will need to be placed in the /mnt/var/user directory inside the manager container. With Kubernetes, the easiest way to do this is to mount a volume to that directory in the container image and put the custom files on that volume.

After making the files available in that directory, reference them from the UI using the manager:/// scheme, for example:

File: /mnt/var/user/hdfs-site.xml -> Manager path: manager:///hdfs-site.xml

Manually creating the metadata databases

Imply Manager automatically attempts to create the databases it needs if they do not exist. There is one database for the Manager and one database for each Imply cluster.

If the provided user does not have authorization to create these databases, you should create them manually prior to running the application. Both Druid and the Imply Manager expect a database with a 'utf8mb4' default character set.

An example statement to create the manager database is as follow:

CREATE SCHEMA `imply_manager` DEFAULT CHARACTER SET utf8mb4;

Updating the Imply Manager software

Helm makes it relatively easy to update the Manager and agent software.

To do so, simply make sure the repository is up to date and run the Helm upgrade command:

helm repo update
helm upgrade -f optional-overrides-file.yaml deployment-name imply/imply

If you have customized your deployment chart, you may need to merge your changes to the updated Imply chart before running the command.

Kubernetes applies the upgrade in a rolling manner, allowing you to avoid downtime.

4.0

Imply Managed

Imply Private