Managing Imply Clusters
An Imply cluster consists of the processes that make up Druid, including data servers, query servers, and master servers, along with Pivot.
The way you create and scale a cluster differs depending on how you are running Imply. In an Imply Cloud installation, you can add clusters from the Imply Manager home page, as described in the Quickstart.
With Kubernetes, you can define the cluster to be created using a Helm chart. Servers are automatically added to the cluster when the Kubernetes scheduler creates new pods. For more information on cluster management with Kubernetes, see Deploy with Kubernetes.
Create a cluster
Imply Manager users with the ManageClusters permission can create clusters.
To create a cluster, from the Imply Manager home page, click the New cluster button. The new cluster configuration page appears, as follows:
Configure the general settings and choose the instance types for the cluster.
For a highly available cluster, you must have at least 3 master servers. For more high availability information, see Druid High availability.
In the Advanced config section, default settings appear for deep storage and metadata store settings. Zookeeper settings are configurable as well in Imply Private deployments. They are not available for modification in Imply Cloud.
For Imply Private, you can enter Druid server configuration settings in the Server properties fields. Available properties are in Druid configuration reference. In Imply Cloud and GCP enhanced, appropriate values are calculated for you at cluster creation time based on your chosen instance types.
When finished, click Create cluster and confirm the operation to submit the cluster for creation. It may take fifteen to twenty minutes to start up resources for the cluster.
You can track progress from the cluster Overview page for the new cluster. For details, click the number next to messages in the STATUS section of the page. The message log appears:
When the deployment is finished, Running appears next to the cluster state in the overview page. See Viewing cluster status and capacity usage for more information about cluster status.
With Imply Cloud, behind the scenes, clusters are created by a set of Amazon CloudFormation scripts built for you by the Imply team. You shouldn't need to access or change the scripts manually, but it can be helpful to know of their operation behind the scenes, particularly if asked to access the CloudFormation scripts in your AWS account by Imply support in certain troubleshooting scenarios.
Also note that Imply Cloud by default deploys servers across multiple availability zones (AZs) in a single region. This helps to ensure that the services are highly available.
Connect to your cluster
Accessing Pivot
In Imply Cloud and Enhanced GCP, users can access Pivot by clicking Open from the cluster actions menu found on the cluster's overview page.
On Imply Private, users can access Pivot at http://localhost:9095.
Accessing Druid Web Console
Click Load Data in the Pivot window to open the Druid Web Console. See the Quickstart for information on using the Web Console.
API Access
In Imply Cloud and Enhanced GCP, to get API connection information for the cluster, from the respective cluster's overview page in the Imply Manager UI, click the API tab.
Note that if using Imply Cloud or have deployed to a cloud host, these can only be accessed from within the VPC.
For Imply Private deployments, see the Druid API reference for API access information.
Viewing cluster status and capacity usage
The cluster overview page lets you view the status and health of the cluster at a glance. The top of the page shows the current status of the cluster and the server types that make up the cluster.
The disk utilization of the cluster appears in the Used column of the clusters list. The utilization value reflects the disk usage of the cluster, or more specifically, the segment cache usage compared to the segment cache size for the cluster as a whole. This information helps you determine when you need to scale the cluster or take other remediation actions.
In considering disk usage, note that a certain amount of disk space is reserved for internal Imply operations, and is therefore shown as occupied in the disk usage graph. Specifically, about 25 GB is reserved for purposes of temporary storage space for ingestion, heap dumps, log files, and for other internal processes.
For more information about cluster status and operations, you can access the process logs from across the cluster from the Manager UI:
Updating a Cluster
Imply Manager users with the ManageClusters permission can update clusters.
To update a cluster, navigate to the Setup tab of the corresponding cluster, and change the parameters that you would like. The type of updates permissible depends on the deployment mode. The following shows the upgrade options for AWS.
Click the Apply Changes button. Before any changes are made, the manager UI informs you of the type of update that will be made.
After confirming the change, you will be brought to the cluster overview page, where you can monitor the progress of the update. Click the Changes button to see the list of changes being made in this update.
When the update finishes successfully the cluster state will show as RUNNING.
Stopping a Cluster
Imply Manager users with the ManageClusters permission can stop clusters.
To stop a cluster, navigate to the corresponding cluster's overview page, and click the Stop button. You will be prompted to confirm the operation.
Once the operation completes, the cluster state will show as STOPPED.
You can restart the cluster from the overview page by pressing the Start button
Terminating a Cluster
Imply Manager users with the ManageClusters permission can terminate clusters. Note, this operation is irreversible; once a cluster is terminated, it cannot be used again.
To terminate a cluster, navigate to the corresponding cluster's overview page, and click the Terminate button. You will be prompted to confirm the operation.
Once the operation completes, the cluster state will show as TERMINATED.