Enhanced Imply Private on Google Kubernetes Engine is a Beta feature. For more information, see Experimental features.
This document describes how to deploy and manage Imply Private on GKE using the enhanced installation mode. This solution offers users the simplest set up and management process when running Imply on Google Cloud Platform.
Google Cloud components and sample architecture
The following diagram shows the resulting resources that are created in this deployment mode.
As shown the following resources will be created within a given Google Cloud project
- VPC - The network that resources will communicate over. An existing VPC can be used, or Imply will create one for you.
- Public GKE cluster - The Kubernetes cluster which within all compute resources will run. Several Node pools will also be created for the cluster to use when deploying Druid deployments. An existing cluster cannot be used, the cluster will be created by Imply.
- Cloud SQL instance - The data store used as the default metadata store for Druid deployments, as well as for the Imply control plane. An existing Cloud SQL instance can be used, or Imply will create one for you.
- Cloud Storage bucket - The storage bucket used as the default Deep Storage for all druid deployments. An existing cloud storage bucket can be used, or Imply will create one for you.
- Load balancer - Used to access the Imply control plane.
Google Cloud machine types
- n2-standard-2: 2 vCPU / 8 GB RAM
- n2-standard-8: 8 vCPU / 32 GB RAM"
- n2-standard-16: 16 vCPU / 64 GB RAM
- c2-standard-4: 4 vCPU / 16 GB RAM
- c2-standard-8: 8 vCPU / 32 GB RAM
- c2-standard-16: 16 vCPU / 64 GB RAM
- c2-standard-30: 30 vCPU / 120 GB RAM
- n2-highmem-4: 4 vCPU / 32 GB RAM / 750 GB disk
- n2-highmem-8: 8 vCPU / 64 GB RAM / 1500 GB disk
- n2-highmem-16: 16 vCPU / 128 GB RAM / 3000 GB disk
- n1-highmem-32: 32 vCPU / 208 GB RAM / 6000 GB disk
- n2-highmem-48: 48 vCPU / 384 GB RAM / 9000 GB disk
- n2-highmem-80: 80 vCPU / 640 GB RAM / 3000 GB disk
Before proceeding please ensure the following requirements are fulfilled.
GCP Resource Requirements
You will need the following Google Cloud resources
- A Google Cloud account
- A Google Cloud project
- A GCP User with sufficient permissions
- Option 1. User with
- Option 2. A minimally privileged User with all the permissions of the following GCP Predefined Roles.
- Cloud SQL Admin
- Compute Network Admin
- Kubernetes Engine Admin
- Storage Admin
- DNS Administrator (if configuring ingress)
- Option 1. User with
- A Monitoring workspace containing the respective project. Allow Google to create the workspace with the same name as your project.
Ensure that the following APIs are enabled prior to execution:
- Compute Engine API
- Kubernetes Engine API
- Service Networking API
- Cloud DNS API (if configuring ingress)
To enable these APIs please navigate to the
APIs & Services section of the Google cloud Web Console, search for them, and enable them.
The resources are deployed by a setup script which depends on the following tools:
These tools are included by default in the Google cloud shell, and for that reason we recommend running the setup script from there.
The Enhanced Imply Manager installer requires Internet connectivity to complete the installation process. After installation, the Imply cluster will require Internet access as well, to download assets from https://hub.docker.com and https://static.imply.io, for example.
If your policies prevent such access, we recommend using Imply Manager on Kubernetes instead.
Creating an Imply Deployment
The Imply GKE setup script guides you through the deployment process. To download the setup script, run the command
curl -O https://static.imply.io/onprem/setup. We encourage users to run the setup script in the Google cloud shell, as it contains all the dependent tools that the setup script needs, by default.
Authorize with Google Cloud
Before starting, you need to authorize gcloud to access the Cloud Platform with Google user credentials. To do so, run the command
gcloud auth login and follow the prompts shown. Ensure that you authorize as a user with sufficient permissions.
If you are running the setup script from the Google cloud shell, you are prompted to authorize cloud shell when you start the script.
Start the Script
To start the script, run
sh setup. You will see output similar to the following:
Welcome to the Imply GCP Installer. (build 50aa6b9) This installer will guide you through setting up the required Google Cloud Platform resources for running a managed Imply install in GCP on Kubernetes.
Next you will be prompted to select the Google project that you would like to work in.
1) golden-bonbon-283506 2) pure-episode-234323 Select the Google Project ID [pure-episode-234323]:
Enter the number corresponding to the project that you want to use and press enter, or press enter with no value to use the default value in brackets.
Next you will be prompted to enter a license key.
Enter your Imply License (blank for trial):
Enter your Imply license here and press enter, or press enter with no value to use a trial license which allows you to use the product for 30 days. If using a trial license, you can update the deployment later with a replacement license if needed.
Next you will be prompted to enter the region for the deployment. This is the region that Imply will be deployed to.
1) asia-east1 2) asia-east2 3) asia-northeast1 4) asia-southeast1 5) australia-southeast1 6) europe-north1 7) europe-west1 8) europe-west2 9) europe-west3 10) europe-west4 11) northamerica-northeast1 12) southamerica-east1 13) us-central1 14) us-east1 15) us-east4 16) us-west1 Select a region [us-central1]:
Enter the number corresponding to the region that you want to use for the deployment, and press enter.
Select HA or Non-HA Deployment
Next you will be prompted whether to deploy the resources across multiple availability zones in the region. When deployed across multiple availability zones, Imply is more resilient to failures in any one particular zone.
Running in multiple zones will create all resources with redundancies in multiple GCP zones. Resources such as optionally created MySQL and the Kubernetes cluster will have redundancy configured across these zones. This is recommended in production. Run in multiple zones? [yes]:
Press enter to accept the default of using multiple zones, or enter no for a single zone deployment, and press enter. We recommend using multiple zones.
Next you will be prompted whether to create a new VPC to use, or to use an existing VPC, and will be asked details about the CIDR block to use for the deployment. This is the network that the Imply deployment will use when deploying resources.
Creating a separate VPC is recommended but if you require the Imply cluster to run in an existing VPC to be able to access resources that cannot otherwise be access you can select an existing VPC. Note: When using an existing VPC please ensure that private service networking is enabled to allow the Imply cluster to communicate with GCP services. Create a new VPC [yes]: CIDR Range for the Imply Cluster to use [10.128.0.0/16]:
Press enter to take the default of creating a new VPC, and you can take the default CIDR range specified, or specify your own. If you want to use an existing VPC, enter
no and follow the prompts. Ensure that the CIDR range that you specify in this case is available for use.
Select Deep Storage Bucket
Choose whether to create a Deep Storage bucket or to use an existing bucket. This bucket is used as the default deep storage for all Imply deployments.
Choose a GCS bucket to store your data (deep storage). If you type 'New' (or leave the input blank) a new bucket will be created for you. If you wish to use an existing bucket, use the form: gs://my-bucket-1. Note: If a bucket is created for you, it will be removed on uninstall. ALL DATA WILL BE REMOVED AS WELL. Existing GCS bucket or New [New]:
Press enter to take the default of creating a new storage bucket, or enter
no to use an existing bucket, and follow the prompts to give details to identify the bucket.
Select Metadata Store
Choose whether to create a Cloud SQL store for storing metadata about your Imply deployment or to use an existing database.
Choose a MySQL database to serve as the metadata store. It is recommended that you create a new database for use by Imply. If you require Imply to use an existing MySQL database, you may enter those details here instead. Note: Your SQL connection information may be stored in plaintext in a GCS bucket in your account by Terraform. Note: If a database is created for you, it will be removed on uninstall. ALL DATA WILL BE REMOVED AS WELL.
Press enter to take the default of creating a new Cloud SQL instance, or enter
no to use an existing database, and follow the prompts to give details to identify the database.
About Cloud SQL sizing
Imply enables disk autoscaling for this metadata store instance. Disk autoscaling is a GCP feature described in Automatic storage increase of the Google Cloud documentation.
The Imply configuration does not itself impose a limit on the disk size. However, in practice, the disk size is limited by the size of the default instance type used by Imply for the metadata store, which is db-n1-standard-2.
The disk size automatically starts scaling up in 6 GB increments initially, and scales up linearly to 25 GB increments for disk sizes above 500 GB. The increment sizes determine when scaling is needed, that is, when the amount of free space reaches 6 GB or less, the autoscaling system scales up disk by 6 GB.
Note that storage does not scale down, other than by manually changing the underlying instances.
Choose whether to set up GKE Ingress for the Imply cluster, allowing secure connections to the Imply Manager. This requires that you have at least one Cloud DNS managed zone configured.
Ingress can be automatically configured using Cloud DNS and Google Managed Certificates to allow secure connections to the Imply Manager. This feature requires that a Cloud DNS managed zone already exist. For more information on setting up a Cloud DNS zone see: https://cloud.google.com/dns/docs/zones Note: Security Policy changes may take a few minutes to be reflected and access to the Imply Manager may be interrupted. It will not affect any running clusters. Automatically setup ingress? [no]: yes Determining available Cloud DNS Managed Zones 1) gcp-imply-io Select a zone: 1
If you do not have Cloud DNS managed zones configured, press enter to take the default selection,
no, as this feature is not supported without Cloud DNS managed zones. Otherwise, to use this feature, enter
yes and enter the number corresponding to the Cloud DNS zone that you want to use for the deployment, and press enter. This results in the control plane being hosted in a subdomain of the Cloud DNS zone that you selected.
Attach CloudArmor policy
Next choose whether to have a CloudArmor Policy applied to the services in the cluster. If you select yes, you are shown a list of existing CloudArmor policies. Select one of them to apply.
A Cloud Armor Security Policy can be attached to the Ingress that will be created. This feature requires that a Security Policy already exist. For more information on Security Policies see: https://cloud.google.com/armor/docs/configure-security-policies Configure a Security Policy? [no]: yes Determining available Security Policies 1) imply-sp Select a policy: 1
Next review and confirm the details of the deployment. If correct, press enter and proceed with the deployment. Otherwise, enter
no to restart the script.
Wait for deployment to complete
The deployment will take around 30 minutes. Wait for it to finish deploying resources. When deployment finishes, you will receive a message similar to the following.
helm_release.manager: Creation complete after 3m3s [id=imply] Apply complete! Resources: 32 added, 0 changed, 0 destroyed Outputs: bucket = cidr = 10.128.0.0/16 dns_host = imply.gcp.imply.io dns_zone = gcp-imply-io gke_id = imply-gke license = project_id = pure-episode-234323 region = us-central1 sql_endpoint = sql_password = sql_username = vpc = zone = You can access your Imply Manager at: https://imply.gcp.imply.io Note: It may take 15-20 minutes for the SSL certificate to be generated and during that time you may see a certificate error
The details will include instructions for how to access your deployment.
Create an Imply Manager admin user
Use the access instructions given at the end of the setup script to access your deployment. Please refer to Imply Manager users for how to add users.
Please refer here for how to manage your Imply clusters.
Connect to your cluster
Please refer here for how to connect to your cluster.
Loading data into your cluster
This will take you through an example of loading data into your cluster from a Google Cloud storage bucket. Other ingestion methods are available. Please refer to the Druid ingestion documentation for other ingestion types.
By default, the service accounts that are being used on the Druid service pods only have access to read and write data to the deep storage bucket configured. In order to ingest data from another storage bucket, you will need to explicitly give the agent service account the required permissions needed to access that bucket. Please see Google's IAM documentation. The agent service account is named
Give Service Account Permission to Data Bucket
First you will need to give the service account access to the resource from which you would like to ingest data. In this example, we will be ingesting data from Google Storage Bucket
Use the Google Cloud Web Console to navigate to your service account, we'll need to look up the email address of the agent service account. Find the agent service account, which is named
imply-default-asa, and copy the email address for the account.
In the Google Cloud web console, navigate to the bucket details for the bucket that you would like to ingest data from, and click the permissions tab.
Add the service account with role
storage admin using the service account email.
Now the service account has the permission needed to read and write to the bucket.
Ingest Data Using Druid Web Console
Navigate to the Druid Web Console, click
new ingestion spec and choose the
Google Cloud Storage ingestion type.
Follow the on-screen instructions for providing the path to the data that you want to ingest, and any filtering or aggregation options that you'd like to apply.
You can also download some sample data to try out:
Upload the file to your Google Cloud bucket and provide the path to the file inside the Druid console.
After the data is ingested successfully, you will see a corresponding ingestion task listed in the
Ingestion tab with status,
Once Druid loads all of the segments data, the corresponding datasource will show as
Fully available under the
Query the Data
Now the datasource is ready to be queried. Please see the query tutorial documentation.
Updating an Imply Deployment
To update an existing Imply deployment, run the setup script as described in Creating an Imply Deployment. Give the selection for the deployment that you'd like to update, and when asked if you'd like to update, enter
Select the Google Project ID [zachsherman]: 1) New 2) default Select existing deployment to manage or New to create one [New]: 2 An existing deployment was found, update? [yes]:
You will be given the details of the deployment, and prompted whether the details are correct. If you are using an updated version of the script, and don't want to make any explicit changes to your deployment, enter
yes. If you would like to make explicit changes to your deployment enter
License: Project ID: zachsherman Region: us-central1 Zone: [Multizone] VPC: [Created] CIDR: 10.128.0.0/16 Bucket: [Created] MySQL: [Created] Ingress: no Is the above information correct [yes]:
If you entered
no, from there, you will be permitted to modify the deployment. Follow the steps as described earlier in this document.
Deleting an Imply Deployment
To delete an existing Imply deployment, run the setup script as described in Creating an Imply Deployment. Give the selection for the deployment that you'd like to update, and when asked if you'd like to update, enter
no. You will then be given the option to uninstall the deployment.
Follow the prompts from here.
The following section describes advanced use cases of the deployment.
Direct access Pivot
Direct access Pivot is supported in GKE Enhanced but has some differences in how it works in Imply Cloud. All of the configuration options are the same, however, so reviewing the Direct access Pivot docs is a good first step.
The main difference is in how the URL for Pivot access field interacts with the Pivot configuration and underlying Kubernetes resources. For example, setting the field to a name such as
pivot.company.com at installation time will result in an Ingress that points to Pivot on port 80 of the Nginx Ingress Controller. This means that once DNS has been configured to point to the public or private load balancers, you can use the name provided.
Using the same domain as configured to access your Imply Manager without a URL path could render your Manager inaccessible.
If your URL for Pivot access field contains a path as well (e.g.,
imply.mycompany.com/10257016-8bfd-433a-9e98-f62dd23bce1e/pivot) then Pivot will be automatically configured to use the serverRoot from the path configured. This means that if you had configured an Ingress during installation with a domain name, you could use a path to enable routing with the same domain and certificate that have already been configured. The easiest way to configure this would be to copy the public load balancer path and append
/pivot to the end of it, as in the example above.
If the field is configured with a different domain than the Manager, it will still work, but you will need to configure the domain and TLS certificate separately. To configure this from the GCP Console, select Network Service, then Load balancing. You can choose whether to make it available publicly on the internet or only within your VPC. From there you will want to configure the Backend configuration. If you have many services available, it may be hard to determine the correct one to select. To figure out the correct service in a new GCP Console tab, select Kubernetes Engine, Services & Ingresses, and then select the Imply cluster named
imply-default-gke, by default. Then select the
ingress-nginx-controller. On the Details page under Annotations, you should see it displayed as the Network Endpoint Group name. It should look something like
k8s1-31d25674-default-ingress-nginx-controller-80-c05a40bc, which you can select in the Backend configuration screen.
Once that is completed, select the Frontend configuration. Here you can configure options like reusing an existing TLS certificate or having one generated, as well as SSL policies. Review the Google Cloud Load Balancing documentation for a full list of configuration options.
Note that when accessing Direct access Pivot any configured CloudArmor policy will automatically apply as well.
The setup script can be used to create multiple Imply deployments. If you do not currently have any deployments, you will not be prompted for a deployment name, and instead a name of
default will be automatically given for your first deployment. If you have at least one deployment, you will be prompted to select which deployment to use or can create a new deployment at this time
Select the Google Project ID [pure-episode-234323]: 1) New 2) default Select existing deployment to manage or New to create one [New]:
If using a non-default deployment, the name of the agent service account will be
The following section goes over some common issues that you may face in the field and how to resolve them.
Deployment Terraform State Locked
The setup script uses terraform to create, update, or delete resources in Google Cloud for the Imply deployment. If an error occurs while terraform is in the process of modifying resources, such as the user's terminal window being closed, the next time the setup script is run, terraform will complain that it was not able to acquire a lock on the state for the deployment. Such a message would look something like the following:
To resolve this situation, ensure that no other operations are being performed on the deployment, and unlock the state file using terraform force-unlock. If this fails, delete the lock file from the Google storage bucket noted in the message.
Cluster Create or Update Fails to Scale-up Node Pool
You may encounter an issue when creating or updating an existing cluster, where a failure occurs because a particular node pool fails to scale to the required number of nodes, as shown below:
This can be caused by quota limits in the account being hit. To fix this issue, you will need to increase the corresponding quota.