Docker makes it easy to deploy the Manager and servers that make up an Imply cluster. If you are new to deploying Imply with Docker, see the following quickstart to get started.
Imply is provided as a pair of Docker images available on Docker Hub:
The following quickstart describes how to get an Imply up and running with Docker quickly for learning and investigation scenarios.
To install Imply using Docker, you need:
We strongly recommend installing on a Linux host to avoid virtualization overhead. Running the manager on an OSX host is suitable for the quickstart, but not recommended for a production deployment. Windows hosts are not supported.
Imply Manager requires ZooKeeper and a MySQL server to operate. In this quickstart, we will run these dependencies as additional Docker containers on your local machine.
To ensure that our containers can communicate with each other, create a user-defined bridge network with the following command:
docker network create imply
Start ZooKeeper:
docker run --name zk --network imply -d --rm zookeeper:3.4.14
Start MySQL:
docker run --name mysql --network imply -e MYSQL_ROOT_PASSWORD=imply -d --rm mysql:5.7.26
Create a file named env.list
with the following contents, substituting {LICENSE_KEY}
with the license key provided by Imply:
IMPLY_MANAGER_LICENSE_KEY={LICENSE_KEY}
IMPLY_MANAGER_STORE_TYPE=mysql
IMPLY_MANAGER_STORE_HOST=mysql
IMPLY_MANAGER_STORE_PORT=3306
IMPLY_MANAGER_STORE_USER=root
IMPLY_MANAGER_STORE_PASSWORD=imply
IMPLY_MANAGER_STORE_DATABASE=imply_manager
imply_defaults_zkType=external
imply_defaults_zkHosts=zk:2181
imply_defaults_zkBasePath=imply
imply_defaults_metadataStorageType=mysql
imply_defaults_metadataStorageHost=mysql
imply_defaults_metadataStoragePort=3306
imply_defaults_metadataStorageUser=root
imply_defaults_metadataStoragePassword=imply
imply_defaults_deepStorageType=local
imply_defaults_deepStorageBaseLocation=/mnt/var/druid/storage
Start the manager container:
docker run --name imply-manager --network imply -p 9097:9097 --env-file env.list -d --rm imply/manager
Start an agent container:
docker run --network imply -p 9095:9095 -p 8888:8888 -e "IMPLY_MANAGER_HOST=imply-manager" -d --rm imply/agent
Once the manager container is running, you can access the Imply Manager in your web browser at: http://localhost:9097
.
Open your browser to that address and follow these steps:
At this point, the cluster definition has been submitted but no resources have been assigned to the cluster. You may have to wait a few seconds for the cluster to be ready, at which point you should see a link, Start by adding a server:
The Cluster Overview screen should show the cluster in an UPDATING state. The update operation may take a few minutes. Once the update is finished, the cluster's state will change to RUNNING.
Congratulations! Your cluster is now up and running.
You can now access the Imply web interfaces at:
http://localhost:8888
http://localhost:9095
See Using the Imply Manager to learn more about using the Imply Manager.
To learn about Pivot, continue with the Load data file section of the Quickstart.
Imply is provided as a pair of Docker images on Docker Hub. There is one image for the Manager container, which includes coordination services and the Manager application, and another image serves as the agent containers. Agents are instantiated as Druid cluster nodes.
Currently, a deployment can only have a single manager. Multiple managers for high availability is not currently supported.
The images are available in the Docker hub repositories here:
The recommended way to configure the manager container is to create a configuration file that can be passed to the container using the --env-file
Docker run argument. The remainder of this documentation will assume this file is named env.list
.
The following items are required:
IMPLY_MANAGER_LICENSE_KEY={license_key}
IMPLY_MANAGER_STORE_TYPE=mysql
IMPLY_MANAGER_STORE_HOST={mysql_hostname}
IMPLY_MANAGER_STORE_PORT={mysql_port}
IMPLY_MANAGER_STORE_USER={mysql_user}
IMPLY_MANAGER_STORE_PASSWORD={mysql_password}
IMPLY_MANAGER_STORE_DATABASE=imply_manager
The database name (IMPLY_MANAGER_STORE_DATABASE
) should not contain any spaces or dashes.
The Imply Manager application and the Druid clusters require an external MySQL database, version 5.7.21 or later. Ensure that this database will be reachable from the manager container and all agent containers.
You will also require a valid license key, which can be obtained by contacting a representative at Imply.
Druid clusters require:
You can provide default values for these external dependencies so that they don't need to be specified at cluster creation time.
To do so, add parameters as shown by the following example to your env.list
file. Supply values appropriate for your external systems.
imply_defaults_zkType=external
imply_defaults_zkHosts={zk-host-1}:2181,{zk-host-2}:2181,{zk-host-3}:2181
imply_defaults_zkBasePath=imply
imply_defaults_metadataStorageType=mysql
imply_defaults_metadataStorageHost={mysql_hostname}
imply_defaults_metadataStoragePort={mysql_port}
imply_defaults_metadataStorageUser={mysql_user}
imply_defaults_metadataStoragePassword={mysql_password}
imply_defaults_deepStorageType={'s3' | 'hdfs' | 'azure' | 'google' | 'local'}
imply_defaults_deepStoragePath={storage_path} # e.g., "s3://{bucket}/{prefix}", azure://{container_name}/{prefix}, OR
# gs://{bucket}/{path}, etc
imply_defaults_deepStorageUser={account_name} # i.e., {s3_access_key} OR {azure_account_name}
imply_defaults_deepStoragePassword={account_password} # i.e., {s3_secret_key} OR {azure_password} OR {google_json_key}
The agent containers only require a single configuration - setting the environment variable IMPLY_MANAGER_HOST
to the hostname of the manager container. The agent will register with the manager and will pull any additional configurations that are required.
It is important that IMPLY_MANAGER_HOST
be a name that is resolvable and routable from the agent containers. Appropriate values to use here depends on what Docker networking configuration you plan to use. For Linux-host deployments not utilizing a container orchestration system (Kubernetes, Docker Swarm, etc.), running a single container per host and starting the container with the --network host
configuration is a common setup. In this case, IMPLY_MANAGER_HOST
would be the externally addressable name or address of the host machine running the manager container. Note that --network host
is not a supported option on OSX. See the Networking section for suggestions on how to run the container in OSX.
Your Docker run command will vary based on the networking configuration chosen. Note the following important ports:
For proper cluster coordination, the manager and agent containers should be able to talk to each other. In most default networking configurations, containers can talk to each other unrestricted. If you need to allow access on specific ports, the ones used for cluster coordination are:
The recommended deployment strategy if you are not using a container orchestration system and you are using a Linux host is to use host networking. In this configuration, the container will use the host's network stack, which will provide the best performance as there is no need for packets to be routed to a container's separate network stack. You would access the Imply Manager on port 9097 of the host system.
Do not run multiple containers using host networking on a single host. You will encounter port conflicts and the agents will fail to properly register with the manager.
On a non-Linux host, you are unable to use host networking since the containers do not run natively on the host but run in a virtual machine. The recommended alternative is to use a user-defined bridge network. To create a bridge network, run:
docker network create imply
You will then be able to use this network when running containers by specifying --network imply
and will be able to address a container from other containers by its container name.
Your Docker run command will also vary depending on if you plan to use Docker volumes or bind mounts. Note that by default, Docker runs containers on the host's
primary storage device, which may not have sufficient space, particularly for Imply data nodes. You can either configure Docker to move these containers to a
secondary device on the host, or use volumes/binds to mount another location inside the container. By default, Imply writes temporary files, logs, and caches to
the /mnt
directory, so this is a good path to mount a larger volume. See the Docker volume documentation for more
information.
When considering disk space, particularly for data nodes, keep in mind that there needs to be enough free space on the volume for internal processes. Ensure that about 25 GB is available on the volume in addition to the configured segment cache (see segment cache configuration) for use for internal purposes such as temporary storage space for ingestion, heap dumps, and log files.
As an example, running the manager and agent using host networking, with a bind mount on /mnt
, and using a configuration file named env.list
would look like this. The commands must be run on different host machines:
docker run --rm --network host --env-file env.list --mount type=bind,source=/mnt,target=/mnt imply/manager
docker run --rm --network host -e "IMPLY_MANAGER_HOST=imply-manager" --mount type=bind,source=/mnt,target=/mnt imply/agent
As another example, running the manager and agent using a user-defined network named imply
with a configuration file named env.list
and the IMPLY_MANAGER_HOST
property set to match a manager container name of 'imply-manager' would look like this:
docker run --rm --network imply --name imply-manager -p 9097:9097 --env-file env.list imply/manager
docker run --rm --network imply -p 9095 -p 8888 -e "IMPLY_MANAGER_HOST=imply-manager" imply/agent
Once the manager container is running, you can access the Imply Manager application at http://<managerHost>:9097
See the TLS Docs for information on how to generate certificates and general TLS information.
By default the manager looks for the certificate files in the following locations:
/run/secrets/imply-ca.crt
/run/secrets/imply-ca.key
To enable TLS with Docker the certificates should be passed as volumes to Docker.
To run the manager providing the certificates as volumes the command would look like:
docker run --rm --network host --name imply-manager --env-file env.list -v path/to/ca.crt:/run/secrets/imply-ca.crt -v path/to/ca.key:/run/secrets/imply-ca.key imply/manager
And to provide the certificate to the agent:
docker run --rm --network host -e "IMPLY_MANAGER_HOST=imply-manager" -v path/to/ca.crt:/run/secrets/imply-ca.crt imply/agent
When the containers start you should see TLS: Enabled
in the logs.
When using Docker Swarm we can use Docker secrets. To create the secret, use the following command:
docker create secret imply-ca.crt path/to/ca.crt
docker create secret imply-ca.key path/to/ca.crt
Then we can create the service and allow it access to the secret as follows:
docker service create --name imply-manager --env-file env.list --secret imply-ca.crt --secret imply-ca.key imply/manager
docker service create --name imply-agent -e "IMPLY_MANAGER_HOST=imply-manager" --secret imply-ca.crt imply/agent
You can verify that TLS was successfully enabled by running:
docker service logs imply-manager
And verify that TLS: Enabled
shows in the logs.
See the Authentication Docs for information on enabling authentication will affect your cluster.
By default the manager looks for the authentication token in the following location:
/run/secrets/imply-auth-token
This file should contain one line, the shared secret to use. To create this file you could use the following command:
echo "supersecret" > path/to/auth-token
Creating the authentication token in this way will leave the token in your history and is meant as an example only. Consider creating this file with an editor and ensure the permissions are as restricted as possible to keep it safe.
To enable authentication with Docker the authentication token should be passed as volumes to Docker.
To run the manager providing the authentication token as a volume the command would look like:
docker run --rm --network host --name imply-manager --env-file env.list -v path/to/auth-token:/run/secrets/imply-auth-token imply/manager
And to provide the authentication token to the agent:
docker run --rm --network host -e "IMPLY_MANAGER_HOST=imply-manager" -v path/to/auth-token:/run/secrets/imply-auth-token imply/agent
When the containers start you should see Authentication: Enabled
in the logs.
When using Docker Swarm we can use Docker secrets. To create the secret you can use the following command:
echo "supersecret" | docker create secret imply-auth-token -
Passing the authentication token in this way to agent will leave the token in your history and is meant as an example only. Please consult the Docker secrets documentation for secure ways to create the secret.
Then we can create the service and allow it access to the secret by us a command like:
docker service create --name imply-manager --env-file env.list --secret imply-auth-token imply/manager
docker service create --name imply-agent -e "IMPLY_MANAGER_HOST=imply-manager" --secret imply-auth-token imply/agent
You can verify that authentication was successfully enabled by running:
docker service logs imply-manager
And verify that Authentication: Enabled
shows in the logs.
Custom user files can be pushed by the manager to the nodes of the Imply cluster by specifying them in the Setup / Advanced Config section of the manager. These files can be written to the following locations:
/opt/imply/user
)Additionally, these files can be processed as follows:
tar -x
)chmod +x
)There are two ways to make these files available to the manager:
manager:///
schemeTo make the files available to the manager locally, they will need to be placed in the /mnt/var/user
directory inside the manager container. There are two ways to do this:
docker cp
to copy the files from the host machine into the container/mnt/var/user
. If using a bind mount during docker run
, the argument would look similar to:--mount type=bind,source=/myfiles,target=/mnt/var/user
Once the files are available in that directory, reference them using the manager:///
scheme. As an example:
Imply Manager automatically attempts to create the databases it needs if they do not exist. There is one database for the Manager and one database for each Imply cluster.
If the provided user does not have authorization to create these databases, you should create them manually prior to running the application. Both Druid and the Imply Manager expect a database with a 'utf8mb4' default character set.
An example statement to create the manager database is as follow:
CREATE SCHEMA `imply_manager` DEFAULT CHARACTER SET utf8mb4;
To upgrade the manager via Docker, simply stop the running manager and start a new one with the updated container. For example:
docker stop imply-manager
docker pull imply/manager:latest
docker run --name imply-manager --network imply -p 9097:9097 --env-file env.list -d --rm imply/manager
You can update the agents in a similar manner. Note, however, that updating can result in brief downtime as the update is applied.