Security is of the utmost importance for any mission-critical application, and at Imply we know this is especially true for applications deployed in the cloud. This document details the security features offered by Imply Cloud. If you have any further questions beyond what is described in this document, please contact your Imply representative.
Imply Cloud is built on the secure foundation of Amazon Web Services (AWS). Your Imply cluster is private and dedicated to you. It is deployed completely within your own AWS account, using a dedicated virtual private network (VPC). This gives you full control over your machines and your data.
Imply also operates user-facing login and management software (“Imply Cloud Portal”), allowing you to manage and use your clusters through a web interface at https://implycloud.com/. The Imply Cloud Portal is provisioned on Imply servers and does not run in your AWS account.
Data that you load into your Imply cluster is never stored on Imply servers — it is only stored on EC2 instances and S3 buckets in your own AWS account. However, you may query your cluster through the Imply Cloud Portal, in which case query requests and responses are transferred through Imply's network. Imply's network merely transfers these query responses to you, and does not store them. This communication is encrypted using TLS; for more details, see data in transit below.
Imply Cloud requires two linkages between your AWS account and Imply:
Imply Cloud identifies each authorized user of your account by a unique login tied to their email address. Imply Cloud offers a role-based access control (RBAC) model that allows you to ensure that your users have exactly the permissions they need to do their work.
In particular, the RBAC model can be used to grant or restrict the following permissions on Imply Cloud authorized users:
In addition to authorized users, Imply Cloud lets you define API users that can call Druid APIs on your Imply cluster directly. API users can be used for building your own apps on top of the Druid API, automating cluster management tasks, and many other functions. The Druid API is protected by TLS encryption and HTTP Basic authentication. API users can be granted permissions tailored to the access they require, including:
Imply Cloud uses Transport Layer Security (TLS) for end-to-end encryption of data in transit. In particular, TLS is used to secure communications including the following:
Your Imply cluster also uses ZooKeeper for certain cluster coordination tasks. This traffic is not encrypted, since ZooKeeper support for TLS not yet stable as of this writing. However, Imply Cloud still safeguards this traffic using network segmentation through AWS EC2 Security Groups. Furthermore, this traffic is restricted to cluster coordination: your data is not at risk of exposure in these communications.
Data that you load into your Imply cluster is never stored on Imply servers. Your data is only stored on EC2 instances and S3 buckets in your own AWS account, and you have full control over it.
Imply Cloud supports encryption of your data in S3 at rest, through the use of Server-Side Encryption with Amazon S3-Managed Keys (SSE-S3), if you configure this option for your S3 buckets linked to Imply Cloud. The choice of whether or not to use this option is wholly within your control. If you would like to enable this, you should do it before launching your Imply cluster.
Imply Cloud optionally supports encryption of data at rest on your EC2 instances. The mechanism varies based on instance type:
Imply Cloud uses 6 subnets across 3 availability zones to provide resiliency in case of an AWS data center outage (each AZ has one subnet for dynamic addresses and one for static ones). These subnets are configured with a route to an internet gateway to provide egress access to, and optionally ingress access from the internet.
Three security groups are set up for use by EC2 and ELB instances: one managed group containing security rules specified by Imply, and initially empty unmanaged groups for the EC2 and ELB instances respectively intended for custom rules provided by you. By default, there are no internet- facing rules configured and all ingress from the internet is blocked.
To facilitate communication with the Imply Cloud Manager, the VPC created in your AWS account is linked through a peering connection with the Cloud Manager VPC in Imply's account. This allows your Imply cluster to be addressable by the Manager using only private IP addresses without requiring any form of public accessibility. The only ingress rules that are added to the managed security group are the ones necessary for the Cloud Manager to communicate with the various Druid services (ports 8081-8084, 8090-8091, and 9095) and port 22 for support and cluster maintenance.
Each cluster node's TLS certificate is signed by a self-signed root certificate from Imply. In order to access your cluster nodes over TLS, you'll need to download the Imply root certificate and ensure that it is trusted by your client software.
The download link for this root certificate can be found in your cluster's API view, at the Root certificate field under the Security section:
This example command uses curl
to access the /status
endpoint on the coordinator node of the cluster shown in the image above, over TLS:
curl -u admin:bWm4+3iDF+URWmDV7zXTpQ== --cacert f1503f07-f919-47f5-942f-e6a5f42b2d57.crt https://ip-10-1-0-5.ec2.internal:8281/status
f1503f07-f919-47f5-942f-e6a5f42b2d57.crt
is the root certificate downloaded from this cluster's API view, specified with the --cacert flag.
For other clients such as web browsers, please refer to your client software's documentation for instructions on how to add a trusted root TLS certificate.
In this tutorial, you'll create and manage a Druid API user, defining its authentication credentials, roles, and permissions.
Druid API users are configured through an endpoint on the coordinator service. The coordinator API endpoint can be found under a cluster's API view:
This view provides both the private and public endpoints for the coordinator. Note the IP addresses of the coordinator in the Coordinator server fields.
A user account with sufficient privileges is needed to access the API user configuration endpoints. A default admin user with superuser privileges is provided. The username and initial password for this default admin account is located in the Security section of a cluster's API view, contained in the Initial Druid API user and Initial Druid API pass fields, respectively.
The default admin account should be used in the following tutorial examples.
To create a new user, issue a POST to https://<coordinator-ip>:8281/druid-ext/basic-security/authentication/db/basic/users/<username>
.
The following curl command creates a user named tutorial
. Be sure to replace <coordinator-ip>
with the private or public address of the coordinator, as shown in your cluster's API view.
curl -u admin:password -XPOST https://<coordinator-ip>:8281/druid-ext/basic-security/authentication/db/basic/users/tutorial
To verify that the new user was created, issue a GET to https://<coordinator-ip>:8281/druid-ext/basic-security/authentication/db/basic/users
.
For example, the following curl command:
curl -u admin:password https://<coordinator-ip>:8281/druid-ext/basic-security/authentication/db/basic/users
should return the following output on a new cluster, indicating that the tutorial
user was created:
["admin","tutorial"]
You can also verify that the new user was created by requesting information for a specific user.
To do so, issue a GET to https://<coordinator-ip>:8281/druid-ext/basic-security/authentication/db/basic/users/<username>
The following curl command retrieves information for the tutorial
user:
curl -u admin:password https://<coordinator-ip>:8281/druid-ext/basic-security/authentication/db/basic/users/tutorial
It should return the following output:
{"name":"tutorial","credentials":null}
Note that the tutorial
user does not have any credentials information set. The next step will show you how to assign a password for the tutorial
user.
Let's assign a password to the tutorial
user now.
To do so, you will need to POST a password update in JSON format to https://<coordinator-ip>:8281/druid-ext/basic-security/authentication/db/basic/users/<username>/credentials
The following example request body shows the format of the JSON password update object:
{"password": "foobar123"}
The following curl command assigns a password to the tutorial
user, where pass.json
contains JSON data in the format show above.
curl -u admin:password -H'Content-Type: application/json' -XPOST --data-binary @pass.json https://<coordinator-ip>:8281/druid-ext/basic-security/authentication/db/basic/users/tutorial/credentials
If you retrieve the tutorial
user's information now, you should see that credentials have been assigned, e.g.:
curl -u admin:password https://<coordinator-ip>:8281/druid-ext/basic-security/authentication/db/basic/users/tutorial
{"name":"tutorial","credentials":{"salt":"O1777dLwNzgZVNkNBTbnl6nQxi60nmGawb9E33qernE=","hash":"02pH8/wZ8rUGv8GdokAkxbIXGad5bxRMbX9fd54EeeN5AdwWAV31FF54V0p3cBSWqQZhm/6cw14T9/bwxSgsUA==","iterations":10000}}
We've now configured a user that can authenticate with your Druid cluster. The next section will show how you can define roles and permissions for the new user.
Please note that the role-based access control system is a separate module from the basic HTTP authentication module. Therefore, it's necessary to create a new user in the role-based access control system separately from the user that you configured in the previous section. The RBAC user name must be identical to the authentication user name.
To create a new user, issue a POST to https://<coordinator-ip>:8281/druid-ext/basic-security/authorization/db/basic/users/<username>
.
The following curl command creates a user named tutorial
.
curl -u admin:password -XPOST https://<coordinator-ip>:8281/druid-ext/basic-security/authorization/db/basic/users/tutorial
To verify that the new user was created, issue a GET to https://<coordinator-ip>:8281/druid-ext/basic-security/authorization/db/basic/users
.
For example, the following curl command:
curl -u admin:password https://<coordinator-ip>:8281/druid-ext/basic-security/authorization/db/basic/users
should return the following output on a new cluster, indicating that the tutorial
user was created:
["admin","tutorial"]
You can also verify that the new user was created by requesting information for a specific user.
To do so, issue a GET to https://<coordinator-ip>:8281/druid-ext/basic-security/authorization/db/basic/users/<username>
The following curl command retrieves information for the tutorial
user:
curl -u admin:password https://<coordinator-ip>:8281/druid-ext/basic-security/authorization/db/basic/users/tutorial
It should return the following output:
{"name":"tutorial","roles":[]}
To create a new role, issue a POST to https://<coordinator-ip>:8281/druid-ext/basic-security/authorization/db/basic/roles/<rolename>
.
curl -u admin:password -XPOST https://<coordinator-ip>:8281/druid-ext/basic-security/authorization/db/basic/roles/tutorial_role
To verify that the new role was created, issue a GET to https://<coordinator-ip>:8281/druid-ext/basic-security/authorization/db/basic/roles
.
For example, the following curl command:
curl -u admin:password https://<coordinator-ip>:8281/druid-ext/basic-security/authorization/db/basic/roles
should return the following output on a new cluster, indicating that the tutorial_role
role was created:
["admin","tutorial_role"]
You can also verify that the new role was created by requesting information for a specific role.
To do so, issue a GET to https://<coordinator-ip>:8281/druid-ext/basic-security/authorization/db/basic/roles/<rolename>
The following curl command retrieves information for the tutorial_role
role:
curl -u admin:password https://<coordinator-ip>:8281/druid-ext/basic-security/authorization/db/basic/roles/tutorial_role
It should return the following output:
{"name":"tutorial_role","permissions":[]}
Let's now set permissions for the newly created tutorial_role
.
To set permissions for a role, issue a POST to https://<coordinator-ip>:8281/druid-ext/basic-security/authorization/db/basic/roles/<roleName>/permissions
, where the request body contains a JSON list of permission definitions.
An example list of permission definitions is show below:
[
{
"resource": {
"name": ".*",
"type": "STATE"
},
"action": "READ"
},
{
"resource": {
"name": ".*",
"type": "STATE"
},
"action": "WRITE"
}
]
This list of permission definitions grants a role READ and WRITE access to all resources of type STATE. For more details on permission definitions, please refer to Defining permissions
The following curl command assigns permissions to tutorial_role
, where perms.json
contains the JSON list of permissions shown above.
curl -u admin:password -H'Content-Type: application/json' -XPOST --data-binary @perms.json https://<coordinator-ip>:8281/druid-ext/basic-security/authorization/db/basic/roles/tutorial_role/permissions
Retrieving information about the tutorial_role
should now show that it has the newly assigned permissions:
curl -u admin:password https://<coordinator-ip>:8281/druid-ext/basic-security/authorization/db/basic/roles/tutorial_role
{
"name": "tutorial_role",
"permissions": [
{
"resourceAction": {
"resource": {
"name": ".*",
"type": "STATE"
},
"action": "READ"
},
"resourceNamePattern": ".*"
},
{
"resourceAction": {
"resource": {
"name": ".*",
"type": "STATE"
},
"action": "WRITE"
},
"resourceNamePattern": ".*"
}
]
}
Now that we have set permissions for tutorial_role
, let's assign tutorial_role
to the tutorial
user that you previously created.
To assign a role to a user, issue a POST to https://<coordinator-ip>:8281/druid-ext/basic-security/authorization/db/basic/users/<username>/roles/<rolename>
The following curl command assigns the role tutorial_role
to the tutorial
user:
curl -u admin:password -XPOST https://<coordinator-ip>:8281/druid-ext/basic-security/authorization/db/basic/users/tutorial/roles/tutorial_role | jq
Retrieving the information for tutorial
now shows that the user has the tutorial_role
assigned.
curl -u admin:password https://<coordinator-ip>:8281/druid-ext/basic-security/authorization/db/basic/users/tutorial
{"name":"tutorial","roles":["tutorial_role"]}
Now that the tutorial
user has authentication and access control configured, you can test this new user account by issuing a GET to the /status
endpoint on the coordinator. The STATE READ permission grants the tutorial
access to this endpoint.
curl -u tutorial:foobar123 https://<coordinator-ip>:8281/status
This command should produce output similar to the following:
{"version":"0.11.0-iap1-SNAPSHOT","modules":[{"name":"io.druid.storage.s3.S3StorageDruidModule","artifact":"druid-s3-extensions","version":"0.11.0-iap1-SNAPSHOT"},{"name":"io.druid.firehose.s3.S3FirehoseDruidModule","artifact":"druid-s3-extensions","version":"0.11.0-iap1-SNAPSHOT"},{"name":"io.druid.indexing.kafka.KafkaIndexTaskModule","artifact":"druid-kafka-indexing-service","version":"0.11.0-iap1-SNAPSHOT"},{"name":"io.druid.query.aggregation.histogram.ApproximateHistogramDruidModule","artifact":"druid-histogram","version":"0.11.0-iap1-SNAPSHOT"},{"name":"io.imply.clarity.emitter.ClarityEmitterModule"},{"name":"io.druid.security.basic.BasicSecurityDruidModule","artifact":"druid-basic-security","version":"0.11.0-iap1-SNAPSHOT"},{"name":"io.druid.examples.ExamplesDruidModule","artifact":"druid-examples","version":"0.11.0-iap1-SNAPSHOT"},{"name":"io.druid.query.aggregation.datasketches.theta.SketchModule","artifact":"druid-datasketches","version":"0.11.0-iap1-SNAPSHOT"},{"name":"io.druid.query.aggregation.datasketches.theta.oldapi.OldApiSketchModule","artifact":"druid-datasketches","version":"0.11.0-iap1-SNAPSHOT"},{"name":"io.druid.metadata.storage.mysql.MySQLMetadataStorageModule","artifact":"mysql-metadata-storage","version":"0.11.0-iap1-SNAPSHOT"},{"name":"io.druid.server.lookup.namespace.NamespaceExtractionModule","artifact":"druid-lookups-cached-global","version":"0.11.0-iap1-SNAPSHOT"},{"name":"io.druid.parsers.ParserRouteModule","artifact":"druid-parser-route","version":"0.11.0.5"},{"name":"io.druid.https.SSLContextModule","artifact":"simple-client-sslcontext","version":"0.11.0-iap1-SNAPSHOT"},{"name":"io.imply.security.ImplySecurityDruidModule"}],"memory":{"maxMemory":536870912,"totalMemory":536870912,"freeMemory":213291048,"usedMemory":323579864}}
For more information see the documentation for the Druid basic authentication and authorization extension.