Security

Security is of the utmost importance for any mission-critical application, and at Imply we know this is especially true for applications deployed in the cloud. This document details the security features offered by Imply Cloud. If you have any further questions beyond what is described in this document, please contact your Imply representative.

Secure foundation

Imply Cloud is built on the secure foundation of Amazon Web Services (AWS). Your Imply cluster is private and dedicated to you. It is deployed completely within your own AWS account, using a dedicated virtual private network (VPC). This gives you full control over your machines and your data.

Imply also operates user-facing login and management software (“Imply Cloud Portal”), allowing you to manage and use your clusters through a web interface at https://implycloud.com/. The Imply Cloud Portal is provisioned on Imply servers and does not run in your AWS account.

Data that you load into your Imply cluster is never stored on Imply servers — it is only stored on EC2 instances and S3 buckets in your own AWS account. However, you may query your cluster through the Imply Cloud Portal, in which case query requests and responses are transferred through Imply's network. Imply's network merely transfers these query responses to you, and does not store them. This communication is encrypted using TLS; for more details, see data in transit below.

Imply Cloud requires two linkages between your AWS account and Imply:

1. You must grant Imply permissions through IAM for management operations such as launching and terminating instances. This occurs during initial setup of your Imply Cloud account, which you can read about at: https://imply.io/cloud-signup-instructions
2. The Imply Cloud VPC in your AWS account will be peered with an Imply-operated VPC containing the Imply Cloud Portal. This peering allows the Imply Cloud service to communicate with your cluster through private IP addresses, without requiring access over the internet. For increased security, the standard Imply Cloud configuration blocks ingress from the internet to your Imply Cloud VPC.

User authentication and authorization

Imply Cloud identifies each authorized user of your account by a unique login tied to their email address. Imply Cloud offers a role-based access control (RBAC) model that allows you to ensure that your users have exactly the permissions they need to do their work.

In particular, the RBAC model can be used to grant or restrict the following permissions on Imply Cloud authorized users:

• Ability to manage clusters, datasets, and/or other users.
• Ability to manage data cubes (building blocks of visualizations).
• Ability to manage dashboards.
• Ability to see visualizations (data cube view).
• Ability to query the Imply cluster directly with SQL.
• Ability to load new data into the Imply cluster.

In addition to authorized users, Imply Cloud lets you define API users that can call Druid APIs on your Imply cluster directly. API users can be used for building your own apps on top of the Druid API, automating cluster management tasks, and many other functions. The Druid API is protected by TLS encryption and HTTP Basic authentication. API users can be granted permissions tailored to the access they require, including:

• Ability to read or write cluster configuration.
• Ability to load new data into the cluster (all datasets, or specific datasets).
• Ability to query the cluster (all datasets, or specific datasets).

Data in transit

Imply Cloud uses Transport Layer Security (TLS) for end-to-end encryption of data in transit. In particular, TLS is used to secure communications including the following:

• All communications between your browser and the Imply Cloud Portal at https://implycloud.com/.
• All communications between the Imply Cloud Portal and your private Imply cluster.
• All internal traffic involving your data within your Imply cluster, including data ingestion, persistence, and query requests and responses.
• All Druid API calls you make to your Imply cluster.

Your Imply cluster also uses ZooKeeper for certain cluster coordination tasks. This traffic is not encrypted, since ZooKeeper support for TLS not yet stable as of this writing. However, Imply Cloud still safeguards this traffic using network segmentation through AWS EC2 Security Groups. Furthermore, this traffic is restricted to cluster coordination: your data is not at risk of exposure in these communications.

Data at rest

Data that you load into your Imply cluster is never stored on Imply servers. Your data is only stored on EC2 instances and S3 buckets in your own AWS account, and you have full control over it.

Imply Cloud supports encryption of your data in S3 at rest, through the use of Server-Side Encryption with Amazon S3-Managed Keys (SSE-S3), if you configure this option for your S3 buckets linked to Imply Cloud. The choice of whether or not to use this option is wholly within your control. If you would like to enable this, you should do it before launching your Imply cluster.

Imply Cloud optionally supports encryption of data at rest on your EC2 instances. The mechanism varies based on instance type:

• For EBS-backed instance types (e.g. c5, m5), Imply Cloud can optionally provision encrypted EBS volumes. This is controlled by the "instance encryption" setting in your Imply Cloud cluster configuration, and is enabled by default for clusters created after April 2, 2018. This setting can be changed for existing clusters, but they must be stopped first and then restarted.
• Instances types that use NVMe local storage (e.g. i3, c5d, m5d) are transparently encrypted via hardware on the instance. This is always enabled, regardless of your Imply Cloud "instance encryption" setting. See here for details: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ssd-instance-store.html.

Network setup

Imply Cloud uses 6 subnets across 3 availability zones to provide resiliency in case of an AWS data center outage (each AZ has one subnet for dynamic addresses and one for static ones). These subnets are configured with a route to an internet gateway to provide egress access to, and optionally ingress access from the internet.

Three security groups are set up for use by EC2 and ELB instances: one managed group containing security rules specified by Imply, and initially empty unmanaged groups for the EC2 and ELB instances respectively intended for custom rules provided by you. By default, there are no internet- facing rules configured and all ingress from the internet is blocked.

To facilitate communication with the Imply Cloud Manager, the VPC created in your AWS account is linked through a peering connection with the Cloud Manager VPC in Imply's account. This allows your Imply cluster to be addressable by the Manager using only private IP addresses without requiring any form of public accessibility. The only ingress rules that are added to the managed security group are the ones necessary for the Cloud Manager to communicate with the various Druid services (ports 8081-8084, 8090-8091, and 9095) and port 22 for support and cluster maintenance.

Downloading the root TLS certificate

Each cluster node's TLS certificate is signed by a self-signed root certificate from Imply. In order to access your cluster nodes over TLS, you'll need to download the Imply root certificate and ensure that it is trusted by your client software.

The download link for this root certificate can be found in your cluster's API view, at the Root certificate field under the Security section:

Accessing a cluster node over TLS with curl

This example command uses curl to access the /status endpoint on the coordinator node of the cluster shown in the image above, over TLS:

curl -u admin:bWm4+3iDF+URWmDV7zXTpQ== --cacert f1503f07-f919-47f5-942f-e6a5f42b2d57.crt https://ip-10-1-0-5.ec2.internal:8281/status

f1503f07-f919-47f5-942f-e6a5f42b2d57.crt is the root certificate downloaded from this cluster's API view, specified with the --cacert flag.

Other clients

For other clients such as web browsers, please refer to your client software's documentation for instructions on how to add a trusted root TLS certificate.

Managing Druid API users

In this tutorial, you'll create and manage a Druid API user, defining its authentication credentials, roles, and permissions.

Accessing the coordinator

Druid API users are configured through an endpoint on the coordinator service. The coordinator API endpoint can be found under a cluster's API view:

This view provides both the private and public endpoints for the coordinator. Note the IP addresses of the coordinator in the Coordinator server fields.

Default admin credentials

A user account with sufficient privileges is needed to access the API user configuration endpoints. A default admin user with superuser privileges is provided. The username and initial password for this default admin account is located in the Security section of a cluster's API view, contained in the Initial Druid API user and Initial Druid API pass fields, respectively.

The default admin account should be used in the following tutorial examples.

Authentication

Creating a new user

To create a new user, issue a POST to https://<coordinator-ip>:8281/druid-ext/basic-security/authentication/db/basic/users/<username>.

The following curl command creates a user named tutorial. Be sure to replace <coordinator-ip> with the private or public address of the coordinator, as shown in your cluster's API view.

curl -u admin:password -XPOST https://<coordinator-ip>:8281/druid-ext/basic-security/authentication/db/basic/users/tutorial

List all users

To verify that the new user was created, issue a GET to https://<coordinator-ip>:8281/druid-ext/basic-security/authentication/db/basic/users.

For example, the following curl command:

curl -u admin:password https://<coordinator-ip>:8281/druid-ext/basic-security/authentication/db/basic/users

should return the following output on a new cluster, indicating that the tutorial user was created:

["admin","tutorial"]

Show a single user

You can also verify that the new user was created by requesting information for a specific user.

To do so, issue a GET to https://<coordinator-ip>:8281/druid-ext/basic-security/authentication/db/basic/users/<username>

The following curl command retrives information for the tutorial user:

curl -u admin:password https://<coordinator-ip>:8281/druid-ext/basic-security/authentication/db/basic/users/tutorial

It should return the following output:

{"name":"tutorial","credentials":null}

Note that the tutorial user does not have any credentials information set. The next step will show you how to assign a password for the tutorial user.

Set a password for the user

Let's assign a password to the tutorial user now.

To do so, you will need to POST a password update in JSON format to https://<coordinator-ip>:8281/druid-ext/basic-security/authentication/db/basic/users/<username>/credentials

The following example request body shows the format of the JSON password update object:

{"password": "foobar123"}

The following curl command assigns a password to the tutorial user, where pass.json contains JSON data in the format show above.

curl -u admin:password -H'Content-Type: application/json' -XPOST --data-binary @pass.json https://<coordinator-ip>:8281/druid-ext/basic-security/authentication/db/basic/users/tutorial/credentials

If you retrieve the tutorial user's information now, you should see that credentials have been assigned, e.g.:

curl -u admin:password https://<coordinator-ip>:8281/druid-ext/basic-security/authentication/db/basic/users/tutorial

{"name":"tutorial","credentials":{"salt":"O1777dLwNzgZVNkNBTbnl6nQxi60nmGawb9E33qernE=","hash":"02pH8/wZ8rUGv8GdokAkxbIXGad5bxRMbX9fd54EeeN5AdwWAV31FF54V0p3cBSWqQZhm/6cw14T9/bwxSgsUA==","iterations":10000}}

We've now configured a user that can authenticate with your Druid cluster. The next section will show how you can define roles and permissions for the new user.

Role-based Access Control

Creating a new user

Please note that the role-based access control system is a separate module from the basic HTTP authentication module. Therefore, it's necessary to create a new user in the role-based access control system separately from the user that you configured in the previous section. The RBAC user name must be identical to the authentication user name.

To create a new user, issue a POST to https://<coordinator-ip>:8281/druid-ext/basic-security/authorization/db/basic/users/<username>.

The following curl command creates a user named tutorial.

curl -u admin:password -XPOST https://<coordinator-ip>:8281/druid-ext/basic-security/authorization/db/basic/users/tutorial

List all users

To verify that the new user was created, issue a GET to https://<coordinator-ip>:8281/druid-ext/basic-security/authorization/db/basic/users.

For example, the following curl command:

curl -u admin:password https://<coordinator-ip>:8281/druid-ext/basic-security/authorization/db/basic/users

should return the following output on a new cluster, indicating that the tutorial user was created:

["admin","tutorial"]

Show a single user

You can also verify that the new user was created by requesting information for a specific user.

To do so, issue a GET to https://<coordinator-ip>:8281/druid-ext/basic-security/authorization/db/basic/users/<username>

The following curl command retrives information for the tutorial user:

curl -u admin:password https://<coordinator-ip>:8281/druid-ext/basic-security/authorization/db/basic/users/tutorial

It should return the following output:

{"name":"tutorial","roles":[]}

Creating a role

To create a new role, issue a POST to https://<coordinator-ip>:8281/druid-ext/basic-security/authorization/db/basic/roles/<rolename>.

curl -u admin:password -XPOST https://<coordinator-ip>:8281/druid-ext/basic-security/authorization/db/basic/roles/tutorial_role

List all roles

To verify that the new role was created, issue a GET to https://<coordinator-ip>:8281/druid-ext/basic-security/authorization/db/basic/roles.

For example, the following curl command:

curl -u admin:password https://<coordinator-ip>:8281/druid-ext/basic-security/authorization/db/basic/roles

should return the following output on a new cluster, indicating that the tutorial_role role was created:

["admin","tutorial_role"]

Show a single role

You can also verify that the new role was created by requesting information for a specific role.

To do so, issue a GET to https://<coordinator-ip>:8281/druid-ext/basic-security/authorization/db/basic/roles/<rolename>

The following curl command retrives information for the tutorial_role role:

curl -u admin:password https://<coordinator-ip>:8281/druid-ext/basic-security/authorization/db/basic/roles/tutorial_role

It should return the following output:

{"name":"tutorial_role","permissions":[]}

Set the permissions of a role

Let's now set permissions for the newly created tutorial_role.

To set permissions for a role, issue a POST to https://<coordinator-ip>:8281/druid-ext/basic-security/authorization/db/basic/roles/<roleName>/permissions, where the request body contains a JSON list of permission defintions.

An example list of permission definitions is show belown:

[
{
  "resource": {
    "name": ".*",
    "type": "STATE"
  },
  "action": "READ"
},
{
  "resource": {
    "name": ".*",
    "type": "STATE"
  },
  "action": "WRITE"
}
]

This list of permission defintions grants a role READ and WRITE access to all resources of type STATE. For more details on permission definitions, please refer to Defining permissions

The following curl command assigns permissions to tutorial_role, where perms.json contains the JSON list of permissions shown above.

curl -u admin:password -H'Content-Type: application/json' -XPOST --data-binary @perms.json https://<coordinator-ip>:8281/druid-ext/basic-security/authorization/db/basic/roles/tutorial_role/permissions

Retrieving information about the tutorial_role should now show that it has the newly assigned permissions:

curl -u admin:password https://<coordinator-ip>:8281/druid-ext/basic-security/authorization/db/basic/roles/tutorial_role

{
  "name": "tutorial_role",
  "permissions": [
    {
      "resourceAction": {
        "resource": {
          "name": ".*",
          "type": "STATE"
        },
        "action": "READ"
      },
      "resourceNamePattern": ".*"
    },
    {
      "resourceAction": {
        "resource": {
          "name": ".*",
          "type": "STATE"
        },
        "action": "WRITE"
      },
      "resourceNamePattern": ".*"
    }
  ]
}

Assign a role to a user

Now that we have set permissions for tutorial_role, let's assign tutorial_role to the tutorial user that you previously created.

To assign a role to a user, issue a POST to https://<coordinator-ip>:8281/druid-ext/basic-security/authorization/db/basic/users/<username>/roles/<rolename>

The following curl command assigns the role tutorial_role to the tutorial user:

curl -u admin:password -XPOST https://<coordinator-ip>:8281/druid-ext/basic-security/authorization/db/basic/users/tutorial/roles/tutorial_role | jq

Retrieving the information for tutorial now shows that the user has the tutorial_role assigned.

curl -u admin:password https://<coordinator-ip>:8281/druid-ext/basic-security/authorization/db/basic/users/tutorial

{"name":"tutorial","roles":["tutorial_role"]}

Verifying the new user account

Now that the tutorial user has authentication and access control configured, you can test this new user account by issuing a GET to the /status endpoint on the coordinator. The STATE READ permission grants the tutorial access to this endpoint.

curl -u tutorial:foobar123 https://<coordinator-ip>:8281/status

This command should produce output similar to the following:

{"version":"0.11.0-iap1-SNAPSHOT","modules":[{"name":"io.druid.storage.s3.S3StorageDruidModule","artifact":"druid-s3-extensions","version":"0.11.0-iap1-SNAPSHOT"},{"name":"io.druid.firehose.s3.S3FirehoseDruidModule","artifact":"druid-s3-extensions","version":"0.11.0-iap1-SNAPSHOT"},{"name":"io.druid.indexing.kafka.KafkaIndexTaskModule","artifact":"druid-kafka-indexing-service","version":"0.11.0-iap1-SNAPSHOT"},{"name":"io.druid.query.aggregation.histogram.ApproximateHistogramDruidModule","artifact":"druid-histogram","version":"0.11.0-iap1-SNAPSHOT"},{"name":"io.imply.clarity.emitter.ClarityEmitterModule"},{"name":"io.druid.security.basic.BasicSecurityDruidModule","artifact":"druid-basic-security","version":"0.11.0-iap1-SNAPSHOT"},{"name":"io.druid.examples.ExamplesDruidModule","artifact":"druid-examples","version":"0.11.0-iap1-SNAPSHOT"},{"name":"io.druid.query.aggregation.datasketches.theta.SketchModule","artifact":"druid-datasketches","version":"0.11.0-iap1-SNAPSHOT"},{"name":"io.druid.query.aggregation.datasketches.theta.oldapi.OldApiSketchModule","artifact":"druid-datasketches","version":"0.11.0-iap1-SNAPSHOT"},{"name":"io.druid.metadata.storage.mysql.MySQLMetadataStorageModule","artifact":"mysql-metadata-storage","version":"0.11.0-iap1-SNAPSHOT"},{"name":"io.druid.server.lookup.namespace.NamespaceExtractionModule","artifact":"druid-lookups-cached-global","version":"0.11.0-iap1-SNAPSHOT"},{"name":"io.druid.parsers.ParserRouteModule","artifact":"druid-parser-route","version":"0.11.0.5"},{"name":"io.druid.https.SSLContextModule","artifact":"simple-client-sslcontext","version":"0.11.0-iap1-SNAPSHOT"},{"name":"io.imply.security.ImplySecurityDruidModule"}],"memory":{"maxMemory":536870912,"totalMemory":536870912,"freeMemory":213291048,"usedMemory":323579864}}

Further reading

For more information see the documentation for the Druid basic authentication and authorization extension.