Cloud Manager API tutorial for Imply Hybrid

This tutorial introduces you to the Cloud Manager API for Imply Hybrid and how it can be used to manage the cluster lifecycle. In this tutorial, you will learn how to:

Enable API access for an OAuth client
Create a cluster
Scale up the cluster
Get the cluster version number
View the cluster's previous configuration
Stop the cluster
Delete the cluster

Before getting started, ensure that you have the following:

Access to the Cloud Manager API for your Imply Cloud deployment. Contact your Imply representative to enable it for your organization.
The ManageClusters permission for your user role. A user with the ManageUsers permission can grant this permission for users.

Throughout this tutorial, you use cURL to send requests to the Cloud Manager API. For all examples below, replace the following:

{vanity_domain} with the name of your Imply Cloud account.
{aws_role} with the name of your role in your AWS instance.
{deepstorage_path} with the path to your deep storage, for example, s3://my-bucket/prefix.

Step 1: Enable access for an OAuth client

Follow the instructions for Configuring OAuth client access to create an OAuth client. Before obtaining your token, enable access to the Cloud Manager API through your client as follows:

In the Service Account Roles tab, select your Imply Cloud environment. A list of available roles should appear.
Find the manage-clusters role, and click Add selected to assign this role to your client.
Acquire the bearer token for this client either by downloading it from the Tokens tab or following the instructions in Getting the token by API.

For greater security, you can assign the token to an environment variable on the machine from which the API requests originate. Throughout the rest of this tutorial, the environment variable $IMPLY_TOKEN refers to the bearer token.

Step 2: Create a cluster

Create a cluster by submitting a POST request to the Clusters API. In this example, you create a cluster named clusters-example with three master servers, three query servers, and one data server. Before creating the cluster, you can validate the specification using the command below, changing dryRun=true.

Use this configuration for test cases only. As described in Imply Cloud overview, t2.small nodes are not recommended for production use cases.

Request:

curl --location --request POST 'https://{vanity_domain}.implycloud.com/api/v1?pretty=true&dryRun=false' \
--header 'Authorization: Bearer ${IMPLY_TOKEN}' \
--header 'Content-Type: application/json' \
--data-raw '{
    "metadata": {
        "name": "clusters-example"
    },
    "spec": {
        "version": "2021.10",
        "awsInstanceRole": "{aws_role}",
        "encryptVolumes": true,
        "master": {
            "replicas": 3,
            "instanceType": "t2.small"
        },
        "query": {
            "replicas": 3,
            "instanceType": "t2.small"
        },
        "data": {
            "tier1": {
                "replicas": 1,
                "instanceType": "t2.small"
            }
        },
        "metadataStore": {
            "instanceType": "db.t3.small",
            "diskSize": 20,
            "backupRetention": 14,
            "encrypt": true,
            "monitoring": true
        },
        "deepStorage": {
            "path": "{deepstorage_path}"
        }
    }
}'

Output:

{
    "metadata": {
        "name": "clusters-example",
        "uid": "def4e7c4-bc06-42f3-95cd-e7bff1885808"
    },
    "spec": {
        "version": "2021.10",
        "awsInstanceRole": "{aws_role}",
        "awsTags": [],
        "encryptVolumes": true,
        "commonRuntimeProperties": [],
        "master": {
            "replicas": 3,
            "instanceType": "t2.small",
            "coordinatorRuntimeProperties": [],
            "overlordRuntimeProperties": []
        },
        "query": {
            "replicas": 3,
            "instanceType": "t2.small",
            "brokerRuntimeProperties": [],
            "routerRuntimeProperties": []
        },
        "data": {
            "historicalRuntimeProperties": [],
            "middleManagerRuntimeProperties": [],
            "tier1": {
                "replicas": 1,
                "instanceType": "t2.small",
                "historicalRuntimeProperties": [],
                "middleManagerRuntimeProperties": []
            }
        },
        "metadataStore": {
            "instanceType": "db.t3.small",
            "diskSize": 20,
            "backupRetention": 14,
            "monitoring": true,
            "encrypt": true
        },
        "pivot": {
            "accessMode": "PROXIED"
        },
        "deepStorage": {
            "path": "{deepstorage_path}",
            "encrypt": true
        },
        "druidExtensions": [],
        "featureFlags": [],
        "customFiles": []
    }
}

Step 3: Scale up the cluster

Scale up the Imply cluster by increasing the number of replicas for the data nodes. In this example, you also update the cluster name from "clusters-example" to "clusters-example-scaled". Send the new cluster configuration in a PUT request to replace the cluster configuration. Note that you can also update the cluster configuration by making a PATCH request with a partial cluster specification, as shown in Step 6 for stopping a cluster.

Visit Managing Imply clusters for more information on scaling up or scaling down your Imply clusters. You can also change the instance types for your Imply cluster. See Imply Cloud overview for the available instance types.

Request:

curl --location --request PUT 'https://{vanity_domain}.implycloud.com/api/v1/clusters/def4e7c4-bc06-42f3-95cd-e7bff1885808?pretty=true&dryRun=false&force=true' \
--header 'Content-Type: application/json' \
--header 'Authorization: Bearer ${IMPLY_TOKEN}' \
--data-raw '{
    "metadata": {
        "name": "clusters-example-scaled"
    },
    "spec": {
        "version": "2021.10",
        "awsInstanceRole": "{aws_role}",
        "encryptVolumes": true,
        "master": {
            "replicas": 3,
            "instanceType": "t2.small"
        },
        "query": {
            "replicas": 3,
            "instanceType": "t2.small"
        },
        "data": {
            "tier1": {
                "replicas": 2,
                "instanceType": "t2.small"
            }
        },
        "metadataStore": {
            "instanceType": "db.t3.small",
            "diskSize": 20,
            "backupRetention": 14,
            "encrypt": true,
            "monitoring": true
        },
        "pivot": {                                                                                                                     
            "accessMode": "PROXIED"                                                                                                    
        },
        "deepStorage": {
            "path": "{deepstorage_path}"
        }
    }
}'

Output:

For a successful request, the API returns status code 202 Accepted with the new cluster configuration. Submit a GET request with the cluster ID to verify that the cluster state is UPDATING.

You may get a StateConflict error if you attempt to overwrite a cluster that is not running.

{
    "metadata": {
        "name": "clusters-example-scaled",
        "uid": "def4e7c4-bc06-42f3-95cd-e7bff1885808"
    },
    "spec": {
        "version": "2021.10",
        "awsInstanceRole": "{aws_role}",
        "awsTags": [],
        "encryptVolumes": true,
        "commonRuntimeProperties": [],
        "master": {
            "replicas": 3,
            "instanceType": "t2.small",
            "coordinatorRuntimeProperties": [],
            "overlordRuntimeProperties": []
        },
        "query": {
            "replicas": 3,
            "instanceType": "t2.small",
            "brokerRuntimeProperties": [],
            "routerRuntimeProperties": []
        },
        "data": {
            "historicalRuntimeProperties": [],
            "middleManagerRuntimeProperties": [],
            "tier1": {
                "replicas": 2,
                "instanceType": "t2.small",
                "historicalRuntimeProperties": [],
                "middleManagerRuntimeProperties": []
            }
        },
        "metadataStore": {
            "instanceType": "db.t3.small",
            "diskSize": 20,
            "backupRetention": 14,
            "monitoring": true,
            "encrypt": true
        },
        "pivot": {
            "accessMode": "PROXIED"
        },
        "deepStorage": {
            "path": "{deepstorage_path}"
        },
        "druidExtensions": [],
        "featureFlags": [],
        "customFiles": []
    }
}

Step 4: Get the cluster version number

Make a GET request with the cluster ID to get its current configuration. You can pipe the result through the JSON processor jq to parse the JSON result. The following command obtains the cluster configuration and extracts the resource version number:

Request:

curl --location --request GET 'https://{vanity_domain}.implycloud.com/api/v1/clusters/def4e7c4-bc06-42f3-95cd-e7bff1885808?pretty=true' \
--header 'Authorization: Bearer ${IMPLY_TOKEN}' | jq -r '.status.resourceVersion'

The output of the request is 1. The resource version increments with every change made to the cluster.

Step 5: View the cluster's previous configuration

Imply Cloud Manager stores all past configurations of your cluster. Access these configurations using the cluster resource version.

To view a past cluster configuration, make a GET request to the ClustersHistorical API, specifying the cluster ID as well as the resource version. Our cluster has one former configuration, resourceVersion = 0.

Notice the output matches the cluster configuration you used to create the cluster. Use the resulting cluster definition in the body of a PUT request to restore the cluster to a previous configuration.

Request:

curl --location --request GET 'https://{vanity_domain}.implycloud.com/api/v1/clusters/def4e7c4-bc06-42f3-95cd-e7bff1885808/history/0' \
--header 'Authorization: Bearer ${IMPLY_TOKEN}'

Output:

{
    "metadata": {
        "name": "clusters-example",
        "uid": "def4e7c4-bc06-42f3-95cd-e7bff1885808"
    },
    "spec": {
        "version": "2021.10",
        "awsInstanceRole": "{aws_role}",
        "awsTags": [],
        "encryptVolumes": true,
        "commonRuntimeProperties": [],
        "master": {
            "replicas": 3,
            "instanceType": "t2.small",
            "coordinatorRuntimeProperties": [],
            "overlordRuntimeProperties": []
        },
        "query": {
            "replicas": 3,
            "instanceType": "t2.small",
            "brokerRuntimeProperties": [],
            "routerRuntimeProperties": []
        },
        "data": {
            "historicalRuntimeProperties": [],
            "middleManagerRuntimeProperties": [],
            "tier1": {
                "replicas": 1,
                "instanceType": "t2.small",
                "historicalRuntimeProperties": [],
                "middleManagerRuntimeProperties": []
            }
        },
        "metadataStore": {
            "instanceType": "db.t3.small",
            "diskSize": 20,
            "backupRetention": 14,
            "monitoring": true,
            "encrypt": true
        },
        "pivot": {
            "accessMode": "PROXIED"
        },
        "deepStorage": {
            "path": "{deepstorage_path}",
            "encrypt": true
        },
        "druidExtensions": [],
        "featureFlags": [],
        "customFiles": []
    }
}

Step 6: Stop the cluster

You can send a partial cluster configuration using a PATCH request to update a single property or subset of properties. In the example below, send a minimal JSON body to stop the cluster. Verify the cluster state by making a GET request with the cluster ID.

You can start a cluster that has been stopped using the same type of request, changing the desired state to STARTED.

Request:

curl --location --request PATCH 'https://{vanity_domain}.implycloud.com/api/v1/clusters/def4e7c4-bc06-42f3-95cd-e7bff1885808?pretty=true&dryRun=false' \
--header 'Content-Type: application/merge-patch+json' \
--header 'Authorization: Bearer ${IMPLY_TOKEN}' \
--data-raw '{
  "spec": {
    "desiredState": "STOPPED"
  }
}'

Output:

{
    "metadata": {
        "name": "clusters-example-scaled",
        "uid": "def4e7c4-bc06-42f3-95cd-e7bff1885808"
    },
    "spec": {
        "version": "2021.10",
        "awsInstanceRole": "{aws_role}",
        "awsTags": [],
        "encryptVolumes": true,
        "commonRuntimeProperties": [],
        "master": {
            "replicas": 3,
            "instanceType": "t2.small",
            "coordinatorRuntimeProperties": [],
            "overlordRuntimeProperties": []
        },
        "query": {
            "replicas": 3,
            "instanceType": "t2.small",
            "brokerRuntimeProperties": [],
            "routerRuntimeProperties": []
        },
        "data": {
            "historicalRuntimeProperties": [],
            "middleManagerRuntimeProperties": [],
            "tier1": {
                "replicas": 2,
                "instanceType": "t2.small",
                "historicalRuntimeProperties": [],
                "middleManagerRuntimeProperties": []
            }
        },
        "metadataStore": {
            "instanceType": "db.t3.small",
            "diskSize": 20,
            "backupRetention": 14,
            "monitoring": true,
            "encrypt": true
        },
        "pivot": {
            "accessMode": "PROXIED"
        },
        "deepStorage": {
            "path": "{deepstorage_path}"
        },
        "druidExtensions": [],
        "featureFlags": [],
        "customFiles": []
    }
}

Step 7: Delete the cluster

Send a DELETE request to delete a cluster, specifying the cluster's ID as a path parameter.

Request:

curl --location --request DELETE 'https://{vanity_domain}.implycloud.com/api/v1/clusters/def4e7c4-bc06-42f3-95cd-e7bff1885808' \
--header 'Authorization: Bearer ${IMPLY_TOKEN}' \

The API returns 202 Accepted for a successful request. You may still see the deleted cluster when listing all clusters, however, you can no longer make requests using the deleted cluster ID.

Step 1: Enable access for an OAuth client​

Step 2: Create a cluster​

Step 3: Scale up the cluster​

Step 4: Get the cluster version number​

Step 5: View the cluster's previous configuration​

Step 6: Stop the cluster​

Step 7: Delete the cluster​

Step 1: Enable access for an OAuth client

Step 2: Create a cluster

Step 3: Scale up the cluster

Step 4: Get the cluster version number

Step 5: View the cluster's previous configuration

Step 6: Stop the cluster

Step 7: Delete the cluster