Install Lumi Enterprise with Terraform

AI summary

Explains how to deploy Imply Lumi Enterprise in your AWS environment using Terraform, keeping data within your infrastructure. Covers prerequisites like VPC CIDR selection and Route 53 configuration, walks through Terraform setup and deployment, and describes available configuration options.

About AI summaries.

Imply provides a Terraform module for you to deploy Lumi Enterprise.

This topic walks you through the steps of deploying Lumi Enterprise on AWS using Terraform. After Lumi Enterprise is deployed, administrators will want to learn about the integrations that Lumi supports and how to send events to Lumi. Users can get started with tutorials to familiarize themselves with how to query data.

Prerequisites

Before you begin, make sure you have the following:

Terraform CLI version 1.10.1 or higher.
AWS CLI version 2.27.0 or higher configured with a role that has full administrator permissions for the target AWS account. You need full administrator permissions because the module deploys a wide range of AWS resources on your behalf.

The AWS CLI is often used with profiles, which are a way to manage multiple roles and users across different AWS accounts. If the principal you will be using for the installation is tied to a profile, make note of the profile name. You need this information to configure the Terraform module. For details, see Configuration and credential file settings in the AWS CLI in the AWS documentation.
12-digit AWS account ID for the account where you plan to deploy Lumi Enterprise. You must provide the account ID to your Imply representative so that Imply can grant you access to its private Amazon Elastic Container Registry (ECR).
Imply provides the following arguments during onboarding. Enter these values in main.tf:
- API key
- Customer ID
- Tenant ID

AWS requirements

The Lumi Terraform module deploys a complete Lumi Enterprise environment into an AWS account with all the necessary infrastructure and dependencies. This includes:

VPC and networking resources
Elastic Kubernetes Service (EKS) cluster
Managed Streaming for Apache Kafka (MSK) cluster
RDS Aurora database (metadata store)
S3 buckets (data storage)
Cognito (identity management)

It requires access to AWS Certificate Manager (ACM), Route 53, Identity and Access Management (IAM), Key Management Service (KMS), and Secrets Manager to create the supporting records, certificates, service accounts, and secrets.

As a best practice, we recommend that you deploy Lumi Enterprise into a separate AWS account from other organizational resources to provide a controlled cloud environment and maintain separation of permissions.

Regions

You can deploy Lumi Enterprise in any standard AWS region. Deployment in restricted or specialized partitions, such as China or AWS GovCloud (US) regions, is not supported.

VPC

You need to choose a /19 CIDR block for the VPC that doesn't conflict with any other VPCs in your environment. Administrators often configure private networking such as PrivateLink, Transit Gateway, or VPC peering connections between VPCs and across AWS accounts. Consider these connections and the CIDRs of the linked VPCs when choosing the CIDR block for Lumi Enterprise.

You set the CIDR block using vpc_cidr in your main.tf file.

DNS

Lumi Enterprise requires a Route 53 hosted zone to create DNS records for its deployed services. This hosted zone must already exist in the AWS account where you plan to install the Terraform module.

You can configure a Route 53 hosted zone in one of the following ways:

A primary hosted zone for a domain that may be registered through Route 53 or another registrar.
For example, deploying Lumi Enterprise at a new domain such as lumi-mycompany.com.
A delegated hosted zone for a subdomain of an existing domain.
For example, deploying Lumi Enterprise at the subdomain lumi.mycompany.com, where mycompany.com is managed elsewhere.

There are many ways to set up and manage DNS records, and these vary between organizations.

Configure the main.tf file

After completing the prerequisites, you can begin writing the main Terraform configuration to deploy the Lumi Enterprise module.

Create a directory for your Terraform configuration files.
In the directory, create a new file named main.tf.

The following example provides a starting point for a deployment and includes both required and strongly recommended configuration arguments. Detailed descriptions of the Lumi module arguments are provided in the sections that follow. For a complete reference to the arguments supported in the AWS provider block, see the AWS documentation.

terraform {
  required_version = ">= 1.10.1"

  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = ">= 6.15.0"
    }
  }
}

provider "aws" {
  region  = "us-east-1"
}

data "aws_route53_zone" "lumi_domain" {
  name = "example.com."
}

module "lumi" {
  source                        = "s3::https://s3.amazonaws.com/lumi-enterprise-artifacts/terraform-aws-lumi-enterprise-v1.0.1.tar.gz"

  vpc_cidr                      = "10.92.0.0/19"
  hosted_zone                   = data.aws_route53_zone.lumi_domain

  tenant_id                     = "1234ab56"
  api_key                       = "a1234b5c-def1-2f3c-45bf-678f912345c6"
  customer_id                   = "z12bb34b-df56-4d78-9101-7d12b34567c8"

  admin_email                   = "user1@example.com"
  admin_initial_password        = "InitialPassword1"

  nlb_ingress_cidr_blocks       = ["10.0.0.0/16"]

}

The following arguments are required or strongly recommended when deploying Lumi Enterprise:

source: Location of the Lumi Enterprise Terraform bundle. Use the URL provided unless otherwise directed by Imply.
vpc_cidr: CIDR block for the VPC. This should not overlap with any other VPC in your environment.
hosted_zone: Route 53 hosted zone, provided as an object:
```
  {
    id   = "Z1234567" // Hosted zone ID
    name = "lumi.example.com" // Hosted zone name
  }
```
As shown in the example, you can use an aws_route53_zone data source to look up the hosted zone by the domain name instead of providing an object with the zone ID and name directly.
tenant_id: Tenant ID provided by Imply during onboarding.
api_key: API key provided by Imply during onboarding.
customer_id: Customer ID provided by Imply during onboarding.
admin_email: Email address for the initial administrative user.
admin_initial_password: Password for the initial administrative user. You are prompted to change this password on first login.
nlb_ingress_cidr_blocks: Controls the permitted inbound source IP ranges for the network load balancer's security group in public deployments.
- To allow access from any IP address, set this value to ["0.0.0.0/0"].
- To restrict access to a single IP address, specify the IP in CIDR notation with a /32 suffix, for example: ["203.0.113.45/32"].
If you don't set nlb_ingress_cidr_blocks, inbound access to the Lumi Enterprise UI is blocked. This doesn't prevent data ingestion from external sources. Alternatively, you can set nlb_additional_security_groups to configure additional security groups that define access rules. For example, permitting access only through a VPN server.

For a full list of what you can customize, see the Inputs section.

Create your instance

In your terminal, execute the following commands from the Terraform directory containing main.tf:

To initialize your Terraform workspace, run terraform init.
To create an execution plan, run terraform plan.
To apply the configuration, run terraform apply.

When prompted, review the execution plan and enter yes to apply the changes. Deployment of Lumi Enterprise can take up to an hour to complete.

After deploying Lumi Enterprise, you can access the user interface at the hosted zone's name. If you set the subdomain configuration, use subdomain.hosted_zone.name instead. The application is configured for HTTPS access only, so you need to specify https://.

For example:

If the hosted zone is example.com and subdomain was not set, you can access the application at https://example.com.
If the hosted zone is example.com and subdomain was set to lumi, you can access the application at https://lumi.example.com.

Use the admin_email and admin_initial_password credentials you provided in main.tf to log in to Lumi Enterprise for the first time. You are prompted to update the password after your first login.

Manage your instance

You can update your Lumi Enterprise instance by modifying the main.tf file and then running the Terraform commands terraform plan and terraform apply again.

To remove the Lumi Enterprise instance and destroy all associated infrastructure, run the terraform destroy command.

Inputs

The following table describes the fields that are available in the Lumi Enterprise Terraform module:

Name	Description	Required	Type	Default
`admin_email`	Email address for the initial administrative user.	yes	`string`	none
`admin_initial_password`	Password for the initial administrative user. You are prompted to change this password on first login.	yes	`string`	none
`api_key`	API key provided by Imply.	yes	`string`	none
`aws_profile`	AWS profile to use for authentication. If omitted, Terraform uses the default profile.	no	`string`	`null`
`customer_id`	Customer ID provided by Imply.	yes	`string`	none
`data_instance_count`	Number of data nodes to deploy. The value you provide is per availability zone and is doubled since Lumi Enterprise deploys data nodes to two availability zones.	no	`number`	1
`data_instance_type`	Instance type for data nodes. For details, see Supported instance types.	no	`string`	`i4i.large`
`db_control_plane_enable_high_availability`	Enable high availability for the control plane metadata store database.	no	`boolean`	`true`
`db_control_plane_enable_serverless`	Enable serverless mode for the control plane metadata store database.	no	`boolean`	`true`
`db_control_plane_instance_type`	Instance type for the metadata store database.	no	`string`	`db.r6g.large`
`db_control_plane_max_capacity`	Maximum capacity in Aurora Capacity Units (ACUs) for the control plane metadata store database.	no	`number`	`128`
`db_control_plane_min_capacity`	Minimum capacity in ACUs for the control plane metadata store database.	no	`number`	`0.5`
`db_data_plane_enable_high_availability`	Enable high availability for the data plane metadata store database.	no	`boolean`	`true`
`db_data_plane_enable_serverless`	Enable serverless mode for the data plane metadata store database.	no	`boolean`	`true`
`db_data_plane_instance_type`	Instance type for the metadata store database.	no	`string`	`db.r6g.large`
`db_data_plane_max_capacity`	Maximum capacity in ACUs for the data plane metadata store database.	no	`number`	`128`
`db_data_plane_min_capacity`	Minimum capacity in ACUs for the data plane metadata store database.	no	`number`	`1`
`eks_access_entries`	Map of access entries to add to the cluster.	no	`map(object)`	`{}`
`eks_enable_creator_admin_permissions`	Enable admin permissions for the cluster creator.	no	`boolean`	`true`
`eks_private_access_cidrs`	List of CIDRs that are allowed to access the EKS cluster endpoint.	no	`list(string)`	`[]`
`eks_public_access`	Enable public access to the EKS cluster endpoint.	no	`boolean`	`true`
`enable_telemetry`	Enable sending telemetry data to Imply.	no	`boolean`	`true`
`hosted_zone`	Identifier and name of the Route 53 hosted zone.	yes	`object({ id = string, name = string })`	none
`hot_data_retention_period`	Period to retain data on data nodes. Only applies if virtual nodes are enabled. Must be a valid ISO-8601 duration.	no	`string`	`P30D`
`ingestion_instance_type`	Instance type for the ingestion node. For details, see Supported instance types.	no	`string`	`lumi-peon-5vcpu`
`kong_max_replicas`	Maximum number of Kong replicas for autoscaling.	no	`number`	`10`
`kong_min_replicas`	Minimum number of Kong replicas for autoscaling.	no	`number`	`2`
`coordinator_instance_type`	Instance type for the coordinator node. For details, see Supported instance types.	no	`string`	`m7g.large`
`msk_broker_instance_type`	Instance type for the MSK brokers. All brokers use the same type.	no	`string`	`express.m7g.large`
`msk_brokers_per_az`	Number of broker instances in each availability zone of the MSK cluster.	no	`number`	`1`
`msk_ingress_cidrs`	List of additional CIDRs that are allowed to access the MSK cluster.	no	`list(string)`	`[]`
`name`	Deployment identifier. Must be unique per AWS account to avoid resource naming conflicts.	no	`string`	`lumi`
`nlb_additional_security_groups`	Additional security groups to attach to the NLB.	no	`list(string)`	`[]`
`nlb_egress_cidr_blocks`	CIDR blocks to allow egress traffic from the NLB.	no	`list(string)`	`[“0.0.0.0/0”]`
`nlb_enable_external_access`	Enable external access to the NLB.	no	`boolean`	`true`
`nlb_ingress_cidr_blocks`	CIDR blocks to allow ingress traffic to the NLB.	no	`list(string)`	`[]`
`broker_instance_count`	Number of broker nodes to deploy.	no	`number`	`2`
`broker_instance_type`	Instance type for the broker node. For details, see Supported instance types.	no	`string`	`m7g.xlarge`
`s3_set_block_public_access`	Block public access to the S3 bucket. You should only set this to `false` if you already have a default policy that restricts public access.	no	`boolean`	`true`
`subdomain`	Prefix to append to `hosted_zone.name` to configure Lumi Enterprise at a subdomain under the hosted zone's domain name. If provided, Lumi Enterprise is served from this subdomain. For example, if the hosted zone is `example.com` and `subdomain` is set to `lumi`, the application is served from `lumi.example.com`.	no	`string`	`null`
`tenant_id`	Tenant ID provided by Imply.	yes	`string`	none
`virtual_instance_count`	Number of virtual nodes to deploy. The value you provide is per availability zone and is doubled since Lumi Enterprise deploys virtual nodes to two availability zones.	no	`number`	`0`
`virtual_instance_type`	Instance type for the virtual node. Virtual nodes use the same instance families as data nodes.	no	`string`	`i4i.large`
`vpc_cidr`	CIDR block for the VPC.	yes	`string`	none

Supported instance types

Lumi Enterprise supports both AWS EC2 instances and Lumi-specific instances.

AWS EC2 instances

Lumi Enterprise supports the following AWS EC2 instance types by node type:

Data nodes

i3en.large through i3en.6xlarge
i4i.large through i4i.8xlarge
is4gen.large through is4gen.8xlarge
i7ie.large through i7ie.6xlarge
i8ge.large through i8ge.6xlarge

Coordinator nodes

m7g.large through m7g.4xlarge

Broker nodes

m7g.large through m7g.8xlarge

Lumi-specific instances

Lumi Enterprise supports the following Lumi-specific instance types for ingestion nodes:

peon-xlarge
lumi-peon-5vcpu
lumi-peon-7vcpu

Outputs

The following table describes the outputs of the Lumi Enterprise Terraform module:

Name	Description
`db_security_group_id`	ID of the database security group.
`eks_cluster_name`	Name of the EKS cluster.
`nlb_arn`	ARN of the NLB.
`private_route_table_ids`	IDs of the private route tables.
`public_route_table_ids`	IDs of the public route tables.
`vpc_cidr`	CIDR block of the VPC.
`vpc_id`	ID of the VPC.
`vpc_private_subnet_ids`	IDs of all private subnets in the VPC.

Learn more

See the following topics for more information:

Glossary for definitions of Lumi terms.
Send events to Lumi for details on sending events to Lumi.
Search Lumi events for options to search events in Lumi.
Lumi Management Console for an overview of the Lumi Management Console application.

Prerequisites​

AWS requirements​

Regions​

VPC​

DNS​

Configure the main.tf file​

Create your instance​

Manage your instance​

Inputs​

Supported instance types​

AWS EC2 instances​

Lumi-specific instances​

Outputs​

Learn more​