Install Lumi Enterprise with Terraform

AI summary

Explains how to deploy Imply Lumi Enterprise on AWS using Terraform. Covers prerequisites, configuration steps, and infrastructure requirements. Details VPC setup, DNS configuration, and supported instance types for your deployment.

About AI summaries.

Imply provides a Terraform module for you to deploy Lumi Enterprise.

This topic walks you through the steps of deploying Lumi Enterprise on AWS using Terraform. After Lumi Enterprise is deployed, administrators will want to learn about the integrations that Lumi supports and how to send events to Lumi. Users can get started with tutorials to familiarize themselves with how to query data.

Prerequisites

Before you begin, make sure you have the following:

• Terraform CLI version 1.12.0 or higher.

• AWS CLI version 2.27.0 or higher configured with a role that has full administrator permissions for the target AWS account. You need full administrator permissions because the module deploys a wide range of AWS resources on your behalf.

  The AWS CLI is often used with profiles, which are a way to manage multiple roles and users across different AWS accounts. If the principal you will be using for the installation is tied to a profile, make note of the profile name. You need this information to configure the Terraform module. For details, see Configuration and credential file settings in the AWS CLI in the AWS documentation.

• 12-digit AWS account ID for the account where you plan to deploy Lumi Enterprise. You must provide the account ID to your Imply representative so that Imply can grant you access to its private Amazon Elastic Container Registry (ECR).

• Imply provides the following arguments during onboarding. Enter these values in main.tf:

  • API key
  • Customer ID
  • Tenant ID

AWS requirements

The Lumi Terraform module deploys a complete Lumi Enterprise environment into an AWS account with all the necessary infrastructure and dependencies. This includes:

• VPC and networking resources
• Elastic Kubernetes Service (EKS) cluster
• Managed Streaming for Apache Kafka® (MSK) cluster
• RDS Aurora database (metadata store)
• S3 buckets (data storage)
• Cognito (identity management)

It requires access to AWS Certificate Manager (ACM), Route 53, Identity and Access Management (IAM), Key Management Service (KMS), and Secrets Manager to create the supporting records, certificates, service accounts, and secrets.

As a best practice, we recommend that you deploy Lumi Enterprise into a separate AWS account from other organizational resources to provide a controlled cloud environment and maintain separation of permissions.

Regions

You can deploy Lumi Enterprise in any standard AWS region. Deployment in restricted or specialized partitions, such as China or AWS GovCloud (US) regions, is not supported.

Provide this information to Imply before you deploy Lumi Enterprise.

VPC

Administrators often configure private networking between VPCs and across AWS accounts using PrivateLink, Transit Gateway, or VPC peering. When choosing a CIDR block for Lumi Enterprise, consider these connections and the CIDRs of any connected VPCs.

Select an RFC 1918 compliant CIDR block that doesn't conflict with other VPCs in your environment. The minimum supported size is /21, but we recommend using a /16 CIDR block to ensure you have sufficient IP addresses to scale your deployment.

Set the CIDR block using the vpc_cidr parameter in your main.tf file. The Terraform module creates the VPC and other networking resources within the specified CIDR block.

You can't expand a VPC or add more subnets after creation.

DNS

Lumi Enterprise requires a Route 53 hosted zone to create DNS records for its deployed services. This hosted zone must already exist in the AWS account where you plan to install the Terraform module, and it must be publicly accessible.

You can configure a Route 53 hosted zone in one of the following ways:

• A primary hosted zone for a domain that may be registered through Route 53 or another registrar.
  For example, deploying Lumi Enterprise at a new domain such as lumi-mycompany.com.
• A delegated hosted zone for a subdomain of an existing domain.
  For example, deploying Lumi Enterprise at the subdomain lumi.mycompany.com, where mycompany.com is managed elsewhere.

There are many ways to set up and manage DNS records, and these vary between organizations.

DaemonSets

EKS cluster DaemonSets are scheduled before other workloads. Over-provisioned DaemonSets can leave insufficient resources for Lumi services. To avoid resource conflicts, ensure your DaemonSet resource requests are accurate.

Tags

At a minimum consider creating Creator tag to make it easier to identify the resources that Lumi Enterprise creates in your AWS account. Other helpful tags may include Environment or Project.

Network requirements

Ensure that the following ports are open on your network to allow inbound traffic:

• HTTP 80 for initial connection and redirecting traffic to port 443.
• HTTPS 443 for secure web communications over SSL/TLS.
• TCP 9997 for Splunk® to receive data from forwarders.

For more information, see Ports used in Lumi.

Configure the main.tf file

After completing the prerequisites, you can begin writing the main Terraform configuration to deploy the Lumi Enterprise module.

1. Create a directory for your Terraform configuration files.
2. In the directory, create a new file named main.tf.

The following example provides a starting point for a deployment and includes both required and strongly recommended configuration arguments. Detailed descriptions of the Lumi module arguments are provided in the sections that follow. For a complete reference to the arguments supported in the AWS provider block, see the AWS documentation.

terraform {
  required_version = ">= 1.12.0"

  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = ">= 6.15.0"
    }
  }
}

provider "aws" {
  region  = "us-east-1"

  default_tags {
    tags = {
      Creator = "Lumi"
      Environment = "dev"
    }
  }
}

data "aws_route53_zone" "lumi_domain" {
  name = "example.com."
}

module "lumi" {
  source                        = "s3::https://s3.amazonaws.com/lumi-enterprise-artifacts/terraform-aws-lumi-enterprise-v1.latest.tar.gz"

  vpc_cidr                      = "10.92.0.0/16"
  hosted_zone                   = data.aws_route53_zone.lumi_domain

  tenant_id                     = "1234ab56"
  api_key                       = "a1234b5c-def1-2f3c-45bf-678f912345c6"
  customer_id                   = "z12bb34b-df56-4d78-9101-7d12b34567c8"

  admin_email                   = "user1@example.com"
  admin_initial_password        = "InitialPassword1"

  nlb_ingress_cidr_blocks       = ["10.0.0.0/16"]

}

The following arguments are required or strongly recommended when deploying Lumi Enterprise:

• source: Location of the Lumi Enterprise Terraform bundle. Use the URL provided unless otherwise directed by Imply.

• vpc_cidr: CIDR block for the VPC. This should not overlap with any other VPC in your environment.

• hosted_zone: Route 53 hosted zone, provided as an object:

    {
      id   = "Z1234567" // Hosted zone ID
      name = "lumi.example.com" // Hosted zone name
    }

  As shown in the example, you can use an aws_route53_zone data source to look up the hosted zone by the domain name instead of providing an object with the zone ID and name directly.

• tenant_id: Tenant ID provided by Imply during onboarding.

• api_key: API key provided by Imply during onboarding.

• customer_id: Customer ID provided by Imply during onboarding.

• admin_email: Email address for the initial administrative user.

• admin_initial_password: Password for the initial administrative user. You are prompted to change this password on first login.

• nlb_ingress_cidr_blocks: Controls the permitted inbound source IP ranges for the network load balancer's security group in public deployments.

  • To allow access from any IP address, set this value to ["0.0.0.0/0"].
  • To restrict access to a single IP address, specify the IP in CIDR notation with a /32 suffix, for example: ["203.0.113.45/32"].

  If you don't set nlb_ingress_cidr_blocks, inbound access to the Lumi Enterprise UI is blocked. This doesn't prevent data ingestion from external sources. Alternatively, you can set nlb_additional_security_groups to configure additional security groups that define access rules. For example, permitting access only through a VPN server.

For a full list of what you can customize, see the Inputs section.

Select a module version

The source variable in main.tf defines which Lumi version to install.

To update your Lumi version:

1. Update the version in the source variable in main.tf.
2. Run terraform apply.

For example:

module "lumi" {
  source = "s3::https://s3.amazonaws.com/lumi-enterprise-artifacts/terraform-aws-lumi-enterprise-v1.1.2.tar.gz"
}

Lumi follows semantic versioning, where the version number reflects the type of changes included. The version number consists of three parts:

v[MAJOR].[MINOR].[PATCH]

• MAJOR: Major version number. Incremented for breaking changes that are backward incompatible.
• MINOR: Minor version number. Incremented for new features that are backward compatible.
• PATCH: Patch version number. Incremented for improvements and bug fixes that are backward compatible.

You can reference the Lumi version using version numbers or latest:

terraform-aws-lumi-enterprise-v1.1.2.tar.gz      # pin to specific version
terraform-aws-lumi-enterprise-v1.1.latest.tar.gz # latest patch version
terraform-aws-lumi-enterprise-v1.latest.tar.gz   # latest minor version
terraform-aws-lumi-enterprise-latest.tar.gz      # latest version (not recommended for production)

If you pin to a specific version such as v1.1.2, you must follow the update steps above each time Imply releases a new version. If you use v[MAJOR].[MINOR].latest such as v1.1.latest, you don't need to update main.tf as frequently since Lumi automatically uses the latest patch version.

Use the following diagram to determine the right module version for your deployment:

info

When using latest for your patch version, you may need to run terraform init -upgrade to refresh the Terraform source file before running terraform apply. For example, if terraform-aws-lumi-enterprise-v1.1.latest.tar.gz points to v1.1.2 and Imply releases v1.1.3, you may have v1.1.2 cached locally. This is more likely to occur in manual deployments than in automated pipelines.

Create your instance

In your terminal, execute the following commands from the Terraform directory containing main.tf:

1. To initialize your Terraform workspace, run terraform init.
2. To create an execution plan, run terraform plan.
3. To apply the configuration, run terraform apply.

When prompted, review the execution plan and enter yes to apply the changes. Deployment of Lumi Enterprise can take up to an hour to complete.

After deploying Lumi Enterprise, you can access the user interface at the hosted zone's name. If you set the subdomain configuration, use subdomain.hosted_zone.name instead. The application is configured for HTTPS access only, so you need to specify https://.

For example:

• If the hosted zone is example.com and subdomain was not set, you can access the application at https://example.com.
• If the hosted zone is example.com and subdomain was set to lumi, you can access the application at https://lumi.example.com.

Use the admin_email and admin_initial_password credentials you configured in main.tf to log in to Lumi Enterprise for the first time. Lumi prompts you to update your password after your first login.

Admin user credentials

The values assigned to admin_email and admin_initial_password remain in main.tf. To remove the admin credentials from Terraform, you must disable the system admin user, after which you can set admin_email and admin_initial_password to null.

warning

Before disabling the system admin user, ensure that you have another admin user or an identity provider (IdP) configured. If single sign-on (SSO) becomes unavailable and no other admin user exists, there is no fallback login option until you re-enable the system admin user.

To disable the system admin user, set the enable_admin_user property to false in main.tf, then run terraform apply.

Manage your instance

You can update your Lumi Enterprise instance by modifying the main.tf file and then running the Terraform commands terraform plan and terraform apply again.

To remove the Lumi Enterprise instance and destroy all associated infrastructure, run the terraform destroy command.

Inputs

The following table describes the fields that are available in the Lumi Enterprise Terraform module:

Name	Description	Required	Type	Default
admin_email	Email address for the initial administrative user	yes	string	none
admin_initial_password	Password for the initial administrative user. You are prompted to change this password on first login.	yes	string	none
api_key	API key provided by Imply	yes	string	none
auth_mode	Authentication mode to use. cognito tells Lumi to use Amazon Cognito. Set to external to use an external identity provider.	no	string	cognito
aws_profile	AWS profile to use for authentication. If omitted, Terraform uses the default profile.	no	string	null
broker_instance_count	Number of broker nodes to deploy	no	number	2
broker_instance_type	Instance type for the broker node	no	string	m7g.xlarge
coordinator_instance_type	Instance type for the coordinator node	no	string	m7g.xlarge
customer_id	Customer ID provided by Imply	yes	string	none
data_instance_count	Number of data nodes to deploy. The value you provide is per availability zone and is doubled since Lumi Enterprise deploys data nodes to two availability zones.	no	number	1
data_instance_type	Instance type for data nodes. For details, see Supported instance types.	no	string	i3en.2xlarge
db_control_plane_enable_high_availability	Enable high availability for the control plane metadata store database.	no	boolean	true
db_control_plane_enable_serverless	Enable serverless mode for the control plane metadata store database	no	boolean	true
db_control_plane_instance_type	Instance type for the metadata store database	no	string	db.r6g.large
db_control_plane_max_capacity	Maximum capacity in Aurora Capacity Units (ACUs) for the control plane metadata store database	no	number	128
db_control_plane_min_capacity	Minimum capacity in ACUs for the control plane metadata store database	no	number	0.5
db_data_plane_enable_high_availability	Enable high availability for the data plane metadata store database	no	boolean	true
db_data_plane_enable_serverless	Enable serverless mode for the data plane metadata store database	no	boolean	true
db_data_plane_instance_type	Instance type for the metadata store database	no	string	db.r6g.large
db_data_plane_max_capacity	Maximum capacity in ACUs for the data plane metadata store database	no	number	128
db_data_plane_min_capacity	Minimum capacity in ACUs for the data plane metadata store database	no	number	1
eks_access_entries	Map of access entries permitted to access the EKS cluster	no	map(object)	{}
eks_enable_creator_admin_permissions	Enable admin permissions for the cluster creator	no	boolean	true
eks_private_access_cidrs	List of CIDRs that are allowed to access the EKS cluster endpoint	no	list(string)	[]
eks_public_access	Enable public access to the EKS cluster endpoint. When private networking is configured, set this to false to disable public access.	no	boolean	true
enable_admin_user	Enable the admin user. Before disabling, ensure that another user exists or an IdP is configured.	no	boolean	true
enable_telemetry	Enable sending telemetry data to Imply	no	boolean	true
hosted_zone	Identifier and name of the Route 53 hosted zone	yes	object({ id = string, name = string })	none
kong_max_replicas	Maximum number of Kong replicas for autoscaling	no	number	10
kong_min_replicas	Minimum number of Kong replicas for autoscaling	no	number	2
coordinator_instance_type	Instance type for the coordinator node. For details, see Supported instance types.	no	string	m7g.xlarge
msk_broker_instance_type	Instance type for the MSK brokers. All brokers use the same type.	no	string	express.m7g.large
msk_brokers_per_az	Number of broker instances in each availability zone of the MSK cluster	no	number	1
msk_ingress_cidrs	List of additional CIDRs that are allowed to access the MSK cluster	no	list(string)	[]
name	Deployment identifier. Must be unique per AWS account to avoid resource naming conflicts.	no	string	lumi
nlb_additional_security_groups	Additional security groups to attach to the NLB	no	list(string)	[]
nlb_egress_cidr_blocks	CIDR blocks to allow egress traffic from the NLB	no	list(string)	[“0.0.0.0/0”]
nlb_enable_external_access	Enable external access to the NLB	no	boolean	true
nlb_ingress_cidr_blocks	CIDR blocks to allow ingress traffic to the NLB	no	list(string)	[]
s3_set_block_public_access	Block public access to the S3 bucket. You should only set this to false if you already have a default policy that restricts public access.	no	boolean	true
subdomain	Prefix to append to hosted_zone.name to configure Lumi Enterprise at a subdomain under the hosted zone's domain name. If provided, Lumi Enterprise is served from this subdomain. For example, if the hosted zone is example.com and subdomain is set to lumi, the application is served from lumi.example.com.	no	string	null
tenant_id	Tenant ID provided by Imply	yes	string	none
vpc_cidr	CIDR block for the VPC	yes	string	none

Supported instance types

Lumi Enterprise supports both AWS EC2 instances and Lumi-specific instances.

AWS EC2 instances

Lumi Enterprise supports the following AWS EC2 instance types by node type:

Data nodes

• i3en.large through i3en.6xlarge
• i4i.large through i4i.8xlarge
• is4gen.large through is4gen.8xlarge
• i7ie.large through i7ie.6xlarge
• i8ge.large through i8ge.6xlarge

Coordinator nodes

• m7g.large through m7g.4xlarge

Broker nodes

• m7g.large through m7g.8xlarge

Lumi-specific instances

Lumi Enterprise supports the following Lumi-specific instance types for ingestion nodes:

• peon-xlarge
• lumi-peon-5vcpu
• lumi-peon-7vcpu

Outputs

The following table describes the outputs of the Lumi Enterprise Terraform module:

Name	Description
db_security_group_id	ID of the database security group.
eks_cluster_name	Name of the EKS cluster.
nlb_arn	ARN of the NLB.
private_route_table_ids	IDs of the private route tables.
public_route_table_ids	IDs of the public route tables.
vpc_cidr	CIDR block of the VPC.
vpc_id	ID of the VPC.
vpc_private_subnet_ids	IDs of all private subnets in the VPC.

Learn more

See the following topics for more information:

• Glossary for definitions of Lumi terms.
• Send events to Lumi for details on sending events to Lumi.
• Search Lumi events for options to search events in Lumi.
• Lumi Management Console for an overview of the Lumi Management Console application.