Skip to main content

Lumi Enterprise

AI summary
Explains how to deploy Imply Lumi Enterprise in your AWS environment using Terraform for data sovereignty and compliance. Covers infrastructure setup, event data ingestion through MSK, storage management, and cluster monitoring. Includes deployment architecture and scaling options.

About AI summaries.

Imply Lumi Enterprise is a deployment of Lumi that runs in your AWS environment, keeping your data within your own infrastructure. Lumi Enterprise is well-suited for organizations that have strict data sovereignty, security, and compliance requirements.

Imply provides a Terraform module that simplifies Lumi Enterprise deployment.

The module handles all deployment steps, including the AWS infrastructure and dependent PaaS services:

  • The Imply distribution of Apache® Druid
  • Amazon Managed Streaming for Apache® Kafka (Amazon MSK)
  • Amazon Elastic Kubernetes Service (Amazon EKS)
  • Amazon S3
  • Amazon RDS Aurora MySQL

You don't need to perform any additional installation or configuration outside of the Terraform module.

Lumi uses the Imply distribution of Druid to power its services and MSK to stream event data from various sources.

Lumi infrastructure​

The following table provides a basic overview of nodes that Lumi deploys:

NodeDescription
CoordinatorManages data availability and distribution, enforces data retention rules, and oversees ingestion tasks.
BrokerRoutes queries from external clients to the appropriate nodes. Lumi Brokers are unrelated to MSK brokers.
DataServes frequently accessed data with low latency and high performance. Data nodes power the hot tier, consisting of a persistent cache (hot storage) and dedicated compute resources (hot compute).
VirtualServes infrequently accessed data at lower cost. Virtual nodes power the virtual tier, consisting of an on-demand cache (virtual storage) that loads data when queried and virtual compute resources that spin up to serve queries against virtual storage.
IngestionReads data from source systems, then transforms, indexes, and partitions the data into files. MSK streams event data to ingestion nodes, where the Indexer service processes it.

Event data ingestion​

MSK streams event data to Lumi. The Indexer service on Ingestion nodes runs the tasks. As part of the ingestion process, Lumi indexes and pre-aggregates data to speed up query processing.

Event data storage​

Lumi organizes ingested data into time-based segments. During segment creation, Lumi pre-indexes the data to speed up query processing. This segment-based approach allows Lumi to efficiently store and retrieve data by accessing only the segments that fall within the requested time interval.

Lumi loads segments onto Data nodes by default. You can create a tiering rule that specifies when to remove the data. Data removed from Data nodes remains accessible through Virtual nodes on demand.

Queries​

The Lumi Broker routes your queries to the Data nodes containing the relevant segment if the queried time interval falls within the hot data retention period. Queries for data outside that range are routed to Virtual nodes.

If you query data that's actively streaming to Lumi, the Broker routes your query to the Indexer.

Lumi Management Console​

After you deploy Lumi Enterprise, you can monitor your deployment through the Lumi Management Console hosted by Imply.

As you monitor your deployment, use the Terraform module to scale your cluster to better meet your needs.

Updates​

Imply periodically releases updates to Lumi Enterprise. Application updates are delivered automatically and include new features and bug fixes. These updates are separate from updates to the Lumi Terraform module. When a new version of the Lumi module is available, you'll need to update main.tf and run terraform apply to ensure your infrastructure stays current.

Get started​

When you're ready to get started, learn how to install Lumi Enterprise using Terraform.