Skip to main content

Configure tiering rules

info

This information applies to Lumi Enterprise.

All data you ingest into Imply Lumi is written to object storage by default. To deliver low latency, high concurrency, and consistent query performance, Lumi copies the data to the hot tier. For data that doesn't need always-on infrastructure, Lumi provides a virtual tier, which offers lower costs at the expense of higher latency.

This topic introduces Lumi data tiers and explains how to configure tiering rules. For information on Lumi Enterprise storage metrics, see Storage metrics reference.

Prerequisites​

To configure tiering rules, you need the Admin role. For information on roles and permissions, see Manage roles.

The following sections describe the Lumi data tiers.

Hot tier​

The hot tier is enabled by default and provides low-latency access to frequently queried data. It delivers fast query performance but consumes more infrastructure resources.

The hot tier consists of:

  • Hot storage: High-performance persistent cache.
  • Hot compute: Compute resources that serve queries against hot storage.

Virtual tier​

The virtual tier provides cost-efficient access to infrequently queried data at higher latency than the hot tier. Like the hot tier, the virtual tier caches data, so frequently accessed data may deliver performance comparable to the hot tier, depending on the eviction rate. To monitor the data eviction rate for your virtual tier, see to the Virtual tier churn ratio metric.

When you query data outside the hot tier, Lumi retrieves the necessary data from object storage and loads it into virtual storage, trading some query performance for cost savings. This frees up local disk space and allows you to size your hot tier based on compute needs rather than storage capacity.

The virtual tier consists of:

  • Virtual storage: High-latency more economical transient cache.
  • Virtual compute: Compute resources that load the necessary data on demand and serve queries against virtual storage.

To add a virtual tier, configure the following properties in the main.tf file of the Lumi Terraform module:

  • hot_data_retention_period: Period to retain data in the hot tier, in ISO 8601 format. Retention is based on the event's timestamp, not the time the event was persisted in Lumi. Data that exceeds the retention period is removed from hot storage but remains accessible in object storage. Defaults to P30D.
  • virtual_instance_type: AWS EC2 instance type for virtual nodes. Virtual nodes use the same instance families as data nodes. For more information, see Supported instance types. Defaults to i3en.2xlarge.
  • virtual_instance_count: Number of virtual nodes to deploy. Defaults to 0.

For example:

module "lumi" {
   hot_data_retention_period = "P90D"
   virtual_instance_type = "i3en.3xlarge"
   virtual_instance_count = 2
}

Learn more​

For more information, see the following topics: