Skip to main content

Configure tiering rules

Early access

This is an early access feature. Contact your Imply support representative to request access.

By default, Imply Lumi copies ingested data to the hot tier, which provides persistent storage and compute resources for low latency and high concurrency. For price-performance balance, Lumi provides a virtual tier that loads data and spins up compute resources on demand at a lower cost.

To use the virtual tier, you create a tiering rule that specifies when to remove data from hot storage. Removed data remains in object storage and is loaded into virtual storage on demand when queried.

This topic explains how to create a tiering rule in Lumi.

Prerequisites​

To create a tiering rule, you need the following:

  • Virtual tier enabled for your Lumi account. Contact your Imply support representative to request access.
  • A Lumi user with the Admin role. For information on roles and permissions, see Manage roles.

Additionally, to query data in virtual storage using Splunk® Search Processing Language (SPL), you must enable virtual tier access in the Tiering access field for the IAM key. For more information, see Federated search attributes. Non-SPL integrations, such as Grafana, Tableau, or Model Context Protocol (MCP), can query both hot and virtual tier by default.

Use a virtual tier​

To use a virtual tier, create a virtual storage rule and configure the virtual compute pool.

Create a virtual storage rule​

A virtual storage rule specifies when to evict data from hot storage.

To create a virtual storage rule:

  1. From the Lumi navigation menu, click Data.
  2. On the Virtual storage page, click + Create rule.
  3. In the Create virtual storage rule pane, enter the following:
    • Name: Rule name
    • Description: Optional description of what the rule does.
    • Retention period: Period to retain data in hot storage. Lumi stores data in hot storage for the duration of this period before evicting it. After eviction, the data remains accessible through virtual storage.
    • Index: Index to filter events. Events that match the filter are evicted from hot storage at the end of the retention period. You can preview matching events in the Events matched section.
  4. Click Create.

Configure the virtual compute pool​

When you add your first virtual storage rule, Lumi provisions a single Large-size virtual compute pool with a 20-minute maximum idle time by default. You can create multiple virtual storage rules, but they all share the same virtual compute pool. You incur charges when the virtual compute pool is running, whether it's actively serving queries or idle.

To configure your virtual compute pool settings:

  1. From the Lumi navigation menu, click Data > Virtual compute.
  2. Click Edit next to the virtual compute pool.
  3. In the Virtual compute settings pane, select the appropriate pool size for your workload. We recommend starting with the Large size and scaling up or down based on observed performance.
  4. In the Idle time section, set the maximum time the pool can remain idle before shutting down.
  5. Click Save.

Lumi provides storage metrics to help you understand how your data is distributed across tiers.

Idle time and cold starts​

The first query you run after the virtual compute pool shuts down due to inactivity might be slower because Lumi needs to provision compute resources and load the necessary data into virtual storage. This delay is known as a cold start. To avoid cold start latency for time-sensitive queries, you can warm the virtual tier by issuing a warm-up query in anticipation of running important queries. The warm-up query spins up the compute resources and loads the data into virtual storage, so subsequent queries run faster.

The idle time setting controls how long the virtual compute pool remains available before shutting down. When configuring the idle time for your virtual compute pool, consider cold starts:

  • A shorter idle time reduces costs, but leads to more cold starts.
  • A longer idle time keeps the pool available longer, but incurs more charges.

Conflicting rules​

If you have conflicting tiering rules, Lumi keeps data in hot storage for the longest retention period. For example, if one rule evicts data after 14 days and another after 30 days, Lumi keeps the data in hot storage for 30 days.

Deletion rules take precedence over tiering rules. For example, if a tiering rule says to keep data in hot storage for 30 days but a deletion rule says to delete it after 14 days, the deletion rule applies.

Manage tiering rules​

You can edit, duplicate, or delete a tiering rule:

  1. From the Lumi navigation menu, click Data > Virtual storage.
  2. Click the ellipsis next to the rule name.
  3. Select Edit, Duplicate, or Delete.
  4. If editing or duplicating, make the necessary changes, then click Save.
  5. If deleting, confirm your selection.

Learn more​

For more information, see the following topics: