Skip to main content

Configure a virtual tier

AI summary
Explains how to create tiering rules in Imply Lumi to move data out of hot storage. Describes configuring virtual compute pools to balance cost and performance. Covers managing cold starts and handling conflicting rules.

About AI summaries.
Early access

Virtual tier is an early access feature. If you don't have access to the virtual tier, contact your Imply support representative.

Imply Lumi supports two tiers of data storage for ingested data:

  • Hot tier: High-performance cache layer optimized for low-latency queries. Lumi stores all ingested data in the hot tier by default.
  • Virtual tier: On-demand cache layer that loads data when queried at a lower cost than the hot tier, allowing you to retain more data while balancing performance and cost.

This topic explains how to configure a virtual tier. For more information about data tiers, see Virtual tier overview.

Prerequisites​

To use the virtual tier feature, you need the following:

  • Virtual tier enabled for your Lumi account. The virtual tier is an early access feature and may not be available for your account. If you don't have access to the virtual tier, contact your Imply support representative.
  • A Lumi user with the Admin role. For information on roles and permissions, see Manage roles.

Create a virtual compute pool​

To query data outside the hot tier, you must first create a virtual compute pool. The first virtual compute pool you create becomes the default pool.

You can have up to 10 virtual compute pools per company account, each processing queries in isolation.

To create a virtual compute pool:

  1. From the Lumi navigation menu, click Data > Virtual compute pools.
  2. Click + Create pool.
  3. In the Create virtual compute pool pane, enter the following:
    • Name: Pool name
    • Description: Optional description of the pool.
    • Keys: IAM keys with the federated search integration that can route queries to this pool. If the IAM key does not exist yet, leave the Keys field blank and assign this pool when you create the key. For more information, see IAM key access.
    • Size: Compute resources allocated to virtual storage queries. For more information, see Size.
    • Idle time: Maximum time the virtual compute pool can remain idle before shutting down. Idle time must be between 15 minutes and 3 days.
  4. Click Create.

Once the pool is created, you can configure tiering rules to control when data moves out of hot storage.

Size​

The size of a virtual compute pool is one of the factors that determine how much compute power is available to process a query. Other factors include the volume of data scanned, the shape of the data, and the complexity of the query.

Lumi measures the pool size in t-shirt sizes, such as Small, Medium, and Large.

The following table shows the default t-shirt sizes and their corresponding query concurrency:

SizeMax concurrency
X-small, Small, Medium4
Large8
X-large, 2X-large, 3X-large12

Lumi Enterprise customers can define custom t-shirt sizes in Terraform. For more information, see Configure custom t-shirt sizes for Lumi Enterprise.

When selecting a t-shirt size, consider the following general guidelines:

  • Number of concurrent queries: The more concurrent queries you run, the more compute resources you may need.
  • Data size and composition: Larger and more complex datasets may require more compute resources to query.
  • Query latency: If low latency is required, you may need a larger t-shirt size for faster query execution and a longer idle time to avoid cold start delays.

Idle time and cold starts​

When the first query spins up a virtual compute pool, Lumi provisions compute resources and loads the necessary data into virtual storage before serving the query. This process is known as a cold start. To avoid cold start latency for time-sensitive queries, you can warm the virtual tier by issuing a warm-up query in anticipation of running important queries. The warm-up query spins up the compute resources and loads the data into virtual storage, so subsequent queries run faster.

The idle time setting controls how long the virtual compute pool remains available before shutting down. When configuring the idle time for your virtual compute pool, consider the following:

  • A shorter idle time reduces costs, but leads to more cold starts.
  • A longer idle time keeps the pool available longer, but incurs more charges.

Create a tiering rule​

A tiering rule determines how long your data remains in the hot tier. You must create at least one virtual compute pool before you can add a tiering rule.

To create a tiering rule:

  1. From the Lumi navigation menu, click Data > Tiering rules.
  2. Click + Create rule.
  3. In the Create rule pane, enter the following:
    • Name: Rule name. The name is limited to 20 alphanumeric characters and can't be modified once set.
    • Description: Optional description of what the rule does.
    • Move to virtual tier after: Period of time Lumi retains data in the hot tier before evicting it.
    • Index: Index to filter events (for example, index=demo_web_lumi). Events that match the filter are evicted from hot storage at the end of the retention period. You can preview matching events in the Events matched section.
  4. Click Create.

Manage the virtual tier​

To modify a tiering rule:

  1. From the Lumi navigation menu, click Data > Tiering rules.
  2. Click the ellipsis next to the rule name.
  3. Select Edit, Duplicate, or Delete.
    • If editing or duplicating, make the necessary changes, then click Save.
    • If deleting, confirm your selection, then click Delete.

To modify a virtual compute pool:

  1. From the Lumi navigation menu, click Data > Virtual compute pools.
  2. Click the ellipsis next to the pool name.
  3. Select Edit or Delete.
    • If editing, make the necessary changes, then click Save.
    • If deleting, confirm your selection, then click Delete.
info

Updating IAM keys assigned to a virtual compute pool can take up to two minutes to take effect.

Learn more​

For more information, see the following topics: