Virtual tier overview
AI summary
About AI summaries.
Virtual tier is an early access feature. If you don't have access to the virtual tier, contact your Imply support representative.
By default, Imply Lumi copies ingested data to the hot tier, a high-performance cache layer optimized for low-latency queries. The hot tier consists of a persistent cache, called hot storage, and persistent compute resources that serve queries. The hot tier is best suited for frequently accessed data that needs to be available immediately without spin-up time or data loading delays.
Because the hot tier requires persistent compute resources that are always running, keeping large volumes of data in the hot tier can be cost-prohibitive, especially for data that you query only occasionally or data that is no longer frequently accessed. To balance performance and cost, Lumi provides a virtual tier that loads data and spins up compute resources on demand at a lower cost.
Virtual tier
The virtual tier is a cost-efficient alternative to the hot tier that is suitable for infrequently accessed data, such as historical investigations, audits, and archival data. The virtual tier consists of the following components:
- Virtual storage: Transient cache where data is loaded on demand. Subsequent queries can read from the same cache if the cache already contains the required data.
- Virtual compute: Transient compute resources that serve queries against virtual storage. When a query targets data outside hot storage, Lumi spins up a virtual compute pool to serve the query. The pool shuts down automatically after it has been inactive for the maximum idle time you configure.
The virtual tier may deliver performance comparable to the hot tier for frequently accessed data if the virtual compute pool is running and the required data is already cached.
Tiering rules
How long data remains in hot storage depends on your deployment model:
- Lumi Cloud: Data remains in hot storage for up to 90 days based on when the event was created. You can use tiering rules to move data out of hot storage sooner.
- Lumi Enterprise: Data remains in hot storage until it exceeds the retention period you configure using tiering rules.
When you create a tiering rule, you specify the index of data and the hot storage retention period. When the retention period expires, data that matches the specified index is removed from hot storage and remains accessible through virtual storage on demand.
The following diagram illustrates the relationship between tiers:
For information on how to create a tiering rule, see Configure a virtual tier.
Conflicting tiering rules
If two tiering rules target the same index, Lumi applies the rule with the longest retention period.
The following diagram shows tiering rules A and B that target the same index. Lumi applies rule B because it has a longer retention period.
Rule precedence
Deletion rules take precedence over tiering rules.
The following diagram shows a tiering rule and deletion rule that target the same index. Lumi applies the deletion rule.
Data residency
Configuring tiering rules doesn't change where your data resides:
- In Lumi Cloud, data resides in the Lumi AWS environment.
- In Lumi Enterprise, data resides in your AWS environment.
Virtual compute pools
A virtual compute pool represents ephemeral compute resources that Lumi provisions to serve queries that target data outside hot storage. Lumi shuts down the virtual compute pool after it reaches the maximum idle time you set.
To query data outside the hot tier, you must first create a virtual compute pool. The first virtual compute pool you create becomes the default pool, which Lumi uses to serve queries across both hot and virtual storage as needed. You can modify the default pool, but you can't delete it.
You can have up to 10 virtual compute pools per company account, each processing queries in isolation. For information on how to create a virtual compute pool, see Configure the virtual compute pool.
IAM key access
When you create an IAM key for the federated search integration, you specify virtual tier access for the key. You can use the pre-selected default pool or choose a different pool if you have multiple pools configured.
Application integrations that don't use federated search, such as Grafana, Tableau, or Model Context Protocol (MCP), always route queries to the default pool.
You can assign multiple IAM keys to a virtual compute pool, but a key can only have one pool.
To restrict virtual tier access, select No virtual tier access when configuring the IAM key. Keys without virtual tier access can only query data in the hot tier.
Status
Because virtual compute resources spin up on demand and shut down when idle, the status of the virtual compute pool changes based on its current state:
- Starting: Virtual compute pool is starting. Lumi is provisioning compute resources to serve queries.
- Running: Virtual compute pool is running and serving queries.
- Stopped: Virtual compute pool is inactive. All compute resources are shut down.
To see the current status of your virtual compute pool, go to Data > Virtual compute.
Troubleshoot virtual tier queries
If queries against data in virtual storage are timing out, check the following:
- Query timeout: Check the query timeout attribute on the IAM key. If the timeout value is too short, increase it to give queries more time to complete.
- Idle time: If the query times out because of the time it takes to spin up a virtual compute pool, you may be experiencing a cold start issue. Consider increasing the idle time for the pool to reduce cold starts. For more information, see Idle time and cold starts.
- Pool size: If you are querying a large amount of data, ensure that your compute pool is configured with sufficient resources for your workload. Consider increasing the pool size to handle larger query loads.
Learn more
For more information, see the following topics:
- Configure a virtual tier to create and manage tiering rules and virtual compute pools.
- Configure deletion rules to set up automatic data deletion.