Virtual tier overview
AI summary
About AI summaries.
By default, Imply Lumi copies ingested data to the hot tier, a high-performance cache layer optimized for low-latency queries. The hot tier consists of a persistent cache, called hot storage, and persistent compute resources that serve queries. The hot tier is best suited for frequently accessed data that needs to be available immediately without spin-up time or data loading delays.
Because the hot tier requires persistent compute resources, keeping large volumes of data in the hot tier can be cost-prohibitive, especially for data that you query only occasionally. To balance performance and cost, Lumi provides the virtual tier, which loads data and spins up compute resources on demand at a lower cost.
Virtual tier
The virtual tier is a cost-efficient alternative to the hot tier that is suitable for infrequently accessed data, such as historical investigations, audits, and archival data. The virtual tier consists of the following components:
- Virtual storage: Transient cache where data is loaded on demand. Subsequent queries can read from the same cache if the cache already contains the required data.
- Virtual compute: Transient compute resources that serve queries against virtual storage. When you query data outside hot storage, Lumi spins up the assigned virtual compute pool unless the pool is already running. The pool shuts down automatically after reaching the maximum idle time you set.
Queries against the virtual tier can incur additional latency compared to the hot tier. Several factors contribute to this latency, including the pool's size, max concurrency, and status, and whether data is already loaded. For more information, see Idle time and cold starts.
Virtual compute pools
A virtual compute pool represents transient compute resources that Lumi provisions to serve queries that target data outside the hot tier.
To query data outside the hot tier, you must first create a virtual compute pool. The first virtual compute pool you create becomes the default pool, which Lumi uses to serve queries across both hot and virtual storage as needed. You can modify the default pool, but you can't delete it.
You can have up to 10 virtual compute pools per company account, each processing queries in isolation. For information on how to create a virtual compute pool, see Manage virtual compute pools.
Tiering rules
Tiering rules control when data moves out of the hot tier and becomes accessible through the virtual tier. When you create a tiering rule, you specify the index of data and the hot tier retention period. When the retention period expires, data that matches the specified index is removed from the hot tier and remains accessible through the virtual tier on demand. For information on how to create a tiering rule, see Manage tiering rules.
The following diagram illustrates the relationship between tiers:
In Lumi Cloud, data remains in the hot tier for up to 90 days based on when the event was created. To move data out of the hot tier sooner, configure a tiering rule. In Lumi Enterprise, data remains in the hot tier until a tiering rule that you configure moves it out of the hot tier.
If multiple tiering rules target the same index, Lumi applies the rule with the longest retention period. Deletion rules always take precedence over tiering rules. For more information, see How Lumi applies tiering rules.
Troubleshoot virtual tier queries
If your queries against data in virtual storage are timing out, check the following:
- Query timeout: Check the query timeout attribute on the IAM key. If the timeout value is too short, increase it to give queries more time to complete.
- Idle time: If the query times out because of the time it takes to spin up a virtual compute pool, you may be experiencing a cold start issue. Consider increasing the idle time for the pool to reduce cold starts. For more information, see Idle time and cold starts.
- Pool size: If you are querying a large amount of data, ensure that your compute pool is configured with sufficient resources for your workload. Consider increasing the pool size to handle larger query loads.
Learn more
For more information, see the following topics:
- Manage virtual compute pools to create and manage virtual compute pools.
- Manage tiering rules to create and manage tiering rules.
- Manage deletion rules to set up automatic data deletion.