Virtual tier overview
AI summary
About AI summaries.
By default, Imply Lumi copies ingested data to the hot tier, a high-performance cache layer optimized for low-latency queries. The hot tier consists of a persistent cache, called hot storage, and persistent compute resources that serve queries. The hot tier is best suited for frequently accessed data that needs to be available immediately without spin-up time or data loading delays.
Because the hot tier requires persistent compute resources, keeping large volumes of data in the hot tier can be cost-prohibitive, especially for data that you query only occasionally. To balance performance and cost, Lumi provides the virtual tier, which loads data and spins up compute resources on demand at a lower cost.
Virtual tier
The virtual tier is a cost-efficient alternative to the hot tier that is suitable for infrequently accessed data, such as historical investigations, audits, and archival data. The virtual tier consists of the following components:
- Virtual storage: Transient cache where data is loaded on demand. Subsequent queries can read from the same cache if the cache already contains the required data.
- Virtual compute: Transient compute resources, called virtual compute pools, that serve queries against virtual storage and external data.
When you query data outside the hot tier, Lumi starts the virtual compute pool for that query, unless the pool is already running. The pool shuts down automatically after reaching the maximum idle time you set.
New Lumi Cloud accounts include a default virtual compute pool. In Lumi Enterprise and existing Lumi Cloud accounts, the first pool you create becomes the default pool. For more information, see Manage virtual compute pools.
Tiering rules
Tiering rules control when data moves out of the hot tier and becomes accessible through the virtual tier. When you create a tiering rule, you specify the index of data and the hot tier retention period. When the retention period expires, data that matches the specified index is removed from the hot tier and remains accessible through the virtual tier on demand. For information on how to create a tiering rule, see Manage tiering rules.
The following diagram illustrates the relationship between tiers:
In Lumi Cloud, data remains in the hot tier for up to 90 days based on when the event was created. To move data out of the hot tier sooner, configure a tiering rule. In Lumi Enterprise, data remains in the hot tier until a tiering rule that you configure moves it out of the hot tier.
If multiple tiering rules target the same index, Lumi applies the rule with the longest retention period. Deletion rules always take precedence over tiering rules. For more information, see How Lumi applies tiering rules.
Optimize virtual tier query performance
Queries against the virtual tier can perform comparably to the hot tier when the virtual compute pool is running and the required data is already cached. Additional latency occurs mainly when Lumi needs to start the pool or load data into virtual storage. Other factors, such as the pool size and max concurrency, also affect query performance.
To improve query performance or resolve timeouts, consider the following:
- Query timeout: If queries time out before they complete, increase the query timeout attribute on the IAM key. For more information, see Federated search attributes.
- Idle time: If queries are slow because the pool has to start, increase the idle time so the pool stays running longer between queries. For more information, see Idle time and cold starts.
- Pool size: If you are querying a large amount of data, ensure that your compute pool is configured with sufficient resources for your workload. Consider increasing the pool size to handle larger query loads.
Learn more
For more information, see the following topics:
- Manage virtual compute pools to create and manage virtual compute pools.
- Manage tiering rules to create and manage tiering rules.
- Manage deletion rules to set up automatic data deletion.