Monitor federated search performance
AI summary
About AI summaries.
When you use federated search to query Imply Lumi events in Splunk®, you can use the Splunk job inspector to examine aspects of the search. The job inspector includes components specific to Lumi to help you troubleshoot federated queries of Lumi events.
Access the Splunk job inspector
To access the job inspector:
-
In Splunk Web, perform a search against a Lumi federated index. For supported commands and query examples, see Federated search reference.
-
Below the search bar, click Job and select Inspect job. You can open the job inspector while the search is still running or after it completes.

The top section of the job inspector window shows the number of results, the number of events scanned, the duration of the search, and the search ID (SID).
-
The Execution costs section shows details for each search component, including its duration in seconds, the number of invocations, and input and output counts. Use this information to troubleshoot and improve slow queries. Scroll down to see the components that start with
lumi. For details, see Lumi components. -
To see the properties of the search and examine details of a specific search ID, click Jobs Detail Dashboard in the job inspector dialog.
-
Close the window when you've finished.
Lumi components
The following components in the job inspector are specific to Lumi.
Performance metrics
The following components measure query execution and data volume.
| Component | Description |
|---|---|
lumi.bytesProcessed.in | Total number of bytes scanned to run a query. |
lumi.bytesProcessed.out | Total number of bytes returned after filtering and processing a query. |
lumi.endToEnd | Total time to complete a query and send results from Lumi to Splunk. |
lumi.timeToFirstResult | Time taken to produce the first batch of results. |
Virtual compute metrics
The following components show how a query used a virtual compute pool to access data outside the hot tier.
| Metric | Description |
|---|---|
lumi.virtualCompute.dispatch | Indicates that the query ran on a virtual compute pool. Lumi adds this metric when the first results return. |
lumi.virtualCompute.poolWait.launching | Time the query waited for the virtual compute pool to start. |
lumi.virtualCompute.poolWait.capacity | Time the query waited for capacity in a running virtual compute pool. |
lumi.vsf.load | Number of bytes Lumi had to load into virtual storage on demand to run the query. |
Signals
The following components indicate conditions encountered during query processing.
| Component | Description |
|---|---|
lumi.signal.datamodel_requested_but_unavailable | Data models object is null. |
lumi.signal.datamodel_parse_failed | Data models object exists but couldn't be parsed. |
lumi.signal.no_datasets_in_datamodel | Data model parsed successfully but contains no datasets. |
lumi.signal.dataset_not_found | Requested dataset doesn't exist in the data model. |
lumi.signal.query_to_disallowed_index | Query references an index not in the allowed list. |
lumi.signal.index_missing_in_query | Non-data model query has no explicit index specified. |
lumi.signal.exception_in_transparent_mode | Exception occurred while processing a query in transparent federated search mode. |
Internal components
The following components are used internally by Imply.
lumi.intermediate.deserializelumi.resultGenlumi.writeToWire
Troubleshoot virtual tier queries
Use the virtual compute metrics to troubleshoot queries against the virtual tier. What counts as an acceptable wait depends on your workload, so compare the values to what you consider reasonable for your queries.
The query didn't use a virtual compute pool
If you don't see lumi.virtualCompute.dispatch in the job inspector, the query didn't run on a virtual compute pool.
Possible reasons include the following:
- The query only needed data in the hot tier.
- The IAM key doesn't have virtual tier access.
- The query hasn't sent any results yet. Lumi sets this metric only when the first results return, so it doesn't appear during a cold start.
The query incurred a cold start
If lumi.virtualCompute.poolWait.launching is higher than you expect, the query incurred a cold start.
Consider the following options:
- To reduce cold starts, increase the pool's idle time so the pool stays running between queries.
- To avoid timeouts that can occur during a cold start, increase the query timeout on the IAM key.
The query waited for capacity
If lumi.virtualCompute.poolWait.capacity is greater than zero, the pool was already running its maximum number of concurrent queries when the query was submitted.
The value shows how long the query waited for capacity to become available.
If the wait is longer than acceptable for your workload, consider the following options:
- Increase the maximum number of clusters for the pool. Lumi starts additional clusters when queries need more capacity.
- Choose a larger pool size with a higher max concurrency.
The query waited for data to load
If lumi.vsf.load is higher than you expect, the query had to load data that wasn't already in virtual storage.
Consider the following options:
- To reduce the amount of data the query needs, make the query more selective, for example by narrowing the time range or filtering on an index.
- To reduce how often Lumi reloads data, choose a larger pool size. A larger pool holds more data in virtual storage, so Lumi evicts less data.
Learn more
See the following topics and third-party documents for more information:
- Search events with Splunk to configure federated search of Lumi events within Splunk.
- Splunk documentation on search job properties.
- Splunk Job inspector tips and tricks blog post.