Skip to main content

Monitor federated search performance

AI summary
Explains how to use the Splunk® job inspector to troubleshoot federated queries of Imply Lumi events. Covers Lumi-specific components for measuring query performance, virtual compute pool usage, and data access patterns. Provides guidance on interpreting metrics and resolving common issues.

About AI summaries.

When you use federated search to query Imply Lumi events in Splunk®, you can use the Splunk job inspector to examine aspects of the search. The job inspector includes components specific to Lumi to help you troubleshoot federated queries of Lumi events.

Access the Splunk job inspector​

To access the job inspector:

  1. In Splunk Web, perform a search against a Lumi federated index. For supported commands and query examples, see Federated search reference.

  2. Below the search bar, click Job and select Inspect job. You can open the job inspector while the search is still running or after it completes.

    Job inspector

    The top section of the job inspector window shows the number of results, the number of events scanned, the duration of the search, and the search ID (SID).

  3. The Execution costs section shows details for each search component, including its duration in seconds, the number of invocations, and input and output counts. Use this information to troubleshoot and improve slow queries. Scroll down to see the components that start with lumi. For details, see Lumi components.

  4. To see the properties of the search and examine details of a specific search ID, click Jobs Detail Dashboard in the job inspector dialog.

  5. Close the window when you've finished.

Lumi components​

The following components in the job inspector are specific to Lumi.

Performance metrics​

The following components measure query execution and data volume.

ComponentDescription
lumi.bytesProcessed.inTotal number of bytes scanned to run a query.
lumi.bytesProcessed.outTotal number of bytes returned after filtering and processing a query.
lumi.endToEndTotal time to complete a query and send results from Lumi to Splunk.
lumi.timeToFirstResultTime taken to produce the first batch of results.

Virtual compute metrics​

The following components show how a query used a virtual compute pool to access data outside the hot tier.

MetricDescription
lumi.virtualCompute.dispatchIndicates that the query ran on a virtual compute pool. Lumi adds this metric when the first results return.
lumi.virtualCompute.poolWait.launchingTime the query waited for the virtual compute pool to start.
lumi.virtualCompute.poolWait.capacityTime the query waited for capacity in a running virtual compute pool.
lumi.vsf.loadNumber of bytes Lumi had to load into virtual storage on demand to run the query.

Signals​

The following components indicate conditions encountered during query processing.

ComponentDescription
lumi.signal.datamodel_requested_but_unavailableData models object is null.
lumi.signal.datamodel_parse_failedData models object exists but couldn't be parsed.
lumi.signal.no_datasets_in_datamodelData model parsed successfully but contains no datasets.
lumi.signal.dataset_not_foundRequested dataset doesn't exist in the data model.
lumi.signal.query_to_disallowed_indexQuery references an index not in the allowed list.
lumi.signal.index_missing_in_queryNon-data model query has no explicit index specified.
lumi.signal.exception_in_transparent_modeException occurred while processing a query in transparent federated search mode.

Internal components​

The following components are used internally by Imply.

  • lumi.intermediate.deserialize
  • lumi.resultGen
  • lumi.writeToWire

Troubleshoot virtual tier queries​

Use the virtual compute metrics to troubleshoot queries against the virtual tier. What counts as an acceptable wait depends on your workload, so compare the values to what you consider reasonable for your queries.

The query didn't use a virtual compute pool​

If you don't see lumi.virtualCompute.dispatch in the job inspector, the query didn't run on a virtual compute pool. Possible reasons include the following:

  • The query only needed data in the hot tier.
  • The IAM key doesn't have virtual tier access.
  • The query hasn't sent any results yet. Lumi sets this metric only when the first results return, so it doesn't appear during a cold start.

The query incurred a cold start​

If lumi.virtualCompute.poolWait.launching is higher than you expect, the query incurred a cold start. Consider the following options:

  • To reduce cold starts, increase the pool's idle time so the pool stays running between queries.
  • To avoid timeouts that can occur during a cold start, increase the query timeout on the IAM key.

The query waited for capacity​

If lumi.virtualCompute.poolWait.capacity is greater than zero, the pool was already running its maximum number of concurrent queries when the query was submitted. The value shows how long the query waited for capacity to become available. If the wait is longer than acceptable for your workload, consider the following options:

  • Increase the maximum number of clusters for the pool. Lumi starts additional clusters when queries need more capacity.
  • Choose a larger pool size with a higher max concurrency.

The query waited for data to load​

If lumi.vsf.load is higher than you expect, the query had to load data that wasn't already in virtual storage. Consider the following options:

  • To reduce the amount of data the query needs, make the query more selective, for example by narrowing the time range or filtering on an index.
  • To reduce how often Lumi reloads data, choose a larger pool size. A larger pool holds more data in virtual storage, so Lumi evicts less data.

Learn more​

See the following topics and third-party documents for more information: