How Lumi works
AI summary
About AI summaries.
Observability data has distinctive characteristics that make it challenging to manage efficiently. Events from disparate systems, like web hosts, app servers, microservices, and remote devices, can emit uniquely structured and verbose logs. The variable structure of the event data from these systems makes it difficult to predefine a schema that suits all observability use cases over time.
Imply Lumi is a data layer purpose-built to handle the challenges of observability workflows with the following design principles and goals:
Schema on persist: Lumi optimizes its schema based upon incoming data to ensure both high performance and adaptability.
Efficient storage: Lumi compresses incoming events using proprietary algorithms to greatly reduce your data footprint, and lets you tier data to balance cost and performance.
Fast querying: Lumi search is powered by the Imply distribution of Apache® Druid which queries the compressed data and returns results faster.
Fits existing workflows: Native integration with Splunk® and other tools means you can keep using your existing workflows.
Schema on persist
Transactional database systems that require predefined schemas do well with query optimization and performance. This model is commonly referred to as "schema on write" because the data must fit the schema before being stored. The schema does not change according to real-time data structure differences, which are common in observability and security data.
To address the need for flexibility, many observability tools use a "schema on read" model which lets the user make schema choices at search time. However, this model requires you to perform extractions and transformations at query time which can hinder search performance.
To solve for flexibility and performance, Lumi uses a schema on persist model. This model lets Lumi adapt its schema to the structure of event data as it prepares to index events for storage. When Lumi detects schema changes coming from different sources, it updates the schema to be most efficient for all the events.
Efficient storage
Lumi uses an enhanced proprietary compression algorithm to significantly reduce event storage size on disk depending on your data. Even though Lumi works with numerous data sources, it detects similarly structured events and compresses the data more efficiently and economically regardless of the source. As the data grows, the compression improves.
To balance cost and performance, Lumi keeps frequently accessed data in the hot tier for low-latency queries. For data you query less often and where performance is less critical, you can use the virtual tier. The virtual tier loads data and spins up compute on demand, at a lower cost than the hot tier.
If your data already lives in Amazon S3, you can connect it as external data instead of ingesting it. Lumi indexes the data in your S3 and queries it through the virtual tier, which can greatly reduce ingestion costs and the expense of keeping data in the hot tier.
Fast querying
Schema on persist, advanced log compression, and the Imply query engine enable Lumi to query compressed logs at very fast speeds. Queries that require scanning long data ranges and sorting perform very well.
Fits existing workflows
Observability teams already have tools and pipelines in place. Lumi is built to fit into them rather than replace them, on both the ingestion and query sides, so you can adopt Lumi without rebuilding how you collect or analyze data.
On the ingestion side, Lumi accepts data from the sources you already run, including Splunk® universal and heavy forwarders, the OpenTelemetry Collector, HTTP endpoints such as HEC and OTLP, and Amazon S3. You keep collecting logs the way you do today. Adopting Lumi requires only minor configuration changes to send that data to Lumi.
On the query side, you can search Lumi events from the tools your teams already use. Connect Grafana as a Loki data source and query with LogQL, or query in natural language from AI agents such as Claude Code, Claude Desktop, Cursor, and VS Code through the Model Context Protocol (MCP). The deepest integration allows you to query Lumi data from Splunk.
Query Lumi from Splunk
With federated search, Splunk sends queries to Lumi and returns the results alongside your Splunk data. This means there is no need for you to move or duplicate your Splunk data. You keep writing SPL, and your existing searches, dashboards, and alerts keep working with minor configuration changes.
A federated search query reaches Lumi in one of two modes:
- Transparent: Query Lumi indexes with standard SPL and no prefix, as if they were native Splunk indexes. Supports data model queries and lookups, and is the easiest way to query Lumi directly from Splunk.
- Standard: Prefix queries with
federated:and route them through a Splunk federated index that maps to a Lumi index. Gives you explicit control over which indexes are queried, and lets you use different index names in Splunk than in Lumi.
Both modes read from the same Lumi indexes. They differ only in how Splunk addresses them.
You also use transparent federated search to query external data.