How Lumi works

Observability data has distinctive characteristics that make it challenging to manage efficiently. Events from disparate systems, like web hosts, app servers, microservices, and remote devices, can emit uniqely structured and verbose logs. The variable structure of the event data from these systems makes it difficult to predefine a schema that suits all observability use cases over time.

Imply Lumi is a data layer purpose-built to handle the challenges of observability workflows with the following design principles and goals:

Schema on persist: Lumi optimizes its schema based upon incoming data to ensure both high performance and adaptability.
Efficient storage: Lumi compresses incoming events using proprietary algorithms to greatly reduce your data footprint.
Fast querying: Lumi search is powered by Imply's distribution of Apache Druid which queries the compressed data and returns results faster.
Fits existing workflows: Native integration with Splunk® and other tools means you can keep using your existing workflows.

Schema on persist

Transactional database systems that require predefined schemas do well with query optimization and performance. This model is commonly referred to as "schema on write" because the data must fit the schema before being stored. The schema does not change according to real-time data structure differences, which are common in observability and security data.

To address the need for flexibility, many observability tools use a "schema on read" model which lets the user make schema choices at search time. However, this model requires you to perform extractions and transformations at query time which can hinder search performance.

To solve for flexibility and performance, Lumi uses a schema on persist model. This model lets Lumi adapt its schema to the structure of event data as it prepares to index events for storage. When Lumi detects schema changes coming from different sources, it updates the schema to be most efficient for all the events.

Efficient storage

Lumi uses an enhanced proprietary compression algorithm to significantly reduce event storage size on disk depending on your data. Even though Lumi works with numerous data sources, it detects similarly structured events and compresses the data more efficiently and economically regardless of the source. As the data grows, the compression improves.

Fast querying

Schema on persist, advanced log compression, and the Imply query engine enable Lumi to query compressed logs at very fast speeds. Queries that require scanning long data ranges and sorting perform very well.

Fits existing workflows

Lumi integrates natively with your existing observability tools to take advantages of their logging capabilities. For example, you can use SPL from within Splunk Web or with the REST API. You can also use the drill down feature in Grafana. All your existing dashboards and alerts continue to work with minor configuration changes.