Skip to main content

External data overview

AI summary
Explains how Imply Lumi external data connections let you query data in Amazon S3 without ingestion. Covers how Lumi indexes your S3 data and serves queries using the virtual tier. Describes when to use external data to reduce ingestion costs while maintaining query performance for production workloads.

About AI summaries.

External data is data that stays in your own Amazon S3 storage instead of being ingested into Imply Lumi. The external data feature is also known as Loglake. Use an external data connection to query data in your S3 without ingesting it, from production workloads you query continuously to archival logs and audit data you query occasionally.

Lumi indexes your data and writes the index to your S3 storage, then serves queries through the virtual tier on virtual compute pools that you size and configure.

How an external data connection works​

When you connect external data, you point Lumi at two S3 locations:

  • A logs location that holds your source data. Lumi only reads from this location and never writes to it.
  • An index location where Lumi writes its index of your data. Lumi reads from and writes to this location.

Lumi indexes your source data and writes the index to the index location. When you query external data, Lumi runs the query on a virtual compute pool, starting the pool if it isn't already running, and loads only the data needed to serve the query into virtual storage. These copies are temporary. Lumi holds them in virtual storage while the pool is active and discards them when the pool shuts down. Your source data and its index stay in your S3.

The following diagram shows how data and queries flow through an external data connection:

External data connection process flow

Lumi virtual tier is required for external data because it relies on virtual compute pools to serve queries. For information on how the virtual tier loads and serves data, see Virtual tier overview.

Deletion and tiering rules apply to managed data only, so they don't apply to external data.

When to connect external data​

Lumi provides the following options for storing and querying data. Select an option based on whether you want to store your data in Lumi or in your own storage, and where you query it from.

OptionBest forWhere data livesHow you query it
Hot tierFrequently accessed data that needs the lowest latency and highest concurrencyIngested into LumiLumi search, Splunk® federated search, and other query integrations
Virtual tierData you query infrequently, such as historical or archival dataIngested into Lumi, evicted from the hot tier by tiering rulesLumi search, Splunk federated search, and other query integrations
External dataData you want to keep in your own storage. Accommodates both production workloads you query continuously and archival data you query occasionallyYour S3 bucket; never ingested into LumiLumi search and Splunk federated search

Connect external data when your data lives in your S3 and you would rather keep it there than ingest it. Because your data stays in S3, you can greatly reduce ingestion costs and the expense of keeping data in the hot tier.

Because you control the virtual compute pools that serve your queries, you can match the compute to the workload:

  • Security operations: Run detections, dashboards, and investigations from Splunk against security logs that stay in your S3, such as AWS WAF, firewall, and CloudTrail logs. Query the data with SPL through Splunk transparent federated search. Your existing SPL searches, dashboards, and detections can query the data without being rewritten in another query language. Keep a dedicated pool running so queries avoid cold starts. See Run production workloads on external data.
  • Audit and compliance: Keep logs in your own S3 for as long as your retention policy requires, and search them on demand for audits and investigations. Use a pool with a short idle time, so the pool shuts down between queries and reduces compute costs.

You query external data through Lumi search or Splunk transparent federated search. Like any query served by a virtual compute pool, an external data query can incur a cold start if the pool isn't running or the data isn't loaded into virtual storage. For the query steps, see Connect external data.

Run production workloads on external data​

When a virtual compute pool is running and the data it needs is loaded, query performance is comparable to the hot tier. To serve production queries on external data with consistent performance, configure the connection and its pools as follows:

  • Use a dedicated pool: Create a pool for the production workload and assign it the IAM keys that query the connection. Federated search queries run on the pool assigned to the IAM key, so a key left on the default pool doesn't use your dedicated pool. See Keys and pools.
  • Size the pool for your workload: Choose a size with enough resources and max concurrency for your query volume. See Size.
  • Set an idle time that keeps the pool running: Set the idle time longer than the longest expected gap between queries for a session, so the pool doesn't shut down between them. A pool that stays running incurs charges for the whole time it runs. See Idle time.
  • Use eager indexing: In eager mode, Lumi indexes new files before you query them, so queries don't wait for indexing. See Set the index location.

Learn more​

For more information, see the following topics: