External data overview
AI summary
About AI summaries.
External data is data that stays in your own Amazon S3 storage instead of being ingested into Imply Lumi. The external data feature is also known as Loglake. Use an external data connection to query data in your S3 without ingesting it, from production workloads you query continuously to archival logs and audit data you query occasionally.
Lumi indexes your data and writes the index to your S3 storage, then serves queries through the virtual tier on virtual compute pools that you size and configure.
How an external data connection works
When you connect external data, you point Lumi at two S3 locations:
- A logs location that holds your source data. Lumi only reads from this location and never writes to it.
- An index location where Lumi writes its index of your data. Lumi reads from and writes to this location.
Lumi indexes your source data and writes the index to the index location. When you query external data, Lumi runs the query on a virtual compute pool, starting the pool if it isn't already running, and loads only the data needed to serve the query into virtual storage. These copies are temporary. Lumi holds them in virtual storage while the pool is active and discards them when the pool shuts down. Your source data and its index stay in your S3.
The following diagram shows how data and queries flow through an external data connection:

Lumi virtual tier is required for external data because it relies on virtual compute pools to serve queries. For information on how the virtual tier loads and serves data, see Virtual tier overview.
Deletion and tiering rules apply to managed data only, so they don't apply to external data.
When to connect external data
Lumi provides the following options for storing and querying data. Select an option based on whether you want to store your data in Lumi or in your own storage, and where you query it from.
| Option | Best for | Where data lives | How you query it |
|---|---|---|---|
| Hot tier | Frequently accessed data that needs the lowest latency and highest concurrency | Ingested into Lumi | Lumi search, Splunk® federated search, and other query integrations |
| Virtual tier | Data you query infrequently, such as historical or archival data | Ingested into Lumi, evicted from the hot tier by tiering rules | Lumi search, Splunk federated search, and other query integrations |
| External data | Data you want to keep in your own storage. Accommodates both production workloads you query continuously and archival data you query occasionally | Your S3 bucket; never ingested into Lumi | Lumi search and Splunk federated search |
Connect external data when your data lives in your S3 and you would rather keep it there than ingest it. Because your data stays in S3, you can greatly reduce ingestion costs and the expense of keeping data in the hot tier.
Because you control the virtual compute pools that serve your queries, you can match the compute to the workload:
- Security operations: Run detections, dashboards, and investigations from Splunk against security logs that stay in your S3, such as AWS WAF, firewall, and CloudTrail logs. Query the data with SPL through Splunk transparent federated search. Your existing SPL searches, dashboards, and detections can query the data without being rewritten in another query language. Keep a dedicated pool running so queries avoid cold starts. See Run production workloads on external data.
- Audit and compliance: Keep logs in your own S3 for as long as your retention policy requires, and search them on demand for audits and investigations. Use a pool with a short idle time, so the pool shuts down between queries and reduces compute costs.
You query external data through Lumi search or Splunk transparent federated search. Like any query served by a virtual compute pool, an external data query can incur a cold start if the pool isn't running or the data isn't loaded into virtual storage. For the query steps, see Connect external data.
Run production workloads on external data
When a virtual compute pool is running and the data it needs is loaded, query performance is comparable to the hot tier. To serve production queries on external data with consistent performance, configure the connection and its pools as follows:
- Use a dedicated pool: Create a pool for the production workload and assign it the IAM keys that query the connection. Federated search queries run on the pool assigned to the IAM key, so a key left on the default pool doesn't use your dedicated pool. See Keys and pools.
- Size the pool for your workload: Choose a size with enough resources and max concurrency for your query volume. See Size.
- Set an idle time that keeps the pool running: Set the idle time longer than the longest expected gap between queries for a session, so the pool doesn't shut down between them. A pool that stays running incurs charges for the whole time it runs. See Idle time.
- Use eager indexing: In eager mode, Lumi indexes new files before you query them, so queries don't wait for indexing. See Set the index location.
Learn more
For more information, see the following topics:
- Connect external data to set up a connection.
- Virtual tier overview for how the virtual tier serves queries against data outside the hot tier.
- Set up Splunk transparent federated search to query external data.