Skip to main content

Connect external data

AI summary
Explains how to set up and manage external data connections in Imply Lumi to query data in your Amazon S3 storage. Covers creating connections, configuring logs and index locations, and querying through Lumi or Splunk® federated search. Details how IAM keys route queries to virtual compute pools.

About AI summaries.

An external data connection lets Imply Lumi query data that stays in your own Amazon S3 storage without ingesting it into Lumi. This topic explains how to create, manage, and query an external data connection. For how the external data feature works and when to use it, see External data overview.

Prerequisites​

To connect external data, you need the following:

  • Virtual compute pool. See Create a virtual compute pool. The first pool you create becomes the default pool.
  • Lumi user with the Admin role. For information on roles and permissions, see Use roles to control access.
  • Source logs in an S3 bucket and an S3 location where Lumi can write its index.
  • AWS user with permission to create IAM roles and policies.

Connect external data​

You connect external data through the connection wizard. Lumi saves your progress as a draft as you go, so you can leave the wizard and finish later.

To start, go to External data in the Lumi navigation menu, then click Create connection to open the wizard. If you don't see External data in the menu, contact your Imply support representative.

Name the connection and attach keys​

On the Naming & access page, name the connection and attach the IAM keys for Splunk federated search:

  1. Set the connection details:
    • Name: Name to identify the connection. Use 3 to 40 characters: lowercase letters, digits, and hyphens. You can't change this after you create the connection. The name is the index attribute that Lumi adds to your logs. To search the external data, add this attribute to your search query.
      For example, if the index attribute is parquet-logs, search with index=parquet-logs. Lumi sets this attribute on every event and replaces any index field in your data.
    • Description: Optional description to identify the connection.
  2. In Attach IAM keys, create or assign the keys that can query the data from Splunk. You only need a key to query the data through Splunk federated search. You can search the data in Lumi without a key. We recommend creating a new IAM key for each connection, and using that key to provision a federated provider in Splunk®. For how keys route to pools and which keys you can attach, see Keys and pools.
  3. Click Next.

Create an external data connection

Set the logs location​

On the Logs location page, point Lumi to your source data. Lumi only reads from this location and never writes to it.

If you're connecting to DDSS storage, use Indexes in Splunk Web to identify the indexes that use self-storage. The Self Storage column shows the S3 bucket name and path for each index.

  1. Under Location, set the following:

    • Region: AWS region where your logs are stored.

    • Bucket: S3 bucket that holds your logs.

    • Path: Path to your logs within the bucket. You can use glob patterns to match multiple paths or objects. For example:

      path/to/index/**
  2. Configure an IAM role in your AWS account that Lumi can use to read your logs. You can create a role using the policies from the wizard, or add the equivalent configuration to an existing role:
    1. Copy the Trust policy from the wizard and add it to the role. The trust policy lets Lumi assume the role, secured by a unique external ID.
    2. Copy the Inline permissions from the wizard and add them to the role. Lumi uses these permissions to read your logs in the specified S3 bucket. If your logs bucket uses AWS KMS server-side encryption (SSE-KMS), also add the kms:Decrypt permission for the bucket's KMS key.
    3. Copy the role ARN.
  3. Enter the Role ARN.
  4. Click Verify access. Once Lumi verifies access, File preview lists the files at the path.
  5. Click Next.

Set the index location​

On the Indexing page, set the S3 location where Lumi writes its index of your data. Unlike the logs location, Lumi reads from and writes to this location.

note

Lumi may overwrite existing files at the index path. Use an empty path that only Lumi writes to.

  1. Under Location, set the following:

    • Region: AWS region for the index location.
    • Bucket: S3 bucket where Lumi writes the index.
    • Path: Path to the index within the bucket. A path is required, and it can't be the bucket root. If the index and your logs share a bucket, set the index path outside the logs path so Lumi doesn't read index files as logs.

  2. Configure an IAM role in your AWS account that Lumi can use to write the index. You can create a role using the policies from the wizard, or add the equivalent configuration to an existing role:

    1. Copy the Trust policy from the wizard and add it to the role. The trust policy lets Lumi assume the role, secured by a unique external ID.
    2. Copy the Inline permissions from the wizard and add them to the role. Lumi uses these permissions to read and write the index. If your index bucket uses AWS KMS server-side encryption (SSE-KMS) as its default encryption, also add the kms:Decrypt and kms:GenerateDataKey permissions for the bucket's KMS key. We recommend adding these permissions whenever you use SSE-KMS with the index bucket.
    3. Copy the role ARN.

  3. Enter the Role ARN.

  4. Click Verify access. Once Lumi verifies access, File preview lists the files at the path.

  5. Select an Indexing mode:

    • Eager: Lumi checks for new files every 10 minutes and indexes them before you query them.
    • Lazy: Lumi indexes data when you query it.

    Lumi doesn't re-index files it has already indexed when the contents of your logs location change. For more information, see Edit a connection.

  6. Click Next.

Transform and connect​

On the Transformation page, tell Lumi how to read your files. The File type you select determines how Lumi reads your data, which fields it extracts, and how you search the data. Lumi shows a sample preview so you can check the result before you complete the connection.

  1. In File type, select the format of your files.
  2. Configure the options for your file type. For the settings for each file type, see External event formats reference.
  3. Check the Sample preview to confirm Lumi parses your data as expected. Preview sampling reads the first 100 files or 10,000 events, whichever comes first.
    • The Parsed events tab shows the events Lumi extracts. Click an event to see its details and attributes. Preview an external data connection
    • The Files tab shows the source files sampled.
  4. Click Connect.

Lumi creates the connection. In eager mode, Lumi begins indexing your data. In lazy mode, Lumi indexes the data when you first query it.

As Lumi indexes, the status on the connection's page moves from Indexing in progress to Indexed. On the main External data page, the Indexing status column shows In progress and then Done.

External data connections

Query unavailable in the Keys column means the connection has no IAM keys attached, so you can't query it through Splunk federated search. You can still query the data in Lumi. To query the data from Splunk, attach a key.

Manage connections​

After you create a connection, you can edit or delete it from the External data page.

Edit a connection​

To edit a connection, go to External data and click the connection's name. If you enter the name of an existing connection in the connection wizard, Lumi opens that connection in edit mode.

You can perform the following actions:

  • Update the transformation options, including the file type and configuration, and run a new sample preview.
  • Change the indexing mode.
  • Change the index attribute Lumi adds to your logs. Update any searches that use the previous index attribute.
  • Attach and remove IAM keys.

When you save changes to the connection, Lumi re-indexes the data. Lumi only re-indexes when the connection configuration changes, not when the contents of your logs location change. For example, if you add files to the logs location of a connection in eager mode, Lumi indexes only the new files.

You can't edit any other connection details. To change anything else, delete the connection and create a new one.

Delete a connection​

To delete a connection, go to External data, click the ellipsis in the connection's row, then click Delete.

Deleting a connection doesn't affect the source logs in your S3 bucket. Lumi doesn't delete the index files at the index location.

Query external data​

You can query external data in Lumi and through Splunk federated search. Lumi serves external data queries through the virtual tier, so each query runs on a virtual compute pool.

The first query after the pool has been idle may incur a cold start while Lumi provisions compute and loads the data it needs to serve the query. Lumi loads this data into virtual storage, not into your S3, and discards it when the pool shuts down. For more information, see Idle time and cold starts.

You can query external data in the Lumi UI without attaching an IAM key.

  1. Go to Explore.
  2. Select the connection from the drop-down at the top of the page. Selecting the connection limits your search to its data, so you don't need to include the index attribute in your query.
  3. Set the time range and enter your query.

Lumi search queries on external data run on the default virtual compute pool. Lumi runs these queries with your Lumi user credentials, so the IAM keys attached to the connection and their pools don't apply.

For example:

External data Lumi search

The Attributes panel lists the fields Lumi extracted from your data. The fields available depend on the connection's file type. See Supported file types for more information.

For supported operators and syntax, see Lumi query syntax reference.

To query external data from Splunk, use the IAM key attached to the connection to provision a federated provider in Splunk. For the provider setup steps, see Set up Splunk transparent federated search.

Then query using the connection's index attribute. The following query finds events with index attribute parquet-logs for the past 14 days with the specified status_code, level, and method:

index=parquet-logs status_code=200 level=WARN method=POST earliest=-14d

External data federated search

Federated search queries run on the virtual compute pool assigned to the IAM key you query with.

note

A single IAM key can't query both external and managed data. Create separate keys for each, and provision a separate federated provider in Splunk for each key. See Which keys you can attach for the rules Lumi enforces.

Keys and pools​

Each key routes the external data connection's federated search queries to one virtual compute pool. Lumi search queries always run on the default pool. When you create or attach a key, you select the pool it routes queries to. If the key already serves another connection, changing its pool also changes the pool for that connection's queries.

You can assign a key to only one pool, but a pool can serve multiple keys. Name your pools by intended use. Pools that serve managed data and external data appear together in the Lumi UI, so a clear naming convention helps you tell them apart. For more information, see IAM key access.

To run different workloads on different pools, attach a separate key for each workload. For example, an interactive team could query through a larger pool with a longer idle time, while a scheduled job queries the same external data through a smaller pool.

Which keys you can attach​

You can only attach a key if either of the following is true:

  • The key has no integrations enabled.
  • The key has the federated search integration and is already attached to an external data connection.

This restriction protects keys that already query managed data through federated search. Attaching such a key to an external data connection could reassign its pool or integration and break those existing queries.

If you attach a key that has no integrations enabled, Lumi adds the federated search integration to it automatically. You don't need to add the integration yourself first.

Learn more​

For more information, see the following topics: