Send events with S3 ingest actions

You can send events to Imply Lumi using a Splunk® ingest action for routing to S3. This protocol routes events to an S3-compatible Lumi endpoint. Use this integration when you send events to Splunk using a heavy forwarder in your own environment (Splunk Enterprise).

For a tutorial on this integration, see How to send events with an S3 ingest action. For other approaches to sending events, see Send events to Lumi.

This topic provides details on configuring event forwarding using S3 ingest actions.

Prerequisites

To send events to Lumi using an S3 ingest action, you need the following:

• Access to Lumi with the Data manager role or higher. For information on roles and permissions, see Manage roles.

• Lumi IAM key. See Create an IAM key for details.

  The IAM key authenticates connections to send events to Lumi. It also enriches incoming events with the system attributes for environment and team.

• Splunk heavy forwarder configured in your environment for routing events to an S3 destination. For more information, see Use forwarders to get data into Splunk Enterprise.

• Port 443 open for outbound TCP traffic on the machine running the Splunk heavy forwarder.

Configure event forwarding

This section describes how to set up S3 ingest actions. To configure Splunk, you can use the outputs.conf configuration file and REST API, or use Splunk Web, the graphical user interface for the Splunk platform.

To configure event forwarding, complete the following steps:

1. From the Lumi navigation menu, click Integrations > S3 ingest actions.
2. Select or create an IAM key.
3. In Splunk, create an S3 remote storage destination. See the following sections for detailed instructions.
4. In Splunk, create a ruleset to route to the destination. See the following sections for detailed instructions.

The integration page lists the information required to create the S3 destination:

• S3 bucket name: Suggested bucket name resembling imply-f590a036-3590-4a6f-b5a5-da8d37c53463
• S3 endpoint: Lumi endpoint resembling https://splunk-s3.us1.api.lumi.imply.io
• Access key ID: IAM key ID in UUID format
• Secret access key: IAM key token in UUID format

Unique bucket name

Splunk may require you to have unique bucket names for S3 destinations. Lumi generates a new bucket name with the format imply- followed by a randomly generated UUID. The S3 bucket name changes every time you access the S3 ingest actions integration page. You're not required to use the suggested bucket name, but ensure that your S3 bucket name is unique.

Splunk configuration file and API

Configure an S3 destination directly on the heavy forwarder by editing the outputs.conf file. Use the Splunk REST API to create a ruleset. If you have multiple heavy forwarders, configure the destination and apply the ruleset on each one.

Create an S3 destination

Manually create a remote storage destination for S3 in Splunk.

1. Update or add the remote file system (RFS) stanza in outputs.conf to include the following settings:

   [rfs:logs_lumi]
   path = s3://LUMI_BUCKET/
   remote.s3.endpoint = LUMI_ENDPOINT
   remote.s3.access_key = IAM_KEY_ID
   remote.s3.secret_key = IAM_KEY_TOKEN

2. Replace the following values provided by Lumi:

   • LUMI_BUCKET: S3 bucket name suggested by Lumi
   • LUMI_ENDPOINT: Lumi S3 endpoint
   • IAM_KEY_ID: IAM key ID
   • IAM_KEY_TOKEN: IAM key token

3. Configure any other output RFS properties based on your specific setup and requirements.

4. Restart your forwarder for the changes to take effect.

To view Splunk recommendations and requirements, see Heavy forwarders managed through a deployment server.

Create a ruleset

Use the Splunk API endpoint /services/data/ingest/rulesets to create a ruleset.

Your request should resemble the following:

curl -k https://HOST:PORT/services/data/ingest/rulesets\?output_mode=json -d name=RULESET_NAME -d sourcetype=SOURCETYPE -d rules="[
          {
            \"name\": \"RULE_NAME\",
            \"action\": \"route\",
            \"clone\": false,
            \"dest\": \"DESTINATION\",
            \"cond\": {
              \"type\": \"eval\",
              \"expr\": \"true()\"
            }
          }
        ]"

Replace the following values:

• HOST: Hostname or IP address of the machine running the heavy forwarder
• PORT: Port used by Splunk REST API interface
• RULESET_NAME: Name of the ruleset in Splunk
• SOURCETYPE: Sourcetype in Splunk
• RULE_NAME: Name of the rule in Splunk
• DESTINATION: S3 destination for the ingest action

Depending on the authentication method, you might need to include your login credentials in the API request. For required parameters and usage examples, see Input endpoint descriptions.

Splunk Web

In Splunk Web, follow the steps to create an S3 destination and a ruleset to route to the destination.

Create an S3 destination

Create a remote storage destination for S3 in Splunk. After you create or select an IAM key, Lumi populates the details in the Configure event forwarding pane with your specific IAM key ID and token.

In Splunk Web, go to Settings > Ingest actions > Destinations > New destination > S3.

1. Enter the following fields:
   • S3 bucket name: S3 bucket name suggested by Lumi
   • S3 endpoint: Endpoint provided by Lumi
2. Complete all other fields based on your specific setup and requirements.
3. Click Next.
4. Enter the authentication details for Lumi:
   • Authentication method: Access key and secret key
   • Access key ID: IAM key ID
   • Secret access key: IAM key token
5. Click Test connection to validate the connection.
   A valid connection returns the message, "Successfully connected to the server."
6. Click Save.

Splunk Cloud

If you don't see options for "S3 endpoint" or "Authentication method," you may be on Splunk Cloud. Please contact Imply support for assistance.

Create a ruleset

Create a ruleset in Splunk that routes events to Lumi at ingest time. The ruleset references the S3 destination you created in the previous section.

1. While still on the Ingest actions page in Splunk, go to Ruleset > New ruleset.
2. Under Event stream, set sourcetype to the type that matches your data.
   For all other fields, use the default settings.
3. Click Add rule > Route to destination.
4. In the field for Immediately send to, enter the name of the S3 destination created in the previous step.
5. Click Apply.
6. Click Save.
7. Restart your heavy forwarder for the changes to take effect.

Check Imply Lumi for events

Once you configure event forwarding and send events, you can preview the incoming data in Lumi:

1. From the Lumi navigation menu, click Integrations > S3 ingest actions.

2. In Select or create an IAM key, select your key.

3. In Preview incoming data, view the events coming in to Lumi. Lumi automatically refreshes the preview pane to display the latest events.

4. Click Explore events to see more events associated with the IAM key. The explore view populates the search bar with your IAM key ID and the receiver type. For example:

   #iamKeyId=229a2561-0000-0000-0000-bc433de16f89 #receiver=splunk.s3

   Adjust the time filter to choose the range of data displayed.

Once events start flowing into Lumi, you can search them. See Search events with Lumi for details and information on supported search syntax.

Learn more

See the following topics for more information:

• Send events to Lumi for other options to send events.
• IAM keys reference for details on IAM keys.

For information on ingest actions in Splunk, refer to the following Splunk documentation:

• Configure forwarding with outputs.conf
• Routing data to S3
• Create an S3 destination