Skip to main content

How to send events with the OTel collector

In this tutorial, you learn how to send data to Imply Lumi using the OpenTelemetry (OTel) collector. You'll use the Lumi integration for the Splunk® HTTP event collector (HEC). To use the OTel collector with the OpenTelemetry Protocol, see Send events with OTLP.

The steps show you how to:

  • Configure the collector to export data to Lumi using its HEC endpoint.
  • Parse fields from raw events.
  • Attach metadata fields to events.
  • Preview the data and view events in Lumi.

To complete the steps, you use sample web traffic data from a fictional online store. For details about the dataset and its format, see the tutorial data overview.

The following diagram summarizes the end-to-end process of sending events to Lumi using the OTel collector. Yellow shaded boxes represent steps taken within Lumi, and blue shaded boxes represent steps taken outside Lumi. Click any box in the diagram to jump to that step.

Prerequisites​

To complete the tutorial, you need the following:

  • Access to Lumi with the Data manager role or higher.
    For information on roles and permissions, see Manage roles.

  • OTel collector distribution with the Splunk® extension. You can use one of the following:

    • Splunk distribution of the OTel collector. For installation instructions, refer to the Splunk documentation.

    • Contributor distribution of the OTel collector. Download the distribution from the OpenTelemetry collector release page. Click the link to show all assets to find the distribution for your operating system. Be sure to download the contributor distribution, prefixed by otelcol-contrib. Note that the core distribution doesn't include the filelog receiver nor the splunk_hec exporter.

  • Port 443 open for outbound TCP traffic on the machine running the OTel collector.

If you're already using the OTel collector to send data to Splunk, you can follow along using your existing setup. You may need to restart the collector after making modifications for Lumi.

1. Create an IAM key​

In this section, you create an IAM key and set global and HEC-specific attributes.

  1. From the Lumi navigation menu, click Integrations > Splunk HEC.

    Integration page

  2. Click Select or create key > Create key.

    HEC create IAM key

  3. Enter the following information in the Create IAM key dialog:

    • Name: tutorial-otel
      Only the Name field is required to create the IAM key.
    • Description: IAM key for OTel tutorial
    • Environment: tutorial
    • Team: learning

    When you provide Environment and Team, Lumi assigns those values to the system attributes env and team, respectively. For more information, see System attributes.

  4. Enter the following HEC attributes.
    Lumi enriches events with these values as user attributes. You can view and search events using these attributes. See Set HEC attributes for the default values.

    • Source: tutorial-otel-file
    • Source type: access_combined
    • Default index: demo
    • Allowed indexes: demo

  5. Click Create.

  6. Leave the HEC page open. You'll copy the IAM key token and endpoint and preview incoming data on this page.

    HEC page with OTel configuration

2. Configure the OTel collector​

In this section, you configure the OTel collector to forward logs to Lumi. You create a configuration file for the collector to receive data from a log file, process the events, and export to Lumi.

If you want to skip the details, you can use the full configuration file in the expander at the end of this section.

The following diagram shows an overview of the configuration components:

OTel collector configuration diagram

  1. In a text editor, create and open a file named lumi_hec_config.yaml.

  2. Copy and paste the following section to add the receiver.

    The receivers section determines where to collect the telemetry data and how to parse it. Within that section, you add the filelog receiver to track a file called site_visitors.log. You generate this file in a later step.

    receivers:
        filelog:
            include: [ logs/site_visitors.log ]

  3. Copy and paste the following section to add parsing rules for the events.

    The example data uses the Apache server log combined format. The regex_parser operator parses each event using the specified regular expression. For details on the extracted fields, see Example data for tutorials.

            operators:
              - type: "regex_parser"
                regex: '(?m)^(?P<clientip>[^ ]*) (?P<identd>[^ ]*) (?P<user>[^ ]*) \[(?P<time>[^\]]*)\] "(?P<method>\S+)(?: +(?P<uri>[^\"]*?)(?: +(?P<protocol>\S+))?)?" (?P<status>[^ ]*) (?P<bytes>[^ ]*)(?: "(?P<referer>[^\"]*)" "(?P<useragent>[^\"]*)")?$'

    Note that you can use a free regex parser such as Regex101 to test regular expressions before you add them to a processor.

  4. Copy and paste the following section to add the processors.

    The processors section determines how to transform or enrich the collected data, such as by adding metadata or renaming fields. In this configuration, you add the batch processor to compress the events into batches. You also add metadata attributes for region and myhost.

    processors:
        batch:
        attributes/add_metadata:
            actions:
                - key: region
                  value: "eu-west-1"
                  action: insert
                - key: myhost
                  value: "192.0.2.1"
                  action: insert

  5. Copy and paste the following section to add the exporter.

    The exporters section determines where to send the processed data. Within that section, you add the splunk_hec exporter named logs_lumi and supply the HEC connection information from Lumi.

    The HEC attributes you set on the exporter—source, sourcetype, and index—overwrite the HEC attributes on the IAM key. The exporter assigns the host value from the metadata field myhost, which you added in the processors section. The resulting events contain the host attribute but not myhost. For details on the priority of user attributes, see Event model.

    exporters:
        splunk_hec/logs_lumi:
            endpoint: "LUMI_ENDPOINT"
            token: "IAM_KEY_TOKEN"
            source: "logs/site_visitors.log"
            sourcetype: "access_combined"
            index: "demo"
            otel_attrs_to_hec_metadata:
              host: "myhost"

    1. Replace LUMI_ENDPOINT with your Lumi endpoint. Access the endpoint and your token from the HEC integration page.

    2. Replace IAM_KEY_TOKEN with the token of your IAM key. For the purposes of this tutorial, you can include the token directly in the configuration file. However, for production deployments, consider more secure ways of storing and reading the token, such as an environment variable.

  6. Copy and paste the following section to add the service section.

    The service section enables the preceding components in a logs pipeline. If you create a component and don't add it to service, the collector doesn't use the component.

    service:
        pipelines:
            logs:
                receivers: [ filelog ]
                processors: [ batch, attributes/add_metadata ]
                exporters: [ splunk_hec/logs_lumi ]

  7. Your configuration should resemble the following. Save the file in the directory where the OTel collector is located, then close the file.

    Full configuration file for the OTel collector
    receivers:
        filelog:
            include: [ logs/site_visitors.log ]
            operators:
              - type: "regex_parser"
                regex: '(?m)^(?P<clientip>[^ ]*) (?P<identd>[^ ]*) (?P<user>[^ ]*) \[(?P<time>[^\]]*)\] "(?P<method>\S+)(?: +(?P<uri>[^\"]*?)(?: +(?P<protocol>\S+))?)?" (?P<status>[^ ]*) (?P<bytes>[^ ]*)(?: "(?P<referer>[^\"]*)" "(?P<useragent>[^\"]*)")?$'

    processors:
        batch:
        attributes/add_metadata:
            actions:
                - key: region
                  value: "eu-west-1"
                  action: insert
                - key: myhost
                  value: "192.0.2.1"
                  action: insert

    exporters:
        splunk_hec/logs_lumi:
            endpoint: "LUMI_ENDPOINT"
            token: "IAM_KEY_TOKEN"
            source: "logs/site_visitors.log"
            sourcetype: "access_combined"
            index: "demo"
            otel_attrs_to_hec_metadata:
              host: "myhost"

    service:
        pipelines:
            logs:
                receivers: [ filelog ]
                processors: [ batch, attributes/add_metadata ]
                exporters: [ splunk_hec/logs_lumi ]

  8. Start the OTel collector and provide the custom configuration lumi_hec_config.yaml.

    The way you start the collector depends on your distribution. See the OpenTelemetry docs for details for each operating system and architecture. For example, on MacOS, run the collector in a terminal with the following command:

    ./otelcol-contrib --config=lumi_hec_config.yaml

    If you're not able to start the collector successfully, see the troubleshooting section.

3. Send events​

In this section, you update the file monitored by the OTel collector to export events in the file to Lumi.

  1. In the same directory that contains lumi_hec_config.yaml, create a folder called logs.

  2. Download the example data, site_visitors.log. Save the file in the logs folder in the location you have the OTel collector and configuration file. For details on events in the log file, see Example data for tutorials.

  3. Revisit where you have the OTel collector running. It should print information about watching the file:

    OTel collector log for successful file monitoring
    2025-07-22T17:17:35.598-0500	info	fileconsumer/file.go:267	Started watching file	{"resource": {"service.instance.id": "44f58df0-1a11-2345-b67c-aab776f65043", "service.name": "otelcol-contrib", "service.version": "0.130.1"}, "otelcol.component.id": "filelog", "otelcol.component.kind": "receiver", "otelcol.signal": "logs", "component": "fileconsumer", "path": "logs/site_visitors.log"}

    Since you have the collector running, the file that you downloaded should already be received and exported by the collector.

4. Preview data​

In this section, you preview the data you sent to Lumi and view the events in the explore view.

  1. Return to the HEC integration page in Lumi. In the section to Preview incoming data, view the newly added events.

    Preview incoming data

  2. Select Explore events. Lumi takes you to the explore view and applies search filters for your IAM key and the HEC integration. You may need to edit the time range or refresh the page to see the events.

    Explore events from OTel

  3. Select an event to view the event and its attributes.

User attributes can provide additional context or help you drill down into query results.

Event with attributes

The Message section shows the raw event message, a line in the log file.

The System attributes section shows fields determined by Lumi as well as the global attributes you set on the IAM key. Learn more about system attributes.

The User attributes section shows three categories of attributes:

  • HEC attributes such as source
  • Fields the OTel collector parsed such as method
  • The metadata attribute the OTel collector added, region

Learn more about user attributes.

For information on searching events and filtering on attributes, see Search events with Lumi.

Troubleshoot the OTel collector​

If your operating system can't verify the identity and security of the OTel collector, it may prevent you from starting the application. Go to your system's security settings and allow use of the collector.

If the collector fails and raises the error error decoding 'exporters': unknown type: "splunk_hec", you may have installed the core distribution, which doesn't have the Splunk-specific components. Revisit the prerequisites to install the contributor distribution or a distribution that includes the Splunk component.

If you receive the warning no files match the configured criteria, that may appear because the file doesn't exist. However, the collector will be able to recognize and monitor the file when you create it.

If you receive the error regex pattern does not match, open and confirm that the downloaded log file isn't corrupt. It should be a text file that contains Apache HTTP server access logs with a single log per line.

Learn more​

For more information, see the following topics: