Skip to main content

Troubleshoot data ingestion

When you send events to Imply Lumi, ingestion errors such as unparsable events can occur. Lumi raises ingestion errors when it can't convert the raw events into a valid format.

Unparsable events can occur for various reasons. For example, an application crash corrupts event data before it reaches Lumi. Or a forwarding agent doesn't have the proper configuration, resulting in the event body having missing fields or malformed JSON.

Lumi detects unparsable events and places them in a separate queue for you to investigate further. This prevents your datasets from being polluted with invalid data and alerts you to potential upstream issues.

This topic explains how to view and analyze ingestion errors in Lumi.

For a list of ingestion errors, see Ingestion errors reference. To understand the current scope of Lumi, review the known limitations.

View ingestion errors​

When you send events to Lumi, you can learn about ingestion errors through error messages from your client, the Keys page described in the preceding section, or the Usage page. On the Usage page, you can view general ingestion statistics as well as unparsable events organized by IAM key or integration.

Usage unparsable events

You can view ingestion metrics for the past 24 hours for each IAM key on the Keys page. Select a key to go to its details page. Lumi shows the number of successfully ingested and unparsable events in green and yellow bars, respectively. The details page also shows the associated error codes for any unparsable events.

IAM key details

Drill into unparsable events​

For events Lumi can't parse, access the Unparsable events pane to drill into the details. Lumi stores unparsable events for three days, although the retention window is subject to change. You can still see the aggregated metrics for unparsable events over a larger time window on the Usage page.

To view unparsable events:

  1. Go to the Keys page.
  2. Select your IAM key.
  3. Click the ellipsis for the ingestion integration.
  4. Select View unparsable events.
  5. Lumi displays the Unparsable events pane. The pane displays the ingestion error types and number of events with each error. Unparsable pane
  6. Select the time range selector to adjust the time period of the results.
  7. In the Raw events section, use the search bar to filter your results. You can search text from the raw events.
  8. Hover over any event's content, and select Open to view the contents of the raw unparsable event. Unparsable pane contents

You can also view the unparsable events from the instructions page of an ingestion integration. Be sure to select the IAM key you used to send events.

Each IAM key and receiver combination has a separate pool of unparsable events. Note that different ingestion integrations can have the same receiver. When you view unparsable events from one integration's instructions, it can include unparsable events from another integration that shares the same receiver. For example, both tcpout and httpout S2S integrations share the same receiver. If you use the same IAM key for both integrations, the unparsable events for both integrations enter the same pool.

Resolve ingestion errors​

If you know the IAM key and receiver associated with ingestion errors, you can access its unparsable events. Each unparsable event includes an error code for why Lumi couldn't ingest it. For details on specific ingestion errors and resolution strategies, see the Ingestion errors reference.

Contact Lumi support​

For additional assistance, contact Lumi customer support.

Learn more​

For more information, see the following topics: