Troubleshoot data ingestion

AI summary

Explains how Imply Lumi detects and handles ingestion errors when events can't be parsed due to corrupted data or configuration issues. Shows how to view, analyze, and resolve these errors.

About AI summaries.

When you send events to Imply Lumi, ingestion errors such as unparsable events can occur. Lumi raises ingestion errors when it can't convert the raw events into a valid format.

Unparsable events can occur for various reasons. For example, an application crash corrupts event data before it reaches Lumi. Or a forwarding agent doesn't have the proper configuration, resulting in the event body having missing fields or malformed JSON.

Lumi detects unparsable events and places them in a separate queue for you to investigate further. This prevents your datasets from being polluted with invalid data and alerts you to potential upstream issues.

This topic explains how to view and analyze ingestion errors in Lumi.

For a list of ingestion errors, see Ingestion errors reference. To understand the current scope of Lumi, review the known limitations.

Ingestion metrics overview

Ingestion metrics describe the number of events successfully ingested or unable to be parsed by Lumi. Lumi reports ingestion metrics for a given receiver and IAM key.

Access the Usage page for an overview of ingestion metrics. On the Unparsable events tab, you can filter results by IAM key, integration (receiver), or error type.

Note that different ingestion integrations can have the same receiver. When you view unparsable events from one integration's instructions, it can include unparsable events from another integration that shares the same receiver. For example, both tcpout and httpout S2S integrations share the same receiver. If you use the same IAM key for both integrations, the unparsable events for both integrations enter the same pool. For a list of relationships between integrations and receivers, see Receivers reference.

Drill into unparsable events

For events Lumi can't parse, access the Unparsable events pane to drill into the details. On this pane, you can view the total count of unparsable events, any error codes raised, and the raw events that raised the errors.

Lumi stores unparsable events for three days, although the retention window is subject to change. You can still see the aggregated metrics for unparsable events over a larger time window on the Usage page.

To access the Unparsable events pane:

1. Go to the Keys page, and select your IAM key.

2. In the Integrations section, locate your integration. Lumi shows the number of successfully ingested and unparsable events in green and yellow bars, respectively.

   

3. Click the ellipsis, and select View unparsable events.

   

4. Lumi displays the Unparsable events pane. You can view the metrics associated with this receiver and IAM key.

   

5. Use the time range selector to adjust the time period of the results.

6. In the Raw events section, use the search bar to filter your results. You can search text in the raw events.

7. Hover over any event's content, and select Open to view the contents of the raw unparsable event.

   

You can also view the unparsable events pane from the instructions page of an integration tile. After you select your IAM key, click View unparsable in the Preview incoming data pane.

Troubleshoot missing events

If your events ingest successfully but seem to be missing, check the event timestamps. The Keys details page and the Preview incoming data pane only show ingested events with timestamps from the past 24 hours.

When no events appear in the preview, the events may have timestamps older than 24 hours. To confirm that the data was ingested, search for the events in the explore view. Adjust the time range selector to span the time range of your events. To access the explore view filtered by a given receiver and IAM key, take one of the following approaches:

• Go to the Keys details page, click the ellipsis on the integration tile, then select Search in explorer.
• Go to the integration instructions page, select your IAM key, and click Explore events in the Preview incoming data pane.

For example, suppose you configured a backfill ingestion using S3 pull. You ingest events from your S3 bucket that were from the past, such as January 1 through January 14. Filter the results for that time period in your search. The following screenshot shows an example search with filters for the IAM key, receiver, and time range:

Note that if Lumi can't detect the timestamp on an event, it assigns the timestamp the event was received. For more information, see Timestamp handling.

For more details on searching in the explore view, see Search events with Lumi.

Resolve ingestion errors

If you know the IAM key and receiver associated with ingestion errors, you can access its unparsable events. Each unparsable event includes an error code for why Lumi couldn't ingest it. For details on specific ingestion errors and resolution strategies, see the Ingestion errors reference.

Contact Lumi support

For additional assistance, contact Lumi customer support.

Learn more

For more information, see the following topics:

• Ingestion errors reference
• Monitor usage
• View IAM key metrics