Skip to main content

File upload reference

AI summary
Explains requirements for uploading log files to Imply Lumi. Describes how Lumi extracts the timestamp, event message, and attributes. Shows how to search events only from a specific file.

About AI summaries.

To quickly get started with Imply Lumi, you can upload log files. For a step-by-step guide using file upload, see the Quickstart.

This topic describes the requirements for files you plan to upload to Lumi.

Prerequisites​

To upload files to Lumi, you need the following:

  • Access to Lumi with the Data manager role or higher. For information on roles and permissions, see Manage roles.
  • Files in JSON or Splunk® CSV format. For more information, see File formats.

File size limit​

Lumi supports a maximum file size of 200 MB. This limit refers to the size of the file transmitted by the browser or HTTP client.

Timestamp and message​

Lumi assigns the timestamp and message for uploaded events as described in this section. In the file upload UI, Lumi automatically parses the data and shows a preview of the events.

CSV format​

For CSV files exported from Splunk, Lumi maps the event message as follows:

  • Event timestamp, taken from the _time field.
  • Event message, taken from the _raw field.

File upload CSV

JSON format​

For a newline-delimited JSON file, specify the fields that map to the timestamp and message. These fields don't appear as user attributes on the events.

  • Event timestamp: Enter a field for Time, and select your time format and time zone. For supported formats and examples, see Time formats.

  • Event message: Enter a field for Message. You can specify a single field or combine multiple fields to make up the event message.

    To combine multiple fields, enter the field names separated by a comma. For example, key1, key2. Lumi joins the values in the specified order with a space between each value. For example, val1 val2.

File upload JSON

Attributes​

All events, including those added to Lumi through file upload, contain system attributes and user attributes.

System attributes​

System attributes define metadata related to the source of the event and system information about Lumi. The following system attributes apply to file uploads:

  • receiver is always imply.file.
  • file stores the name of the uploaded file.

If you supply values for env and team in the file upload UI, Lumi stores them as system attributes on events. Otherwise, the events don't have those system attributes. If the CSV file contains columns for env or team, Lumi stores your supplied values as user attributes instead.

You can't set the env and team system attributes with JSON upload; however, you can assign them as user attributes.

For details on additional system attributes, see the event model reference.

User attributes​

User attributes can provide context to events and help you drill down into query results.

When you upload a CSV file exported from Splunk, Lumi stores all the fields as user attributes. The fields _raw and _time aren't stored as user attributes since they map to the event message and timestamp, respectively.

You can configure source, sourcetype, or index for CSV file uploads. If the attributes are already defined in the source file, Lumi retains the values from the file. In other words, if source is defined in the CSV file, you can't override it during the file upload.

When you upload a JSON file, Lumi stores all the JSON fields as user attributes by default. To only store a subset of the fields, go to Allow list > Specify fields, and enter a comma-separated list of fields to retain.

Note that you can modify user attributes on uploaded events using pipelines. In a pipeline, you can add attributes from static values, other attribute values, or parsed text. You can also remove or replace user attributes. To learn more, see Transform events using pipelines.

For more information on user attributes, see the event model reference.

Search events from a file​

To explore events related only to the file, use the file system attribute in your search. You can also filter by a user attribute specific to the file. For example, add the following filter to your query, and set the appropriate time range:

#file=site_visitors_quickstart.csv

Search by file name

You can search all events with a specific file extension using a wildcard. For example:

#file=*.csv

You can search all events Lumi received by file upload. For example:

#receiver=imply.file

Learn more​

For more information, see the following topics: