Skip to main content

File upload reference

To quickly get started with Imply Lumi, you can upload log files. For a step-by-step guide using file upload, see the Quickstart.

This topic describes the requirements for files you plan to upload to Lumi.

Prerequisites​

To upload files and search events, you need the Data manager role or higher. For more information on roles and permissions, see Manage roles.

Supported file formats​

Lumi supports uploads from the following file formats:

  • CSV files exported from Splunk®. For information about exporting data from Splunk, see Export data using Splunk Web.
  • Newline-delimited JSON files. This format uses one row of JSON data per line. For details, see the NDJSON specification.

The following formats are not supported:

  • A single JSON object that contains all the data.
  • JSON files exported from Splunk.

File size limit​

Lumi supports a maximum file size of 200 MB. This limit refers to the size of the file transmitted by the browser or HTTP client.

Timestamp and message​

Lumi assigns the timestamp and message for uploaded events as described in this section. In the file upload UI, Lumi automatically parses the data and shows a preview of the events.

CSV format​

For CSV files exported from Splunk, Lumi maps the event message as follows:

  • Event timestamp, taken from the _time field.
  • Event message, taken from the _raw field.

File upload CSV

JSON format​

For newline-delimited JSON files, Lumi maps the event message as follows:

  • Event timestamp: Enter your time field in Time and specify the time format. For supported time formats, see Time formats.

  • Event message: Enter your message field in Message.

    You can combine multiple fields to make up the event message. Separate field names with a comma. For example, log1, log2. Lumi combines the values in order and separates them with a space character.

File upload JSON

Attributes​

All events, including those added to Lumi through file upload, contain system attributes and user attributes.

System attributes​

System attributes define metadata related to the source of the event and system information about Lumi. The following system attributes apply to file uploads:

  • observedAt represents the time the file uploaded. Events in the same file have the same observedAt timestamps.
  • receiver is always imply.file.
  • file stores the name of the uploaded file.

If you supply values for env and team in the file upload UI, Lumi stores them as system attributes on events. Otherwise, the events don't have those system attributes. If the CSV file contains columns for env or team, Lumi stores your supplied values as user attributes instead.

You can't set the env and team system attributes with JSON upload; however, you can assign them as user attributes.

For details on additional system attributes, see the event model reference.

User attributes​

User attributes can provide context to events and help you drill down into query results.

When you upload a CSV file exported from Splunk, Lumi stores all the fields as user attributes. The fields _raw and _time aren't stored as user attributes since they map to the event message and timestamp, respectively.

You can configure source, sourcetype, or index for CSV file uploads. If the attributes are already defined in the source file, Lumi retains the values from the file. In other words, if source is defined in the CSV file, you can't override it during the file upload.

When you upload a JSON file, Lumi stores all the JSON fields as user attributes by default. To only store a subset of the fields, go to Allow list > Specify fields, and enter a comma-separated list of fields to retain.

Note that you can modify user attributes on uploaded events using pipelines. In a pipeline, you can add attributes from static values, other attribute values, or parsed text. You can also remove or replace user attributes. To learn more, see Transform events with pipelines.

For more information on user attributes, see the event model reference.

Search events from a file​

To explore events related only to the file, use the file system attribute in your search. You can also filter by a user attribute specific to the file. For example, add the following filter to your query, and set the appropriate time range:

#file=site_visitors_quickstart.csv

Search by file name

You can search all events with a specific file extension using a wildcard. For example:

#file=*.csv

You can search all events Lumi received by file upload. For example:

#receiver=imply.file

Learn more​

For more information, see the following topics: