Skip to main content

File upload

AI summary
Describes requirements for uploading log files to Imply Lumi, including file size limits and format specifications. Explains how Lumi assigns timestamps and messages for uploaded events. Covers system and user attributes that Lumi applies to file uploads.

About AI summaries.

To quickly get started with Imply Lumi, you can upload log files. For a step-by-step guide using file upload, see the Quickstart.

This topic describes the requirements for files you plan to upload to Lumi.

Prerequisites​

To upload files to Lumi, you need the following:

  • Access to Lumi with the Data manager role or higher. For information on roles and permissions, see Manage roles.
  • Files in CSV or JSON format. For more information, see Event formats.

File size limit​

Lumi supports a maximum file size of 200 MB. This limit refers to the size of the file transmitted by the browser or HTTP client.

Event parsing​

This section describes how Lumi parses events from uploaded files. To transform the event timestamp, message, or user attributes after upload, see Transform events using pipelines.

CSV​

For generic CSV format, Lumi parses events as follows:

  • Event timestamp: Detected from the logs, else the current timestamp
  • Event message: Whole CSV row
  • User attributes: Each CSV field is a user attribute. You can customize CSV parsing if you use a delimiter other than comma or if you want to provide your own column headers (attribute names). You can also assign Splunk default fields in the UI.
  • System attributes: At least the following—
    • receiver: imply.file
    • file: FILENAME, where FILENAME is the name of the file
    • env: any value you entered for Env in the UI
    • team: any value you entered for Team in the UI

Splunk® CSV​

For Splunk CSV format, Lumi parses events as follows:

  • Event timestamp: Assigned from _time field
  • Event message: Assigned from _raw field
  • User attributes: All CSV columns except _time and _raw You can also assign Splunk default fields in the UI.
  • System attributes: At least the following—
    • receiver: imply.file
    • file: FILENAME, where FILENAME is the name of the file
    • env: any value you entered for Env in the UI
    • team: any value you entered for Team in the UI

File upload CSV

JSON​

For JSON format, including Splunk HEC, specify the fields that map to the timestamp and message.

  • Event timestamp: Field you designate in Time. Also select your time format and time zone. For supported formats and examples, see Timestamp handling.

  • Event message: Field you designate in Message. You can specify a single field or combine multiple fields to make up the event message.

    To combine multiple fields, enter the field names separated by a comma. Lumi joins the values in order with a space character. For example, for {"key1": "val1", "key2": "val2"}, the Message key1,key2 stores the event message val1 val2.

  • User attributes: All other fields besides the ones used for the timestamp and message. To only store a subset of the fields, enter the fields you want to retain as a comma-separated list in Allow list > Specify fields.

  • System attributes: At least the following—

    • receiver: imply.file
    • file: FILENAME, where FILENAME is the name of the file

File upload JSON

Splunk default fields​

You can configure the Splunk default fields source, sourcetype, and index for CSV file uploads. If the source file already defines the attributes, Lumi assigns the values from the file. If you don't assign the fields in your data or in the UI, Lumi doesn't create the user attributes.

Search events from a file​

To explore events related only to the file, use the file system attribute in your search. You can also filter by a user attribute specific to the file. For example, add the following filter to your query, and set the appropriate time range:

#file=site_visitors_quickstart.csv

Search by file name

You can search all events with a specific file extension using a wildcard. For example:

#file=*.csv

You can search all events Lumi received by file upload. For example:

#receiver=imply.file

Learn more​

For more information, see the following topics: