Ingest from files

This topic describes the process to upload and ingest data from files into tables in Imply Polaris.

You do not need to create a table before ingesting data from files. You may choose to create a table before ingestion, such as to control the table type or schema mode. For information on tables, including automatically created tables, see Introduction to tables.

Prerequisites

The following permissions are required to batch ingest from files:

• ManageTables
• ViewTables
• ManageFiles
• ManageIngestionJobs

Visit Permissions reference for more information on permissions.

File staging area

View the staging area from the Sources tab of the left navigation tree. In the staging area, you can view uploaded files or upload files to use in ingestion. The file staging area displays the name, size, and upload date of files. Click on the ellipsis menu for any file to view its MD5 checksum, select it as the ingestion source for a job, or to delete the file.

You can upload a file while setting up an ingestion job or do so beforehand in the staging area. To upload a file in the staging area, click Create source > Select files from computer.

See Upload files by API for a guide on uploading files using the Polaris API.

The following screenshot shows the file staging area:

Ingest data from uploaded files

1. From the Jobs tab, select Create job > Insert data. Select the table to ingest data into. If the table doesn’t exist, click New table and provide the name of the table.
   Polaris determines the appropriate table type and schema mode from the ingestion job. For more details, see Create a table.

2. Select Files from the batch ingestion options.

   

3. Select the files you want to ingest. You can choose the files you uploaded in the staging area or upload new files.

   Polaris can load data from multiple files in a single batch ingestion job as long as the files have the same format. To ingest data from different formats into the same table, create a separate ingestion job for each format—for example, one job to ingest newline-delimited JSON data and a separate job to ingest CSV data. See Supported source data formats for the data formats supported by Polaris.

4. After you select your files to ingest, Polaris automatically detects the file type and samples your data in the Parse data step. In this step, you can change the input format, correct the data types of input fields, and add any missing input fields. Select Continue.

   

5. Next, you can map input fields to table columns. You can refine the schema further with the following actions:

   • Add or delete columns
   • Update a column name
   • Change a column's data type
   • Set a different source column

   

6. Click Start ingestion to begin the ingestion job.

You can also complete batch ingestion from files using the Polaris API. To learn how, see Ingest data from files by API.

Learn more

See the following topics for more information:

• Ingestion sources for data sources for Polaris ingestion jobs.
• Upload files by API for uploading files using the Polaris API.
• Ingest data from files by API for batch ingestion using the Polaris API.