Ingest using SQL

You can define an ingestion job to load data into Polaris using SQL data manipulation language (DML). With SQL-based ingestion, you describe the source data, destination table, and any data transformations in a single SQL statement. For reference on SQL syntax for ingestion jobs in Polaris, see SQL ingestion reference.

SQL-based ingestion is supported in the UI and the API. When using the UI, you can switch to SQL-based ingestion at any stage. However, any changes you make to the SQL statement do not persist when you switch back to the load data flow.

Prerequisites

  • A table. For details on creating a table, see Create a table.

  • A batch ingestion source. You can only use SQL-based ingestion for batch ingestion. For available batch ingestion sources, see Ingestion sources overview.

  • The ManageIngestionJobs user permission. For more information on permissions, visit Permissions reference.

Ingest using SQL

To ingest using SQL, follow the same ingestion process in Ingest data. At every step, Polaris displays the option to Switch to custom SQL, as shown in the screenshot below:

Select ingestion source

When you switch to custom SQL, Polaris fills in as much information as it can for the SQL query:

Custom SQL job

You can switch back to the basic ingestion process at any stage, although Polaris does not retain any changes you make in the SQL editor.

When viewing ingestion jobs, Polaris displays both the job request payload that you can use directly in the API as well as the SQL statement itself.

View SQL job

Learn more

See the following topics for more information:

← Create an ingestion jobJob auto-discovery →