Ingest using SQL
You can define an ingestion job to load data into Polaris using SQL data manipulation language (DML). With SQL-based ingestion, you describe the source data, destination table, and any data transformations in a single SQL statement. For reference on SQL syntax for ingestion jobs in Polaris, see SQL ingestion reference.
SQL-based ingestion is supported in the UI and the API. When using the UI, you can switch to SQL-based ingestion at any stage. However, any changes you make to the SQL statement do not persist when you switch back to the load data flow.
Prerequisites
A table. For details on creating a table, see Create a table.
A batch ingestion source. You can only use SQL-based ingestion for batch ingestion. For available batch ingestion sources, see Ingestion sources overview.
The
ManageIngestionJobs
user permission. For more information on permissions, visit Permissions reference.
Ingest using SQL
To ingest using SQL, follow the same ingestion process in Ingest data. At every step, Polaris displays the option to Switch to custom SQL, as shown in the screenshot below:
When you switch to custom SQL, Polaris fills in as much information as it can for the SQL query:
You can switch back to the basic ingestion process at any stage, although Polaris does not retain any changes you make in the SQL editor.
When viewing ingestion jobs, Polaris displays both the job request payload that you can use directly in the API as well as the SQL statement itself.
Learn more
See the following topics for more information:
- SQL ingestion reference for reference on SQL syntax in Polaris.
- Ingest using SQL by API for how to submit a SQL-based ingestion job using the Polaris API.
- Create an ingestion job for how to create an ingestion job in the UI.