Skip to main content

Ingest using SQL

You can define an ingestion job to load data into Polaris using SQL data manipulation language (DML). With SQL-based ingestion, you describe the source data, destination table, and any data transformations in a single SQL statement. For reference on SQL syntax for ingestion jobs in Polaris, see SQL ingestion reference.

SQL-based ingestion is supported in the UI and the API. When using the UI, you can switch to SQL-based ingestion at any stage. However, any changes you make to the SQL statement do not persist when you switch back to the load data flow.

Prerequisites​

  • A batch ingestion source. You can only use SQL-based ingestion for batch ingestion. For available batch ingestion sources, see Ingestion sources overview.

  • The ManageIngestionJobs user permission. For more information on permissions, visit Permissions reference.

Ingest using SQL​

To ingest using SQL, follow the same ingestion process in Ingest data. At every step, Polaris displays the option to Switch to custom SQL, as shown in the screenshot below:

Select ingestion source

When you switch to custom SQL, Polaris fills in as much information as it can for the SQL query:

Custom SQL job

info

You can switch back to the basic ingestion process at any stage, although Polaris does not retain any changes you make in the SQL editor.

Select Edit context in the menu bar to set context parameters for your job. For example:

{
  "sqlTimeZone": "America/Los_Angeles",
  "sqlJoinAlgorithm": "sortMerge"
}

Edit context in SQL job

View SQL query​

In the Job details page of an ingestion job, Polaris displays both the ingestion job spec as well as the SQL query itself. You can use the spec or SQL as the basis for new ingestion jobs.

For more details, see Manage ingestion jobs.

View SQL job

Learn more​

See the following topics for more information: