Skip to main content

Release notes

These release notes provide information about the latest features and improvements to Imply Polaris. For older release notes, see the archive.

info

Some of the features listed here depend on a database upgrade, which can take one to three days to roll out to all customers. After the database upgrade completes, such features will be available in Polaris.

For information about Imply Enterprise, see Imply Enterprise and Hybrid release notes.

2023.11.28​

Feature highlights​

Three value logic​

Polaris is adopting some SQL-compliance improvements from Apache Druid. Beginning with the 2023-11-28 release, new projects in Polaris will use SQL three value logic. The new behavior adheres to the SQL standard. Inequality operators will only match rows with values not equal to the compared value, disregarding NULL values. For example, SELECT * FROM my_table WHERE dim <> 'some value' matches all rows with values not equal to some value. NULL values are ignored and will not be returned in query results.

For existing projects, the logic remains the same. For example, SELECT * FROM my_table WHERE dim <> 'some value' matches all rows with values not equal to 'some value' as well as NULL values.

Imply will enable the new three value logic behavior for all existing projects as part of the 01/31/24 release. If you need additional time before we enable this capability, please contact Polaris customer support.

Window functions (beta)​

Window functions are now available in Polaris. You can enable window functions in your queries by setting context parameter enableWindowing to true.

Improvements and bug fixes​

  • Added support for scheduling reports to the flat table and gauge visualizations (id: 34900)
  • Added a progress bar to visually track the status of batch ingestion jobs (id: 34633)
  • Added the ability to minimize the left-hand page panel of a dashboard to a minimal length (id: 33333)
  • Added the optional CLOUD property to embedding URLs. See Access a restricted link for more information (id: 39097)
  • Enabled the casting of literals in specific Druid SQL functions (id: 39070)
  • Fixed an issue where auto-discovery was applied to streaming ingestion jobs with strict schema enforcement table mode (id: 39548)
  • Fixed an issue where data cubes were refreshing at intervals inconsistent with the configured refresh rate (id: 39597)
  • Fixed an issue where applying exclude filters on multi-value dimension columns displayed the total count instead of individual values (id: 35604)
  • Fixed an issue that prevented data downloads from the flat table visualization (id: 38619)
  • Fixed an issue where nested GROUP BY queries failed when an ORDER BY clause was present in the outer query (id: 38610)
  • Fixed an issue with inconsistent lookup state after multiple JSON updates (id: 35149)
  • Improved the performance of the HLLSketchMerge aggregator(id: 38408)
  • Polaris now automatically deletes completed batch ingestion jobs older than 90 days when a project exceeds 5000 finished batch ingestion jobs (id: 39236)
  • Replaced the visualizationType property in the viewDescription object of the Embedding v1 API with the type property (id: 39519)
  • Updated the UI to display only rollup granularity options. Previously, the Time granularity drop-down also included partitioning granularities (id: 39292)

2023.11.08​

Improvements and bug fixes​

  • Added support for the Asia/Calcutta time zone (id: 39233)
  • Added support for the all rollup granularity option (id: 38167)
  • Added data visualization drop-down picker that is organized by visualization category (id: 38225)
  • Improved UX around split limits for data cubes (id: 38841)
  • Improved error messages of task reports of an unknown type of error (id: 38764)
  • Improved S3 wildcard pattern validation. Previously, Polaris returned an HTTP 500 status code if the wildcard pattern provided in the search string was invalid. Polaris now returns an HTTP 400 status code indicating that the problem is the wildcard pattern (id: 39078)
  • Fixed an issue with SQL tab retaining state when switching between projects (id: 37777)
  • Fixed an issue with the project deletion dialog not showing errors for required fields (id: 39049)
  • Fixed an issue with filters not persisting in embedding applications (id: 38523)
  • The Embedding v1 API now returns the embedding link URL in the HTTP Location response header (id: 38580)
  • The flat table visualization now supports bucketing for continuous dimensions and filtering by measure (id: 38664, 38911)
  • You can specify the time zone and origin time when setting the rollup granularity for a table (id: 37797)
  • When creating a table using a SQL ingestion, Polaris now uses the partitioning specified by the PARTITIONED BY clause (id: 38677)

2023.10.25​

New features​

  • Ingest from multiple Kafka topics: Polaris now supports ingesting data from multiple Kafka topics through a single connection. You can provide a regular expression that identifies the topics for the connection. For more information, see Multiple topics per connection (id: 35108)
  • Kinesis ingestion metrics: The Metrics export API now returns Kinesis lag and ingestion throughput metrics to enable alerting on ingestion issues for Kinesis jobs. See Kinesis metrics for more information (id: 38025)

Improvements and bug fixes​

  • Added the ability to create project-scoped OAuth access tokens through the API. For more information, see Authenticate with OAuth (id: 38049)
  • Added the Time format measure format type, which lets you specify a time format to use in a measure (id: 36818)
  • Added the ability to Clear all selections when filtering a data cube by a dimension with string format (id: 36879)
  • Added the Query timeout override property to the data cube options, to optionally override the query timeout and client timeout (id: 37770)
  • Fixed an issue where unparseable rows in a Kafka topic caused errors during ingestion (id: 35570)
  • Fixed an issue where a data cube owner could not access the data cube if the Only added people can access permission was selected (id: 35555)
  • Fixed an issue where requests to delete a file with forward slashes in the filename couldn't be processed because Polaris wasn't escaping the forward slashes (id: 37145)
  • Improved exported metrics by exposing the project_id label. For more information, see Labels (id: 35272)
  • Improved alert emails by including the dimension values that met the alert condition (id: 38321)
  • Improved custom colors so that they now apply to all components of a dashboard (id: 36823)
  • Improved pop-up notifications (id: 34092)

2023.10.11​

New features​

  • Updated Polaris UI: We've updated the look and feel of the Polaris UI with new colors, fonts, and icons. Access and manage your favorites on the home page. Use the central project selector to manage and switch between projects. Access the administration menu from every page (id: 33037)
  • Gauge visualization: The gauge visualization displays a summary of a selected aggregate as a gauge. For more information, see the Visualizations reference (id: 35915)
  • Flat table visualization: The flat table visualization is similar to the table visualization, but it displays flattened data instead of nested data. For more information, see the Visualizations reference (id: 35915)
  • Unmet alert evaluations (alpha): You can now view unmet evaluations for an alert in the Polaris UI—times when Polaris evaluated alert criteria but didn't trigger an alert because not all critera were met. Imply can enable this feature for you on request. For more information, see Set up alerts (id: 31794)
  • Non-showable dimensions in embedded visualizations: You can now select dimensions to exclude from the show bar of an embedded visualization (id: 36264)
  • Conditional formatting update: You can now create a condition based on a comparison. For more information, see Conditional formatting (id: 31468)
  • Measure types: Added the longStringPair measure data type. When defining an aggregate column that uses EARLIEST_BY or LATEST_BY, set the column type to longStringPair instead of string (id: 37714)
  • Project capacity metrics: The Metrics export API now returns project_max_size_bytes and project_current_size_bytes project size metrics to enable alerting on project capacity. See Project capacity metrics for more information (id: 38021)

Improvements and bug fixes​

  • Added AccessMonitoring read-only permission. AccessMonitoring grants read-only access to the User queries, Streaming, and Detailed metrics pages. AccessMetrics is now an organization-level permission. See Permissions reference for more information (id: 38240)
  • You can now scope API key permissions to specific projects. See Authenticate with API keys for more information (id: 37293)
  • You can now override the default relative time filter settings in the Polaris UI. By default, relative time filters are inclusive of the start bound and exclusive of the end bound. You can click the toggle to include or exclude the start and end bounds of the filter. For more information, see Relative time filters (id: 37603)
  • When you enable query monitoring in a data cube's settings, you can now view query details and open the SQL in query view. See Manage data cubes for more information (id: 37803)
  • ManageApiKey is now a project-level permission. See Permissions reference for more information (id: 37948)
  • Fixed an issue where data cube Duplicate button was visible to users without the AdministerDataCubes permission. For information on permissions, see Permissions reference (id: 36590)
  • Fixed an issue with editing or disabling alerts. Previously, alerts could not be edited or disabled if the underlying data cube was deleted (id: 36482)
  • Fixed an issue with reports not showing the correct decimal points (id: 35834)
  • Fixed an issue with dragging and dropping dimensions and measures in flat tables (id: 36595)

Other updates​

  • Support for project-scoped access tokens is now planned for October 25, 2023

2023.09.27​

New features​

  • Multiple projects: You can now create multiple projects in a single region and across multiple regions in a Polaris organization. Having multiple projects per region allows you to separate out developer, staging, and production environments for a given use case or application. Members of the Organization Admin group and authorized users can switch between projects and operate each environment independently. For more information, see Create and manage projects (id: 12490)
  • Automatically created tables: You no longer have to create a table prior to starting an ingestion job. If the table does not exist, Polaris automatically determines the table type suitable for your ingestion and creates it before ingesting your data. For more information, see Automatically created tables (id: 33540)
  • Credit-based trials: For all new users, Polaris now offers a free trial with US$500 in credits to use on small project sizes and a 30-day period to evaluate a full-featured version of Polaris (id: 21102)
  • Starter project size: Polaris now includes the starter project size. The starter project size is best suited for functional evaluation of Polaris during trial or proof of concept. It offers 25 GB storage capacity and a minimum of 4 vCPU and 16 GB RAM (id 36901)
  • Projects API (global URL): You can now call the Projects API to retrieve a list of projects for an organization. Use the base URL https://ORGANIZATION_NAME.api.imply.io/v1/projects (id: 34102)
  • Context parameters: With SQL-based ingestion jobs, you can now set context parameters in the UI to configure query planning (id: 37166)
  • Grafana support: You can now integrate Polaris with Grafana using the Druid-Grafana community plugin. We have made the Druid-Grafana community plugin compatible with Polaris by adding the ability to query and visualize data stored in Polaris. See Integrate with Grafana for more information, instructions, and examples (id: 25401)
  • Advanced data cube properties: You can now configure advanced data cube properties including Latest data strategy and Query caching. See Manage data cubes for more information (id: 36044)

Improvements and bug fixes​

  • Added AccessDataCubes and AccessDashboards read-only permissions. See Permissions reference for more information (id: 31123)
  • You can now have a maximum of 100 active alerts in a project (id: 37243)
  • Improved Email verification, Welcome, and Password reset email templates (id: 36607, 36608)
  • Fixed an issue with group by strategy and PT6H (6 hours) bucketing (id: 36529)
  • Fixed an issue where MSQ jobs could remain stuck in a WAITING state due to stale resources left from prior job failures (id: 37558)
  • Fixed an issue with case sensitive search in user group's permissions list (id: 37700)
  • Fixed unexpected behaviors when toggling Overall rows in the table visualization (id: 35907)
  • Fixed an issue with the "Change Summary" warning displaying on the Project size page unprompted (id: 37240)
  • Fixed an issue with the Add to dashboard flow. Previously, adding a new tile would either not save the tile or result in a conflict error (id: 37640)
  • Fixed an issue with the No results placeholder on the Users page (id: 37641)
  • Removed the AdministerApiKeys permission from the Project Admin group (id: 37706)
  • Removed the ManageProjects permission from the Data Manager group (id: 37175)

Authentication​

To support multiple projects, we've updated how Polaris handles OAuth authentication:

  • For projects created after September 29, 2023, you will be required to use project-scoped access tokens if authenticating with OAuth.
  • Existing OAuth clients will continue to work for projects created before September 29, 2023.

Note that support for project-scoped access tokens is planned for the next Polaris release (October 11, 2023). In the meantime, you can authenticate with API keys, which is the recommended method of authentication in Polaris.

Deprecations and sunsets​

We have introduced a number of structural changes to our regional API resources. These changes do not affect global API resources or the actual semantics of the API. For more information, refer to the migration guide for deprecated APIs.

The following are the key changes you should prepare for:

  • Project ID is now required: Project-less regional API resources have been deprecated and will be removed by the end of September 2024. You must include the project ID in the base URL for all regional API calls in projects created after September 29, 2023.

    For example, update your requests to the following APIs to include the project ID in the URL:

    • /v0/pivot/api/v1/embed is deprecated. Use /v0/projects/PROJECT_ID/pivot/api/v1/embed instead.
    • /v1/events/CONNECTION_NAME is deprecated. Use /v1/projects/PROJECT_ID/events/CONNECTION_NAME instead.
    • /v1/files is deprecated. Use /v1/projects/PROJECT_ID/files instead.

  • API version change: All API requests must reference the v1 version identifier in the URL path, even if the previous API version had a higher version identifier.

    For example, update your requests to the following APIs to use the v1 version identifier and include the project ID in the URL:

    • /v2/connections is deprecated. Use /v1/projects/PROJECT_ID/connections instead.
    • /v2/jobs is deprecated. Use /v1/project/PROJECT_ID/jobs instead.
    • /v2/tables is deprecated. Use /v1/projects/PROJECT_ID/tables instead.

  • Project API is deprecated: The Project v1 API has been deprecated and will be removed by the end of September 2024. Use the Projects v1 API instead.

  • JDBC connection string change: Applications that integrate with Polaris using JDBC, including BI tools such as Tableau and Looker, must specify the project ID in the connection URL. Support for previously established JDBC connections will be removed by the end of September 2024. For information on how to update your connection string, see the migration guide for deprecated APIs.

For backward compatibility, you can continue to use the aforementioned deprecated resources, including v2 APIs, on projects created prior to September 29, 2023. Support for these resources will be removed by the end of September 2024. We recommend that you review the migration guide for deprecated APIs for detailed instructions on how to migrate to new API resources.

2023.09.13​

New features​

  • Added a data visualization component to the Job details page of the Polaris UI. The component shows a line chart with input data from the past four weeks, displayed weekly (id: 28384)

Improvements and bug fixes​

  • Fixed an issue where alerts using the "Any" condition failed if the query returned an empty data set (id: 35914)
  • Fixed an issue where removing a favorite item connected to another user's favorites returned an HTTP 500 status code (id: 37146)
  • Fixed an issue with a customized logo not changing immediately after a new image file was uploaded (id: 37161)
  • Improved handling of save conflicts when multiple users edit the same data cube or dashboard (id: 36421)
  • Improved dynamic time filter evaluation. Relative time filters now have inclusive and exclusive start bounds (id: 37119)

2023.09.05​

New features​

  • Added the ability to display emojis in dashboards, tile names, and titles (id: 36878)
  • Implemented a data range tile displaying the minimum and maximum timestamps for a table. Additionally, this feature can detect, display, and suggest fixes for warnings for common data issues (id: 35116)
  • With schema discovery, Polaris now ingests Boolean values as long types (0 false or 1 true) instead of string types (id:36740)
  • You can now view Created by and Edited by user details for all asset types in Collections (id: 33752)
  • You can now select dimensions to exclude from the show bar of an embedded visualization (id: 36264)

Improvements and bug fixes​

  • Improved sign-up and first login experience (id: 36176)
  • Improved communication about data retention (id: 36876)
  • Improved Add payment page (id: 36749)
  • Fixed an issue where clicking Start ingestion from the SQL editor opens an Exit modal (id: 35911)
  • Fixed an issue with report creation date missing from Home and Collections pages (id: 37105)
  • Reduced frequency of data cube and dashboard save conflicts (id: 36421)

Deprecations and sunsets​

We have removed the following deprecated properties on API Tables objects:

  • createdBy. Use createdByUser instead.
  • createdTimestamp. Use createdOnTimestamp instead.
  • lastModifiedBy. Use modifiedByUser instead.
  • lastUpdateTimestamp. Use modifiedOnTimestamp instead.

2023.08.16​

Improvements and bug fixes​

  • Added the ability to edit and remove the __count column (id: 34873)
  • Added the ability to set the data retention policy per table. This allows for automatically deleting data older than some period. The default policy is to retain all data until explicitly deleted (id: 33213)
  • In the ingestion wizard, Polaris now warns about atypical timestamps, such as timestamps from the future. The warning may be bypassed but can indicate issues with parsing the source data (id: 35115)
  • Increased allowed Kafka secret length to 1 KB (id: 36450)
  • Improved downloading data from an embedded visualization (id: 35320)
  • Improved format detection during sampling (id: 35948)
  • Polaris UI improvements:
    • Improved account sign-up UI (id: 36494)
    • Improved project states UI (id: 35871)
    • Improved UI for adding columns to aggregate tables (id: 35548)
  • The JSON spec editor is now available in steps 2 and 3 of the ingestion wizard. It was previously only available in step 1 (id: 35644)

Improved error handling​

  • Improved handling of Amazon SDK exceptions (id: 34961)
  • Improved error message for invalid Snappy and Parquet files (id: 35784)
  • Improved error message for cases when an ingestion would change the aggregation used for a declared measure (id: 35566)

Documentation changes​

  • The docs now include a troubleshooting guide that you can reference to diagnose common ingestion issues in Polaris. For more information, see Troubleshoot data ingestion.

2023.08.02​

New features​

  • Alerts: Alerts now contain the query in the alert payload (id: 31383)
  • Improved filtering: The Jobs page now allows for quick filtering by active and failed job states (id: 35266)
  • Polaris support: You can now click the chat icon in the top right corner to access the live chat function. See Contact support for more information (id: 35734)
  • Ingestion job spec:
    • Added the ability to view the SQL within the job spec for batch ingestion jobs in the Polaris UI (id: 32690)
    • Added the ability to submit and edit an API spec using the Polaris UI (id: 34878)

Improvements and bug fixes​

  • Added the ability to handle dimensionless groupBy queries with partitioning and multiple workers (id: 36033)
  • Improved table name validation. Table names cannot include the following characters: ;, {, }, $ (id: 34226)
  • You can now create a flexible table with clustering columns (id: 34793)
  • Fixed an issue where a column could not be created if the table had an active filter (id: 27687)
  • Fixed an issue with the Sources page displaying an incorrect count of active jobs for a specific connection (id: 34767)
  • Fixed an issue with creating an embedding link for a street map visualization (id: 35831)
  • Fixed an issue where clicking the browser back button in the Parse data step did not reset the sampler state (id: 34786)
  • Fixed an issue with missing dimensions for undeclared columns that were mapped when schema auto-discovery is disabled (id: 35473)
  • Fixed an issue with filtered aggregate expressions incorrectly evaluating to NULL for cases where the filter was based on the result of a function (id: 36096)

Improved error handling​

  • Improved the /jobs endpoint error handling such that the errors relay the message from the response back to the user (id: 35068)
  • Polaris now retains the details of the original QueryRuntimeFault error message (id: 35796)
  • When a data cube encounters a SQL parse exception, Polaris now returns an HTTP 400 error (id: 35188)

Deprecations and sunsets​

The following deprecated properties on API Tables objects will be removed on August 16, 2023:

  • createdBy. Use createdByUser instead.
  • createdTimestamp. Use createdOnTimestamp instead.
  • lastModifiedBy. Use modifiedByUser instead.
  • lastUpdateTimestamp. Use modifiedOnTimestamp instead.

2023.07.19​

Breaking changes​

  • The TIMESERIES aggregator, interpolation, and time-weighted average time series functions now produce a binary representation of the time series. To convert the binary output to JSON format, wrap the outermost function with the TIMESERIES_TO_JSON function (id: 34848)
  • Polaris now creates tables in flexible schema mode by default (id: 35120, 35460)

New features​

  • Region-aware deep linking: Polaris URLs now include the project ID, making it possible to share links to data cubes, dashboards, and other project assets in a specific region (id: 33896)
  • SQL clause validation: Polaris now validates that the PARTITIONED BY and CLUSTERED BY clauses match the time partitioning and the clustering columns set on the target table. For more information, see SQL ingestion reference (id: 34541)
  • Time series functions (alpha): Polaris now supports additional time series processing functions. You can use the time series processing functions to transform or augment a time series (id: 34848)

Improvements and bug fixes​

  • For batch jobs, the describeJob API response now includes a query property containing the SQL generated to run the job (id: 33706)
  • Added the ability to apply date formats to time data cube dimensions (id: 28776)
  • Added the ability to remove columns from declared schema for flexible tables. This does not delete the column data, but lets you drop validation on the column going forward (id: 34714)
  • Added support for changing the table schema enforcement mode in the Polaris UI (id: 35118)
  • Added the ability to remove the __count column from an empty aggregate table (id: 34630)
  • Added a warning message to the Load custom SQL page to indicate that manual changes to the SQL query won't be persisted if you switch back to basic (id: 35107)
  • Improved UI for aggregate tables. Removed the split view and combined all columns into a single scrollable area (id: 34004)
  • Improved handling of streaming ingestion jobs that have offset errors (id: 35464)
  • Improved messaging for canceled subscriptions (id: 35331)
  • Fixed an issue where the __time column could be removed during the Map source to table step of the ingestion flow in the Polaris UI (id: 34463)
  • Fixed an issue with downloading a file to CSV if the data has a null value. Polaris now uses two consecutive commas to represent null values (id: 34859)
  • Fixed an issue with alert preview incorrectly showing that the alert will always trigger regardless of data and conditions (id: 35007)
  • Fixed an issue where filtering on a non-bucketed numeric dimension causes a partial query error (id: 35329)
  • Removed restriction for APPROX_COUNT_DISTINCT_BUILTIN (id: 35463)
  • Updated row count metrics for ingestion jobs. See Ingestion status reference for details (id: 35067)

Improved error handling​

  • Fixed a bug that prevented users from seeing the underlying error messages of batch jobs in some cases (id: 35462)

2023.07.05​

New features​

  • Multi-region support: Polaris now supports ap-south-1 Asia Pacific (Mumbai) AWS region (id: 30071)
  • Collections: You can now use the Collections API in addition to the Polaris UI to group Polaris assets into collections for easy access (id: 30064)
  • Polaris now enforces a limit to the number of files returned for the GET /v1/projects/PROJECT_ID/files request. By default, the Files API now returns 1000 files per page of results. To access subsequent pages of results, use the paginationOffset query parameter in your request (id: 34963)

Improvements and bug fixes​

  • Polaris UI now displays additional details about an ingestion job's health status (id: 34551)
  • Mapping an aggregate expression to a dimension in an aggregate table now fails with an error message (id: 33600)
  • Modified the alert occurrence list UI to emphasize the time frame of the occurrence over the time that the alert was triggered (id: 31799)
  • Improved dynamic time filter clause evaluation when maxTime is midnight (id: 34453)
  • The login page now displays SVG logos correctly (id: 35409)
  • The new Query precision setting in data cube options allows you to control the precision of TopN and COUNT DISTINCT queries (id: 33500)
  • The new Query timeout override setting in data cube general properties allows you to enter a number of milliseconds to override the default 40 second query timeout (id: 33229)
  • The Polaris API now supports SQL aliases for data types: BIGINT for long and VARCHAR for string. See Data types for more information (id: 31345)
  • Fixed an issue with the street map dashboard tile not rendering data points correctly when panning or zooming (id: 34932)
  • Fixed an issue with a data cube failing to download if a group filter was enabled (id: 33345)
  • Fixed an issue where Kinesis ingestion was not handling aggregated Kinesis records properly (id: 34727)

2023.06.21​

New features​

  • Home page: The Polaris home page now displays your favorites in the center pane. If you have no favorites, you'll see recently created Polaris assets—alerts, dashboards, data cubes, embedding links, reports, and tables. See Navigate the console for more details (id: 30064)
  • Collections: You can now group Polaris assets into collections for easy access. See Collections for more details (id: 30064)
  • Embedding visualizations:
    • You can now select dimensions to exclude from the filter bar of an embedded visualization (id: 33342)
    • You can now enable the download button on an embedded visualization and set the maximum number of rows a user can download. See Embed visualizations for more information (id: 34070)
  • Quantiles sketches: You can now use the UI to create Quantiles sketches and query Quantiles sketch expressions (id: 31344)

Improvements and bug fixes​

  • Fixed an issue with embedding street map visualization (id: 34309)
  • You can now add alerts and reports to Favorites by clicking the star icon on the relevant page (id: 34557)
  • Fixed an issue with data cubes being incorrectly marked as "missing underlying table" (id: 32561)
  • Fixed a table visualization issue where duplicate measures rendered values in a non-deterministic order when sorted (id: 34546)
  • Increased table resolution timeout from 5 seconds to 15 seconds and 4 retries (id: 34456)
  • Fixed an issue with the Measures tab not displaying any elements for pending requests (id: 32861)
  • Fixed an issue with the Jobs API not returning results for the job IDs specified in the searchString query parameter (id: 33377)
  • Improved sampling errors (id: 33611)
  • Improved data type declarations in flexible tables (id: 34345)
  • Added the ability to view undeclared columns when editing a flexible table (id: 33988)
  • UI improvements:
    • Added responsiveness support for small screens (id: 34584)
    • Updated the sign-up view (id: 34475)
    • Added the ability to switch between any of the detected input sources for the __time column (id: 33535)

Deprecations and sunsets​

  • The following Files API query parameters are now deprecated: cont, limit, and pattern. We recommend using paginationOffset, paginationLimit, and searchString query parameters instead.

    Even though Polaris does not currently enforce a limit to the number of files returned for the GET /v1/projects/PROJECT_ID/files API call, we will enforce a limit in the future. Please verify your usage of the GET /v1/projects/PROJECT_ID/files API call and use the paginationOffset, paginationLimit, and searchString query parameters to limit the number of files returned in the response.

2023.06.07​

Improvements and bug fixes​

  • Exposed the expand button on the dashboard tile that allows users to move from the tile into a cube and back (id: 33511)
  • Fixed an issue with a table-to-table ingestion job failing if it ran into any deletion markers (tombstones) added from REPLACE data jobs (id: 34096)

2023.05.24​

New features​

  • JDBC: You can now establish a connection to Polaris over JDBC using API keys. For more information, see Connect to Imply Polaris over JDBC (id: 26802)
  • Ingestion:
    • Polaris now supports SQL-based ingestion for ingesting into tables with declared schemas (id: 32855)
    • You can ingest using SQL in the UI (id: 33710)
  • IP addresses: You can ingest, store, and query IP addresses and IP prefixes with specialized data types and SQL functions (id: 31458)
  • A.01 project size: Polaris now offers a smaller project size for streaming analytics. See Billing structure overview for available project sizes.
  • Metrics export API (GA): The Metrics export API is now generally available.
  • Street map: Polaris now supports street level mapping. You can use the street map visualization to pinpoint events to precise locations on a map. See Street map for more information (id: 33072)

Improvements and bug fixes​

  • Fixed an issue with alert trigger time recorded inconsistently (id: 31798)
  • Fixed an issue with auto-filled dimension expressions (id: 33288)
  • Fixed an issue with alert backfilling not using correct max times (id: 33331)
  • Added the ability to send alert email notifications to external email addresses (id: 31178)
  • Added checkFrequency and timeFrame properties to alert payload (id: 30543)

Deprecations and sunsets​

  • The week time partitioning granularity is deprecated. Polaris no longer supports week partitioning for new tables or changing time partitioning to week for existing tables. We recommend using hour, day, or month partitioning instead. The week partitioning does not properly align with other segment sizes, causing occasional bugs (id: 33253)

2023.05.10​

New features​

  • Ingestion:

    • You can define an ingestion job to load data into Polaris using SQL data manipulation language (DML). With SQL-based ingestion, you describe the source data, destination table, and any data transformations in a single SQL statement. For more information, see Ingest using SQL (id: 30675)
    • You can manually reset the offset from which Polaris ingests streaming data. For more information, see Reset streaming job offset (id: 31334)
    • Polaris now supports Kafka event keys, event headers, and timestamps in addition to the event payload when the source is a Kafka, MSK, or Confluent Cloud connection (id: 32222)

  • Flexible table mode: You can set the schema enforcement mode when creating a table in Polaris. The schema mode can be strict or flexible. A strict mode requires that you set a declared schema on the table before ingesting data, while a flexible mode allows you to ingest data even when the table does not have a declared schema (id: 30437)

  • Multi-region support: Polaris now supports us-west-2 US West (Oregon) and eu-west-1 Europe (Ireland) AWS regions (id: 33193)

  • Quantiles sketches: Polaris now supports quantiles sketches (id: 31344)

  • Customize Polaris: You can now customize the application name, logos, UI colors, and visualization colors for your organization. For more information, see Customize Polaris and Customizations API (id: 25136)

  • The Tables API now exposes segmentCompactedBytes and segmentTotalBytes as part of the table information (id: 21856)

Improvements and bug fixes​

  • Fixed an issue with dashboards crashing when the value for Possible bucketing was set to Never bucket on a numeric dimension (id: 32056)
  • Fixed an issue with preview not rendering correctly when a contains or regex filter was used with a string dimension (id: 33126)
  • Instead of displaying the JobResponse in the job spec UI, Polaris now displays the actual spec. This improves the display of the Job spec tab so that it more closely matches what was submitted (id: 32809)
  • When you edit a dashboard with multiple pages, the edit mode now opens on the page you have selected. Previously, the edit mode would open on the first page even when you selected editing from another page (id: 33046)
  • You can now set maxParseExceptions on an ingestion job in the UI. By default, Polaris continues to ingest data when it encounters rows it cannot parse (id: 32811)

2023.04.26​

Improvements and bug fixes​

  • You can now specify a threshold for parsing exceptions for an ingestion job. You set this limit in maxParseExceptions when creating an ingestion job (id: 29582)

  • You can now trigger an alert without ingesting new data (id: 10662)

  • Polaris now preserves JSON dimension type during sampling if explicitly specified (id: 32911)

  • When you ingest from CSV or TSV files that don't have header rows, Polaris now samples the columns in order (id: 28448)

  • Fixed an issue with the Open interactive report button in the report email (id: 32490)

  • Fixed an issue where the job status remains in deleting even when delete data job fails (id: 32495)

  • Fixed an issue with _time fields in input data being assigned string data type (id: 32744)

  • When you create a Kafka/MSK connection in the UI, you can now specify the following:

    • Client rack: A Kafka rack ID, if you want Kafka to try to connect to a broker matching a specific rack.
    • Truststore certificates: One or more SSL server certificates for Polaris to trust, based on your networking setup.

    See Ingest from Kafka and Amazon MSK for more information (id: 32442)

  • UI improvements:

    • Empty table state preview now displays at least one column (id: 27788)
    • Improved ingestion job health statuses. For more information, see Ingestion status reference (id: 31342)
    • You are no longer required to enter a password when preliminary testing an existing connection with secrets (id: 32117)
    • Fixed spacing on the Job details page (id: 32901)
    • Enabled tab indices on Add new connection and Edit connection dialogs (id: 32491)

2023.04.12​

New features​

  • Kafka connections:
    • Polaris can now connect to Kafka clusters that return self-signed certificates (id: 31741)
    • Polaris now supports rack awareness for Kafka clusters (id: 31903)
  • Restricted embedding links: You can now create restricted embedding links in addition to public links. Restricted links can only be accessed when a signature created with a secret key is attached to the URL. You can create signatures programmatically using the secret key and additional parameters. See Embed visualizations and Restrict an embedding link for more information.
  • Alerts:
    • If an alert evaluation cycle is delayed into the time frame of the next evaluation due to technical failures, Polaris now identifies and backfills skipped evaluation cycles in an orderly manner (id: 31796)
  • Ingestion:
    • Polaris now supports LATEST_BY and EARLIEST_BY during ingestion (id: 30605)
    • Polaris now supports wildcard-based S3 ingestion (id: 31531)

2023.03.29​

New features​

  • Time series functions (alpha): Polaris now supports time series functions. You can use the TIMESERIES aggregation function to generate a time series from raw data. You can then transform or augment the time series using the time series processing functions. See Time series functions for more information (id: 30013)
  • Metrics export API (beta): You can now use the Metrics export API to emit ingestion and query performance metrics into third-party monitoring tools. See Monitor performance metrics for more information (id: 29590)
  • AWS IAM: Polaris now supports AWS IAM authentication mode for Amazon MSK (id: 29592)
  • S3 ingestion UI: You can now select all objects in an S3 connection (id: 31785)

Improvements and bug fixes​

  • Added the ability to select all objects in an S3 connection
  • Added the ability to view and copy an ingestion job spec from the UI. See View ingestion jobs for more information (id: 30088)
  • Fixed an issue with aggregate table-to-table ingestion using the source table's aggregator instead of the provided aggregation (id: 30953)
  • Fixed an issue with table-to-table ingestion not validating the consistency of existing table columns on a target table (id: 31907)
  • Fixed an issue with facet exports ignoring the specified row limit (id: 32039)
  • Fixed an issue with UI errors on the Job details page crashing the view (id: 31684)
  • Fixed an issue with the sunburst visualization rendering a blank spot (id: 31784)
  • You can use the Ingestion source reference to compare the various features for Polaris ingestion sources

2023.03.15​

Improvements and bug fixes​

  • Fixed an issue where users were prompted to select schema registry for Avro OCF (id: 31576)
  • Added the ability to mark individual data cubes and dashboards as favorites. Favorites are listed at the top of the data cube and dashboard pages (id: 30931)
  • Added data cube option View essence to display the JSON structure for a data cube (id: 30288)
  • Added the ability to swap axes for table and sparkline visualizations (id: 30172)
  • Improved error handling for color legends of multi-value dimensions (id: 29907)
  • Improved performance of IP dimensions (id: 29444). Update any existing IP dimensions to benefit from these improvements. To do this, edit the IP dimension and:
    1. Remove IP_STRINGIFY from the custom formula. The following example shows the change for a column named ipaddress:
      • Old custom formula: IP_STRINGIFY("t"."ipaddress")
      • New custom formula: "t"."ipaddress"
    2. Change the dimension type from String to IP or IP prefix as appropriate.

2023.03.01​

New features​

  • External ID: When you create a connection to an AWS resource, Polaris provides an external ID that you can include in your trust policy to control role assumption (id: 30601)
  • Dual axis: The line chart visualization now supports multiple scales and lines. You can display two continuous metrics on the same chart with two axes. See Visualizations reference for more information (id: 30240)
  • S3 wildcard ingestion: Polaris now supports wildcard-based S3 ingestion by API. See Ingest data from Amazon S3 by API for more information (id: 29833)
  • Kafka ingestion: You can now ingest event data from Apache Kafka streams, including from Amazon Managed Streaming for Apache Kafka (MSK), into Polaris from the Polaris UI (id: 27484)

Improvements and bug fixes​

  • You can specify the delimiter to use when parsing multi-value dimensions (id: 30585, 30586)
  • Fixed an issue with data cube visualizations not loading when the default measure is removed (id: 29737)
  • Improved ingestion flow error messages (id: 27792)
  • Fixed an issue where the report access dropdown didn’t correctly display some usernames (id: 29635)
  • Fixed an issue where changing the size of a sparkline visualization would also change the width of the sparkline (id: 30174))
  • Fixed an issue with escape sequences treated as literal strings for CSV and TSV input formats (id: 30596)
  • Fixed an issue where parsing CSV data with custom headers returned NULL rows (id: 30679)
  • Fixed an issue where long table names didn’t correctly truncate in the list of tables (id: 30746)
  • Fixed a formatting issue for table row counts (id: 30894)
  • UI improvements:
    • Moved the SQL page to the Data section (id: 30753)
    • Increased the size of visualization assets (id: 30748)
    • Added ability to drag/resize columns (id: 30558)
    • Updated column editor (id: 27795)
  • User management improvements:
    • Members of the Organization Admin group can add new users to the organization without sending an invite email. See Invite users to your organization for information on how to add new users.
    • You can resend invite emails from the Users page. To resend the invitation, click the Menu button next to the user name and select Resend invite.

2023.02.21​

New features​

  • Ingesting pre-computed sketches: You can now ingest pre-computed theta and HLL sketches. For more information, see Ingest pre-computed data sketches (id: 27457)

  • Schema registry: You can create a connection to Confluent Schema Registry from the Polaris UI. For more information, see Connect to Confluent Schema Registry (id: 26367)

  • SQL UNNEST (beta): You can unnest arrays with the SQL UNNEST function. The UNNEST function for SQL allows you to unnest arrays by providing a source array expression using the following syntax:

    UNNEST(source_expression) AS table_alias_name (column_alias_name)

    For more information, see UNNEST (id:20363)

Improvements and bug fixes​

  • Fixed an issue where changing the delimiter for data containing header rows would result in spurious NULL rows (id: 30309)
  • Removed Show metadata option from asynchronous downloads (id: 29328)
  • Fixed measure and dimension conversions for chained filter expressions (id: 29233)
  • Fixed an issue with the Open interactive report button not working in scheduled report emails (id: 28856)
  • Improved user experience for parsing data in the Polaris UI (id: 27867)
  • Fixed an issue where using a table visualization with over 1000 values would sometimes result in an incorrect sort order (id: 26933)
  • Updated the aggregate table view of dimensions and measures—includes styling and layout changes and an improved column editing experience (id: 20829)

2023.02.01​

New features​

  • Kafka ingestion: You can now ingest event data from Apache Kafka streams, including from Amazon Managed Streaming for Apache Kafka (MSK), using the API. For more information, see Ingest from Kafka and MSK (id: 27482)

  • Data schema for Avro and Protobuf: When ingesting data from Apache Avro or Protocol Buffers data formats, you must specify the input schema and provide the schema for the data in your ingestion job request. For specifying this schema, Polaris now supports two new sources:

    • An inline specification in which you set the schema definition directly within the payload of the ingestion job request.
    • Confluent Schema Registry, a storage layer for storing, organizing, and retrieving schemas.

    For more information, see Specify data schema by API.

  • Embedding visualizations: You can now use the Embedding API to manage your links to embedded visualizations. See the Embedding API documentation for details (id: 30195)

Improvements and bug fixes​

  • Increased the maximum number of columns per table to 1000 (id: 30154)
  • Improved parsing of CSV files with semicolon delimiters (id: 29964)
  • Fixed an issue with the Interpolate values measure fill option (id: 29857)
  • You can now ingest batch data into an existing datasource after specifying clustering columns (id: 29903)
  • You can now retrieve a job's logs using the Jobs API (id: 29872)

2023.01.18​

New features​

  • Identity management API: You can now create and manage users and groups through the API. See the Identity management API documentation for more information. (id: 10896)
  • Polaris UI now has a new section called Administration to deliver a seamless experience between interfaces. Use the Administration console to access User management, API access (API keys and OAuth clients), and Usage and Billing pages (id: 10896)
  • Multi-region support: Multi-region support is now promoted to General Availability. You can choose between the following regions when creating a project:
    • us-east-1: US East (N. Virginia)
    • eu-central-1: Europe (Frankfurt)
  • You can now change the appearance of a table visualization to a “flat” layout that displays a column for each dimension. See Visualizations for more information (id: 24697)
  • Ingestion with filters: You can use filters to ingest a subset of your data in the Polaris UI. See Filter data to ingest for more information (id: 27229)
  • You can now create a URL that links directly to a Polaris data cube or dashboard. See Query parameters reference for details (id: 28067, 28068)
  • You can now create embedded links for dashboards in the Polaris UI. See Embed visualizations for more information (id: 29127)

Improvements and bug fixes​

  • Added support for periods . in Confluent Cloud connection (id: 29705)
  • Fixed an issue where changing visualization would reset dimension selection in the data cube view (id: 29538)
  • Optimized IP address queries (id: 29444)
  • Fixed an issue with downloading raw data (id: 29404)
  • Improved alert creation workflow for users without the ManageAlertWebhooks permission (id: 29231)
  • Fixed an issue where setting the Overall dimension setting to Hide made pie chart and sunburst visualizations disappear (id: 28283)
  • Improved handling of reports for data cubes (id: 26073)
  • In data cube and dashboard options, you can now use the Query cache option to enable or disable the query cache for all dashboards and data cubes (id: 23747)
  • Embedding improvements and fixes:
    • When creating a link, you can now select Visualization only or Entire view to include or exclude the left navigation panel and header bar (id: 28741)
    • Interaction is now turned off for Visualization only embedded views (id: 29429)
    • A warning icon now appears in Polaris when you create a link for a data cube or dashboard that has access controls (id: 29330)
    • Improved error messaging for embedded link creation (id: 27602)
    • Improved error messaging when landing on an invalid embedded link (id: 29236)
    • Fixed an issue with embedded links failing to load without an essence hash (id: 29517)