Release notes

The following release notes provide information on features, improvements, and bug fixes up to Imply STS release 2022.04.

Read all release notes carefully, especially the Upgrade and downgrade notes, before upgrading to 2022.04.

If you are upgrading by more than one version, read the intermediate release notes too.

On October 12, 2021, all 3.x versions of Imply entered end-of-support status. See Lifecycle Policy.

See Previous versions for information on older releases.

Imply evaluation

New to Imply? Get started with an Imply Hybrid (formerly Imply Cloud) Free Trial or start a self-hosted trial at Get started with Imply!

With Imply Hybrid, the Imply team manages your clusters in AWS, while you control the infrastructure and own the data. With self-hosted Imply, you can run Imply on *NIX systems in your own environment or cloud provider.

Changes in 2022.04

Druid highlights

The 2022.04 release version of Imply bundles version 2022.04.0-iap of the Imply distribution of Apache Druid. Before your upgrade to 2022.04, read the Upgrade and downgrade notes.

Pivot highlights

The 2022.04 release introduces numerous UX enhancements for Pivot 2.0, including several bug fixes to improve the Pivot 2.0 experience.

• Pivot 2.0 now loads and renders query results in larger increments in Pivot 2.0, leading to a significant improvement in UX on interactive queries.
• Support for customizing the Pivot favicon
• Support for filtering on data by clicking on dimension values
• Support for copying data from visualizations in Pivot 2.0
• Introduces a new config option for customizing the property Pivot references to map OIDC groups to roles. This enables role mapping using certain OIDC providers such as Auth0, which does not support a standard 'groups' scope.

Other changes in 2022.04

Druid changes

• Add feature flag for Kinesis changes requiring listShards (id: 18994)
• All columns in the ingestion spec are queryable (id: 18765)
• Include amazon-kinesis-client library in Druid Kinesis extension packaging (id: 18793)
• Improve error messages for insert statements consistent with select statements (#12414) (id: 19399)
• Improve skipCompact metrics for auto compaction to include segments skipped by skipOffsetFromLatest (id: 19321)
• Fix Boolean types not handled properly in transform expressions (id: 19457)
• Fix zulu8 setup Dockerfile for Hadoop and Hadoop 3 in Hadoop ingestion tutorial (id: 19438)
• Fix Null Columns: includeAllDimensions with flattenSpec fails to save null columns (id: 19007)
• Fix messageGap metric (id: 18880)
• Fix OOM failures in dimension distribution phase of parallel indexing (id: 18634)
• Update inQueryThreshold, convert to query context parameter(id: 18635)
• • Security fixes

Pivot changes

• Add support for customizing Pivot's OIDC groups property name (id: 18400)
• Add support for loading in 'larger' increments in Pivot 2.0 (id: 17947)
• Add ability to filter the data in the cube by clicking on dimension values (id: 17477)
• Add support for copying selected data on visualizations in Pivot 2.0 (id: 16841)
• Add support for customizing the Pivot favicon (id: 2056)
• Fix visualization option tabs do not have associated tooltips (id: 19115)
• Fix APPROX_COUNT_DISTINCT measures not considered additive by Pivot 2.0 visualizations (id: 18881)
• Fix "batch dashboard updates" UI breaks when dashboard already has a 2.0 tile (id: 18876)
• Fix token mismatch error message (id: 18832)
• Fix hierarchical fills not rendering in Pivot 2.0 (id: 18180)
• Fix info button being shown twice in Pivot 2.0 data cube (id: 18168)
• Fix Pivot 2.0 dashboard tile not filtering on the right dimension value when clicking on the corresponding measure (id: 18130)
• Fix time bucketing by month produces invalid query parameters (id: 18056)
• Fix color for icons in data panels (id: 18032)
• Fix can not drag on overflowing show bar (id: 17983)
• Fix cannot see most recent time points when time buckets have 'fine granularity' (id: 17834)
• Fix Pivot is not usable without changing URL after unhandled error (id: 17727)
• Fix Pivot getting stuck with graph logo in the middle of the screen (id: 17478)
• Fix webhooks fail when using a self signed cert (id: 17195)
• Fix measure pill control buttons inactive when overflow container expanded (id: 16920)
• Fix alert not visible to users that have been added as "admins" but not as "recipients" (id: 9812)

Changes in 2022.03

Druid highlights

The 2022.03 release version of Imply bundles version 2022.04.0-iap of the Imply distribution of Apache Druid. Before your upgrade to 2022.04, read the Upgrade and downgrade notes.

Pivot highlights

Pivot 2.0 enhancements

This release introduces numerous UX enhancements for Pivot 2.0, including several bug fixes to improve the Pivot 2.0 experience.

• Improved ‘Filter By Measure’ functionality
  • Filter by Measure in dashboard tiles
  • Unbounded filtering on either side
  • Exclusion via icon.

Pivot Classic improvements

Fixed the regression in Plywood filter token behavior: filter tokens which do not have a $ character being parsed as SQL expressions.

Other changes in 2022.03

Druid changes

• Update push value range and set index get operations into BitmapIndex (id: 18085)
• Update to disallow filter explosion while converting a filter to CNF (id: 18055)
• Fix an issue with FileUtils.copyLarge losing content from input stream on retry (id: 17936)
• Reduce lock contention on Druid lookups (id: 17874)
• Improve performance for topN queries on expressions and lookups (id: 17873)
• Fix an capacity issue with the Coordinator (id: 17766)
• Fix Hadoop 3 profile (id: 17601)
• Improve Protobuf ingestion speeds (id: 17428)
• Add shared access storage token support to Azure extension (id: 17427)
• Improve speed for stream ingestion of json records when you use jq expressions for field extraction (id: 17246)
• Vectorize Time_Shift function (id: 16014)
• Fix an issue where closed shards possibly affect work assignment among Kinesis workers (id: 15962)
• Fix an issue with with parallel indexing where subtasks could deadlock and stop all ingestion (id: 6975)

Pivot changes

• Fix an issue that makes Pivot unable to auto-fill dimensions for data source with complex data types (id: 18269)
• Fix "drag to select" issue with line charts in Firefox (id: 17967)
• Update loading indicator to be consistent across all visualizations (id: 17655)
• Fix expanded show bar doesn't close when you click outside the bar (id: 17651)
• Fix an issue where Pivot hangs with graph logo in middle of screen (id: 17481)
• Fix table visualization crashing when the same dimension is present in both rows and columns (id: 17193)
• Fix Sparkline maxing out CPU causing app to crash (id: 17038)
• Fix issue with dashboards not being scrollable when the cursor is over a scrollable tile (id: 16842)
• Add support for improved 'Filter By Measure' capabilities (id: 16747)
• Enable filter by measure in the dashboard tiles (id: 15964)
• Fix Plywood filter tokens that do not contain "$" character being parsed as SQL (id: 16248)
• Fix gray color for comparison in Pivot 2.0 (id: 15216)
• Add support to pass custom headers when you configure a custom webhook on an alert (id: 7869)

Platform changes

• Update default Self-hosted Manager (GKE, K8s) configuration to rotate logs more often within the container (id: 18110)

Changes in 2022.02

Druid highlights

The 2022.02 release version of Imply bundles version 2022.02.0-iap of the Imply distribution of Apache Druid. Before your upgrade to 2022.04, read the Upgrade and downgrade notes.

Pivot highlights

Pivot 2.0 enhancements

This release introduces numerous post GA UX enhancements for Pivot 2.0, including several bug fixes to improve the Pivot 2.0 experience

• Support for Sankey chart visualizations
• Support for improved query loading indicator in Pivot 2.0 visualizations
• Support for improved error handling in Pivot 2.0 visualizations
• Fixed a bug with synchronous downloads in Pivot 2.0

Other changes in 2022.02

Druid changes

• Fix an issue with multi-value string array expression (id: 16745)
• Fix exception when using useApproximateCountDistinct = false for aggregation query with identity projection (id: 16438)
• Remove duplicate Broker ServerInventoryView, improve HttpServerInventoryView logging (id: 16240)
• Fix an issue where MySQL transient exception does not exist (id: 16208)
• Disable auto compaction config cleanup by default (id: 16167)
• Fix stringAny queries against multi-value dimensions where all values are null (id: 16106)
• Enable auto-kill unused segments (id: 16066)
• Enable metadata store cleanup automatically (id: 16065)
• The 'status' API should not return the 'UNDETERMINED' state for unknown queries (id: 15991)
• ProvisionerService should be closed on overlord, when it stops being the leader (id: 15948)
• Leader overlord may have workExec shutdown and not be able to assign work to Peons / Middle Managers (id: 15911)
• Fix an issue with HTTP input source retry (id: 12459)
• Fix scenario where users are required to set AWS region settings to successfully ingest data to Google Storage (id: 11933)
• Fix segment loading for HTTP mode where load-drop-load cycle causes last load to be ignored and segments not being loaded (id: 10089)

Pivot changes

• Fix bubble chart crashing after adding a split (id: 16863)
• Fix Pivot 2.0 data cubes do not gracefully handle error states (id: 16456)
• Fix synchronous exports fail when using Pivot 2.0 (id: 16439)
• Fix facet bubbles have the wrong position on scrollable Dashboards (id: 16189)
• Disable default auto-collapse of dimensions for multi-dim pivot tables (id: 16170)
• Fix SQL error when column starts with number (id: 16071)
• Create new data cube flow still creates a 'Classic' data cube (id: 16061)
• Fix incorrect calculation of axes constraints on Pivot 2.0 visualization charts (id: 16055)
• Fix sparkline height and width behavior (id: 16018)
• Fix hover state for multi-dim treemaps (id: 16017)
• Fix display of total on split measures in vertical charts (id: 16001)
• Fix scroll bar disappears when duplicating dashboard tile (id: 15971)
• Fix cannot rename or duplicate dashboard page when clicking OK (id: 15969)
• In Pivot 2.0, Filter by Measure boundary settings affect chart centering (id: 15965)

Platform changes

• Imply Enterprise K8s Istio mutual TLS support (id: 13756)

Known issues

Plywood filter tokens behavior

We have discovered a regression in Plywood filter tokens, where token expressions which do not contain the $ character are being incorrectly interpreted as SQL expressions. For example, some roles may have a filter expression of ‘true’ to allow some users to see all data in a filtered cube. To work around this issue, you can replace these with a statement that always evaluates to true, e.g $count != 0. A fix is currently in progress for inclusion in a future release.

Changes in 2022.01.1

Platform changes

• Fix a permissions issue that prevents Imply Manager from creating or updating new clusters in GKE (id: 16211)

Changes in 2022.01

Druid highlights

The 2022.01 release version of Imply bundles version 2022.01.0-iap of the Imply distribution of Apache Druid. Before your upgrade to 2022.04, read the Upgrade and downgrade notes.

Support for ZooKeeper 3.4 is deprecated

Support for ZooKeeper 3.4 is deprecated and will be removed in the 2022.01-LTS release. Imply will be migrating to new ZooKeeper versions over upcoming releases. If you are currently running ZooKeeper 3.4, please consider upgrading your infrastructure.

Impact

• Imply Hybrid: Starting with the 2021.10 release, Cloud Manager will run ZooKeeper 3.5. Customers who upgrade to the 2021.10 release (or later) will be upgraded to ZooKeeper 3.5 automatically.
• Enhanced Imply Enterprise on GKE and Imply Enterprise on Kubernetes: Customers who use their own ZooKeeper 3.4 (or earlier) must update their version of ZooKeeper to 3.5 (or later) before upgrading to Imply 2022.01-LTS.
• Imply Enterprise on Linux and Unmanaged Imply: Customers must update their version of ZooKeeper to 3.5 (or later) before upgrading to Imply 2022.01-LTS.

Pivot highlights

Pivot 2.0 now GA

Pivot 2.0 is now GA. This release introduces numerous changes to enhance Pivot 2.0 and bring it closer to feature parity with Pivot Classic. The changes affect data cubes and dashboards, and include, among others, the following changes:

• Additional visualization types for Pivot 2.0 data cubes, including spot matrix, geo shade, and bubble visualizations.
• Support for setting a default data cube view.
• Support for new and improved filtering by measure.
• Support for dimension URLs.
• Ability to customize the color palette by cluster.
• Support for brush and link filtering for dashboards (that is, the ability to click on visualizations and have that filter applied to all other views that share the filtered dimension).

The following describes the impact for new deployments compared to upgraded deployments:

• For new installations, Pivot 2.0 and Pivot SQL are now on by default. An administrator can still disable them using a feature flag. Disabling them results in new data cubes using Pivot Classic by default.
• For existing installations that have been upgraded from a previous version, there are no changes to existing data cubes or dashboards. Administrators can enable Pivot 2.0 and Pivot SQL as feature flags. If enabled, users have the ability to choose whether to use Pivot 2.0 or Pivot Classic when creating data cubes.

Pivot SQL now GA

Pivot SQL is now generally available (GA). This release brings in additional capabilities to bring Pivot SQL to GA status:

• Support for subset filters with Pivot SQL expressions
• Support for filter tokens with Pivot SQL expressions
• Deprecation of support for customAggregations in Plywood

Pivot SQL is on by default for new deployments. For the impact on upgraded clusters, see Pivot 2.0 now GA.

Pivot Classic and Plywood data cubes deprecated

Pivot Classic and Plywood data cubes are now deprecated. Support will be terminated in a future release.

Async downloading GA in Imply Enterprise

Asynchronous downloading is now GA for Imply Enterprise (formerly Imply Private) deployments. It remains in beta status in other deployment types.

Other changes in 2022.01

Pivot changes

• Prevent Total from showing up on the axis for vertical bar chart (id: 15138)
• Fix padding for display text on regex filters (id: 15085)
• Make empty visualizations look consistent (id: 15081)
• Fix dropdown label to say "measure" instead of "Without" (id: 15080)
• Add support to enable check/uncheck upon clicking on text for filter menu (id: 15079)
• Apply style adjustments in filter menus (id: 14803)
• Adjust min and max heights selection pane for dimensions and measures (id: 14734)
• Fix legend on dashboard tile overlaps tile drop shadow and rounded corner (id: 14663)
• Fix typo when attempting to show a geo visualization with no geo dimensions (id: 14662)
• Fix clicking visualization legend results in a crash when viewing tile in data cube (id: 14611)
• Fix large downloads failing in async downloads (id: 14588)
• Fix error when dragging a dimension on the show bar while displaying the "Totals" visualization (id: 14480)
• Fix new measure modal issue affecting aggregation (id: 14435)
• Add code to handle errors when async downloads are disabled in Druid (id: 13378)
• Remove "slow" label from download options gated by DownloadLargeData permission when using async downloads (id: 13136)
• Fix "Add total row to file" doesn't work for measures other than Count (id: 12985)
• Add support for being able to view the status of an async download anywhere in Pivot (id: 12764)
• Change verbose-gated log messages to use the "debug" method (id: 12657)
• Surface OIDC role name mismatch warning messages only in verbose mode (id: 11771)
• Production support for visualization auto-selection and suggestions (id: 9901)
• Add icons to the header in resource views; indicate the color theme (id: 9479)
• Fix grid view when filter limit is set high enough to return no results (id: 8915)

Druid changes

• Fix an issue with incorrect column types during ingestion (id: 15344)
• Fix array type strategy write size tracking (id: 15315)
• Update EARLIEST_BY/LATEST_BY to make EARLIEST/LATEST function signatures less ambiguous (id: 15259)
• Fix PostAggregator interface to make it backward compatible (id: 15199)
• Update log4j2 to 2.17.1 (id: 15149)
• Fix delegated smoosh writer and some new facilities for segment write-out medium (id: 15132)
• Improve configurable Clarity emitter HTTP client worker pool size (id: 14626)
• Add support for histogram filtering in sessionization (id: 13552)
• Add support for pruning segments on all dimensions in the multi-dim partitioning scheme (id: 12914)
• Fix an issue with DataSketches not recognizing the same IPv6 address in different representations (id: 12901)
• Fix an issue with overlapping segment intervals causing automatic compaction to fail (id: 12671)
• Distinguish between query metrics for async queries and sync queries (id: 10949)
• Add a guardrail that prevents creating too many time chunks by mistake during batch ingestion (id: 10485)
• Fix an issue with grouping keys unexpectedly reduced in SQL planner (id: 7780)
• Fix an issue where SQL queries with floor(<integer>) return wrong results (id: 6698)
• Security fixes

Platform changes

• Fix an issue with helm upgrade command failing to upgrade a cluster in a Kubernetes environment. (id: 15236)

Upgrade and downgrade notes

Before you upgrade or downgrade, review the following items to avoid potential version-related issues.

Minimum supported version for rolling upgrade

Rolling upgrades have been verified for 2022.01 and later. Clusters older than 2021.10 need to upgrade to 2021.10 first before upgrading to the latest version.

Airline dependency upgrade for Druid extensions

Applies to versions 2022.03 and later.

The Airline dependency for Apache Druid has been upgraded to Airline 2. Airline is a Java library providing an annotation-based framework for parsing command line interfaces. This is a backwards-incompatible change. Any extensions that rely on Druid's CliCommandCreator must also be updated.

Kinesis indexing service requires ListShards permission

Applies to versions 2022.02 and later.

To use the Kinesis indexing service, the IAM user that connects to Kinesis must have the ListShards permission. If the user doesn't have the ListShards permission, the extension throws a "Not authorized" error.

Coordinator automatically cleans up unused segments and stale metadata

Applies to versions 2022.02 and later.

The Coordinator automatically kills unused segments older than 90 days. If you want to change this behavior, set druid.coordinator.kill.on to false. The property killAllDataSources is removed and is no longer respected. If you had killAllDataSources set to false and killDataSourceWhitelist set to empty with the intention of retaining unused segments for all datasources, you must now set druid.coordinator.kill.on=false to keep unused segments older than 90 days.

Druid also removes audit records, supervisor records, rule records, and datasource records older than 90 days. You can configure the retention period by setting the durationToRetain field for each property. See Disable automated metadata cleanup for instructions on disabling the automated cleanup.

Changed behavior for null values and empty arrays in multi-value string dimensions

Applies to versions 2022.02 and later.

The version of Apache Druid released with Imply version 2022.02 fixed a bug regarding how null values and empty arrays were treated in multi-value string dimensions. Null values were treated in multi-value string functions as an array of null, [null], rather than a null value itself. Similarly, empty arrays [] were treated as [null]. Null values are now treated as null and empty arrays are treated as []. This may change the output of some multi-value string functions. If you rely on the past behavior, set druid.expressions.homogenizeNullMultiValueStringArrays=true in the commmon.runtime.properties file.

Upgrade to Imply Manager 2022.01 first

Applies to versions 2022.01 and later.

If you are on an Imply Manager version prior to 2021.09, you must upgrade Imply Manager to version 2022.01 (or later) before upgrading the rest of the Imply platform to version 2022.01 or later. Imply Manager 2022.01 supports Imply distributions that only include the MariaDB connector.

Upgrade to Imply Agent v5 for Python 3.9 support

Applies to versions 2022.01 and later.

If you are using Python 3.9, you must upgrade to the Imply Agent version v5.

EARLIEST_BY and LATEST_BY functions replace EARLIEST/LATEST with custom time parameter

Applies to versions 2022.01 and later.

Usage of EARLIEST/LATEST functions with a custom time column has been removed from the Imply distribution of Apache Druid. This removes the cognitive complexity with the coexistence of the maxBytesPerString parameter as well as timeColumn. You can use the following new functions to set a custom time column: EARLIEST_BY/LATEST_BY. For more information, see Aggregation functions.

Enable async download after Imply upgrade

Applies to versions 2022.01 and later.

If you use an Imply version that's older than 2022.01 and upgrade to version 2022.01 or later, you cannot enable async download during the upgrade. Enable the async download extension in a separate step after your cluster has finished updating.

Deprecation notices

CrossTab view is deprecated

The CrossTab view feature is deprecated. It is replaced by Pivot 2.0, which incorporates the capabilities of CrossTab view.