Release notes

The following release notes provide information on features, improvements, and bug fixes up to Imply STS release 2021.03. Read these release notes carefully before upgrading to 2021.03, as well as intermediate release notes if you are upgrading by more than one version.

The current version of Imply bundles version 0.21.0-iap5 of the Imply distribution of Druid.

Imply evaluation

New to Imply? Get started with an Imply Cloud Free Trial or start a self-hosted trial at Get started with Imply!

With Imply Cloud, the Imply team manages your clusters in AWS, while you control the infrastructure and own the data. With self-hosted Imply, you can run Imply on *NIX systems in your own environment or cloud provider.

Changes in 2021.03

Druid highlights

Compaction improvements

Druid compaction now preserves the query granularity for compacted segments by default. For manual compaction, you can specify a query granularity for compacted segments. For example, if you ingest your data with hour query granularity, but after two months, you only need day granularity, you can configure a compaction task to use day granularity for compacted segments older than two months. See Data handling with compaction.

Increased security for HTTP and HDFS input sources

You can now configure Druid to apply additional restrictions on the allowed protocols for URIs with HTTP and HDFS ingestion sources. This feature allows cluster operators more control over how users can load data into the cluster. The two new settings are druid.ingestion.hdfs.allowedProtocols and druid.ingestion.http.allowedProtocols, and default to ["hdfs"] and ["http", "https"] respectively. If ingestion tasks are using other protocols than these defaults, configure this setting to include them when upgrading to prevent tasks from failing. See Druid upgrade notes for more information.

Configurable option for result fetch size

A new configuration property, druid.sql.avatica.minRowsPerFrame, a counterpart to druid.sql.avatica.maxRowsPerFrame, gives cluster operators more control over the size of result ‘fetch’ size. This indirectly reduces the overall number of fetch requests required and can improve JDBC performance with large result sets because they can require many thousands of separate fetches to transfer. See SQL in the configuration reference.

Ingestion support for Confluent Schema Registry

Beta. This feature improves Druid support of Apache Kafka + Apache Avro + Confluent Schema Registry with updated library dependencies, providing support for authorization headers and Schema Registry client configuration. This release also introduces experimental support for Kafka + Protobuf + Schema Registry. Both Avro and Protobuf Schema Registry support is limited to the older "parser" based ingestion specifications instead of the "input format" which has replaced it. See the documentation for Protobuf.

Other changes

Pivot changes

  • Add support for navigating between the Crosstab view and the regular data cube explorer visualizations--visualizations now persist across Crosstab and regular views
  • Fix wrong query created when both filter by measure and calculation of change is applied
  • Add support for configuring Pivot to show a persistent custom header and/or footer, such as "Confidential," to display for all your Pivot users
  • Fix data exports can have incorrect column headers when transformed measures are present
  • Fix alert setup form inaccurately renders comparison as "Previous period" after the "current period" has been redefined
  • Fix alert comparison period is not correctly preserved by link from alert occurrence to data cube view
  • Add support for adding multiple filters against string dimensions--you can now copy and paste to add many string filters simultaneously
  • Fix CSV/TSV exports incorrectly show some column values as "undefined"--a new config setting lets you override the default maximum column/row export value
  • Add support for adjusting the width of string filter menus in the data cube explorer so that users can view longer values
  • Security updates

Druid changes

  • Fix the Web Console Services tab to display which Coordinator and Overlord nodes are currently serving as the leader, providing additional insight to cluster operators
  • Fix Service view actions when grouping in the Web Console
  • Fix runtime error when IndexedTableJoinMatcher matches long selector to unique string index
  • Introduce primitive-typed bucketStart, increment methods for granularity to improve performance
  • Remove namespace property that does not exist from JDBC lookup for the Web Console
  • Add a parser per InputEntityReaderCsvInputFormat
  • Support for Confluent schema registry with authentication. Experimental
  • Fix streaming ingestion failure caused by empty or null rows in the topic
  • Fix a Java runtime exception error while applying rule DruidQueryRule(AGGREGATE) post upgrade from Imply-3.2.9 TO Imply-4.0.4
  • Fix maxBytesInMemory for heap overhead of all sinks and hydrants check
  • Add JDBC handler config for minimum number of rows per frame to improve performance
  • Add SQL functions to support bitwise operations on LONG and DOUBLE typed values
  • Support specifying queryGranularity for compaction
  • Preserve queryGranularity metadata during compaction by default
  • Add a feature to improve query planning for correlated subqueries. To turn on this experimental feature, set enableJoinLeftTableScanDirect=true
  • Fix OvershadowableManager inefficiently handles large numbers of segments in a single time chunk (#10892)
  • Fix historicals can use more disk than configured for the segment cache
  • Security updates

Platform changes

  • Support for a larger number of segments per node for Imply Private on Kubernetes, including for Enhanced Imply Private on Google Kubernetes Engine
  • The Manage Data link is now disabled in Imply Cloud when using direct access Pivot
  • Improved support for managing cluster configuration declaratively with the ability to specify Imply versions for clusters in Helm charts

Druid upgrade notes

  • Because of the enhanced security of this release around HTTP and HDFS input sources, if you use an HDFS input source for anything other than HDFS, for example WebHDFS, or an HTTP input source for anything other than HTTP or HTTP, the task will fail upon upgrade. Either update your ingestion spec to conform or update the value for the corresponding configuration property: druid.ingestion.hdfs.allowedProtocols or druid.ingestion.http.allowedProtocols.

Changes in 2021.02

Pivot changes

  • Fix preserving the columns in a crosstab when accessed through a URL
  • Fix default values on global filters are not applied when loading a dashboard
  • Fix Pivot SQL timeout after 60000 ms regardless of configured timeout
  • Fix using Postgres as a session or state store is broken due to an incompatibility introduced by a library upgrade
  • Fix TLS 1.0 and TLS 1.1 are not correctly supported for Pivot metadata storage connections
  • Fix alert conditions with percent delta cannot be inputted correctly
  • Fix error shown when adding a data cube to favorites
  • Fix error shown when attempting to change user profile properties
  • Fix global dashboard filters are not updated correctly
  • Add support for multiple connections when creating a data cube from Pivot SQL
  • Fix reset password link does not include serverRoot value
  • Fix filters are not respected when compare or multi-range time selections are used in Pivot SQL
  • Fix alert setup form always defines “previous period” as “previous day”
  • Improve UX when adding permissions to user roles
  • Fix CSV exports on time-series visualizations have incorrect column headers

Druid changes

  • Improve error reporting for InputSource errors during sampling
  • Add vectorized theta sketch aggregator and rework of VectorColumnProcessorFactory
  • Add Bitwise math function expressions to the Druid native expression system: bitwiseAnd, bitwiseComplement , bitwiseOr, bitwiseShiftLeft, bitwiseShiftRight, bitwiseXor. Add bitwiseConvertDoubleToLongBits and bitwiseConvertLongBitsToDouble to allow use with double typed columns (#10605)
  • Fix the Web Console to treat null as not defined in AutoForm (#10751)
  • Improve handling of damaged segments such that the historical unloads damaged segments automatically when lazy on start
  • Fix to correctly deserialize empty keys in a map
  • Add the latest AWS Web Identity Token Support for Druid to support IRSA
  • Fix the Web Console to use new example manifest
  • Fix an issue where the Broker starts before SQL metadata view is fully initialized even when awaitInitializationOnStart=true
  • Fix SegmentAnalyzer to properly close column after retrieving it
  • Improve query execution to retain order of AND, OR filter children
  • Fix cardinality estimation to calculate numShards independently of partition dimensions
  • Fix an issue where the Broker doesn't start if druid.extensions.useExtensionClassloaderFirst is set to true and extensions load duplicate jars
  • Better handling of Kinesis errors and message gap metric for Kinesis ingestion
  • Fix java.lang.IllegalStateException: 'other' must be a different instance from 'this' error for some vectorized queries using OrFilter

Platform changes

  • Add Ubuntu 18.04 as a supported platform for Imply Private on Linux
  • Add support for non-default Kubernetes scheduler via schedulerName Helm configuration
  • Fix using authentication in Imply Private on Kubernetes breaks custom file download
  • Fix security section does not appear for a cluster under API in the Cloud Manager UI
  • Fix placement of security parameters in the Pivot configuration file
← DesignQuickstart →