Release notes · 2021.09

The following release notes provide information on features, improvements, and bug fixes up to Imply STS release 2021.09. Read these release notes carefully before upgrading to 2021.09.

Before your upgrade to 2021.09, read the Upgrade and downgrade notes.

If you are upgrading by more than one version, read the intermediate release notes too.

See Previous versions for older releases.

Imply evaluation

New to Imply? Get started with an Imply Cloud Free Trial or start a self-hosted trial at Get started with Imply!

With Imply Cloud, the Imply team manages your clusters in AWS, while you control the infrastructure and own the data. With self-hosted Imply, you can run Imply on *NIX systems in your own environment or cloud provider.

Changes in 2021.09

Druid highlights

The current version of Imply bundles version 2021.09.0-iap of the Imply distribution of Druid. Before your upgrade to 2021.09, read the Upgrade and downgrade notes.

Asynchronous query downloads (alpha)

Asynchronous query downloads (async downloads) lets you run longer-executing queries and retrieve the results. It solves problems caused when timeouts cause interruptions in the connection between query clients and the Druid cluster. The user experience with async download APIs is similar to using the synchronous APIs except that it requires three API calls instead of one. See Async SQL downloads.

SQL query cancelling (alpha)

You can use the HTTP DELETE method to cancel a SQL query on either the Router or the Broker. When you cancel a query, Druid handles the cancellation in a best-effort manner. It marks the query canceled immediately and aborts the query execution as soon as possible. However, your query may run for a short time after your cancellation request. See HTTP DELETE.

Query functions to format numbers into human readable format

There are three new functions for Druid SQL and corresponding native functions to format numbers to different styles: human_readable_binary_byte_format, human_readable_decimal_byte_format, and human_readable_decimal_format. See Numeric functions.

Dynamic auto scaling for Kinesis (alpha)

You can now configure autoscaling for the number of ingestion tasks used for the Kinesis indexing service based upon lag time metrics. See Task Autoscaler Properties.

Improved Lookup UI in the Web Console

The Lookup UI in the Web Console includes several changes to make it more user friendly:

Poll period and Summary are added as additional info columns.
The table is ordered on Lookup name by default.
Updated form validation makes Poll period required for extraction lookups to avoid issues with lookups coming out of sync.
Improved error reporting.
The script to extract SQL functions from the docs has been made more robust.

Web Console lookup UI

Pivot highlights

Alert configuration options added

When setting up an alert, you can specify how often the alert criteria are evaluated as well as the time frame of the data evaluated. The menus for configuring these setting now include options for Week and Month, which supplement the existing options for a day, 4 hours, an hour, and a minute.

Report attachments include total/overall value

Currently, if you download data as a CSV or XLS file from the data cube view, the first row in the result set shows the total/overall value. Report attachments, however, do not include this value. You can now configure report attachments to include the total/overall values as a delivery option. This option is available if you have enabled attachments for the report with format CSV, TSV, or XLS (not JSON). See Scheduled reports for details.

Pivot SQL is now beta

Pivot SQL is now a beta feature, which means that it should be suitable for production after testing in a staging environment and in closely-monitored scenarios. It was previously an alpha feature. For more information, see the Pivot SQL documentation.

Imply Cloud highlights

Additional region and instance support

The us-east-2 region has been added to Imply Cloud AWS region support.

Data server instance types now include c5ad.2xlarge, c5ad.4xlarge, and m5ad.8xlarge. See Imply Cloud overview for more information.

AWS EBS gp3 type supported

Imply Cloud now uses Amazon EBS gp3 for root and attached volumes. Previously, only gp2 was supported. Note that the root volume is where the OS, system, and Imply processes are written and executed; this does not affect segment cache storage, which uses NVMe for most instance types.

The change to gp3 takes affect as follows:

Root volumes: As of 2021.09, all root volumes use gp3; previous Imply versions will still use gp2. As you upgrade clusters to 2021.09, the cluster root volumes will be upgraded to gp3.
Attached volumes: All attached volumes use gp3 for all Imply versions. This change takes effect the next time the cluster is updated.

Imply Private on Linux highlights

Customizable Imply user and group

Previously, the Imply Private on Linux installer created a user and group named imply by default. You can now customize this default user for Imply at installation time, allowing you to use another name or group. See Install Imply Private on Linux for more information.

Other changes

Pivot changes

Fix ending an impersonation session does not correctly redirect the impersonating user to the homepage (id: 8328)

Druid changes

Improved robustness for batch ingestion in some use cases. See druid.indexer.task.batchProcessingMode in the Druid Configuration reference (id: 8557)
Add ability to route a SQL query to a specific Broker tier in the query context (id: 8799)
Add the option to use deep storage for intermediate data to support auto scaling (id: 8623)
Adds configurable maxStreamLength for DoublesSketch (id: 10390)
Increments the Parquet version from 1.11.1 to 1.12.0 (id: 2394)
Fix VirtualColumnRegistry to take account of value type when reusing a virtual column (id: 10391)
Support custom coordinator duties to extend Coordinator capabilities (id: 8984)
Fix intermittent error with Variance Aggregator when druid.genericuseDefaultValueForNull=false (id: 7697)
Fix an issue when rollup is enabled that Druid throws a warning message that no metricsSpec is included even if it is (id: 10083)

Changes in 2021.08.1

Druid changes

Fix a regression where using both period and timezone objects in a granularity spec causes batch ingestion to fail (id: 2491)

Changes in 2021.08

Druid highlights

The current version of Imply bundles version 2021.08.0-iap of the Imply distribution of Druid. Before your upgrade to 2021.09, read the Upgrade and downgrade notes.

String aggregation

This release introduces the STRING_AGG string aggregator function. STRING_AGG is similar to the ARRAY_AGG function, except it returns a string instead of an array and it ignores null values. For example:

SELECT STRING_AGG(DISTINCT continent, ', ')
AS "continent list string",
ARRAY_AGG(DISTINCT continent)
AS "continent list array"
FROM "kttm"

returns:

continent list string                                          continent list array
South America, Asia, Europe, Africa, North America, Oceania    ["South America",null,"Asia","Europe","Africa","North America","Oceania"]

Avro extension

The Avro extension supports inputFormat for ingestion configuration, superseding the legacy parseSpec.

Dynamic config provider for Confluent Schema Registry

You can use a dynamic config provider for the Confluent Schema Registry. See the following:

Batch ingestion for out of order data

This release improves upon robustness for handling out of order data in batch ingestion.

Pivot highlights

Data cube defaults (GA)

This release introduces GA support for an improved Pivot UX for setting the default view on a data cube. Users with edit permissions on a data cube can now easily set the default visualization, splits, and more from the data cube view. For more information, see here.

Data cube access control lists for downloading data

In addition to granting view and edit access to a data cube, data cube administrators can now set up download access using a separate list of users and/or roles. Previously, all users with "view" access to a data cube would be able to export data from the cube. The default value for this setting can be defined in the "Advanced" settings tab, similarly to view and edit access.

Support for creating dashboards

Pivot 2.0 now supports creating dashboards. A dashboard must have tiles either from Pivot Classic or from Pivot 2.0. A dashboard cannot contain tiles from both Pivot Classic and Pivot 2.0.

New "Totals" visualization

Pivot 2.0 now supports a visualization for displaying total values.

Platform highlights

Client authentication

You can now configure client authentication from Imply to the metadata store from the Imply Manager console. For details, see Enabling TLS connectivity.

Other changes

Pivot changes

Fix downloads can be limited to 500 rows even when more are available (id: 9043)
Fix creating an alert with dimension values can throw an unhandled error when switching data cubes (id: 8428)
Fix user-selected datasource is not remembered when switching between "From datasource" and "From SQL query" when creating or editing a Pivot SQL data cube (id: 7789)
Fix users should not be able to convert a Plywood data cube to SQL when the "Create SQL data cubes" feature flag is disabled (id: 7778)
Fix users should not be able to save a SQL data cube's settings with an empty query (id: 7775)
Fix time bucketing is not correctly encoded in the report configuration preview and attachments (id: 8546)
Security updates (id: 8511)

Druid changes

Add a new metric for query/segments/count that is not emitted by default (id: 8655)
Add ability for kill task to mark segments as unused (id: 8785)
Add query error logging improvements (id: 8971)
Fix count aggregation input in the data loader in the web console (id: 8956)
Fix hostname validation not skipping with druid.client.https.validateHostnames=false in java 8u275 and later (id: 8902)
Fix Druid supervise scripts give spurious JVM version error when DRUID_JAVA_HOME/JAVA_HOME contains spaces (id: 8323)
Fix DruidSchema.buildDruidTable can block query processing (id: 8057)
Fix for incorrect "IS NOT NULL" filter rewrite in join query (id: 6578)
Fix TopN query correctness bug when querying indexed table selectors (id: 7200)
Fix error messages for task failures in Overlord (id: 8309)
Fix error messages for task failures in Middle Managers and Indexers (id: 8331)
Fix unbounded in-memory usage due to compaction/status API retaining status for datasources that no longer exist (id: 8819)
Fix druid.request.logging.type to allow noop value (id: 8577)
Fix parse exception handling for stream parsers (id: 8968)
Fix ingestion task failure when no input split to process (id: 8969)
Security updates:
- CVE-2021-35515
- CVE-2021-36090

Druid known issues

STRING_AGG does not support single quote ' as the separator.
STRING_TO_ARRAY creates an array based upon single bytes, not based upon the separator.
Array data type incorrectly casts double or float constant values as a bigint in an array.
ARRAY_OFFSET and ARRAY_ORDINAL return wrong data types in certain cases.

Changes in 2021.07

Druid highlights

The current version of Imply bundles version 2021.07.0-iap of the Imply distribution of Druid. Before you upgrade production systems to 2021.07.0-iap, read the Upgrade and downgrade notes.

Pivot highlights

Default data cube views

You can now specify particular dimensions, measures, visualizations, and filters as the default view when users open a data cube. In other words, you can alter how the data cube looks in any way that you like, and then save this altered version as the default view.

Platform highlights

Imply Cloud Auth enhancements

Imply Cloud Auth, the unified authentication and authorization service for Imply Cloud, is now beta status. This release enhances Imply Cloud Auth by adding support for the following features:

LDAP integration, which allows for federated user data synchronization between Imply Cloud and external LDAP systems.
Druid API client authentication using JSON Web Tokens.

In addition, Imply Cloud Auth users can now access the Imply Cloud Auth user console from their profile menus in the Imply Manager. For more information, see Imply Cloud Auth documentation.

Other changes

Pivot changes

Add support for sending a test delivery when configuring a new report (id: 7791)
Add support for creating scheduled reports for users without the downloadData permission (id: 7623)
Fix changeTimezone functionality is broken in iframe embedded Pivot (id: 7422)
Add support for finer grained report delivery times to include hourly and half hourly (id: 3320)
Fix alert occurrence links should be generated against "maxTime" and not alert evaluation time (id: 8341)
Fix alert and report occurrence links should not apply data cube defaults (id: 8340)
Fix non-functional time dimensions should not be created when creating a SQL data cube from a query without a time value (id: 7893)
Fix Geo-fill bubble doesn't account for zooming or panning in Pivot 2.0 (id: 7865)
Fix measure validation shows SQL parse errors on a Plywood data cube (id: 7657)
Add support for filter by measure in Pivot SQL nested aggregations (id: 7565)
Add support for simplified visualization configuration UX in Pivot 2.0 (id: 7524)
Fix cropped labels on vertical axes in Pivot 2.0 (id: 7294)
Improve support for setting up default views for data cubes (id: 7248)

Druid changes

Remove support for IE11, see upgrade note (id: 8108)
Fix Supervisor metadata auto cleanup failing as missing Guice injection (id: 8357)
Fix compaction/status API retains status for datasources that no longer existed causing in-memory used to grow unbounded (id: 8313)
Fix Start and End filters do not work when clicking on dates in Druid console segment view (id: 8277)
Fix cluster deployment fails because of the Druid extension loading error (id: 8125)
Fix PredicateFilteredDimensionSelector returns incorrectly encoded rows (id: 7838)
Improve resiliency of auto-compaction when there is lock contention (id: 6580)
Add capability for S3 ingestion to use AWS AssumeRole to enable cross-account access
Improve efficiency in auto-compaction (id: 8111)
Add query functionality to support certain bitwise functions (id: 8131)
Improve performance for group by queries that operate on time_floor expressions (id: 8418)
Fix an issue where the invalid time zone "America/Los Angeles" returned the default time zone instead of throwing an error (id: 8356)
Address CVE-2021-27568 (id: 8405)
Security fixes

Platform changes

Fix Data StatefulSet specified incorrect serviceName in Helm chart (id: 8278)
Fix ZooKeeper pods can get stuck pending in GKE mode (id: 8017)
Fix Imply Manager HTTP TLS and Manager metadata store TLS cannot be enabled at the same time (id: 7812)

Changes in 2021.06

Druid highlights

The current version of Imply bundles version 2021.06.0-iap of the Imply distribution of Druid. Before you upgrade production systems to 2021.06.0-iap, read the Upgrade and downgrade notes.

Platform highlights

Machine-specific Druid runtime properties on Imply Private for Linux

The agent for Imply Private for Linux now supports per-machine Druid environment variables through the /etc/opt/imply/agent.conf file. Among other abilities, this feature allows you to set the Druid host to a specific IP or hostname if the target machine has more than one network interface.

For more information, see the machine-specific runtime property documentation.

Other changes

Pivot changes

Update geo visualization country labelling to match current international standards
Show detailed alert information in custom webhook
Pivot SQL: Add support for double aggregate measures
Pivot SQL: Add support for rate-based measures using PIVOT_MILLISECONDS_IN_INTERVAL
Add Month as a default option in time filters
Improve error messaging when users attempt to use aggregation functions when creating a custom Plywood dimension
Fix for scheduled report attachments having HTML characters
Fix for not bucketing on integer field gives an error
Fix CSV downloads return an empty file when the report contains a derived time dimension
Fix corrupted Excel report attachment downloads
Fix invalid theme settings can cause the Settings UI to become unavailable
Fix error reports from the browser are being sent to an invalid endpoint
Fix download options in the UI are not correctly masked by the maxDownloadLimit data cube config
Fix grid view fails with a query error when applying a non-default sort on a shown dimension
Fix time bucketing is not correctly encoded in the report configuration
Fix report update endpoints accept invalid delivery days
Fix OIDC "discovery request" error messages are not guaranteed to be logged
Pivot 2.0: Fix incorrect labelling on measure comparisons when delta is 0
Pivot SQL: Fix error with compare functionality on data cubes created from SQL queries
Pivot SQL: Fix query introspection on SQL data cubes does not work for JOIN queries

Druid changes

Add documentation about operational changes for metadata store maintenance with datasource churn
Fix for queries failing when high/low (hilo) query laning strategy is used and druid.query.default.context.priority property is set
Fix for aggregator expressions on real-time index with string columns always producing 0 values
Fix native parallel batch with dynamic partitioning can produce non-contiguous segment partitions. See Downgrades with parallel batch ingestion and dynamic partitioning.
Fix to have roles and permissions cached to lessen the performance impact on queries and other time-sensitive operations in Druid
Fix to adjust HadoopIndexTask temp segment renaming to avoid potential race conditions
Fix SQL IS NULL and IS NOT NULL return wrong results with null values
Fix TaskQueue not respecting useLineageBasedSegmentAllocation in default taskContext
Fix array_prepend(), array_append(), array_concat(), array_slice(), string_to_array() do not support toArray type
Fix Overshadowed Segments still appear in Druid console Segment Timeline
Fix Druid goes into an infinite loop trying to plan certain queries
Fix ARRAY_AGG() returns a Java Object address when aggregation filter produces 0 records
Fix Index Hadoop failing with index.zip is not a valid DFS filename

Platform changes

Add missing user permissions in Imply Private for GKE
Fix new GKE installations fail to start due to auto-scaling issue
Prevent kube-dns from scaling up to the point of starving the manager and Nginx processes in Imply Private for GKE

Clarity changes

Fix "Severity" and "Description" dimension titles are blank in the UI
Fix malformed links in Clarity alert emails

Changes in 2021.05-1

Pivot changes

Fix for application error tracking
Pivot 2.0 now only works with Pivot SQL and not Plywood

Druid changes

Revert fix introduced in 2021.05 to address Hadoop race condition
Fix bug preventing updating, adding, or deleting auto-compaction configurations when one or more auto-compaction configurations created prior to 2021.04 exist
Fix bug preventing changing auto-compaction maxCompactionTaskSlots and compactionTaskSlotRatio settings when one or more auto-compaction configurations created prior to 2021.04 exist

Changes in 2021.05

Druid highlights

Automated metadata cleanup

You can configure automated cleanup to remove records from the metadata store after you delete delete some entities from Apache Druid:

segments records
audit records
supervisor records
rule records
compaction configuration records
datasource records created by supervisors

This feature helps maintain performance when you have a high datasource churn rate, meaning you frequently create and delete many short-lived datasources or other related entities. You can limit the length of time to retain unused metadata records to prevent your metadata store from filling up. See Automated cleanup for metadata records.

Array aggregation

You can use the ARRAY_AGG function to flatten an array of strings at query time. It enables reporting and data-export scenarios that require flattening of an array of strings. See ARRAY_AGG in the list of Aggregation functions.

Expression aggregator

The Expression aggregator is a native Druid aggregator function that lets you use Druid native expressions to perform fold or reduce operations on any number of input columns. This adds significant flexibility to develop functionality unavailable with the default native Druid aggregation functions. It is the underlying technology for aggregate functions such as ARRAY_AGG.

Additional highlights

Faster inner joins and HyperLogLog (HLL) aggregators
Improved ingestion reliability for large files with dynamic partitioning

Pivot highlights

GA support for query cancellation

If the user navigates away from a running query in a data cube or dashboard, Pivot now cancels the queries and saves server processing time. This also reduces query load by stopping a running query when the user has modified the data cube by, for example, changing the filter or the shown dimensions.

Platform highlights

Platform updates in this release focus on enhancements to Enhanced Imply Private on Google Kubernetes Engine. Be aware that upgrading to this release may require system downtime for Enhanced Imply Private on GKE deployments. For details, and additional upgrade considerations, see the following notes.

Node capacity increase for Enhanced Imply Private on GKE

For deployments based on Enhanced Imply Private on Google Kubernetes Engine, the number of supported managed nodes has been increased to 1,000 nodes across all clusters for a given Imply Manager instance (assuming default CIDR settings of GKE).

This change requires a network configuration change that requires a hard update. This means Imply deployments will not be accessible for about 30 minutes during the upgrade. Please plan your upgrade during downtime of your infrastructure or have a planned maintenance outage.

Previously, due to the default address reservation configuration and subtracting IPs required for system use, the maximum amount of nodes that could be started by the Imply Manager across all managed clusters was 61.

The reservation has been changed to limit the capacity to 8 pods per node, and 16 pods per node on system nodes. As a result, support for managed node has been increased to around 1000 nodes across all clusters within the Manager, given the default CIDR settings.

ZooKeeper per cluster for Enhanced Imply Private on GKE

Previously, Enhanced Imply Private on Google Kubernetes Engine used a shared ZooKeeper ensemble for all clusters. There is now a ZooKeeper ensemble per cluster, which helps prevent the possibility of a cluster overwhelming ZooKeeper.

See Configure existing ZooKeeper ensembles.

GCP metadata store autoscaling documented

The Cloud SQL instance used as the metadata store for Enhanced Imply Private on Google Kubernetes Engine supports autoscaling. This feature was previously undocumented. The documentation has been enhanced to describe this behavior.

For details, see the GCP metadata store documentation.

Imply user management with SSO support (alpha)

This release introduces a new user management interface in Imply Cloud with support for external identity providers, as an alpha feature. The new Imply user management provides for a single sign-on (SSO) experience for users, with support for SAML and OIDC identity systems, and a new, central user management console for administrators.

For details, see User management with SSO support.

Other changes

Pivot changes

Fix SQL-based data cubes fail against datasources with a '.' in the name
Fix data sketch measures don't get translated to SQL properly in SQL data cubes
Fix casting string dimensions to integer does not work in SQL data cubes
Fix deleted roles should not be shown in ACLs
Fix users should be able to select and copy value from role id field
Fix alert emails should not render the link into Pivot when linkHostName is not set
Fix alert filter previews do not correctly query against the selected data cube instance
Fix zero values are incorrectly highlighted in red in the table view
Fix Plywood incorrectly casts Strings to Numbers using float instead of double
Remove non-functional "terms delegate" checkbox from "special columns" UI
Add support for milliSecondsInInterval equivalent functionality in Pivot SQL
Improve support for emitting tracking event metadata when an export fails

Clarity changes

Fix Clarity alerts can fire incorrectly when configured against newly created clusters

Druid changes

Fix incorrectly disabled Apply button in the Connect tab for the Kinesis UI data loader
Update Jetty to v9.4.40 to address a bug with Jetty v9.4.39 that breaks some SSL queries
Fix missing virtual columns in a count() query
Added options to automatically clean up metadata database
Fix HadoopIndexTask temp segment renaming race condition resulting in zip.1 files

Platform changes

Fix JVM Configs not getting set in Enhanced Imply Private on GKE
Fix preStop hook failures with invalid state.
Fix changing machine types does not result in a rolling update and could cause an outage in Enhanced Imply Private on GKE
Add support for overriding Auto Repair, Auto Upgrade and max node pool size in Enhanced Imply Private on GKE

Changes in 2021.04-3

Druid changes

Update to Jetty v9.4.40 to address a bug with Jetty v9.4.39 that breaks some SSL queries

Changes in 2021.04-1

Platform change

Fix unable to edit user roles in the Imply Manager for Imply Private

Changes in 2021.04

Druid highlights

Row and column level security (alpha)

You can use the SQL Views feature of Imply Druid to limit user access to rows or columns within a datasource. The Imply View Manager lets you define views that expose a subset of data for access to a type of user. You can assign permissions for specific views to user roles to enable or disable user access. See the following:

Configure segment granularity in automatic compaction (alpha)

You can configure automatic compaction to go from a finer granularity to a coarser granularity in the automatic compaction granularitySpec.

Pivot highlights

Pivot 2.0 (alpha)

Pivot 2.0 is a new visualization engine that simplifies and accelerates building visualizations from data that you query. Pivot 2.0 aims to be highly configurable and highly scalable. For more information, see Pivot 2.0.

Platform highlights

New GCP machine types and regions

The machine types available for Enhanced Imply Private on GKE (beta) have been extended to include the following:

n2-highmem-80, consisting of 80 vCPU, 640 GB RAM, and 3000 GB disk.
n2-highmem-48, consisting of 48 vCPU, 384 GB RAM, and 9000 GB disk

See Google Cloud machine types for more information.

Additional regions include:

asia-east2
asia-northeast1
australia-southeast1
europe-north1
europe-west2
europe-west3
northamerica-northeast1
southamerica-east1
us-west1

New AWS machine type

AWS machine types now include c5ad.8xlarge (32 vCPU, 64 GB RAM, and 1200 GB disk). See Imply Cloud Instance Types for more information.

GKE Public access and Ingress security policy support

The Enhanced Imply Private on GKE (beta) now allows you to secure public access to Enhanced Imply Private by specifying a Cloud Armor Security Policy to restrict access based on IP. See Imply Cloud Instance Types for more information.

Enabling GKE Ingress now provides for public access, that is access to Imply endpoints from outside the GKE network. The public endpoints are shown in the API tab of the Imply Manager for the cluster.

Other changes

Pivot changes

Fix reports with multiple recipients send emails with corrupted attachments
Fix data cubes and dashboards continue to refresh when the browser is not in focus
Fix "all rows" option for report attachments is not correctly saved when creating or editing a report
Fix error message does not show up when entering an invalid dimension or measure formula
Fix invalid inputs can crash the "Advanced" tab in the data cube editor
Fix help text incorrectly indicates sunburst visualization requires 2 dimensions
Fix dimensions of type "Time" need to be cast to VARCHAR when using Pivot SQL
Fix incorrect compare wrapping on compound measure expressions
Fix comparisons cannot be disabled in alwaysShowCompareControl mode
[Pivot SQL] Fix issue with incorrect results when comparisons are used against a measure containing COUNT(*) FILTER (WHERE ...)
Add support for showing role identifier as a read-only field on the role edit form
Security updates

Druid changes

Add the ability to configure a batch ingestion task to wait for newly indexed segments to become available for query on Historical services before completing.
Fix compaction no longer works if there are overlapping intervals
Update the web console to only query required or visible columns
Enable multiple distinct aggregators in same query
Fix an issue with compaction where, if you run compaction twice, the second run yields better compaction
Fix an issue where Kinesis lag continues to increase when there is remaining data for the Kinesis stream
Enforce allow list for JDBC properties by default
Improve bitmap vector offset to report contiguous groups
Vectorize 'auto' long decoding
Enable the ability to request logs through Kafka emitter
Support protobuf serialization for Avatica connections
Fix an issue with APPROX_QUANTILE returning the wrong median value when the query plan generates a single TopN/GroupBy query
Improve performance of queries against SYSTEM.SEGMENT tables
Add expression filter support for vectorized query engines
Update protobuf and schema registry InputFormat
Update Avro and schema registry InputFormat
Fix an issue where joins with subqueries fail to plan in default mode but work as expected in SQL compatible mode
Fix an issue where compaction no longer works if there are overlapping intervals group by processor for string output expressions
Fix an issue where columns needed as inputs to TransformSpec on reindexing are not read automatically by default
Fix an issue where Kinesis resharding causes EOS messages to logged as '"Events thrown away"" in metrics'
Fix CAST being ignored when aggregating on strings after cast
Fix issue in column projection
Security fix for CVE-26919

Platform changes

Add sys.kernel.threads-max parameter to Kubernetes helm chart
Allow access to Pivot and Druid console from the Imply Manager during GKE updates
Add Pivot roles to Imply Manager in GKE
Improve error message in Enhanced GKE when deployment fails due to GKE Update
Update java.io.tmpdir to use /mnt/var/tmp in GKE Enhanced
Add HTTP to HTTPS redirection for GKE Enhanced
Fix Javascript error on login page
Fix Imply Manager does not load in GKE mode
Fix GKE installation fails when external SQL and bucket are used

Changes in 2021.03

Druid highlights

Compaction improvements

Druid compaction now preserves the query granularity for compacted segments by default. For manual compaction, you can specify a query granularity for compacted segments. For example, if you ingest your data with hour query granularity, but after two months, you only need day granularity, you can configure a compaction task to use day granularity for compacted segments older than two months. See Data handling with compaction.

Increased security for HTTP and HDFS input sources

You can now configure Druid to apply additional restrictions on the allowed protocols for URIs with HTTP and HDFS ingestion sources. This feature allows cluster operators more control over how users can load data into the cluster. The two new settings are druid.ingestion.hdfs.allowedProtocols and druid.ingestion.http.allowedProtocols, and default to ["hdfs"] and ["http", "https"] respectively. If ingestion tasks are using other protocols than these defaults, configure this setting to include them when upgrading to prevent tasks from failing. See Druid upgrade notes for more information.

Configurable option for result fetch size

A new configuration property, druid.sql.avatica.minRowsPerFrame, a counterpart to druid.sql.avatica.maxRowsPerFrame, gives cluster operators more control over the size of result ‘fetch’ size. This indirectly reduces the overall number of fetch requests required and can improve JDBC performance with large result sets because they can require many thousands of separate fetches to transfer. See SQL in the configuration reference.

Ingestion support for Confluent Schema Registry

Beta. This feature improves Druid support of Apache Kafka + Apache Avro + Confluent Schema Registry with updated library dependencies, providing support for authorization headers and Schema Registry client configuration. This release also introduces experimental support for Kafka + Protobuf + Schema Registry. Both Avro and Protobuf Schema Registry support is limited to the older "parser" based ingestion specifications instead of the "input format" which has replaced it. See the documentation for Protobuf.

Other changes

Pivot changes

Add support for navigating between the Crosstab view and the regular data cube explorer visualizations--visualizations now persist across Crosstab and regular views
Fix wrong query created when both filter by measure and calculation of change is applied
Add support for configuring Pivot to show a persistent custom header and/or footer, such as "Confidential," to display for all your Pivot users
Fix data exports can have incorrect column headers when transformed measures are present
Fix alert setup form inaccurately renders comparison as "Previous period" after the "current period" has been redefined
Fix alert comparison period is not correctly preserved by link from alert occurrence to data cube view
Add support for adding multiple filters against string dimensions--you can now copy and paste to add many string filters simultaneously
Fix CSV/TSV exports incorrectly show some column values as "undefined"--a new config setting lets you override the default maximum column/row export value
Add support for adjusting the width of string filter menus in the data cube explorer so that users can view longer values
Security updates

Druid changes

Fix the Web Console Services tab to display which Coordinator and Overlord nodes are currently serving as the leader, providing additional insight to cluster operators
Fix Service view actions when grouping in the Web Console
Fix runtime error when IndexedTableJoinMatcher matches long selector to unique string index
Introduce primitive-typed bucketStart, increment methods for granularity to improve performance
Remove namespace property that does not exist from JDBC lookup for the Web Console
Add a parser per InputEntityReaderCsvInputFormat
Support for Confluent schema registry with authentication. Experimental
Fix streaming ingestion failure caused by empty or null rows in the topic
Fix a Java runtime exception error while applying rule DruidQueryRule(AGGREGATE) post upgrade from Imply-3.2.9 TO Imply-4.0.4
Fix maxBytesInMemory for heap overhead of all sinks and hydrants check
Add JDBC handler config for minimum number of rows per frame to improve performance
Add SQL functions to support bitwise operations on LONG and DOUBLE typed values
Support specifying queryGranularity for compaction
Preserve queryGranularity metadata during compaction by default
Add a feature to improve query planning for correlated subqueries. To turn on this experimental feature, set enableJoinLeftTableScanDirect=true
Fix OvershadowableManager inefficiently handles large numbers of segments in a single time chunk (#10892)
Fix historicals can use more disk than configured for the segment cache
Security updates

Platform changes

Support for a larger number of segments per node for Imply Private on Kubernetes, including for Enhanced Imply Private on Google Kubernetes Engine
The Manage Data link is now disabled in Imply Cloud when using direct access Pivot
Improved support for managing cluster configuration declaratively with the ability to specify Imply versions for clusters in Helm charts

Changes in 2021.02

Pivot changes

Fix preserving the columns in a crosstab when accessed through a URL
Fix default values on global filters are not applied when loading a dashboard
Fix Pivot SQL timeout after 60000 ms regardless of configured timeout
Fix using Postgres as a session or state store is broken due to an incompatibility introduced by a library upgrade
Fix TLS 1.0 and TLS 1.1 are not correctly supported for Pivot metadata storage connections
Fix alert conditions with percent delta cannot be inputted correctly
Fix error shown when adding a data cube to favorites
Fix error shown when attempting to change user profile properties
Fix global dashboard filters are not updated correctly
Add support for multiple connections when creating a data cube from Pivot SQL
Fix reset password link does not include serverRoot value
Fix filters are not respected when compare or multi-range time selections are used in Pivot SQL
Fix alert setup form always defines “previous period” as “previous day”
Improve UX when adding permissions to user roles
Fix CSV exports on time-series visualizations have incorrect column headers

Druid changes

Improve error reporting for InputSource errors during sampling
Add vectorized theta sketch aggregator and rework of VectorColumnProcessorFactory
Add Bitwise math function expressions to the Druid native expression system: bitwiseAnd, bitwiseComplement , bitwiseOr, bitwiseShiftLeft, bitwiseShiftRight, bitwiseXor. Add bitwiseConvertDoubleToLongBits and bitwiseConvertLongBitsToDouble to allow use with double typed columns (#10605)
Fix the Web Console to treat null as not defined in AutoForm (#10751)
Improve handling of damaged segments such that the historical unloads damaged segments automatically when lazy on start
Fix to correctly deserialize empty keys in a map
Add the latest AWS Web Identity Token Support for Druid to support IRSA
Fix the Web Console to use new example manifest
Fix an issue where the Broker starts before SQL metadata view is fully initialized even when awaitInitializationOnStart=true
Fix SegmentAnalyzer to properly close column after retrieving it
Improve query execution to retain order of AND, OR filter children
Fix cardinality estimation to calculate numShards independently of partition dimensions
Fix an issue where the Broker doesn't start if druid.extensions.useExtensionClassloaderFirst is set to true and extensions load duplicate jars
Better handling of Kinesis errors and message gap metric for Kinesis ingestion
Fix java.lang.IllegalStateException: 'other' must be a different instance from 'this' error for some vectorized queries using OrFilter

Platform changes

Add Ubuntu 18.04 as a supported platform for Imply Private on Linux
Add support for non-default Kubernetes scheduler via schedulerName Helm configuration
Fix using authentication in Imply Private on Kubernetes breaks custom file download
Fix security section does not appear for a cluster under API in the Cloud Manager UI
Fix placement of security parameters in the Pivot configuration file

Upgrade and downgrade notes

Before you upgrade or downgrade, review the following items to avoid potential version-related issues.

`druid.indexer.task.batchProcessingMode` replaces `batchMemoryMappedIndex`

Applies to versions 2021.09 and later.

The configuration parameter druid.indexer.task.batchProcessingMode replaces batchMemoryMappedIndex, which is now deprecated.

If you were using batchMemoryMappedIndex=true to work around or troubleshoot the memory improvements for batch ingestion in versions 2021.05, 2021.06, 2021.07, or 2021.08, you can set druid.indexer.task.batchProcessingMode=OPEN_SEGMENTS on the Middle Manager to retain the same behavior.

If you had not set batchMemoryMappedIndex or if you had set batchMemoryMappedIndex=false, you can set batchProcessingMode=CLOSED_SEGMENTS to retain the same behavior.

Review the documentation for batchProcessingMode in the Configuration reference to choose the best option for batch ingestion in your environment.

If you later downgrade from 2021.09 or later to a version from 2021.05 to 2021.08, you must revert to batchMemoryMappedIndex.

IE 11 not supported for Druid Console

Applies to 2021.07 and later.

Internet Explorer 11 is no longer supported for access to the Druid Console. Supported browsers include the most recent versions of Chrome, Firefox, Firefox ESR, Edge (Chromium version), Opera, and Safari.

Downgrades with parallel batch ingestion and dynamic partitioning

Applies to downgrades from 2021.06 and later.

The Druid fix for native parallel batch ingestion with dynamic partitioning in 2021.06 and later is compatible with rolling upgrades. However, it may cause native batch jobs with dynamic partitioning to fail during a rollback to version Imply 2021.05.x and previous versions.

Take the following measures to avoid downtime when downgrading:

In the common runtime properties or the Coordinator-Overlord runtime properties, set useLineageBasedSegmentAllocation to "false" for the default taskContext. For example, set the following in common.runtime.properties:

druid.indexer.task.default.context={ "useLineageBasedSegmentAllocation" : false }

Wait for all tasks to finish that don't have useLineageBasedSegmentAllocation explicitly set to "false". The default is "true" for 2021.06.0-iap and later. You can view the task context in the Druid Console.
Use Imply Manager to start the rolling downgrade.

Configure existing ZooKeeper ensembles

Applies to 2021.05 and later.

All new clusters created after updating to 2021.05 use one ZooKeeper per cluster automatically. For existing clusters, you can determine ZooKeeper utilization by its ZooKeeper hosts path under the advanced settings in the Imply Manager. It will most likely be imply-default-zookeeper:2181 but may be imply-<install-name>-zookeeper:2181 if a secondary cluster was created via the installer.

To update ZooKeeper, change the ZooKeeper hosts path to c-<first-13-of-the-cluster-id>-zookeeper:2181, where <first-13-of-the-cluster-id> is the first 13 characters of the cluster ID. You can find the cluster ID in the URL on the Setup page. The relevant portion is everything up to the 2nd dash (-):

After updating the path, click Apply. This will result in brief down time for the cluster.

Druid HTTP and HDFS allowed protocols

Applies to 2021.05 and later.

Because of the enhanced security of this release around HTTP and HDFS input sources, if you use an HDFS input source for anything other than HDFS, for example WebHDFS, or an HTTP input source for anything other than HTTP or HTTPS, the task will fail upon upgrade. Either update your ingestion spec to conform or update the value for the corresponding configuration property: druid.ingestion.hdfs.allowedProtocols or druid.ingestion.http.allowedProtocols.