Imply Enterprise and Hybrid release notes

The following release notes provide information on features, improvements, and bug fixes up to Imply STS release 2022.09.1.

Read all release notes carefully, especially the Upgrade and downgrade notes, before upgrading to 2022.09.1.

If you are upgrading by more than one version, read the intermediate release notes too.

On October 12, 2021, all 3.x versions of Imply entered end-of-support status. See Lifecycle Policy.

See Previous versions for information on older releases.

Imply evaluation

New to Imply? Get started with an Imply Hybrid (formerly Imply Cloud) Free Trial or start a self-hosted trial at Get started with Imply!

With Imply Hybrid, the Imply team manages your clusters in AWS, while you control the infrastructure and own the data. With self-hosted Imply, you can run Imply on *NIX systems in your own environment or cloud provider.

Changes in 2022.09.1

Druid changes

Fix an issue with ingestion failures when using a custom value for druid.extensions.hadoopDependenciesDir or loading extensions for Hadoop task (id: 26396)

Changes in 2022.09

Druid highlights

The 2022.09 release version of Imply bundles version 2022.09.0-iap of the Imply distribution of Apache Druid. Before your upgrade to 2022.09, read the Upgrade and downgrade notes.

Pivot highlights

You can now apply conditional formatting to color a dashboard tile as an indication of data severity—green, amber, or red. This feature applies to "Overall" type visualizations with a single measure. See Managing dashboards and Visualizations for details.

When you set a non-UTC timezone as the default, Pivot downloads with SQL connection type now reflect the correct timestamps.

Pivot now notifies you if your license is within 30 days of expiring. This notification is displayed in the UI to users who have either the AdministerClusters or ManageClusters permissions.

Other changes in 2022.09

Druid changes

Add ARRAY_QUANTILE function (#13061) (id: 25890)
Add support for KLL sketches (#12498) (id: 25477)
Add SQL planning time metric(id: 25075)
Add EXTERNAL resource type to support SQL based ingestion (id: 24843)
Add visibility into SQL query latencies on the router (id: 23781)
Add byte size limit to Druid sampler as an alternative to row count limit (id: 20773)
Add support to hide sensitive properties from the /status/properties endpoint (#12950) (id: 25594)
Add a scheduler to support sending metrics to the Prometheus Pushgateway regularly and continuously (#13034) (id: 25869)
Add interpolation to JsonConfigurator (#13023) (id: 25859)
Add cleaner JSON for various input sources and formats (#13064) (id: 25958)
Auto-reload TLS certificates for Druid endpoints (#12933) (id: 25485)
Bump the version of the Druid Docker image from 0.16.0-incubating to latest (#13058) (id: 25903)
Bump the version of Maven in the Dockerfile (#11994) (id: 25597)
Enable MSQ for Docker by default (#13069) (id: 25907)
Enable enableRewriteJoinToFilter feature flag by default
Export com.sun.management.internal when running Druid under JRE11 and JRE17 (#13068) (id: 25939)
Expose HTTP Response headers from SqlResource (#13052) (id: 25930)
Fix broken filtering in the web console status dialog table (id: 26125)
Fix bugs for expressions for casting of literals and literals representing Long.MIN_VALUE (#13037) (id: 25974)
Fix QueryContext race condition (#13049) (id: 25959)
Fix number of expected functions in web console (#13050) (id: 25898)
Fix web console message in MSQ data loader (#12996) (id: 25874)
Fix bug in filtering when calling the /status/properties endpoint (#13045) (id: 25873)
Fix KafkaEmitter not emitting queryType for a native query (#12915) (id: 25493)
Fix serialization in TaskReportFileWriters (#12938) (id: 25490)
Fix bug with S3 storage connector provider (#12948) (id: 25489)
Fix pagination add error delimiters in the web console (#12969) (id: 25484)
Fix bug with type conversion in variance queries (#12967) (id: 25480)
Fix an issue with search query to include virtual columns in cache key to improve caching performance (id: 24944)
Fix a race condition in the Overlord which can cause a task to get relaunched after completing (id: 24856)
Fix an issue where reading array values for numeric dimensions caused tasks to fail (id: 18307)
Fix an issue with PendingSegments table (id: 18051)
Fix json_keys() with a negative array index to return null instead of java.lang.ArrayIndexOutOfBoundsException (id: 22183)
Fix select json_keys() returns NPE on a literal value (id: 22178)
Fix an issue where json_object(key X value json_value()) returns an object with wrong data type in value (id: 22109)
Fix an issue with a filterless query on an empty inline row set in a timeseries query causes out of memory or an infinite loop (id: 21055)
Fix hasBuiltInFilters for joins (id: 24846)
Fix join to filter conversion to not create filters on right side table columns (id: 25062)
Fix Avro objects leaking from parser (id: 25067)
Fix an issue with "too many redirects" error during Overlord leader changes for RPC client (id: 24676)
Fix accounting of bytesAdded in ReadableByteChunksFrameChannel (#12988) (id: 25472)
Fix bug for fresh Docker installations due to hardcoded conf/druid/single-server configurations in docker-compose.yml (#12963) (id: 25475)
Improve SQL error message for unsupported queries (#12903) (id: 25474)
Improve parse_json behavior (id: 25138)
Improve compatibility with Java 17 (#12987) (id: 25601)
Improve expression error messaging, and add transform name to transform error messages (id: 25650)
Improve detection for arrays containing objects in web console (#13077) (id: 25973)
Improve configuration of Prometheus emitter (#13028) (id: 25872)
Improve nested column serializer (#13051) (id: 25871)
Improve storage efficiency of stringFirst and stringLast (#12879) (id: 25860)
Improve array handling and fix bug with array_slice output type inference (#12914) (id: 25487)
Remove misleading logging on router for JDBC queries (id: 25074)
Remove dependency on jvm-attach (#12989) (id: 25473)
Enable useNativeQueryExplain by default (#12936) (id: 25492)
Update query history dialog to use ace components (id: 24369)
Update run examples when CDPATH environment variable is set where cd command returns current dir (id: 24857)
Update Json ObjectMapper for tojsonstring to include parsejson expressions (id: 25059)
Update postgresql from 4233 to 4241 (id: 25069)
Update ORC to 176 (id: 25157)
Update behavior for parsejson add tryparsejson to be more strict and remove tojson (id: 25134)
Upgrade the web console to use node 16 (#13017)
Upgrade Curator to 5.3.0 (#12939) (id: 25478)

Pivot changes

Add support for conditional formatting in dashboard tiles for Overall visualization type (id: 25698)
Add ability to set minimum auto-refresh rate for data cubes (id: 23352)
Add support for seconds in Pivot date picker (id: 19768)
Add dashboard export button to Pivot Classic (id: 24536)
Add customization of OIDC end session endpoint (id: 23647)
Add support for multi-value dimension filtering in dashboards (id: 24691)
Extend Pivot data querying to apply group-based filters wherever filter tokens are used (id: 23535)
Fix auto-fill dimensions/measures not working for custom SQL (id: 25905)
Fix downloads formatting not working (id: 25847)
Fix unresponsive app when dashboard is edited and vertical bar graph added (id: 25456)
Fix node_sqlite3.node file not verified on MacOS in Imply distribution (id: 25321)
Fix inability to create measures with SQL reserved words in dimension name (id: 25436)
Fix alerts page always loading the entire set of alerts (id: 25289)
Fix new dimensions and measures can be added with query errors (id: 25209)
Fix measure transform header in Pivot 2 downloads (id: 25160)
Fix Pivot 2 download bar to show correct download progress (id: 25159)
Fix horizontal (x-axis) legend missing in vertical bars visualization, in data cube and dashboard tile (id: 25158)
Fix async downloads not properly bucketing when auto bucket option is applied in visualization (id: 25105)
Fix async downloads headers displaying internal names for dimensions (id: 25104)
Fix empty buckets not being filled properly in Pivot 2 (id: 25032)
Fix time ranges not being reloaded when timezone is changed (id: 24760)
Fix reports breaking if data cube dimension is broken (id: 24549)
Fix line chart comparison in Pivot 2 dashboards (id: 24383)
Fix color legend for two splits sunbursts (id: 24158)
Fix vertical scroll positioning in Pivot 2 tiles (id: 24051)
Fix query limits on scheduled reports (id: 23865)
Fix user has to manually change dimension type to set/string when creating multi-value dimension data cube from SQL query source (id: 23757)
Fix creating an alert or report from a Pivot 2 data cube doesn't port the entire data cube configuration (id: 23615)
Fix time bounds in queries don't reflect the provided timezone (id: 22331)
Fix wrong granularity shown in graph legend when bucketing by day (id: 21968)
Fix Pivot 2 facet bubble showing wrong "previous" date with a comparison (id: 21661)
Fix Pivot 2 downloads missing headers if more than one measure is used (id: 21650)
Fix bar chart, etc segments too big for numerical axes (id: 20420)
Fix cell height being improperly calculated when horizontal scrollbar is visible in Pivot 2 (id: 19036)
Fix restored measure and dimension have plywood-style description (id: 19371)
Fix for Imply Private, inactive users receiving emails for reports and alerts (id: 22985)
Harden Pivot Classic to Pivot 2 dashboard conversion tool (id: 24698)
Update Pivot 2 download experience to parity with Pivot Classic (id: 24776)
Update name of "Total" visualization to "Overall" (id: 20418)
Update default records list limit to 200 rows (id: 22940)

Self-Hosted Manager (GKE) changes

Set an environment variable with the node IP to support statsd emitter (id: 25575)
Wait for Druid services to start before signaling that the pod is ready (id: 23359)

Changes in 2022.08

Druid highlights

The 2022.08 release version of Imply bundles version 2022.08.0-iap of the Imply distribution of Apache Druid. Before your upgrade to 2022.08, read the Upgrade and downgrade notes. If you use Firehose ingestion, read about the upcoming removal of firehose support.

Nested columns support doesn't need an extension

You no longer need to load the imply-nested-data extension to use nested columns. The feature is available as part of Druid.

Pivot highlights

If you enable the Enhanced IP support feature and ingest IP data, you can now use the IP and IP prefix dimensions in Pivot to search and filter in your visualizations based on IP addresses and IP prefixes. See the Dimensions and Data cubes documentation for details.

Other changes in 2022.08

Druid changes

Add DictionaryEncodedStringValueIndex implementation toNestedFieldLiteralColumnIndexSupplier (id: 24350)
Add NumericRangeIndex interface and BoundFilter support (#12830) (id: 24222)
Add missing selectors for explicit null columns (id: 24221)
Add defaultOnDiskStorage configuration for Group By (id: 24153)
Add retention for file request logs (id: 24052)
Add detection for time column type with external datasources (id: 23957)
Add metric to measure SQL query latencies on the router (id: 23781)
Add ability to specify S3 Ingestion from non-default endpoints (id: 23642)
Add Druid nested data column type (id: 23580)
Enable lazy initialization of Parquet Extensions Module (id: 22547)
Enable druidprocessingfifo by default (#12571) (id: 24609)
Update Kafka Lookup Extractor to not register consumer group (id: 24620)
Update default setting for maxQueuedBytes to a nonzero value (id: 24347)
Update dictionary encoded column to allow use of a more generic index interface (id: 24138)
Update hardcoded probe delays with helm values (id: 23961)
Update to Jetty 9448v20220622 to resolve CVE (id: 23950)
Update Prometheus version add more labels to PrometheusEmitter (id: 23607)
Improve tick formatting (id: 23597)
Remove Kafka lookup records when a record is tombstoned (id: 24615)
Fix failure to get shardSpec for interval (id: 24742)
Fix JsonNode leaking from JSON flattener (id: 24613)
Fix IN filter null behavior to match selector (id: 24610)
Fix NPE when applying a transform that outputs to time (id: 24570)
Fix nested column SQL operator return type inference (id: 24407)
Fix get task may be null (id: 24395)
Fix bugs with nested column jsonpath parser (id: 24349)
Fix string first/last aggregator comparator (id: 24269)
Fix a bug with nested column missing index type required by search query (id: 24174)
Fix issues with the Avatica JDBC driver (id: 24139)
Fix result merging in SamplingGroupBy queries (id: 24026)
Fix when IP_STRINGIFY on an IPv4 column returns IPv6 data (id: 23991)
Fix ParseException obscures the cause Exception (#12810) (id: 23907)
Fix an issue with connection balancing with Avatica connection balancer (id: 23855)
Fix CONTAINS_STRING using IP_STRINGIFY returns no results (id: 23846)
Fix an issue with GREATEST function incorrectly specifying that it cannot return null (id: 23752)
Fix an NPE in real-time queries with ipAddress or ipPrefix complex types for Kafka streaming (id: 23717)
Fix reindex flow to take order from Druid (id: 23603)
Fix bug in ObjectFlattenerstoMap which caused null values in avrostreamavroocfparquetorc to be converted to {} instead of null in the Druid console sampler UI (id: 23598)
Fix segment timeline bar chart (id: 23596)
Fix a bug where RealtimeMetricsMonitor, TaskRealtimeMetricsMonitor never unregistered on Indexers (id: 22653)
Fix a bug with json_object and Multi-stage query architecture (id: 22175)
Fix JSON SQL operators lose complex type information in SQL result type header (id: 22173)
Fix bug with broken Clarity metrics due to query context authorization (id: 19267)
Fix bug where the Druid web-console does not encode ';' character properly (id: 15196)
Fix bug where Peons reported CPU metrics for the same host (id: 15145)
Fix bug where the Kinesis supervisor shows unhealthy and tasks are not running when one or more partition is empty (id: 2955)

Pivot changes

Add IP address data type and icon to Pivot (id: 23541)
Add IP prefix data type and icon in Pivot (id: 23542)
Add ability to filter IP prefix dimensions by IP address in visualizations (id: 23547)
Add ability to filter IP address dimensions by IP prefix in visualizations (id: 23544)
Add "type ahead" support when filtering by IP address dimension (id: 23545)
Add "type ahead" support when filtering by IP prefix dimension (id: 23546)
Add IP address & IP prefix complex data type to auto-inspection (id: 23990)
Add IP_SEARCH and IP_MATCH functions as Plywood expressions (id: 23543)
Add a legend for sunburst and pie chart visualizations (id: 20041)
Add support to explain SQL queries using embedded virtual tables (id: 23484)
Use lazy loading of data cubes/dashboards from summary objects (id: 23831)
Enable query cache for axis and facet queries (id: 23747)
Improve the experience of adding a widget to a dashboard that has existing widgets (id: 22632)
Improve the ability to reach all GUI controls on a small screen (id: 19638)
Fix tile separator lines on classic dashboard in dark mode (id: 23433)
Fix UX disparity between Pivot 2.0 and Druid after changing dimension type (id: 21772)
Fix user with View access able to download data cube (id: 23500)

Platform changes

MySQL 5.7.14 not starting in GKE (id: 24333)
GKE Enhanced still uses extensions/v1beta1 ingress if ingress is configured (id: 23926)
Support TLS connections to Cloud SQL for Imply Manager on GKE (id: 23886)
Enable GKE user to bypass creation of service IAM roles (id: 20295)

Changes in 2022.07

Druid highlights

The 2022.07 release version of Imply bundles version 2022.07.0-iap of the Imply distribution of Apache Druid. Before your upgrade to 2022.07, read the Upgrade and downgrade notes.

Enhanced IP support (beta) Enhanced IP support allows you to store and query IPv4 and IPv6 addresses and prefixes. You need a valid Imply license to enable the enhanced IP feature. See Enhanced IP support for examples and more information.

Nested columns (beta)

Nested columns for the Imply distribution of Apache Druid is now beta. see Nested columns.

Pivot highlights

The 2022.07 release introduces numerous UX enhancements for Pivot 2.0, including several bug fixes to improve the Pivot 2.0 experience:

Nested columns

You can now view and query your ingested nested data in Pivot. This feature is in beta stage and has known limitations. For more information, see Nested columns.

Multi-value dimensions

You can now query and filter on multi-value dimensions in Pivot. This feature has known limitations around applying filters on multi-value dimensions on dashboard filters. For more details see the Data cubes documentation.

Other Pivot highlights

Pivot now supports pie chart visualizations
Improved scrolling experience on record list visualization
Pivot now sends emails with the X-Auto-Response-Suppress header by default, to suppress replies to messages
Pivot now allows you to configure password invite expiration from the default 7 days

Other changes in 2022.07

Druid changes

Add experimental monitor to report distribution of number of rows in segments (#12730) (id: 23424)
Add metrics into StatsD reporter (#12762) (id: 23295)
Add EIGHTHOUR into possible list of granularities (#12717) (id: 23094)
Add virtual column support to search query (#12720) (id: 23049)
Add support for IN_INTERVAL function (id: 22135)
Add a complex datatype for IPv4 and IPv6 address prefixes (id: 21226)
Add support for IP_PREFIX_MATCH function (id: 21227)
Add support for IP_SEARCH function (id: 21227)
Allow adding Calcite rules from extensions (id: 23126)
Automatically size GroupBy dictionaries (#12763) (id: 23372)
Consolidate the client and indexing TaskStatus classes (#12765) (id: 23373)
Disable automatically killing segments by default (#12693) (id: 22870)
Expose the sampled query engine via SQL (id: 20190)
Fix JSON column bug with indexes matching null values (id: 23274)
Fix bug with random access reads on JSON compressed raw blob column (id: 23102)
Fix filtering on json_value with mixed type column doesn't behave consistently with regular columns (id: 23101)
Fix CVE-2022-33915 log4j vulnerability (id: 22958)
Fix skipTests build flag (#12716) (id: 22923)
Fix bug with json_value vector selector when casting non-numeric to numeric (id: 22915)
Fix OVERLAPS for wide target segments (#12600) (id: 22867)
Fix memory utilization of GET /tasks API in overlord (id: 17962)
Improve join performance by pushdown join filter with right side referencing columns (id: 23127)
Lazy initialization of Orc extensions module (#12663) (id: 22360)
Preserve column order in DruidSchema SegmentMetadataQuery (#12754) (id: 23296)
Surface more information about task status in tests (#12759) (id: 23486)

Pivot changes

Add support to make password invitation configurable from default 7 days (id: 21656)
Add support for X-Auto-Response-Suppress header in outgoing alert and report email notifications (id: 21393)
Add multi-value column support in Pivot 2.0 (id: 22207)
Add support for multi-value dimensional filters to plywood/druid-query-toolkit (id: 22208)
Add support for pie chart visualization in Pivot 2.0 (id: 19279)
Add ability to select entire column in heatmap by clicking axis label (id: 19777)
Add visibility for log messages with bearer token in (id: 22836)
Fix colors not persisted properly in line chart (id: 23356)
Fix Pivot 2.0 measure filter not being migrated properly (id: 23053)
Fix group by strategy incorrectly wraps non-overlap clauses (id: 22814)
Fix poor performance when using compare in Pivot 2.0 (id: 22812)
Fix incorrect auto-bucketing for exclusion filters (id: 22756)
Fix exact end date entered in the field is not getting selected for date picker (id: 22683)
Fix comparison uses wrong measure when multiple measures are shown in Pivot 2 (id: 22568)
Fix generated queries for multidimensional filters to Pivot server (id: 22210)
Fix displaying measure values in treemap visualization (id: 21929)
Fix exclude filter doesn't exclude the selected value range (id: 21424)
Fix record list visualization not displaying results when user scrolls beyond a certain point (id: 19061)
Fix table hover giving different feedback than selection (id: 19776)
Fix experience with date picker in Pivot (id: 19659)
Fix global update when adjusting a single separator line on a Pivot 2.0 dashboard (id: 20631)
Fix pinned numeric dimension in vertical bars visualization not being selectable (id: 21365)
Fix app crashing while switching from 'Line Chart' to 'Totals' (id: 21362)
Fix disabled users visible in report and alerts recipient list (id: 22987)
Improve multi-value dimension filter to only show selected values (id: 22211)
Improve records visualization (id: 18084)

Platform changes

Switch to Java 11 for self-hosted Kubernetes and GKE (id: 21282)

Changes in 2022.06.2

Druid changes

Revert a change to SqlSegmentsMetadataQuery to address a performance issue

Changes in 2022.06.1

Druid changes

Fix a StackOverflowError when running topN queries that group on a string dimensions and include an aggregator that refers to an expression virtual column

Changes in 2022.06

Druid highlights

The 2022.06 release version of Imply bundles version 2022.06.0-iap of the Imply distribution of Apache Druid. Before your upgrade to 2022.06, read the Upgrade and downgrade notes.

Nested columns

The Imply distribution of Druid for the 2022.06 release introduces support for storing and querying nested JSON data in Druid. This feature is in alpha stage and has known limitations. For more information, see Nested columns.

Multi-Stage Query Engine

The Multi-Stage Query Engine (MSQE) is now available as a preview feature for the Imply distribution of Apache Druid. MSQE supports larger queries, querying external data, and ingestion through SQL INSERT and REPLACE queries. Instead of creating JSON-based ingestion specs, you can generate SQL queries to ingest data. For more information, see MSQE.

Platform highlights

Imply Enterprise (on Linux) now supports Ubuntu 20.04.
The supported Kubernetes versions have changed:
- Kubernetes: 1.19 or later
- Rancher: No longer supported
The Imply Helm chart now requires Helm 3.2.0 or later.
The minimum required Imply Manager version for 2022.06 STS is 2022.06.

Other changes in 2022.06

Druid changes

Add sampling rate function for sessionization (id: 21904)
Add is_active to system segments table (id: 21162)
Add direct UTF-8 access for IN filters (id: 21382)
Add query context parameter setProcessingThreadNames to turn off renaming processing threads (id: 20956)
Add metric to track usage of dropExisting flag in ingestion specs (id: 16446)
Add Cache-Control and Content-Security-Policy response headers (id: 21740)
Add default query context to SQL query logs (id: 16225)
Add Zstandard compression library (id: 21480)
Allow update of existing roles using initialRoleMappingFile (id: 20397)
Enable virtual columns to cache their outputs (id: 21546)
Enable vectorized virtual column processing by default to improve query performance (id: 20842)
Fix regression with ipv4_match function when matching prefixes (#12542) (id: 21606)
Fix behavior of null values in lists (id: 21544)
Fix virtual column cycle bug, which would prevent some expressions that did not contain cycles (id: 21492)
Fix a missing call to DruidExpression.visit in the visitAll code (id: 21492)
Fix query context timeout from being altered in ScanQueryEngine (id: 21163)
Fix filters used in links to segments view in web console (id: 21160)
Fix date comparison for metadata store (id: 20457)
Fix estimation of row count by the index_parallel task resulting in too few segments (id: 5506)
Fix a missing property maxColumnsToMerge in automatic compaction (id: 21381)
Improve task stability when there are many (1000+) tasks queued (id: 20876)
Increase GZIP buffer size (id: 21545)
Make explicit null columns backwards compatible (id: 21602)
Make query context changes backwards compatible (id: 21419)
Update LZ4 compression library to version 1.8.0 (id: 21380)
Update default inputSegmentSizeBytes in compaction from Long.MAX_VALUE to ~100 TB (id: 21168)
Upgrade core Apache Kafka dependencies to version 3.2.0 (id: 21164)
Upgrade to Apache DataSketches version 3.2.0 (id: 20813)
Upgrade Apache ORC library to version 1.7.4 (id: 21476)
Improve logging and error messages (ids: 21165, 21807, 21376, 21673, 21223, 20955)
Security fixes

Pivot changes

Fix data cubes display in Dashboard edit options (id: 19490)
Fix error with UNION ALL based SQL statement in data cube creation (id: 17344)
Fix changeTimezone functionality in iframe-embedded Pivot so it no longer resets all filters (id: 9718)

Changes in 2022.05

Druid highlights

The 2022.05 release version of Imply bundles version 2022.05.0-iap of the Imply distribution of Apache Druid. Before your upgrade to 2022.05, read the Upgrade and downgrade notes.

Pivot highlights

The 2022.05 release introduces numerous UX enhancements for Pivot 2.0, including several bug fixes to improve the Pivot 2.0 experience:

Smoother experience for dashboard loading in Pivot 2.0.
Improved handling of “Others” in legends for Pivot 2.0 visualizations.

Platform highlights

You can now assign additional data tiers for your Imply instance. Previously, you could have up to 3 data tiers. Adding tiers depends on how you deployed Imply:
- GKE Enhanced: You can have up to 40 data tiers. Configure them using the Data instance tiers dropdown on the Setup page in Imply Manager.
- Kubernetes deployments: Requires version 9 or later of the agent. You can verify the version in the values file for the Helm chart: images.agent.tag. You can have any number of data tiers. For information about adding tiers, see Adding additional data tiers.
- Linux deployments: Requires version 6 of the agent or later. For information about adding tiers, see Install agents.
There are new options related to TLS connections for agent servers for Imply Enterprise on Linux: IMPLY_AGENT_FQDN and IMPLY_AGENT_IP_ADDRESS. These values are used for certificate generation and are only required if you use custom DNS entries for the agent servers. You can configure them in /etc/opt/imply/agent.conf.
The minimum required Imply Manager version for 2022.05 STS is 2022.05.

Other changes in 2022.05

Druid changes

Add IPAddress Java library as dependency and migrated IPv4 functions to use the new library (#11634) (id: 20734)
Add ability for Coordinator to kill segments whose interval end date is in the future (#10877) (id: 20681)
Add button to web console to get out of restricted mode (#12503) (id: 20513)
Add daily stats to web console (#12329) (id: 20512)
Add metric task/pending/time to measure how long a task is pending in queue (#12492) (id: 20408)
Add authentication call before cleaning up intermediate files in Hadoop ingestion (#12030) (id: 20368)
Add metrics for Java Shenandoah GC (#12369) (id: 20126)
Add metric for groupBy, scan, and time series queries to distinguish vectorized queries (id: 20045)
Add vectorized metric dimension (id: 19941)
Add support for authorizing query context parameters (id: 9404)
Add feature flag enableTimeBoundaryPlanning to control time boundary SQL planning (id: 20254)
Add millisecond precision SQL timestamp literals (#12407) (id: 20244)
Enable ARM builds (#12451) (id: 20195)
Fix miscellaneous table issues in web console (#12489) (id: 20445)
Fix AWS WebIdentityToken so that Druid can connect to S3 on EKS (#12339) (id: 20444)
Fix ingestion when a user specifies a URI containing underscores (id: 19033)
Fix auto-compaction for datasources that have segment replacement tombstones (id: 18689)
Fix delay when a lookup load fails (id: 17959)
Improve build performance of modules (#12486) (id: 20360)
Improve performance of groupBy queries (#12474) (id: 20302)
Improve speed of DimensionRangeShardSpec (#12477) (id: 20242)
Improve LATEST aggregation function for strings through vectorization (id: 20086)
Improve LATEST aggregation function for numeric data types through vectorization (id: 19527)
Improve DimensionRangeShardSpec lookup using binary search (#12417) (id: 19790)
Improve query planning for scan queries (id: 16306)
Optimize SELECT MAX(__time) query for Pivot SQL (id: 19911)
Remove H2 database from dependencies (#12447) (id: 19861)
Security fixes

Pivot changes

Add support for improved filter menu in Pivot 2.0 (id: 19770)
Add support for smoother experience for dashboard loading upon "brush & link" in Pivot 2.0 (id: 19750)
Add support for revised visualization loading indication (id: 18029)
Add support for "others" in sunburst (id: 17981)
Add support for interactivity with "others" in legend for Pivot visualizations (id: 17980)
Add support to enable new visualization added to the dashboard to be shown directly in edit mode (id: 14729)
Add support to select and copy text from a table or dashboard (id: 18830)
Change data cube default query mode to SQL (id: 20458)
Fix "Could not identify first certificate" error for webhooks in Imply Enterprise Hybrid (id: 20988)
Fix bubble showing up on every tile in dashboards (id: 20552)
Fix getting stuck on dimension/measure "info" popover in Pivot 2.0 (id: 20263)
Fix incorrect bucket ranges (P1M) causing 0.00 values in Table visualization (id: 20181)
Fix pill dropdown in show bar to handle overflow properly (id: 19851)
Fix APPROX_COUNT_DISTINCT measures not considered additive by Pivot 2.0 visualizations (id: 18881)
Fix timestamp formatting: wrong precision with time bucketing of second/5-second (PT1S, PT5S) (id: 18305)
Fix "Other" appearing in the legend even though there are no others in the visualization (id: 17979)
Fix error with "union all" based SQL statement in data cube creation (id: 17344)
Improve "Others" in Pivot 2.0 visualizations (id: 19824)

Platform changes

Add support for setting customFiles on the defaultCluster in the Helm chart (id: 20449)
Use startupProbes for data-plane deployments (id: 20172)
Fix an issue where TLS between Imply services didn't work if the hostname and fully qualified domain name are different. This issue affected Imply Enterprise on Linux. (id: 17220)

Changes in 2022.04

Druid highlights

The 2022.04 release version of Imply bundles version 2022.04.0-iap of the Imply distribution of Apache Druid. Before your upgrade to 2022.04, read the Upgrade and downgrade notes.

Pivot highlights

The 2022.04 release introduces numerous UX enhancements for Pivot 2.0, including several bug fixes to improve the Pivot 2.0 experience.

Pivot 2.0 now loads and renders query results in larger increments in Pivot 2.0, leading to a significant improvement in UX on interactive queries.
Support for customizing the Pivot favicon
Support for filtering on data by clicking on dimension values
Support for copying data from visualizations in Pivot 2.0
Introduces a new config option for customizing the property Pivot references to map OIDC groups to roles. This enables role mapping using certain OIDC providers such as Auth0, which does not support a standard 'groups' scope.

Other changes in 2022.04

Druid changes

Add feature flag for Kinesis changes requiring listShards (id: 18994)
All columns in the ingestion spec are queryable (id: 18765)
Include amazon-kinesis-client library in Druid Kinesis extension packaging (id: 18793)
Improve error messages for insert statements consistent with select statements (#12414) (id: 19399)
Improve skipCompact metrics for auto compaction to include segments skipped by skipOffsetFromLatest (id: 19321)
Fix Boolean types not handled properly in transform expressions (id: 19457)
Fix zulu8 setup dockerfile for Hadoop and Hadoop 3 in Hadoop ingestion tutorial (id: 19438)
Fix bug for null columns: includeAllDimensions with flattenSpec fails to save null columns (id: 19007)
Fix messageGap metric (id: 18880)
Fix OOM failures in dimension distribution phase of parallel indexing (id: 18634)
Update inQueryThreshold, convert to query context parameter (id: 18635)
Security fixes

Pivot changes

Add support for customizing Pivot's OIDC groups property name (id: 18400)
Add support for loading in 'larger' increments in Pivot 2.0 (id: 17947)
Add ability to filter the data in the cube by clicking on dimension values (id: 17477)
Add support for copying selected data on visualizations in Pivot 2.0 (id: 16841)
Add support for customizing the Pivot favicon (id: 2056)
Fix visualization option tabs do not have associated tooltips (id: 19115)
Fix APPROX_COUNT_DISTINCT measures not considered additive by Pivot 2.0 visualizations (id: 18881)
Fix "batch dashboard updates" UI breaks when dashboard already has a 2.0 tile (id: 18876)
Fix token mismatch error message (id: 18832)
Fix hierarchical fills not rendering in Pivot 2.0 (id: 18180)
Fix info button being shown twice in Pivot 2.0 data cube (id: 18168)
Fix Pivot 2.0 dashboard tile not filtering on the right dimension value when clicking on the corresponding measure (id: 18130)
Fix time bucketing by month produces invalid query parameters (id: 18056)
Fix color for icons in data panels (id: 18032)
Fix can not drag on overflowing show bar (id: 17983)
Fix cannot see most recent time points when time buckets have 'fine granularity' (id: 17834)
Fix Pivot is not usable without changing URL after unhandled error (id: 17727)
Fix Pivot getting stuck with graph logo in the middle of the screen (id: 17478)
Fix webhooks fail when using a self signed cert (id: 17195)
Fix measure pill control buttons inactive when overflow container expanded (id: 16920)
Fix alert not visible to users that have been added as "admins" but not as "recipients" (id: 9812)

Changes in 2022.03

Druid highlights

The 2022.03 release version of Imply bundles version 2022.03.0-iap of the Imply distribution of Apache Druid. Before your upgrade to 2022.03, read the Upgrade and downgrade notes.

Pivot highlights

Pivot 2.0 enhancements

This release introduces numerous UX enhancements for Pivot 2.0, including several bug fixes to improve the Pivot 2.0 experience.

Improved ‘Filter By Measure’ functionality
- Filter by Measure in dashboard tiles
- Unbounded filtering on either side
- Exclusion via icon.

Pivot Classic improvements

Fixed the regression in Plywood filter token behavior: filter tokens which do not have a $ character being parsed as SQL expressions.

Other changes in 2022.03

Druid changes

Update push value range and set index get operations into BitmapIndex (id: 18085)
Update to disallow filter explosion while converting a filter to CNF (id: 18055)
Fix an issue with FileUtils.copyLarge losing content from input stream on retry (id: 17936)
Reduce lock contention on Druid lookups (id: 17874)
Improve performance for topN queries on expressions and lookups (id: 17873)
Fix an capacity issue with the Coordinator (id: 17766)
Fix Hadoop 3 profile (id: 17601)
Improve Protobuf ingestion speeds (id: 17428)
Add shared access storage token support to Azure extension (id: 17427)
Improve speed for stream ingestion of json records when you use jq expressions for field extraction (id: 17246)
Vectorize Time_Shift function (id: 16014)
Fix an issue where closed shards possibly affect work assignment among Kinesis workers (id: 15962)
Fix an issue with with parallel indexing where subtasks could deadlock and stop all ingestion (id: 6975)

Pivot changes

Fix an issue that makes Pivot unable to auto-fill dimensions for data source with complex data types (id: 18269)
Fix "drag to select" issue with line charts in Firefox (id: 17967)
Update loading indicator to be consistent across all visualizations (id: 17655)
Fix expanded show bar doesn't close when you click outside the bar (id: 17651)
Fix an issue where Pivot hangs with graph logo in middle of screen (id: 17481)
Fix table visualization crashing when the same dimension is present in both rows and columns (id: 17193)
Fix Sparkline maxing out CPU causing app to crash (id: 17038)
Fix issue with dashboards not being scrollable when the cursor is over a scrollable tile (id: 16842)
Add support for improved 'Filter By Measure' capabilities (id: 16747)
Enable filter by measure in the dashboard tiles (id: 15964)
Fix Plywood filter tokens that do not contain "$" character being parsed as SQL (id: 16248)
Fix gray color for comparison in Pivot 2.0 (id: 15216)
Add support to pass custom headers when you configure a custom webhook on an alert (id: 7869)

Platform changes

Update default Self-hosted Manager (GKE, K8s) configuration to rotate logs more often within the container (id: 18110)

Changes in 2022.02

Druid highlights

The 2022.02 release version of Imply bundles version 2022.02.0-iap of the Imply distribution of Apache Druid. Before your upgrade to 2022.02, read the Upgrade and downgrade notes.

Pivot highlights

Pivot 2.0 enhancements

This release introduces numerous post GA UX enhancements for Pivot 2.0, including several bug fixes to improve the Pivot 2.0 experience

Support for Sankey chart visualizations
Support for improved query loading indicator in Pivot 2.0 visualizations
Support for improved error handling in Pivot 2.0 visualizations
Fixed a bug with synchronous downloads in Pivot 2.0

Other changes in 2022.02

Druid changes

Fix an issue with multi-value string array expression (id: 16745)
Fix exception when using useApproximateCountDistinct = false for aggregation query with identity projection (id: 16438)
Remove duplicate Broker ServerInventoryView, improve HttpServerInventoryView logging (id: 16240)
Fix an issue where MySQL transient exception does not exist (id: 16208)
Disable auto compaction config cleanup by default (id: 16167)
Fix stringAny queries against multi-value dimensions where all values are null (id: 16106)
Enable auto-kill unused segments (id: 16066)
Enable metadata store cleanup automatically (id: 16065)
The 'status' API should not return the 'UNDETERMINED' state for unknown queries (id: 15991)
ProvisionerService should be closed on overlord, when it stops being the leader (id: 15948)
Leader overlord may have workExec shutdown and not be able to assign work to Peons / Middle Managers (id: 15911)
Fix an issue with HTTP input source retry (id: 12459)
Fix scenario where users are required to set AWS region settings to successfully ingest data to Google Storage (id: 11933)
Fix segment loading for HTTP mode where load-drop-load cycle causes last load to be ignored and segments not being loaded (id: 10089)

Pivot changes

Fix bubble chart crashing after adding a split (id: 16863)
Fix Pivot 2.0 data cubes do not gracefully handle error states (id: 16456)
Fix synchronous exports fail when using Pivot 2.0 (id: 16439)
Fix facet bubbles have the wrong position on scrollable Dashboards (id: 16189)
Disable default auto-collapse of dimensions for multi-dim pivot tables (id: 16170)
Fix SQL error when column starts with number (id: 16071)
Create new data cube flow still creates a 'Classic' data cube (id: 16061)
Fix incorrect calculation of axes constraints on Pivot 2.0 visualization charts (id: 16055)
Fix sparkline height and width behavior (id: 16018)
Fix hover state for multi-dim treemaps (id: 16017)
Fix display of total on split measures in vertical charts (id: 16001)
Fix scroll bar disappears when duplicating dashboard tile (id: 15971)
Fix cannot rename or duplicate dashboard page when clicking OK (id: 15969)
In Pivot 2.0, Filter by Measure boundary settings affect chart centering (id: 15965)

Platform changes

Imply Enterprise K8s Istio mutual TLS support (id: 13756)

Known issues

Virtual columns and Druid native SEARCH queries

Druid native SEARCH queries using any VirtualColumn do not work correctly with caching. This includes VirtualColumn on top of COMPLEX<json>.

Plywood filter tokens behavior

We have discovered a regression in Plywood filter tokens, where token expressions which do not contain the $ character are being incorrectly interpreted as SQL expressions. For example, some roles may have a filter expression of ‘true’ to allow some users to see all data in a filtered cube. To work around this issue, you can replace these with a statement that always evaluates to true, e.g $count != 0. A fix is currently in progress for inclusion in a future release.

Changes in 2022.01.1

Platform changes

Fix a permissions issue that prevents Imply Manager from creating or updating new clusters in GKE (id: 16211)

Changes in 2022.01

Druid highlights

The 2022.01 release version of Imply bundles version 2022.01.0-iap of the Imply distribution of Apache Druid. Before your upgrade to 2022.01, read the Upgrade and downgrade notes.

Support for ZooKeeper 3.4 is deprecated

Support for ZooKeeper 3.4 is deprecated and will be removed in the 2022.01-LTS release. Imply will be migrating to new ZooKeeper versions over upcoming releases. If you are currently running ZooKeeper 3.4, please consider upgrading your infrastructure.

Impact

Imply Hybrid: Starting with the 2021.10 release, Cloud Manager will run ZooKeeper 3.5. Customers who upgrade to the 2021.10 release (or later) will be upgraded to ZooKeeper 3.5 automatically.
Enhanced Imply Enterprise on GKE and Imply Enterprise on Kubernetes: Customers who use their own ZooKeeper 3.4 (or earlier) must update their version of ZooKeeper to 3.5 (or later) before upgrading to Imply 2022.01-LTS.
Imply Enterprise on Linux and Unmanaged Imply: Customers must update their version of ZooKeeper to 3.5 (or later) before upgrading to Imply 2022.01-LTS.

Pivot highlights

Pivot 2.0 now GA

Pivot 2.0 is now GA. This release introduces numerous changes to enhance Pivot 2.0 and bring it closer to feature parity with Pivot Classic. The changes affect data cubes and dashboards, and include, among others, the following changes:

Additional visualization types for Pivot 2.0 data cubes, including spot matrix, geo shade, and bubble visualizations.
Support for setting a default data cube view.
Support for new and improved filtering by measure.
Support for dimension URLs.
Ability to customize the color palette by cluster.
Support for brush and link filtering for dashboards (that is, the ability to click on visualizations and have that filter applied to all other views that share the filtered dimension).

The following describes the impact for new deployments compared to upgraded deployments:

For new installations, Pivot 2.0 and Pivot SQL are now on by default. An administrator can still disable them using a feature flag. Disabling them results in new data cubes using Pivot Classic by default.
For existing installations that have been upgraded from a previous version, there are no changes to existing data cubes or dashboards. Administrators can enable Pivot 2.0 and Pivot SQL as feature flags. If enabled, users have the ability to choose whether to use Pivot 2.0 or Pivot Classic when creating data cubes.

Pivot SQL now GA

Pivot SQL is now generally available (GA). This release brings in additional capabilities to bring Pivot SQL to GA status:

Support for subset filters with Pivot SQL expressions
Support for filter tokens with Pivot SQL expressions
Deprecation of support for customAggregations in Plywood

Pivot SQL is on by default for new deployments. For the impact on upgraded clusters, see Pivot 2.0 now GA.

Pivot Classic and Plywood data cubes deprecated

Pivot Classic and Plywood data cubes are now deprecated. Support will be terminated in a future release.

Async downloading GA in Imply Enterprise

Asynchronous downloading is now GA for Imply Enterprise (formerly Imply Private) deployments. It remains in beta status in other deployment types.

Other changes in 2022.01

Pivot changes

Prevent Total from showing up on the axis for vertical bar chart (id: 15138)
Fix padding for display text on regex filters (id: 15085)
Make empty visualizations look consistent (id: 15081)
Fix dropdown label to say "measure" instead of "Without" (id: 15080)
Add support to enable check/uncheck upon clicking on text for filter menu (id: 15079)
Apply style adjustments in filter menus (id: 14803)
Adjust min and max heights selection pane for dimensions and measures (id: 14734)
Fix legend on dashboard tile overlaps tile drop shadow and rounded corner (id: 14663)
Fix typo when attempting to show a geo visualization with no geo dimensions (id: 14662)
Fix clicking visualization legend results in a crash when viewing tile in data cube (id: 14611)
Fix large downloads failing in async downloads (id: 14588)
Fix error when dragging a dimension on the show bar while displaying the "Totals" visualization (id: 14480)
Fix new measure modal issue affecting aggregation (id: 14435)
Add code to handle errors when async downloads are disabled in Druid (id: 13378)
Remove "slow" label from download options gated by DownloadLargeData permission when using async downloads (id: 13136)
Fix "Add total row to file" doesn't work for measures other than Count (id: 12985)
Add support for being able to view the status of an async download anywhere in Pivot (id: 12764)
Change verbose-gated log messages to use the "debug" method (id: 12657)
Surface OIDC role name mismatch warning messages only in verbose mode (id: 11771)
Production support for visualization auto-selection and suggestions (id: 9901)
Add icons to the header in resource views; indicate the color theme (id: 9479)
Fix grid view when filter limit is set high enough to return no results (id: 8915)

Druid changes

Fix an issue with incorrect column types during ingestion (id: 15344)
Fix array type strategy write size tracking (id: 15315)
Update EARLIEST_BY/LATEST_BY to make EARLIEST/LATEST function signatures less ambiguous (id: 15259)
Fix PostAggregator interface to make it backward compatible (id: 15199)
Update log4j2 to version 2.17.1 (id: 15149)
Fix delegated smoosh writer and some new facilities for segment write-out medium (id: 15132)
Improve configurable Clarity emitter HTTP client worker pool size (id: 14626)
Add support for histogram filtering in sessionization (id: 13552)
Add support for pruning segments on all dimensions in the multi-dim partitioning scheme (id: 12914)
Fix an issue with DataSketches not recognizing the same IPv6 address in different representations (id: 12901)
Fix an issue with overlapping segment intervals causing automatic compaction to fail (id: 12671)
Distinguish between query metrics for async queries and sync queries (id: 10949)
Add a guardrail that prevents creating too many time chunks by mistake during batch ingestion (id: 10485)
Fix an issue with grouping keys unexpectedly reduced in SQL planner (id: 7780)
Fix an issue where SQL queries with floor(<integer>) return wrong results (id: 6698)
Security fixes

Platform changes

Fix an issue with helm upgrade command failing to upgrade a cluster in a Kubernetes environment. (id: 15236)

Upgrade and downgrade notes

Before you upgrade or downgrade, review the following items to avoid potential version-related issues.

Upgrades from a version earlier than 2022.05

The minimum required Imply Manager version for 2022.05 STS is 2022.05.

Minimum supported version for rolling upgrade

Rolling upgrades have been verified for 2022.01 and later. Clusters older than 2021.10 need to upgrade to 2021.10 first before upgrading to the latest version.

Airline dependency upgrade for Druid extensions

Applies to versions 2022.03 and later.

The Airline dependency for Apache Druid has been upgraded to Airline 2. Airline is a Java library providing an annotation-based framework for parsing command line interfaces. This is a backwards-incompatible change. Any extensions that rely on Druid's CliCommandCreator must also be updated.

Kinesis indexing service requires ListShards permission

Applies to versions 2022.02 and later.

To use the Kinesis indexing service, the IAM user that connects to Kinesis must have the ListShards permission. If the user doesn't have the ListShards permission, the extension throws a "Not authorized" error.

Coordinator automatically cleans up unused segments and stale metadata

Applies to versions 2022.02 and later.

The Coordinator automatically kills unused segments older than 90 days. If you want to change this behavior, set druid.coordinator.kill.on to false. The property killAllDataSources is removed and is no longer respected. If you had killAllDataSources set to false and killDataSourceWhitelist set to empty with the intention of retaining unused segments for all datasources, you must now set druid.coordinator.kill.on=false to keep unused segments older than 90 days.

Druid also removes audit records, supervisor records, rule records, and datasource records older than 90 days. You can configure the retention period by setting the durationToRetain field for each property. See Disable automated metadata cleanup for instructions on disabling the automated cleanup.

Changed behavior for null values and empty arrays in multi-value string dimensions

Applies to versions 2022.02 and later.

The version of Apache Druid released with Imply version 2022.02 fixed a bug regarding how null values and empty arrays were treated in multi-value string dimensions. Null values were treated in multi-value string functions as an array of null, [null], rather than a null value itself. Similarly, empty arrays [] were treated as [null]. Null values are now treated as null and empty arrays are treated as []. This may change the output of some multi-value string functions. If you rely on the past behavior, set druid.expressions.homogenizeNullMultiValueStringArrays=true in the commmon.runtime.properties file.

Upgrade to Imply Manager 2022.01 first

Applies to versions 2022.01 and later.

If you are on an Imply Manager version prior to 2021.09, you must upgrade Imply Manager to version 2022.01 (or later) before upgrading the rest of the Imply platform to version 2022.01 or later. Imply Manager 2022.01 supports Imply distributions that only include the MariaDB connector.

Upgrade to Imply Agent v5 for Python 3.9 support

Applies to versions 2022.01 and later.

If you are using Python 3.9, you must upgrade to the Imply Agent version v5.

`EARLIEST_BY` and `LATEST_BY` functions replace `EARLIEST/LATEST` with custom time parameter

Applies to versions 2022.01 and later.

Usage of EARLIEST/LATEST functions with a custom time column has been removed from the Imply distribution of Apache Druid. This removes the cognitive complexity with the coexistence of the maxBytesPerString parameter as well as timeColumn. You can use the following new functions to set a custom time column: EARLIEST_BY/LATEST_BY. For more information, see Aggregation functions.

Enable async download after Imply upgrade

Applies to versions 2022.01 and later.

If you use an Imply version that's older than 2022.01 and upgrade to version 2022.01 or later, you cannot enable async download during the upgrade. Enable the async download extension in a separate step after your cluster has finished updating.

Deprecation notices

CrossTab view is deprecated

The CrossTab view feature is deprecated. It is replaced by Pivot 2.0, which incorporates the capabilities of CrossTab view.

End of support notices

Firehose ingestion

Support for firehose ingestion will be removed in Imply 2022.10, as well as the upcoming LTS release. Firehose has been deprecated since Druid version 0.17. You must transition your ingestion tasks to use inputSource and ioConfig before upgrading to 2022.10.