Imply 3.0.12 includes the following packages:
- Druid 0.15.0-incubating-iap7
- Pivot 0.10.36
- Tranquility 0.8.3
- ZooKeeper 3.4.13
The Imply download includes a 30 day trial evaluation of Pivot.
Full licenses are included with Imply subscriptions — contact us to learn more!
This Imply release is based on Apache Druid 0.15.0-incubating.
For a list of all changes in Druid 0.15, check out the
Apache Druid 0.15.0 release notes.
Major new functionality includes:
- The Druid console now has a new data loader, which greatly simplifies building data ingestion pipelines by providing a
point-and-click UI and automated previews.
- Druid SQL has graduated out of 'experimental' status and is now a first-class way to query Druid.
- Google Cloud Storage is now officially supported as a core extension, both for deep storage and native batch ingestion.
- The Scan query type now supports time ordering and makes Scan the default method used by Druid SQL. This upgrade
avoids the memory use and performance issues associated with using the legacy Select query. Anyone using Select is
encouraged to migrate to Scan. Druid SQL users will see this migration happen automatically due to a planner update,
whereas native query language users can switch to the Scan query type. Read more
In addition, Imply 3.0 adds additional functionality and fixes that have become available since Druid 0.15 was released.
For a full list of these patches, see "Changes in 3.0.0" → "Druid changes" below.
Major additional functionality available in the Imply 3.0 distribution of Druid 0.15 includes:
- Kafka and Kinesis streaming ingest support in the Druid console's data loader.
- Druid SQL's
sys.servers table includes information about all Druid processes, not just segment-handling processes.
- Multi-value dimension support in Druid SQL.
- Generally improved Druid console UX.
This additional functionality has also been contributed upstream and will appear in a future Apache Druid release.
If you are upgrading from a previous Imply release, please take note of the following sections.
When upgrading from Imply 2.9.x, which is based on Apache Druid 0.14.0, please take note of the items in the
"Updating from 0.14.0 and earlier" section of the Druid release notes at:
You may need to take these items into consideration if they are relevant for your deployment.
- The GCP extension has been promoted to a core extension. When deploying Imply 3.0.12, please ensure that
your extensions directory does not have any older versions of the
druid-google-extensions contrib extension.
druid-parser-route extension is now part of Druid core and should be removed from any
loadList as it has been removed.
- Upstream Druid notes mention a new, revamped core ORC extension.
This new extension was included in Imply 2.9, so if you are updating
from that version, you already have it and nothing major will change. If you are updating from Imply 2.8 or earlier,
please ensure that your extensions directory does not have any older versions of the
- Imply 2.9 included a revamped Kinesis indexing extension. If you are updating
from Imply 2.8 or earlier, please see the release notes for
important configuration changes that may impact you. For full documentation on the new Kinesis extension,
please refer to the Druid docs.
- The minimum supported version of Kafka for the Kafka indexing service is now 0.11.0.
- The coordinator configuration for auto segment merge,
druid.coordinator.merge.on, has been removed.
Please use auto compaction instead.
- The segment metadata file
descriptor.json is not stored in deep storage any more. As a result, the
insert-segment-to-db tool has been removed, since it worked based on
descriptor.json files in deep storage.
Please see https://druid.apache.org/docs/0.15.0-incubating/operations/insert-segment-db.html for details.
If you are updating from Imply 2.8.x or earlier, please refer to release notes for the intervening versions as well.
Starting in Imply 3.0, the application formerly named "Imply UI" is now named "Pivot".
The Pivot binary has changed name from
An alias exists for backwards compatibility, but the process will show up in
top as "pivot".
The "Data" tab of Pivot has been dramatically reduced.
All functionality related to manipulating the data in Druid has moved into the Druid console.
The Druid console in turn can now be accessed from Pivot.
Access to the Druid console is controlled by the
The Druid console is hosted by the Druid Router.
For Druid console integration to work in Pivot, the Pivot connection must point at the Router at
xxx:8888 instead of the Broker at
You can make this change before upgrading to ensure a seamless transition.
Due to server-side API changes, seamless rolling upgrades of Pivot clusters are only supported from 2.x to 3.0 when
coming from 2.9.18 and later, and only if your 2.9.18+ and your 3.x version of Pivot have both been configured to
use the ZooKeeper-based routing protocol for seamless rolling updates (see
ZooKeeper requirement below).
You can use any one of these mechanisms to update Pivot from 2.x to 3.0:
- Update with downtime (shut down all Pivot 2.x servers, upgrade them to 3.0, and restart them).
- Do a rolling update to Pivot 2.9.18 first, configure it to use ZooKeeper, and then do
another rolling update to 3.0.
If you chose to do a rolling update the Druid console will not work well during the rolling update when proxied via Pivot.
Note that Druid rolling updates work as normal
from 2.9.x to 3.0, but see the section "Druid upgrade notes" above for items that you may need to keep in mind.
readOnly - The
readOnly config is no longer supported, and if it is set to true, Pivot will not start.
Equivalent functionality can be achieved by granting only read permissions to your users.
purePivot - The
purePivot config has been removed.
Similar functionality can be achieved by granting only
settingsLocation - If you have a
settingsLocation pointed to a MySQL cluster it will no longer be auto migrated to a
sessionStore - The
sessionStore property used to be mandatory now the
stateStore will be used for storing sessions resulting in simpler configs.
servingMode - The
servingMode property, possible values:
A ZooKeeper connection is now required for two Pivot features when running in clustered mode (
- Supporting seamless rolling updates going forwards, and for seamless rolling updates from Imply 2.9.18.
The ZooKeeper quorum should be provided through the
zookeeperConnection property. Additionally, for inter-Pivot
routing to work, the Pivot servers must be able to contact each other on their HTTP serving ports, and their hostnames
must either be routable or overridden via the
ownHostname property. See the
Pivot configuration page for details.
We recommend using the same ZooKeeper quorum that you are using for Druid.
Starting in Imply 3.0, a new license key format is supported that unlocks additional features. Older keys valid for
Imply 2.x will continue to work with Imply 3.0, but may not unlock all available features. Contact your Imply
representative for details.
Starting with Imply 3.0, Tranquility is in maintenance mode.
We are continuing to bundle and support Tranquility, but we recommend using Druid's native Kafka
or Kinesis ingestion instead.
This Imply release is based on Apache Druid 0.15.0-incubating.
Release notes for Druid 0.15.0-incubating can be found at:
In addition, this version adds the following patches:
- Data loader (sampler component) - Kafka/Kinesis samplers (#7566)
- Supervisor list api with states and health (#7839)
- Add errors and state to stream supervisor status API endpoint (#7428)
- Show all server types in sys.servers table (#7654)
- Optimize single string dimension expression selector (#8014)
- Add array_slice and array_unshift function expr (#7950)
- Expression language array constructor and SQL multi-value string filtering support (#7973)
- Multi-value string column support for expressions (#7588)
- Remove keepSegmentGranularity option for compaction (#7747)
- Add bloom filter fallback aggregator when types are unknown (#7719)
- Discard filter when processing subtotalsSpec (#7827)
- SeekableStreamIndexTaskRunner: Lazy init of runner (#7729)
- Add option to use YARN RM as fallback for JobHistory failure (#7673)
- Add simpler permissions option to BasicAuthorizer GET APIs (#7635)
- Add round support for DS-HLL (#8023)
- Switch links from druid.io to druid.apache.org (#7914)
- SupervisorManager: Add authorization checks to bulk endpoints. (#8044)
- Set DRUID_AUTHORIZATION_CHECKED attribute for router endpoints. (#8026)
- Update to sketches-core-0.13.4 (#7666)
- SQL: Enable by default. (#7808)
- SQL: Allow NULLs in place of optional arguments in many functions. (#7709)
- SQL: Add TIME_CEIL function (#8027)
- SQL: Add TIMESTAMPDIFF function (#7695)
- SQL: Add NVL function (#7965)
- SQL: Allow select-sort-project query shapes (#7769)
- SQL: Docs for multi-value string dimensions (#8011)
- SQL: More support for expression array functions (#7974)
- SQL: Allow LOOKUP function to take advantage of injective lookups (#7655)
- SQL: TIME_EXTRACT should have 2 required operands (#7710)
- SQL: Respect default timezone for TIME_PARSE and TIME_SHIFT (#7704)
- SQL: Upgrade avatica to 1.12.0 (#7644)
- SQL: Fix exception with OR of impossible filters. (#7707)
- Web console: Fix segment table filter (#8034)
- Web console: Fix query input undo (#8033)
- Web console: Improve data loader styling, enforce stricter TS types (#8001)
- Web console: add clipboard modal (#7964)
- Web console: Power up the data loader init step (#7947)
- Web console: Add action column to segments view (#7954)
- Web console: add timed button (#7912)
- Web console: prettier code (#7939)
- Web console: more accurate keyword auto complete (#7934)
- Web console: Data loader respects parse spec columns for data preview (#7922)
- Web console: clean auto generated doc file also (#7911)
- Web console: added Query View metadata pane, more query details (#7905)
- Web console: Add download path to SQL Query (#7898)
- Web console: tail log based on task status (#7888)
- Web console: allow semicolon with comment (#7892)
- Web console: update supervisors table (#7799)
- Web console: Support more JODA time formats (#7857)
- Web console: make the data loader more responsive when sampling data with many columns (#7856)
- Web console: switch to use exInterlop (#7853)
- Web console: Simplify task and supervisor detail displays (#7848)
- Web console: Normalize ingestion spec type (#7847)
- Web console: prevent the time parse view from going into a bad state (#7846)
- Web console: switch the web console to use react pure components (#7845)
- Web console: Add minor CSS to make long value copy even easier (#7829)
- Web console: Truncate and allow easy copy of large values in query table (#7816)
- Web console: Fix spaces and make linting more strict (#7812)
- Web console: prevent unnecessary loading of disabled data sources (#7804)
- Web console: Adding a server view that can display all servers (#7770)
- Web console: add hostname to tasks view (#7758)
- Web console: add actions icon to lookups (#7725)
- Web console: add go to editor button to tasks and supervisors view (#7705)
- Web console: add log tailing to task log view (#7703)
- Web console: add resizable split screen layout to tasks and servers views (#7723)
- Web console: add enable/disable actions for middle manager workers (#7642)
- Web console: add tests to dialogs, compnents and views. Add index files to components and dialogs. add nested file structure (#7669)
- Web console, adding Apache Kafka and AWS Kinesis to the data loader (#7643)
- Web console: fixed issue when grouping tasks by different attributes (#7657)
- Web console: friendly printed arrays for json responses (#7805)
- Web console: Add run keyboard short cut to sql-control (#7787)
- Web console: bump deps (#7788)
- Web console: added scrolling to data loader panes (#7605)
- Fix multi-value string expression transformation (#8019)
- Fix expression virtual column selector for expressions which produce array types (#7958)
- Fix ExpressionVirtualColumn capabilities; fix groupBy's improper uses of StorageAdapter#getColumnCapabilities (#8013)
- Fix FileUtils performance regression due to FilterOutputStream (#8024)
- Fix compilation error in CoordinatorBasicAuthorizerResourceTest (#7667)
- Fix some problems reported by PVS-Studio (#7738)
- Fix spotbugs failure in Expr.java (#7944)
- Fix memory problem (OOM/FGC) when expression is used in metricsSpec (#7716)
- Fix timestamp ceil lower bound bug (#7823)
- Fix repeated expr parsing in ExpressionPostAggregation (#7791)
- Fix Kerberos checking of host URI when reading cookies from cookie store (#7825)
- Fix currSize attribute of historical server type (#7706)
- Added bubble chart
- Added preview of alerts
- Added Druid console integration
- Added Zookeeper based protocol for rolling updates
- Do not resolve fqdn via DNS of not needed (not in clustered mode)
- API endpoint can also be enabled in default user mode
- Fix CSUM in table view
- Web console: add missing dynamic coordinator configs (#8090)
- Web console: update dependencies (#8007)
- Web console: added replicated size (#8043)
- Web console: added bulk supervisor actions (#8028)
- Fix metric ordering in visualizations
- Fix checkbox position in grouped metrics in metrics picker
- Add batter error message when trying to access Druid console on broker
- Pass Pivot version to Druid console requests for better cross version working
- Fix time formatting in XLSX exports
- Fix impersonation link in user menu / user page
- Fix enable user UI not showing up
- fix forking task runner task shutdown to be more graceful (#8085)
- WorkerTaskManager to create disk files atomically and ignore task file corruption (#7917)
- force native order when wrapping ByteBuffer since Druid can have it set (#8055)
- fix issue with CuratorLoadQueuePeon shutting down executors it does not own (#8140)
- remove unnecessary lock in ForegroundCachePopulator leading to a lot of contention (#8116)
- Kinesis: Fix getPartitionIds, should be checking isHasMoreShards. (#7830)
- Fix range rendering
- Fix placement of tile editor form in dashboard view
- Fix dimension drag and drop
- Fix axis rendering bug in bubble chart
- Fix download with PII mask
- Fix error log in alert dispatch
- Fix inability to type in
| into regexp field
- Change selection for Sankey to be more persistennt
- Fix bubble chart axis labels
- Fix button in new alert screen
- Fix inlining html into files
- Fix interpolating two values into the same value in the config
- Fix alert creation in a proxied setup
- Add missing maxBytesInMemory in tuningConfig for auto compaction (#8274)
- Fix DeterminePartitionsJob reducer when total rows < targetPartitionSize * SHARD_COMBINE_THRESHOLD (#8273)
- Fix issues with downloads downloading less data than expected
- Allow specifying download filename
- Fix issue with cert interpolation in configs
- Set direct memory if unable to detect JVM config (#7606)
- Add logging for LZ4Factory instance type (#8341)