Release notes

Imply 3.1.5.2 includes the following packages:

Pivot evaluation

The Imply download includes a 30 day trial evaluation of Pivot. Full licenses are included with Imply subscriptions — contact us to learn more!

New Druid features

Vectorized query processing

An experimental 'vectorized' query execution engine is new in 0.16.0, which can provide a speed increase in the range of 1.3–3x for timeseries and group by v2 queries. It operates on the principle of batching operations on rows instead of processing a single row at a time, e.g. iterating bitmaps in batches instead of per row, reading column values in batches, filtering in batches, aggregating values in batches, and so on. This results in significantly fewer method calls, better memory locality, and increased cache efficiency.

This is an experimental feature, but we view it as the path forward for Druid query processing and are excited for feedback. In this release, the following limitations apply, which we plan to address in upcoming releases:

The feature can be enabled by setting vectorize to true in your query context (default is false). This works both for Druid SQL and for native queries. When set to true, vectorization will be used if possible; otherwise, Druid will fall back to its non-vectorized query engine. You can also set it to "force", which will return an error if the query cannot be fully vectorized. This is helpful for confirming that vectorization is indeed being used.

You can control the block size during execution by setting the vectorSize query context parameter (default is 1000).

See also:

GroupBy array-based result rows

groupBy v2 queries now use an array-based representation of result rows, rather than the map-based representation used by prior versions of Druid. This provides faster generation and processing of result sets. Out of the box, this change is backwards-compatible. For Druid SQL, the result format will not change (it will continue to respect the resultFormat setting you have chosen). For native queries, the Broker will translate results to the legacy map-based before sending them back to the client.

If you would like to avoid the overhead of this translation for native queries, and get even faster results, set resultAsArray to true in your query context. The schema of the returned arrays would be as follows, in order:

See also:

Indexer process

The new Indexer process is an alternative to the MiddleManager + Peon task execution system. Instead of forking a separate JVM process per-task, the Indexer runs tasks as separate threads within a single JVM process. The Indexer is designed to be easier to configure and deploy compared to the MiddleManager + Peon system and to better enable resource sharing across tasks.

The advantage of the Indexer is that it allows query processing resources, lookups, cached authentication/authorization information, and much more to be shared between all running indexing task threads, giving each individual task access to a larger pool of resources and far fewer redundant actions done than is possible with the Peon model of execution where each task is isolated in its own process.

Using Indexer does come with one downside: the loss of process isolation provided by Peon processes means that a single task can potentially affect all running indexing tasks on that Indexer. The druid.worker.globalIngestionHeapLimitBytes and druid.worker.numConcurrentMerges configurations (see below) are meant to help minimize this.

You can start using indexing by supplying server indexer as the command-line argument to org.apache.druid.cli.Main when starting the service. To use Indexer in place of a MiddleManager and Peon, you should be able to adapt values from the configuration into the Indexer configuration, lifting druid.indexer.fork.property. configurations directly to the Indexer, and sizing heap and direct memory based on the Peon sizes multiplied by the number of task slots (unlike a MiddleManager, it does not accept the configurations druid.indexer.runner.javaOpts or druid.indexer.runner.javaOptsArray).

The Indexer supports these key configuration properties:

Property Description Default
druid.worker.capacity Maximum number of tasks the Indexer can accept. Number of available processors - 1
druid.worker.globalIngestionHeapLimitBytes Total amount of heap available for ingestion processing. This is applied by automatically setting the maxBytesInMemory property on tasks. 60% of configured JVM heap
druid.worker.numConcurrentMerges Maximum number of segment persist or merge operations that can run concurrently across all tasks. druid.worker.capacity / 2, rounded down
druid.server.http.numThreads Number of threads for HTTP requests. Please see the Server HTTP threads section for more details on how the Indexer uses this configuration. max(10, (Number of cores * 17) / 16 + 2) + 30
druid.processing.numThreads The number of processing threads to have available for parallel processing of segments. Our rule of thumb is num_cores - 1, which means that even under heavy load there will still be one core available to do background tasks like talking with ZooKeeper and pulling down segments. If only one core is available, this property defaults to the value 1. Number of cores - 1 (or 1)
druid.processing.buffer.sizeBytes This specifies a buffer size for the storage of intermediate results. The computation engine in the Indexer processes will use a scratch buffer of this size to do all of their intermediate computations off-heap. Larger values allow for more aggregations in a single pass over the data while smaller values can require more passes depending on the query that is being executed. auto (max 1GB)
druid.processing.numMergeBuffers The number of direct memory buffers available for merging query results. The buffers are sized by druid.processing.buffer.sizeBytes. This property is effectively a concurrency limit for queries that require merging buffers. If you are using any queries that require merge buffers (currently, just groupBy v2) then you should have at least two of these. max(2, druid.processing.numThreads / 4)

See also:

Native batch indexing with shuffle

In 0.16.0, Druid's index_parallel native batch parallel indexing task now supports 'perfect' rollup with the implementation of a 2 stage shuffle process.

Tasks in stage 1 perform a secondary partitioning of rows on top of the standard time based partitioning of segment granularity, creating an intermediary data segment for each partition. Stage 2 tasks are each assigned a set of the partitionings created during stage 1, and will collect and combine the set of intermediary data segments which belong to that partitioning, allowing it to achieve complete rollup when building the final segments. At this time, only hash-based partitioning is supported.

This can be enabled by setting forceGuaranteedRollup to true in the tuningConfig; numShards in partitionsSpec and intervals in granularitySpec must also be set.

The Druid middleManager processes have a new responsibility for these indexing tasks, serving the intermediary partition segments output of stage 1 into the stage 2 tasks, so depending on configuration and cluster size, the middleManager JVM configuration might need to be adjusted to increase heap allocation and HTTP threads. These numbers are expected to scale with cluster size, as all middleManager processes involved in a shuffle will need the ability to communicate with each other, but we do not expect the footprint to be significantly larger than it is currently. Optimistically we suggest trying with your existing configurations, and bumping up heap and HTTP thread count only if issues are encountered.

Native batch partitioning options

The index and index_parallel tasks no longer use the tuning options maxRowsPerSegment, maxTotalRows, numShards, and partitionDimensions to control partitioning. Instead, a new partitionsSpec object has been added to the tuningConfig. There are two types, dynamic offering best-effort rollup and incremental publishing, and hashed offering perfect rollup.

An example of hashed partitioning:

"partitionsSpec" : {
  "type" : "hashed",
  "maxRowsPerSegment" : 5000000,
  "numShards" : null,
  "partitionDimensions" : null
}

An example of dynamic partitioning:

"partitionsSpec" : {
  "type" : "dynamic",
  "maxTotalRows" : 20000000,
  "maxRowsPerSegment" : 5000000
}

Minor compaction

Users of the Kafka indexing service and compaction and who get a trickle of late data, can find a huge improvement in the form of a new concept called 'minor' compaction. Enabled by internal changes to how data segments are versioned, minor compaction is based on the idea of 'segment' based locking at indexing time instead of the current Druid locking behavior (which is now referred to as 'time chunk' locking). Segment locking as you might expect allows only the segments which are being compacted to be locked, while still allowing new 'appending' indexing tasks (like Kafka indexing tasks) to continue to run and create new segments, simultaneously. This is a big deal if you get a lot of late data, because the current behavior results in compaction tasks starving as higher priority realtime tasks hog the locks. This prevention of compaction tasks from optimizing the datasources segment sizes results in reduced overall performance.

To enable segment locking, you will need to set forceTimeChunkLock to false in the task context, or set druid.indexer.tasklock.forceTimeChunkLock=false in the Overlord configuration. However, beware, after enabling this feature, due to required backwards-incompatible changes in segment versioning, there is no rollback path built in, so once you upgrade to 0.16 with minor compaction enabled, you will not be able to roll back. Because of this, we highly recommend confirming that Druid 0.16 is stable in your cluster before enabling this feature.

See also:

SQL IPv4 functions

Druid 0.16.0 also adds specialized SQL operators and native expressions for dealing with IPv4 internet addresses in dotted-decimal string or integer format. The new operators are IPV4_MATCH(address, subnet), IPV4_PARSE(address), and IPV4_STRINGIFY(address), which can match IP addresses to subnets in CIDR notation, translate dotted-decimal string format to integer format, and translate integer format into dotted-decimal string format, respectively.

SQL stddev, variance functions

The druid-stats core extension has been enhanced with SQL support, exposing VAR_POP and VAR_SAMP to compute variance population and sample with the variance aggregator, as well as STDDEV_POP and STDDEV_SAMPto compute standard deviation population and sample using the standard deviation post aggregator. Additionally, VARIANCE and STDDEV functions are added as aliases for VAR_SAMP and STDDEV_SAMP respectively. See SQL documentation and stats extension documentation for more details.

DataSketches extension changes

The druid-datasketches extension, built on top of Apache Datasketches (incubating), has been expanded with 3 new post aggregators, quantilesDoublesSketchToRank which computes an approximation to the rank of a given value that is the fraction of the distribution less than that value, and quantilesDoublesSketchToCDF which computes an approximation to the Cumulative Distribution Function given an array of split points that define the edges of the bins.

Another post aggregation, thetaSketchToString which will print a summary of sketch has been added to assist in debugging. See Datasketches extension documentation to learn more about this and other features.

The HLLSketch aggregator has been improved with a query-time only round option to support rounding values into whole numbers, to give it feature parity with the built-in cardinality and hyperUnique aggregators.

Finally, users of HllSketch should also see a performance improvement due to some changes made which allow Druid to precompute an empty sketch and copy that into the aggregation buffers, greatly decreasing time to initialize the aggregator during query processing.

See also:

Query-time bitmap filter controls

Bitmap indexes are usually a huge performance boost for Druid, but under some scenarios can result in slower query speeds, particularly in cases of computationally expensive filters on very high cardinality dimensions. In Druid 0.16, a new mechanism to provide some manual control over when bitmap indexes are utilized, and when a filter will be done as a row scan are now in place, and available on a per filter, per query basis. Most filters will accept a new property, filterTuning, which might look something like this:

"filterTuning": {
  "useBitmapIndex": true,
  "minCardinalityToUseBitmapIndex": 0,
  "maxCardinalityToUseBitmapIndex": 1000
}

useBitmapIndex if set to false will disallow a filter to utilize bitmap indexes, while minCardinalityToUseBitmapIndex and maxCardinalityToUseBitmapIndex allow using column cardinality to determine if bitmap indexes should be used per segment queried. All properties are optional, and default behavior if filterTuning is not supplied remains unchanged. Note that this feature is not documented in user facing documentation and its API is subject to change in future releases.

See also:

Request log type-based muting

If you would have liked to enable Druid request logging, but use Druid SQL and find them a bit too chatty due to all the metadata queries, you are luck with 0.16 due to a new configuration option that allows selectively muting specific types of queries from request logging. The option, druid.request.logging.mutedQueryTypes, accepts a list of "queryType" strings as defined by Druid's native JSON query API, and defaults to an empty list (so nothing is ignored). For example, the following configuration would mute all request logs for segmentMetadata and timeBoundary queries:

druid.request.logging.mutedQueryTypes=["segmentMetadata", "timeBoundary"]

See also:

New Pivot features

Improved embedding of Pivot visualizations

If you would like to embed and control Pivot visualizations from within your own application, please check out our new tutorial repository, which will walk you through both existing mechanisms to do so, as well as a powerful new way to do this using the HTML5 postMessage API.

Additionally, if you have experimental UI features enabled, you can now easily generate a code snippet to embed a visualization as an <iframe> by clicking on the "Options" menu.

Improved role-based access control

Managing access control for large numbers of users and roles has been made smoother in Imply 3.1. Previously, the ability to share content such as data cubes, dashboards, and alert configurations with other users was controlled by the SeeOtherUsers permission, which exposed all users in the system when sharing content.

Roles now have two new flags:

Each of these can be configured to be hidden, visible only to other members of the role, or public.

Improved user management

It is now possible to view locked out or disabled users in the user admin area, and to view when a user last logged in.

Improved contextual navigation and search

We've redesigned the UI slightly to highlight the actions you can take when on a module home screen, e.g. searching for or creating a new data cube or dashboard.

Miscellaneous improvements

Upgrading from previous releases

If you are upgrading from a previous Imply release, please take note of the following sections.

Druid upgrade notes

When upgrading from Imply 3.0, which is based on Apache Druid 0.15.0, please take note of the following items:

When upgrading from Imply 2.9, which is based on Apache Druid 0.14.0, please additionally take note of the items in the "Updating from previous releases" section of the Imply 3.0 release notes. You may need to take these items into consideration if they are relevant for your deployment.

Changes in 3.1.1

Pivot changes

Changes in 3.1.2

Pivot changes

Changes in 3.1.3

Pivot changes

Changes in 3.1.4

Pivot changes

Druid changes

Changes in 3.1.5

Pivot changes

Druid changes

Changes in 3.1.5.1

Pivot changes

Changes in 3.1.5.2

Pivot changes

Overview

Tutorial

Deploy

Manage Data

Query Data

Visualize

Configure

Special UI Features

Imply Manager

Misc