Apache Druid
  • Imply Documentation

›Operations

Getting started

  • Introduction to Apache Druid
  • Quickstart
  • Docker
  • Single server deployment
  • Clustered deployment

Tutorials

  • Loading files natively
  • Load from Apache Kafka
  • Load from Apache Hadoop
  • Querying data
  • Roll-up
  • Configuring data retention
  • Updating existing data
  • Compacting segments
  • Deleting data
  • Writing an ingestion spec
  • Transforming input data
  • Kerberized HDFS deep storage

Design

  • Design
  • Segments
  • Processes and servers
  • Deep storage
  • Metadata storage
  • ZooKeeper

Ingestion

  • Ingestion
  • Data formats
  • Schema design tips
  • Data management
  • Stream ingestion

    • Apache Kafka
    • Amazon Kinesis
    • Tranquility

    Batch ingestion

    • Native batch
    • Hadoop-based
  • Task reference
  • Troubleshooting FAQ

Querying

  • Druid SQL
  • Native queries
  • Query execution
  • Concepts

    • Datasources
    • Joins
    • Lookups
    • Multi-value dimensions
    • Multitenancy
    • Query caching
    • Context parameters

    Native query types

    • Timeseries
    • TopN
    • GroupBy
    • Scan
    • Search
    • TimeBoundary
    • SegmentMetadata
    • DatasourceMetadata

    Native query components

    • Filters
    • Granularities
    • Dimensions
    • Aggregations
    • Post-aggregations
    • Expressions
    • Having filters (groupBy)
    • Sorting and limiting (groupBy)
    • Sorting (topN)
    • String comparators
    • Virtual columns
    • Spatial filters

Configuration

  • Configuration reference
  • Extensions
  • Logging

Operations

  • Web console
  • Getting started with Apache Druid
  • Basic cluster tuning
  • API reference
  • High availability
  • Rolling updates
  • Retaining or automatically dropping data
  • Metrics
  • Alerts
  • Working with different versions of Apache Hadoop
  • HTTP compression
  • TLS support
  • Password providers
  • dump-segment tool
  • reset-cluster tool
  • insert-segment-to-db tool
  • pull-deps tool
  • Misc

    • Legacy Management UIs
    • Deep storage migration
    • Export Metadata Tool
    • Metadata Migration
    • Segment Size Optimization
    • Content for build.sbt

Development

  • Developing on Druid
  • Creating extensions
  • JavaScript functionality
  • Build from source
  • Versioning
  • Experimental features

Misc

  • Papers

Hidden

  • Apache Druid vs Elasticsearch
  • Apache Druid vs. Key/Value Stores (HBase/Cassandra/OpenTSDB)
  • Apache Druid vs Kudu
  • Apache Druid vs Redshift
  • Apache Druid vs Spark
  • Apache Druid vs SQL-on-Hadoop
  • Authentication and Authorization
  • Broker
  • Coordinator Process
  • Historical Process
  • Indexer Process
  • Indexing Service
  • MiddleManager Process
  • Overlord Process
  • Router Process
  • Peons
  • Approximate Histogram aggregators
  • Apache Avro
  • Microsoft Azure
  • Bloom Filter
  • DataSketches extension
  • DataSketches HLL Sketch module
  • DataSketches Quantiles Sketch module
  • DataSketches Theta Sketch module
  • DataSketches Tuple Sketch module
  • Basic Security
  • Kerberos
  • Cached Lookup Module
  • Apache Ranger Security
  • Google Cloud Storage
  • HDFS
  • Apache Kafka Lookups
  • Globally Cached Lookups
  • MySQL Metadata Store
  • ORC Extension
  • Druid pac4j based Security extension
  • Apache Parquet Extension
  • PostgreSQL Metadata Store
  • Protobuf
  • S3-compatible
  • Simple SSLContext Provider Module
  • Stats aggregator
  • Test Stats Aggregators
  • Ambari Metrics Emitter
  • Apache Cassandra
  • Rackspace Cloud Files
  • DistinctCount Aggregator
  • Graphite Emitter
  • InfluxDB Line Protocol Parser
  • InfluxDB Emitter
  • Kafka Emitter
  • Materialized View
  • Moment Sketches for Approximate Quantiles module
  • Moving Average Query
  • OpenTSDB Emitter
  • Druid Redis Cache
  • Microsoft SQLServer
  • StatsD Emitter
  • T-Digest Quantiles Sketch module
  • Thrift
  • Timestamp Min/Max aggregators
  • GCE Extensions
  • Aliyun OSS
  • Cardinality/HyperUnique aggregators
  • Select
  • Realtime Process
Edit

Web console

Druid include a console for managing datasources, segments, tasks, data processes (Historicals and MiddleManagers), and coordinator dynamic configuration. Users can also run SQL and native Druid queries in the console.

The Druid Console is hosted by the Router process.

The following cluster settings must be enabled, as they are by default:

  • the Router's management proxy must be enabled.
  • the Broker processes in the cluster must have Druid SQL enabled.

The Druid console can be accessed at:

http://<ROUTER_IP>:<ROUTER_PORT>

It is important to note that any Druid console user will have, effectively, the same file permissions as the user under which Druid runs. One way these permissions are surfaced is in the file browser dialog. The dialog will show console users the files that the underlying user has permissions to. In general, avoid running Druid as root user. Consider creating a dedicated user account for running Druid.

Below is a description of the high-level features and functionality of the Druid Console

Home

The home view provides a high level overview of the cluster. Each card is clickable and links to the appropriate view. The legacy menu allows you to go to the legacy coordinator and overlord consoles should you need them.

home-view

Data loader

The data loader view allows you to load data by building an ingestion spec with a step-by-step wizard.

data-loader-1

After selecting the location of your data just follow the series for steps that will show you incremental previews of the data as it will be ingested. After filling in the required details on every step you can navigate to the next step by clicking the Next button. You can also freely navigate between the steps from the top navigation.

Navigating with the top navigation will leave the underlying spec unmodified while clicking the Next button will attempt to fill in the subsequent steps with appropriate defaults.

data-loader-2

Datasources

The datasources view shows all the currently enabled datasources. From this view you can see the sizes and availability of the different datasources. You can edit the retention rules, configure automatic compaction, and drop data. Like any view that is powered by a DruidSQL query you can click View SQL query for table from the ... menu to run the underlying SQL query directly.

datasources

You can view and edit retention rules to determine the general availability of a datasource.

retention

Segments

The segment view shows all the segments in the cluster. Each segment can be has a detail view that provides more information. The Segment ID is also conveniently broken down into Datasource, Start, End, Version, and Partition columns for ease of filtering and sorting.

segments

Tasks and supervisors

From this view you can check the status of existing supervisors as well as suspend, resume, and reset them. The tasks table allows you see the currently running and recently completed tasks. To make managing a lot of tasks more accessible, you can group the tasks by their Type, Datasource, or Status to make navigation easier.

supervisors

Click on the magnifying glass for any supervisor to see detailed reports of its progress.

supervisor-status

Click on the magnifying glass for any task to see more detail about it.

tasks-status

Servers

The servers tab lets you see the current status of the nodes making up your cluster. You can group the nodes by type or by tier to get meaningful summary statistics.

servers

Query

The query view lets you issue DruidSQL queries and display the results as a table. The view will attempt to infer your query and let you modify via contextual actions such as adding filters and changing the sort order when possible.

query-sql

The query view can also issue queries in Druid's native query format, which is JSON over HTTP. To send a native Druid query, you must start your query with { and format it as JSON.

query-rune

Lookups

You can create and edit query time lookups via the lookup view.

lookups

← LoggingGetting started with Apache Druid →
  • Home
  • Data loader
  • Datasources
  • Segments
  • Tasks and supervisors
  • Servers
  • Query
  • Lookups

Technology · Use Cases · Powered by Druid · Docs · Community · Download · FAQ

 ·  ·  · 
Copyright © 2019 Apache Software Foundation.
Except where otherwise noted, licensed under CC BY-SA 4.0.
Apache Druid, Druid, and the Druid logo are either registered trademarks or trademarks of The Apache Software Foundation in the United States and other countries.