Loading data (Kafka)

Getting started

There are two supported mechanisms for ingesting data from Kafka:

  1. Druid Kafka indexing service, a newer Druid extension first available in Imply 1.3.0 / Druid 0.9.1.1. This service offers exactly-once ingestion guarantees as well as the ability to ingest historical data. Additionally, this runs as part of the core Druid services and does not require any additional processes. Setup instructions are available in our Kafka indexing service tutorial and the Druid Kafka indexing service documentation.

  2. Tranquility Kafka, a process which reads messages from Kafka and then pushes these messages into Druid using the Tranquility framework. For more information see the Tranquility documentation.

Both options support partitioning and replication, and can be easily scaled out to achieve the desired ingestion throughput. While we provide support for both ingestion methods, please note that we are still considering the Kafka indexing service to be experimental until it has been thoroughly battle-tested in larger production systems.

Overview

Tutorial

Deploy

Manage Data

Query Data

Visualize

Configure

Special UI Features

Misc