2023.05

2023.05

  • Imply
  • Ingest
  • Query
  • Visualize
  • Administer
  • Deploy

›Stream ingestion

Ingestion

  • Ingestion overview
  • Supported file formats
  • Data model
  • Data rollup
  • Partitioning
  • Ingestion spec
  • Schema design tips
  • Troubleshooting FAQ

SQL-based ingestion

  • Overview
  • Key concepts
  • Examples
  • API
  • Security
  • Reference
  • Release notes
  • Known issues

Stream ingestion

  • Apache Kafka ingestion
  • Apache Kafka supervisor
  • Apache Kafka operations
  • Amazon Kinesis
  • Tranquility
  • Realtime Process

Batch ingestion

  • Native batch
  • Native batch (simple)
  • Native batch: input sources
  • Firehose (deprecated)
  • Hadoop-based
  • Load Hadoop data via Amazon EMR

Ingestion reference

  • Ingestion
  • Data formats
  • Task reference

Realtime Process

Older versions of Apache Druid supported a standalone 'Realtime' process to query and index 'stream pull' modes of real-time ingestion. These processes would periodically build segments for the data they had collected over some span of time and then set up hand-off to Historical servers.

This processes could be invoked by

org.apache.druid.cli.Main server realtime

This model of stream pull ingestion was deprecated for a number of both operational and architectural reasons, and removed completely in Druid 0.16.0. Operationally, realtime nodes were difficult to configure, deploy, and scale because each node required an unique configuration. The design of the stream pull ingestion system for realtime nodes also suffered from limitations which made it not possible to achieve exactly once ingestion.

The extensions druid-kafka-eight, druid-kafka-eight-simpleConsumer, druid-rabbitmq, and druid-rocketmq were also removed at this time, since they were built to operate on the realtime nodes.

Please consider using the Kafka Indexing Service or Kinesis Indexing Service for stream pull ingestion instead.

Last updated on 11/17/2022
← TranquilityNative batch →
2023.05
Key links
Try ImplyApache Druid siteImply GitHub
Get help
Stack OverflowSupportContact us
Learn more
Apache Druid forumsBlog
Copyright © 2023 Imply Data, Inc