Streaming ingestion

Apache Druid can consume data streams from the following external streaming sources:

• Apache Kafka through the bundled Kafka indexing service extension.
• Amazon Kinesis through the bundled Kinesis indexing service extension.

Each indexing service provides real-time data ingestion with exactly-once stream processing guarantee. To use either of the streaming ingestion methods, you must first load the associated extension on both the Overlord and the Middle Manager. See Loading extensions for more information.

Streaming ingestion is controlled by a continuously running supervisor. The supervisor oversees the state of indexing tasks to coordinate handoffs, manage failures, and ensure that scalability and replication requirements are maintained. You start a supervisor by submitting a JSON specification, often referred to as the supervisor spec, either though the Druid web console or using the Supervisor API.