ZooKeeper
Apache Druid uses Apache ZooKeeper (ZK) for management of current cluster state.
Minimum ZooKeeper versions
Apache Druid supports ZooKeeper versions 3.5.x and above.
Note: Starting with Apache Druid 0.22.0, support for ZooKeeper 3.4.x has been removed Starting with Apache Druid 31.0.0, support for Zookeeper-based segment loading has been removed.
ZooKeeper Operations
The operations that happen over ZK are
- Coordinator leader election
- Segment "publishing" protocol from Historical
- Overlord leader election
- Overlord and MiddleManager task management
Coordinator Leader Election
We use the Curator LeaderLatch recipe to perform leader election at path
${druid.zk.paths.coordinatorPath}/_COORDINATOR
Segment "publishing" protocol from Historical and Realtime
The announcementsPath
and liveSegmentsPath
are used for this.
All Historical processes publish themselves on the announcementsPath
, specifically, they will create an ephemeral znode at
${druid.zk.paths.announcementsPath}/${druid.host}
Which signifies that they exist. They will also subsequently create a permanent znode at
${druid.zk.paths.liveSegmentsPath}/${druid.host}
And as they load up segments, they will attach ephemeral znodes that look like
${druid.zk.paths.liveSegmentsPath}/${druid.host}/_segment_identifier_
Processes like the Coordinator and Broker can then watch these paths to see which processes are currently serving which segments.