[ACCEPTED]-Real World Use of Zookeeper-distributed

Free Software Projects Powered by ZooKeeper:

• AdroitLogic UltraESB
• Akka
• Eclipse Communication Framework
• Eclipse Gyrex
• GoldenOrb
• Juju
• Katta
• KeptCollections
• Mesos
• Neo4j
• Norbert
• Talend ESB
• redis_failover

Apache Projects Powered by ZooKeeper:

• Apache Accumulo
• Apache BookKeeper
• Apache CXF DOSGi
• Apache Flume
• Apache Hadoop MapReduce
• Apache HBase
• Apache Hedwig
• Apache Kafka
• Apache S4
• Apache Solr

Source: https://cwiki.apache.org/confluence/display/ZOOKEEPER/PoweredBy

0

HBase uses Zookeeper for coordinating activities 8 its "head node" was responsible 7 for prior to the current version. The move 6 to using Zookeeper means the central control 5 is no longer a single point of failure.

Zookeeper 4 is very versatile; here is an example of 3 using it to build a distributed concurrent 2 queue:

http://blog.cloudera.com/blog/2009/05/building-a-distributed-concurrent-queue-with-apache-zookeeper/

You can of course also use it to create 1 resource locks, etc, in a distributed system.

Old question, but since this page comes 3 up first on a google search for zookeeper 2 use cases, I figured it would be best to 1 give an updated listing

1. wikipedia
2. zookeeper wiki
3. real users

The Apache CXF implementation of DOSGi uses zookeeper for its service 13 registration repository. Individual containers 12 have a distributed software (dsw) bundle 11 that listens for all service events and 10 when a service status changes that has a 9 property indicating distribution. The dsw 8 talks to the discovery bundle which, in 7 the reference implementation case, uses 6 zookeeper to store service as ephemeral 5 nodes. Other instances will look for changes 4 to the node structure and register proxies 3 on their local systems. The end result 2 is you can code to plain OSGi and end up 1 with transparent distribution.

Norbert is a good example from a scalable production 18 system. I general, it integrates Netty, Protocol 17 Buffers and Zookeeper into a lightweight 16 framework for running clustered services. Protocol 15 Buffers are used to specify your service 14 API, Netty implements transport-layer abstractions 13 and Zookeeper is essentially a fault-tolerant 12 discovery service.

Every time a service instance 11 is started Norbert registers it as available 10 instance of a particular service type. From 9 implementation perspective, it creates two 8 Zookeeper trees:

• "/ServiceName/members" which lists all known instances of the service
• "/ServiceName/available" which lists currently available instances of the service

The most important property 7 for each node is the url to use to connect 6 to the corresponding service instance. It 5 enables client-side load balancing - a Norbert 4 client finds the list of urls for a given 3 service name and attempt to connect to one 2 of them is some order (e.g. round-robin 1 or random).

There is a good article ZooKeeper - The King of Coordination about ZooKeeper 7 at Elastic Cloud.

At Found, for example, we 6 use ZooKeeper extensively for discovery, resource 5 allocation, leader election and high priority 4 notifications. In this article, we'll 3 introduce you to this King of Coordination 2 and look closely at how we use ZooKeeper 1 at Found

Solr is also working to integrate ZooKeeper. Here you can see they are 2 using for dynamic config, sharding, SPOF 1 elimination (master/slave election), rebalancing, etc.

• Storm is used by a number of companies (Twitter and Groupon being two of the better known) and relies on Zookeeper.
• Kafka is used by Linkedin and relies on Zookeeper.

Storm uses Zookeeper to store all state so that 10 it can recover from an outage in any of 9 its (distributed) component services.

This 8 allows the component services to be stateless 7 and simply download or sync with the Zookeeper 6 servers when configuration data is needed. If 5 you have ever had to recover a production 4 server you will know what a headache this 3 can be!

Kafka queue consumers can use Zookeeper 2 to store information (high water mark) on 1 what has been consumed from the queue.

Zookeeper was used for many things other 3 than configuration. Here is a official list 2 of implement distributed primitives using 1 zookeeper.

https://zookeeper.apache.org/doc/current/recipes.html

In my case we are storing configuration 4 files in zookeeper ensemble for cluster 3 usage . We are using leader -> follower 2 schema . So when one zookeeper down we are 1 switched for another one (replicated mode)

Neo4j uses Zookeeper their High Availability 1 enterprise server! http://docs.neo4j.org/chunked/milestone/ha.html

datomic uses apache zookeeper to manage riak based 8 data storage.

Because Riak supports only 7 eventual consistency at this time, a Datomic 6 system running on Riak also utilizes Apache 5 ZooKeeper, a highly-available coordination 4 service. Datomic uses ZooKeeper for transactor 3 failover coordination, and for the handful 2 of keys per database that need to be updated 1 with CAS. source: http://blog.datomic.com/2012/11/riak-and-couchbase-support.html

Here's some detail on how HBase uses ZooKeeper, including information on how 4 they intend to use it in future. Generally 3 they use it for eliminating SPOF on the 2 region servers via Leader election implemented 1 using ZooKeeper.