1. zookeeper event listener
1.1. watcher concept:
-
zookeeper provides data publish / subscribe function. Multiple subscribers can listen to a specific subject object at the same time. When the subject object's own state changes, such as the node content changes, the list of child nodes under the node changes, etc., all subscribers will be notified in real time and actively.
-
zookeeper uses Watcher mechanism to realize the function of data publish and subscribe. This mechanism will notify the client asynchronously when the subscribed object changes, so the client does not have to poll for blocking after the Watcher registers, thus reducing the pressure on the client
-
The watcher mechanism event is similar to the observer pattern, which can also be seen as a way to implement the observer pattern in a distributed scenario.
1.1.1. watcher architecture:
The watcher implementation consists of three parts
-
zookeeper server
-
zookeeper client
-
ZKWatchManager object for the client
The client first registers the Watcher with the server and saves the Watcher object to the client's Watch manager. When the data status monitored by Zookeeper server changes, the server will actively notify the client, and then the Watch manager of the client will trigger the relevant Watcher to call back the corresponding processing logic, so as to complete the overall data publish / subscribe process
1.1.2. watcher characteristics:
| characteristic | explain |
|---|---|
| disposable | The watcher is one-time. Once triggered, it will be removed. When it is used again, it needs to be re registered |
| Client sequence callback | The watcher callback is executed sequentially and serially. Only after the callback can the client see the latest data status. One watcher callback logic should not be too many, so as not to affect the execution of other watchers |
| Lightweight | WatchEvent is the smallest communication unit, which only contains notification status, event type and node path, and does not tell the specific content of data nodes before and after changes |
| Timeliness |
The Watcher will be invalid only when the current session is completely invalid. If the session is successfully reconnected within the validity period of the session, the watcher still exists and can still receive notifications; //This 5000 milliseconds is the session validity period ZooKeeper zooKeeper = new ZooKeeper(IP, 5000, new ZkConnectionWatcher()); |
1.1.3. watcher interface design:
Watcher is an interface. Any class that implements the watcher interface is a new watcher. There are two enumeration classes in Watcher: KeeperState and EventType
1.2. Watcher event type:
EventType is the corresponding notification type (node add, delete, modify and query) when the data node changes. When the EventType changes, the keeperstate is always in the SyncConnected notification state; when the keeperstate changes, the EventType is always None.
| Enumeration properties | explain |
|---|---|
| None | nothing |
| NodeCreated | When the data node monitored by Watcher is created |
| NodeDeleted | When the data node monitored by Watcher is deleted |
| NodeDataChanged | When the content of the data node monitored by Watcher changes (whether the data changes or not) |
| NodeChildrenChanged | When the list of child nodes of data nodes monitored by Watcher changes |
1.3 Watcher notification status (KeeperState):
KeeperState is the corresponding notification type when the connection state of the client and the server changes.
| Enumeration properties | explain |
|---|---|
| SyncConnected | When the client is normally connected to the server |
| Disconnected | When the client is disconnected from the server |
| Expired | When session fails |
| AuthFailed | When authentication fails |
1.4 capture corresponding events:
Take the node-x node as an example to illustrate the relationship between the calling registration method and the available listening events:
| Registration method | created | childrenChanged | Changed | Deleted |
|---|---|---|---|---|
| zk.exists("/node-x",watcher) | Can be monitored | Can be monitored | Can be monitored | |
| zk.getData("/node-x",watcher) | Can be monitored | Can be monitored | ||
| zk.getChildren("/node-x",watcher) | Can be monitored | Can be monitored |
1.5. Method of registering watcher
Event type: None
public class ZkConnectionWatcher implements Watcher {
//zookeeper link address
private static final String IP = "192.168.1.7:2181";
//Counter object, blocking main thread, waiting for other threads
private static CountDownLatch countDownLatch = new CountDownLatch(1);
@Override
//When there is an event message on the server, it is passed to the client through watchedEvent
public void process(WatchedEvent watchedEvent) {
Event.KeeperState state = watchedEvent.getState();
if(state == Event.KeeperState.SyncConnected){
// normal
System.out.println("Normal connection");
}else if (state == Event.KeeperState.Disconnected){
// Can use Windows to disconnect the virtual machine network card to simulate
// When session disconnection occurs, disconnection does not mean reconnection is not possible. Reconnection can resume normal within the session timeout period
System.out.println("Disconnect");
}else if (state == Event.KeeperState.Expired){
// If the session times out, you need to recreate the zookeeper link
System.out.println("session time out");
}else if (state == Event.KeeperState.AuthFailed){
// Insufficient permission will appear during operation
System.out.println("privilege grant failed");
}
countDownLatch.countDown();
}
public static void main(String[] args) throws Exception {
// 5000 is the session timeout
ZooKeeper zooKeeper = new ZooKeeper(IP, 5000, new ZkConnectionWatcher());
countDownLatch.await();
// Simulation authorization failed
zooKeeper.addAuthInfo("digest1","itcast1:123451".getBytes());
byte[] data = zooKeeper.getData("/hadoop", false, null);
System.out.println(new String(data));
TimeUnit.SECONDS.sleep(50);
}
}1.6. watcher mechanism exists
//Whether to use the listener of the connection object exists(String path, boolean b) //Using a custom listener exists(String path, Watcher w) //The status returned after the operation on the node. NodeCreated: **node**establish NodeDeleted: **node**delete NodeDataChanged: **node**content
public class ZkExistsWathcer {
private static ZooKeeper zookeeper ;
private static CountDownLatch cou ;
@Before
public void before()throws Exception{
//enumerator object
cou=new CountDownLatch(1);
zookeeper = new ZooKeeper("192.168.1.7", 5000, new Watcher() {
@Override
public void process(WatchedEvent event) {
if(event.getState()==Event.KeeperState.SyncConnected){
System.out.println("Link created successfully!");
//Release blocking
cou.countDown();
}
System.out.println("Node address-->"+event.getPath());
System.out.println("Node change type-->"+event.getType());
}
});
//Block the main thread and wait for the link object to be created successfully
cou.await();
}
@After
public void after() throws Exception {
zookeeper.close();
}
/**
*Use the monitor of the connection object. When new ZooKeeper, the third parameter new Watcher() internal method
* Test: start the service, and then add, delete and modify the wathcer node on the server. It can be monitored here.
* Note: each time you listen, you can only listen to one operation. The next operation must be restarted, and the next operation can be monitored.
*/
@Test
public void test0() throws Exception {
zookeeper.exists("/wathcer",true);
Thread.sleep(50000);
}
/**
* Custom listener.
* Test: same as above
* Note: ditto
*/
@Test
public void test1() throws Exception {
zookeeper.exists("/wathcer1", new Watcher() {
@Override
public void process(WatchedEvent event) {
if(event.getState()==Event.KeeperState.SyncConnected){
System.out.println("Link created successfully!");
//Release blocking
cou.countDown();
}
System.out.println("Node address-->"+event.getPath());
System.out.println("Node change type-->"+event.getType());
}
});
Thread.sleep(50000);
}
/**
* wathcer The mechanism itself is one-off. It can be called after notification zookeeper.exists () method. Reuse back and forth
*/
@Test
public void test2() throws Exception {
Watcher watcher = new Watcher() {
@Override
public void process(WatchedEvent event) {
try {
System.out.println("Node address-->" + event.getPath());
System.out.println("Node change type-->" + event.getType());
//After the notification is completed, call zookeeper.exists Method, listen to the node again
Stat exists = zookeeper.exists("/wathcer1", this);
} catch (Exception e) {
e.printStackTrace();
}
}
};
zookeeper.exists("/wathcer1", watcher);
Thread.sleep(50000);
}
/**
* A node has multiple listener objects. As long as the node is operated, multiple listeners will execute.
*/
@Test
public void test3() throws Exception {
zookeeper.exists("/wathcer1", new Watcher() {
@Override
public void process(WatchedEvent event) {
System.out.println("Monitor 1");
}
});
zookeeper.exists("/wathcer1", new Watcher() {
@Override
public void process(WatchedEvent event) {
System.out.println("Monitor 2");
}
});
Thread.sleep(50000);
}
}1.7. watcher mechanism getData
//Whether to use the listener of the connection object getData(String path, boolean b, Stat stat) //Custom listener getData(String path, Watcher w, Stat stat) //The state value of the operation arc of the node NodeDeleted: **node**delete NodeDataChange: **node**Content changes
Test: copy the existing code above. Just put zooKeeper.exists (path,boolean) to zooKeeper.getData(path,boolean,stat). This test will not be repeated too much.
1.8. watcher mechanism getChildren
//Whether to use the listener of the connection object getChildren(String path, boolean b) //Using a custom listener getChildren(String path, Watcher w) //The status returned after the operation on the child node of the node. NodeChildrenChanged: Child node changes NodeDeleted: Child node deletion
Test: copy the existing code above. Just put zooKeeper.exists (path,boolean) to zooKeeper.getChildren(path,boolean). This test will not be repeated too much.
2. Configuration center case
2.1 idea of configuration center
If the user name and password of the database change, you need to reload the remote storage, which is troublesome. It can be easily completed through Zookeeper. When the database changes, the cache synchronization will be automatically completed. Using the event monitoring mechanism, you can make a simple configuration center
Design ideas
-
Connect to zookeeper server
-
Read configuration information in zookeeper, register watcher listener and store local variables
-
When the configuration information in zookeeper changes, the data change event is captured by the callback method of watcher
-
Get configuration information again
2.2 initialization information
create /config "Mysql config" create /config/url "192.168.1.7:3306" create /config/username "root" create /config/password "123456"
2.3 write case code of configuration center
/**
* Configuration center case
*/
public class ZkWatcherConfig implements Watcher {
//zookeeper link address
private static final String IP = "192.168.1.7:2181";
//Counter object, blocking main thread, waiting for other threads
private CountDownLatch countDownLatch = new CountDownLatch(1);
//zookeeper object
private static ZooKeeper zookeeper ;
//Database connection (provide get/set method)
private String url;
private String username;
private String password;
//Constructor. When the object is created, it initializes the data link information
public ZkWatcherConfig(){
initValue();
}
@Override
public void process(WatchedEvent event) {
try {
Event.KeeperState state = event.getState();
if(event.getType()== Event.EventType.None) {
if (state == Event.KeeperState.SyncConnected) {
System.out.println("Normal connection");
countDownLatch.countDown();
} else if (state == Event.KeeperState.Disconnected) {
System.out.println("Disconnect");
} else if (state == Event.KeeperState.Expired) {
System.out.println("session time out");
} else if (state == Event.KeeperState.AuthFailed) {
System.out.println("privilege grant failed");
}
}else if(event.getType()== Event.EventType.NodeDataChanged){
//If the configuration information changes, reload the initMethod method
initValue();
}else if(event.getType()==Event.EventType.NodeDeleted){
System.out.println("If who deleted the configuration information, please find out the sacrifice to heaven!!!");
}
}catch (Exception e){
e.printStackTrace();
}
}
public void initValue() {
try{
//Connect zookeeper
//It is necessary to judge whether zookeeper is connected. If it is connected, there is no need to connect.
// If you have a slow connection. Then the following test fails because it takes a long time to connect each time. Before reconnecting, the test cycle begins again.
if(zookeeper==null)
zookeeper=new ZooKeeper(IP,5000,this);
//Blocking thread waiting for connection creation to succeed
countDownLatch.await();
this.url=new String(zookeeper.getData("/config/url", true, null));
this.username=new String(zookeeper.getData("/config/username", true, null));
this.password=new String(zookeeper.getData("/config/password", true, null));
System.out.println("Configuration information, load once");
}catch (Exception e){
e.printStackTrace();
}
}
public static void main(String[] args) throws InterruptedException {
ZkWatcherConfig zkConfigWatcher = new ZkWatcherConfig();
for(int i=1;i<=10;i++){
Thread.sleep(2000);
System.out.println("current url: "+zkConfigWatcher.getUrl());
System.out.println("current user: "+zkConfigWatcher.getUsername());
System.out.println("current password: "+zkConfigWatcher.getPassword());
System.out.println("--------------");
}
}
public String getUrl() {
return url;
}
public void setUrl(String url) {
this.url = url;
}
public String getUsername() {
return username;
}
public void setUsername(String username) {
this.username = username;
}
public String getPassword() {
return password;
}
public void setPassword(String password) {
this.password = password;
}
}3. Globally unique ID
Question:
In the past, in the system of single database and single phenotype, the auto of database field can be used_ The increment property automatically generates unique IDs for each record. But after the database and table are divided, it is impossible to rely on the database auto_ The increment property uniquely identifies a record. At this point, we can use zookeeper to generate a globally unique ID in a distributed environment
public class ZKUniqueIDWatcher implements Watcher {
private CountDownLatch countDownLatch=new CountDownLatch(1);
//zookeeper information
private static final String IP="192.168.1.7:2181";
private static ZooKeeper zooKeeper;
//Unique ID
private String uniqueId="/id";
public ZKUniqueIDWatcher(){
try{
//Connect zookeeper
if(zooKeeper==null)
zooKeeper=new ZooKeeper(IP,5000,this);
//Blocking thread waiting for connection creation to succeed
countDownLatch.await();
}catch (Exception e){
e.printStackTrace();
}
}
@Override
public void process(WatchedEvent event) {
try{
if(event.getType()== Event.EventType.None) {
if (event.getState() == Event.KeeperState.SyncConnected) {
System.out.println("Connection created successfully");
countDownLatch.countDown();
} else if (event.getState() == Event.KeeperState.Disconnected)
System.out.println("Disconnect");
else if (event.getState() == Event.KeeperState.Expired) {
System.out.println("session time out");
zooKeeper = new ZooKeeper(IP, 5000, new ZKUniqueIDWatcher());
} else if (event.getState() == Event.KeeperState.AuthFailed) {
System.out.println("Authentication failed");
}
}
}catch (Exception e){
e.printStackTrace();
}
}
private String getUniqueId(){
try{
//Create a temporary ordered node
/**
* arg1:Nodes created
* agr2:Data information of node (only unique ID is generated here, no data is required)
* arg3:Create a permission list for a node
* arg4:CreateMode.EPHEMERAL_SEQUENTIAL Generate temporary ordered ID
*/
return zooKeeper.create(uniqueId, new byte[0], ZooDefs.Ids.OPEN_ACL_UNSAFE, CreateMode.EPHEMERAL_SEQUENTIAL);
}catch (Exception e){
e.printStackTrace();
}
return null;
}
public static void main(String[] args) throws InterruptedException {
ZKUniqueIDWatcher zkUniqueIDWatcher= new ZKUniqueIDWatcher();
for(int i=1;i<=10;i++){
System.out.println(zkUniqueIDWatcher.getUniqueId().substring(3));
}
}
}4. zookeeper distributed lock
Design idea:
-
Each client creates a temporary orderly node under / Locks / Locks/Lock_ After creation, there will be nodes corresponding to each client under / Locks, such as / Locks/Lock_000000001
-
The client obtains / Locks sub nodes and sorts them to determine whether the top one is its own. If its own lock node is in the first place, it means the lock is acquired successfully
-
If your own lock node is not in the first position, listen to your previous lock node. For example, lock the node itself_ 000000002, then listen to Lock_000000001
-
Current one bit lock node (Lock_000000001) the corresponding client is executed. If the lock is released, the listening client (lock) will be triggered_ Logic of 000000002)
-
Listen to the client to re execute the logic in step 2, and judge whether it has obtained the lock
Code implementation:
/**
* zookeeper Distributed lock
* 1,preparation
* 2,In the constructor, create a zookeeper connection
* 3,Create lock
* 4,Acquire lock
* 5,Release lock
* 6,Internal class test distributed lock
*/
public class MyLock {
//zookeeper connection
String IP="192.168.1.7:2181";
//Program counter, used to block the main thread and wait for the completion of zookeeper creation when zookeeper is created
CountDownLatch downLatch=new CountDownLatch(1);
//zookeeper configuration information
ZooKeeper zooKeeper;
//Parent node
private static final String LOCK_ROOT_PATH="/Locks";
//Format all child nodes
private static final String LOCK_NODE_PATH="Lock_";
//Final path generated
private String lockpath;
//In the constructor, open the zookeeper connection
public MyLock(){
try {
zooKeeper=new ZooKeeper(IP, 5000, new Watcher() {
@Override
public void process(WatchedEvent event) {
if(event.getType()==Event.EventType.None){
if(event.getState()==Event.KeeperState.SyncConnected){
System.out.println("zookeeper Connection successful");
//Counter minus one, release block
downLatch.countDown();
}
}
}
});
//Block the main thread and wait for zookeeper to create successfully
downLatch.await();
}catch (Exception e){
e.printStackTrace();
}
}
//Acquire lock
public void acquireLock() throws Exception {
//Create lock node
createLock();
//Attempt to acquire lock
attempLock();
}
//Create lock
private void createLock()throws Exception {
//Determine whether Locks exist or not. Create Locks if they don't exist
Stat stat = zooKeeper.exists(LOCK_ROOT_PATH, false);
if(stat==null){
//Create / Locks as persistent node
zooKeeper.create(LOCK_ROOT_PATH,new byte[0], ZooDefs.Ids.OPEN_ACL_UNSAFE, CreateMode.PERSISTENT);
}
//Create a temporary ordered child node
lockpath=zooKeeper.create(LOCK_ROOT_PATH+"/"+LOCK_NODE_PATH,new byte[0], ZooDefs.Ids.OPEN_ACL_UNSAFE,CreateMode.EPHEMERAL_SEQUENTIAL);
System.out.println("The temporary ordered child node was created successfully:"+lockpath);
}
//Monitor whether the previous node is deleted
private Watcher watcher=new Watcher() {
@Override
public void process(WatchedEvent event) {
//10. Events triggered by deleting a listening node
if(event.getType()==Event.EventType.NodeDeleted){
//11. Listen for the last node to release the lock.
synchronized (this){
//12. Release the thread of the current node
watcher.notifyAll();
}
}
}
};
/**
* Attempt to acquire lock
* The sequence number roughly represents the running steps of the thread.
*/
private void attempLock() throws Exception {
//1. Get all child nodes under Locks node
List<String> children = zooKeeper.getChildren(LOCK_ROOT_PATH, false);
//2. Sort children
Collections.sort(children);
//3. Get the location of the current node / Locks/Lock_0000000000, after interception,
int index=children.indexOf(lockpath.substring(LOCK_ROOT_PATH.length()+1));
//4. If the index is 0, then the current node is the first in the collection, and the lock can be acquired
if(index==0){
System.out.println("Lock acquired successfully");
return;
}else{
//5. else indicates that there is a lock in front of the current node. Then listen for the deletion of the previous node of the current node.
String prePath=children.get(index-1);
Stat stat = zooKeeper.exists(LOCK_ROOT_PATH + "/" + prePath, watcher);
if(stat==null){//6. If stat is null, the previous node just releases the lock, and the current node only needs to go through the current method again. Acquire lock
attempLock();
}else {//7. If it is not null, the previous node either occupies the lock or has not acquired the lock. Then let the current thread wait and wait for the previous node to release the lock
//8. The synchronization block passes the current monitor object.
synchronized (watcher){
//9. The monitor waits. Blocks the current thread. (for steps 10-12, see the wathcer method above)
watcher.wait();
}
//13. Here, the previous node is released, and the current thread is also released in the monitor method. Then the current node gets the lock again.
attempLock();
}
}
}
//Release lock
public void releaseLock() throws Exception {
//Delete temporary ordered nodes. -1 means no version number is specified
zooKeeper.delete(lockpath,-1);
zooKeeper.close();
System.out.println("Lock release");
}
}
//Test the distributed lock class.
class TicketSeller {
private void sell(){
System.out.println("Start selling tickets");
int sleepMills=5000;
try{
//Simulate complex logic
Thread.sleep(sleepMills);
}catch (Exception e){
e.printStackTrace();
}
System.out.println("End of ticket sales");
}
private void sellWithLock() throws Exception{
MyLock zkLock=new MyLock();
zkLock.acquireLock();
sell();
zkLock.releaseLock();
}
public static void main(String[] args) throws Exception {
TicketSeller seller = new TicketSeller();
for(int i=0;i<10;i++){
seller.sellWithLock();
}
}
}