1. What is Zookeeper
1.1 Overview
In my impression, zookeeper can exist as a registry. The previous microservice architecture is more of a combination of Dubbo+zookeeper. Therefore, zookeeper mainly serves distributed systems.
The characteristic of a distributed system is that there will be multiple nodes, real-time perception of the status of each node, and management of each node becomes particularly important. The emergence of zookeeper solved this problem. To understand from the perspective of design patterns: it is a distributed service management framework designed based on the observer pattern. It is responsible for storing and managing the data that everyone cares about, and then accepts the observer's registration. Once the state of these data changes, zookeeper It is responsible for notifying those observers who have been registered on zookeeper and letting them respond accordingly.
Therefore, zookeeper can be summarized as: file system + notification mechanism .
Welfare Welfare Welfare Free to receive Java architecture skills map Note that it is free to send
Get the required +V for free
Working mechanism :
- 1. Register information when the server starts (all temporary nodes are created)
- 2. Get the current online server list and register for monitoring
- 3. The server node goes offline
- 4. Server node offline event notification
- 5. The process will get the server list again and register to monitor
1.2 Features
1. Zookeeper is a design pattern that is a cluster composed of a leader and multiple followers
2. Half principle: As long as more than half of the nodes in the cluster survive, the zookeeper cluster can serve normally
3. The global data is consistent, each server saves a copy of the same data, no matter which server the client is connected to, the data is consistent
4. Data update atomicity: A data update either succeeds or fails.
5. Real-time: In a certain time range, the client can read the latest data.
1.3 Data structure
The zookeeper's data model is very similar to the Linux file system. The whole can be regarded as a tree. Each node is called a znode. By default, it can store 1Mb of data. Each znode can be uniquely identified by its path: 
These nodes can be divided into two categories: temporary and persistent. Therefore, an interview question arises: What are the node types of zookeeper?
- Temporary normal node
- Temporary ordered node
- Persistent normal node
- Persistent ordered node
1.4 What can zookeeper do?
The services provided by zookeeper include: unified naming service, unified configuration management, unified cluster management, soft load balancing, and distributed locks.
1.4.1 Uniform Naming Service
It can be understood as accessing the domain name: www.baidu.com , there are many servers under this domain name: 192.168.1.100, 192.168.1.101, 192.168.1.102, etc. As long as others visit this domain name, zookeeper will forward the request to a registered server. And the user does not need to remember which ip address to visit, only need to know the domain name.
1.4.2 Unified Configuration Management
Extract the public configuration files and distribute them to other systems.
Configuration file synchronization is very common. It is generally required that the configuration information of all nodes in a cluster is consistent, such as a Kafka cluster. After modifying the configuration file, it is hoped that it can be quickly synchronized to each configuration.
1.4.3 Unified cluster management
1) In a distributed environment, it is necessary to grasp the status of each node in real time, which can be adjusted according to the real-time status of the node.
2) Zookeeper can realize real-time monitoring of node status changes.
- Node information can be written to a Znode on zookeeper
- Monitor this znode to get its real-time status changes.
1.4.4 Soft load balancing
The number of visits to each server is recorded in zookeeper, so that the server with the least visits can process the latest client requests.
1.4.5 Distributed Lock
If there are multiple clients a, b, and c to grab a zookeeper distributed lock. The principle is this:
- When we first visited the /lock node, everyone came up and created an ordered node one after another. For example, a system created 001 node. The b system created 002 nodes, and the c system created 003 nodes.
- After the system gets all the nodes, it will compare whether its own node is the smallest, if it is, it will get the lock, if it is not, it will add a listener to the previous node to monitor it.
- As long as the previous node releases the lock, it will be queued to the front, which is equivalent to a queuing mechanism.
The first intention of using temporary nodes: if a client creates a temporary sequence node, it doesn’t matter if it accidentally goes down. Zookeeper senses that the client is down and will automatically delete the corresponding temporary sequence node, which is equivalent to automatic release Lock, or automatically cancel your own queue.
Two, zookeeper installation
2.1 Local mode installation and deployment
1. Steps:
- Configuration modification: zoo_sample.cfg under conf is modified to zoo.cfg
- Create a dataDir folder in the zookeeper directory to store the data storage directory.
- Add the data storage path under zookeeper:
dataDir=/opt/module/zookeeper-3.5.7/zkData
复制代码- start up
[root@hadoop101 zookeeper-3.5.7]# bin/zkServer.sh start
复制代码- Check status
[root@hadoop101 zookeeper-3.5.7]# bin/zkServer.sh status
复制代码- Start the zk service of the client to connect to the specified host
[root@hadoop101 zookeeper-3.5.7]# bin/zkCli.sh -server host:port
复制代码- stop
[root@hadoop101 zookeeper-3.5.7]# bin/zkServer.sh stop
复制代码2.2 Interpretation of configuration parameters
1) tickTime = 2000: the number of communication heartbeats, the heartbeat time of Zookeeper server and client, the basic time used by Zookeeper in milliseconds, the time interval for maintaining heartbeats between servers or between the client and the server, that is, each tickTime time will be sent A heartbeat, the time unit is milliseconds. It is used for the heartbeat mechanism and sets the minimum session timeout time to twice the heartbeat time. (The minimum timeout of session is 2*tickTime)
2) initLimit = 10: LF initial communication time limit The maximum number of heartbeats (the number of tickTimes) that can be tolerated during the initial connection between the Follower server in the cluster and the Leader leader server. Use it to limit the Zookeeper server in the cluster to Leader's time limit.
3) syncLimit = 5: LF synchronization communication time limit The maximum response time unit between Leader and Follower in the cluster. If the response exceeds syncLimit * tickTime, the Leader thinks that the follower is dead and deletes the follower from the server list.
4) dataDir: data file directory + data persistence path is mainly used to save data in Zookeeper.
5) clientPort = 2181: The client connection port monitors the port for client connections
Three, the internal principle of Zookeeper
3.1 Node type (Znode)
Persistent: After the client and server are disconnected, the created node will not be deleted.
1) Ordinary persistent node
2) Persistent node with serial number (sequence number zookeeper maintains itself)
Short: After the client and the server are disconnected, the created node is deleted by itself.
1) Ordinary transient node
2) Short-lived nodes with serial numbers (sequence number zookeeper maintains by themselves)
3.2 Stat structure
Describe the status information of each ZNode
[zk: localhost:2181(CONNECTED) 12] stat /zookeeper
cZxid = 0x0
ctime = Thu Jan 01 08:00:00 CST 1970
mZxid = 0x0
mtime = Thu Jan 01 08:00:00 CST 1970
pZxid = 0x0
cversion = -2
dataVersion = 0
aclVersion = 0
ephemeralOwner = 0x0
dataLength = 0
numChildren = 2
复制代码(1) czxid-the transaction zxid that created the node. Every time the ZooKeeper state is modified, a timestamp in the form of zxid will be received, which is the ZooKeeper transaction ID. The transaction ID is the total sequence of all modifications in ZooKeeper. Each modification has a unique zxid, if zxid1 is less than zxid2, then zxid1 occurs before zxid2.
(2) ctime-the number of milliseconds when the znode was created (since 1970)
(3) mzxid-last updated transaction zxid of znode
(4) mtime-the number of milliseconds last modified by znode (since 1970)
(5) The last updated child node zxid of pZxid-znode
(6) cversion-znode child node change number, znode child node modification times
(7) dataversion-znode data change number
(8) aclVersion-the change number of znode access control list
(9) ephemeralOwner- If it is a temporary node, this is the session id of the znode owner. If it is not a temporary node, it is 0.
(10) dataLength- the data length of znode
(11) numChildren-the number of znode child nodes
3.3 Listener principle
1. Principle
- First, there must be a main() thread
- Create a zookeeper client in the main thread. At this time, two threads will be created, one is responsible for network connection communication (connect), and the other is responsible for listening (listener)
- Send the registered monitoring event to zookeeper through the connect thread
- Add the registered monitoring event to the list in the registered monitoring list of zookeeper
- Zookeeper will send this message when it detects data or path changes
- The process() method is called inside the listener thread
2. Common monitoring
- Monitor changes in node data: get path
- Monitor the changes of child nodes: ls path
3.4 Election mechanism
3.4.1 ZAB Agreement:
An algorithm (protocol) based on message transmission and ensuring data consistency
Agreement goal:
1. Election of leader without leader
2. If there is a leader, try to ensure data consistency as much as possible.
3.4.2 zj cluster
(1) Half mechanism: More than half of the machines in the cluster survive and the cluster is available. So Zookeeper is suitable for installing an odd number of servers.
(2) Zookeeper although Master and Slave are not specified in the configuration file. However, when Zookeeper is working, one node is the leader, and the other is the follower. Leaders are temporarily generated through the internal election mechanism.
3.4.3 Mechanism concept
1. Serverid: server id, for example, there are three servers, the numbers are 1, 2, 3
2. Zxid: Data ID
The largest data ID stored in the server: the larger the value, the newer the data, and the newer the data in the election algorithm, the greater the weight.
3. Epoch: logical clock
Or called the number of votes, the logical clock value in the same round of voting is the same. This data will increase every time a vote is cast, and then compared with the value of the returned voting information, different judgments are made based on different values.
4. Server status: election status
- LOOKing: Campaign
- FOLLOWING: Follower status
- OBSERVING: Observe the status,
- LESDING: Leader status.
3.4.4 The election process
Suppose there is a Zookeeper cluster composed of five servers, their ids are from 1-5, and they are all newly started, that is, there is no historical data, and the amount of data stored is the same.
Voting principles :
- The principle of selfishness (the server will vote for itself when it first registers)
- The principle of letting the wind fall: (If you find that other servers are better than you, vote for other servers) The comparison can be zxid timestamp, and which server has the latest data will vote for it.
Process :
(1) Server 1 starts and initiates an election. Server 1 votes for itself. At this time, server 1 has one vote, which is less than half of the votes (3 votes), the election cannot be completed, and the status of server 1 remains LOOKING;
(2) Server 2 is started and another election is initiated. Servers 1 and 2 respectively vote for themselves and exchange vote information: At this time, server 1 finds that the ID of server 2 is larger than the one currently voted by (server 1), and changes the vote to recommend server 2. At this time, server 1 has 0 votes, server 2 has 2 votes, and there is no more than half of the results, the election cannot be completed, and the status of server 1, 2 remains LOOKING
(3) Server 3 starts and initiates an election. At this time, both servers 1 and 2 will change their votes to server 3. The result of this poll: Server 1 has 0 votes, Server 2 has 0 votes, and Server 3 has 3 votes. At this time, server 3 has more than half of the votes, and server 3 is elected Leader. Server 1, 2 changed status to FOLLOWING, server 3 changed status to LEADING;
(4) Server 4 starts and initiates an election. At this time, servers 1, 2, and 3 are no longer in the LOOKING state, and the vote information will not be changed. The result of the vote information exchange: Server 3 has 3 votes and Server 4 has 1 vote. At this time, server 4 obeys the majority, changes the vote information to server 3, and changes the status to FOLLOWING;
(5) Server 5 starts, and becomes a kid like 4.
3.4.5 Election after leader failure
When the cluster is working, after the leader fails, as long as the number of remaining machines is more than half, the cluster can work normally, but the leader needs to be re-elected. The election process is still voting, because the cluster is at work, so the id of each machine may be different. Then every time a vote (myid, zxid) is cast, the zxid is compared first, and then the myid is compared. Therefore, the leader with the largest zxid among the remaining machines in the cluster is elected. If the zxid is the same, in theory, the largest myid wins.
zxid timestamp, the latest data. In a sense, it can indicate the integrity of the data stored in the current machine.
3.5 Write data flow
1) The client connects to any machine in the zk cluster and sends a write request
2) If the leader is deployed in the zk cluster connected by the client, the current machine will forward the client's write request to the leader
3) When the leader receives the write request, it will construct the current write operation into a transaction, corresponding to a zxid
4) After each follower receives the write operation, it first stores the write operation in the queue and sends feedback to the leader
5) When the leader receives feedback from more than half of the followers in the cluster, it means that the write operation can be performed normally.
The leader will broadcast to each follower again, let the follower commit the write operation (real write data)
Four, finally
The content of zookeeper is much more than that. This article is more about introducing some concepts of zookeeper. A few client commands are put on every time. Today we know that zookeeper's storage node and monitoring mechanism can realize many functions. It is enough to understand this at the current stage. If there is a chance to go deeper, I will write a follow-up article to introduce it.