Detailed explanation of the four script execution files in the bin directory in ZooKeeper

There are the following 4 executable scripts in the bin directory in ZooKeeper:

[root@poc01 ~]# cd /opt/module/zookeeper-3.4.6/bin/
[root@poc01 bin]# ls
zkCleanup.sh  zkCli.sh  zkEnv.sh  zkServer.sh

These scripts are part of ZooKeeper and are used to manage and operate ZooKeeper instances.

zkCleanup.sh：
- Purpose: This script is mainly used to clean up the snapshots and transaction log files in the ZooKeeper data directory for data cleaning and maintenance.
- Usage: Typically run this script when you need to clean up the ZooKeeper data directory, it will delete old snapshots and log files, helping to free up disk space and maintain ZooKeeper data.
zkCli.sh：
- Purpose: zkCli.sh is the client command line tool that comes with ZooKeeper. It is used to connect to ZooKeeper instances and perform various operations, such as creating nodes, reading node data, monitoring node changes, etc.
- Usage: Run the zkCli.sh script, specify connection parameters (such as the address of the ZooKeeper instance), and then enter various ZooKeeper client commands in the command line interface to interact with ZooKeeper.
zkEnv.sh：
- Purpose: The zkEnv.sh script is used to set the environment variables and configuration of ZooKeeper. It can set environment variables such as Java environment, memory configuration, JVM parameters, etc.
- Usage: Usually before starting a ZooKeeper instance, you can use zkEnv.sh to set the environment parameters and configuration of the ZooKeeper runtime.
zkServer.sh：
- Purpose: The zkServer.sh script is used to start, stop, and manage ZooKeeper service instances.
- Usage: The ZooKeeper service can be managed by running the zkServer.sh script and specifying start, stop and other parameters. For example, zkServer.sh start is used to start the ZooKeeper service, and zkServer.sh stop is used to stop the service.

The following will introduce the use, parameters, scenarios and precautions of the four ZooKeeper related scripts in detail one by one.

1. zkCli.sh

1-1. zkCli-usage example

Connect to ZooKeeper:./zkCli.sh -server localhost:2181
Create node:create /path value
Read node data:get /path
Parameter introduction:zkCli.sh Accept connection parameters, such as ZooKeeper server address.
Scenario: Used to interact with ZooKeeper, create, read, update, delete nodes, etc.

zkCli.shThe script is a command line client tool provided by ZooKeeper, which is used to connect to the ZooKeeper server and perform related operations. It allows users to interactively manipulate ZooKeeper data, such as creating nodes, setting node data, obtaining node information, etc. The following is about how to use the zkCli.sh script and some common command examples:

1-2. zkCli-Connection method

zkCli.sh -server server:port
# 例如：zkCli.sh -server 192.168.22.22:2181

-server server:port: Specify the ZooKeeper server address and port number to connect to. The default port is 2181.

1-3. zkCli-basic common commands

Building point：

create /path "data"

Get node data:

get /path

Set update node data:

set /path "new data"

Column starting point information:

ls /path

Exclusion point：

delete /path

Check out Hansuke：

help

1-4. zkCli-Notes

After the connection is successful, you will enter the interactive command line mode, and you can directly enter commands to perform operations.
Exit the interaction by quit or Ctrl + C exiting the zkCli client.
Please exercise caution when performing operations involving data changes such as deleting nodes, as data deletion is irreversible.
Use zkCli.sh to easily connect to the ZooKeeper server for data operations and management, but please be careful when performing important operations to avoid accidentally deleting data or affecting the stable operation of ZooKeeper.

1-5. zkCli-Advanced usage

zkCli.shIt is the basic command line client tool provided by ZooKeeper for interacting with the ZooKeeper server to perform various operations. Although it is a command line tool, it still provides some more complex and advanced functions, which can be used for node monitoring, permission management, Watcher monitoring, etc.

Monitor node status changes

Use stat command to monitor changes in node status and view node details in real time, such as version, data length, creation time, etc.:

[zk: poc01:2181(CONNECTED) 22] stat /data
cZxid = 0x200000006
ctime = Tue Dec 12 10:39:20 CST 2023
mZxid = 0x20000000a
mtime = Tue Dec 12 10:39:41 CST 2023
pZxid = 0x200000006
cversion = 0
dataVersion = 4
aclVersion = 0
ephemeralOwner = 0x0
dataLength = 4
numChildren = 0

cZxid: The transaction ID created by the node.

ctime: Node creation time.

mZxid: The last modified transaction ID of the node.

mtime: The last modification time of the node.

pZxid: The transaction ID modified by the node's parent node.

cversion: The number of child node changes.

dataVersion: The number of node data changes.

aclVersion: The number of ACL changes.

ephemeralOwner: The owner ID of the temporary node.

dataLength: Node data length.

numChildren:The number of child nodes.

statThe status information obtained by can also be obtained by basicget commands, but stat will not be obtainedvalue, only It purely obtains various statuses, so it is more suitable for monitoring.

Set up Watcher to use

When operating on a node, you can set up a Watcher to be notified when the node status changes.
When the status of a node changes, such as a node being created, deleted, data modified, etc., the Watcher will be triggered.
Once the Watcher is triggered, the client can perform corresponding logical processing after receiving the notification, such as re-obtaining node data, updating cache, etc.

Operation in window 1:

# 第一个窗口中创建节点
[zk: poc01:2181(CONNECTED) 2] create /example_node "initial_data"
Created /example_node
# 使用watch方式进行通知监控
[zk: poc01:2181(CONNECTED) 3] get /example_node watch
"initial_data"
cZxid = 0x20000000f
ctime = Tue Dec 12 14:38:56 CST 2023
mZxid = 0x20000000f
mtime = Tue Dec 12 14:38:56 CST 2023
pZxid = 0x20000000f
cversion = 0
dataVersion = 0
aclVersion = 0
ephemeralOwner = 0x0
dataLength = 14
numChildren = 0

Operation in window 2:

# 在新的窗口中去更新节点数据
[zk: poc01:2181(CONNECTED) 27] set /example_node "new_data"
cZxid = 0x20000000f
ctime = Tue Dec 12 14:38:56 CST 2023
mZxid = 0x200000010
mtime = Tue Dec 12 14:39:27 CST 2023
pZxid = 0x20000000f
cversion = 0
dataVersion = 1
aclVersion = 0
ephemeralOwner = 0x0
dataLength = 10
numChildren = 0

Return to window 1 and observe whether there are monitoring information notifications

# 此时发现窗口1中的watch监控生效，提示有状态变化的消息
[zk: poc01:2181(CONNECTED) 4] 
WATCHER::
WatchedEvent state:SyncConnected type:NodeDataChanged path:/example_node

Precautions:

One-time trigger: Watcher is one-time, once triggered, it will become invalid. Watcher needs to be reset after handling Watcher events.

Event notification order: Watcher does not guarantee the strict order of event notification, only the FIFO (first in, first out) order.

Too many Watches: If the client sets a large number of Watchers, it may increase the load on the server. It is very important to use Watcher wisely.

ZooKeeper permission management

ZooKeeper supports ACL (access control list) to manage node permissions. You can use the addauth and setAcl commands to add authentication information and set node permissions< /span>

addauth is used to add authentication information to the session, while setAcl is used to set the node's ACL to control the node's access permissions. In actual use, it is necessary to choose the appropriate authentication method and permission settings according to the specific situation

addauthCommand is used to add authentication information in ZooKeeper

# 格式
addauth scheme auth
# 示例
addauth digest wangting:123456

scheme: Proof of proof, example digest, ip etc.

auth: Authentication information, which can be user name and password or IP address depending on the authentication scheme.

The authentication information added using the addauth command is valid for the current session and will automatically expire after the session ends, so no special commands are needed to delete this information.

Once the session ends (for example, the client disconnects), the authentication information added by the addauth command will also become invalid and will no longer take effect. If re-authentication is required, you can use the addauth command again in a new session to add authentication information.

setAclThe command is used to set the access control list (ACL) of the node.

# 格式
setAcl /path acl
# 示例
create /example_node "data" # 创建节点
setAcl /example_node digest:wangting:123456:crwda

/path: The node path to which the ACL is to be set.

acl: ACL information, including permissions and authentication information.

If the node with the ACL set is no longer needed, you can delete the node directly. After the node is deleted, the ACL settings will also be deleted:delete /example_node

By resetting the node's ACL, you can change the node's permission settings. You can use the setAcl command to modify the ACL of a node:setAcl /example_node acl

crwda 电影（create、read、update、delete）

ZooKeeper's ACL permission management is relatively complex and requires careful operation to avoid affecting node access permissions.

2. zkServer.sh

2-1. zkServer-Usage Example

Start ZooKeeper:./zkServer.sh start
Stop ZooKeeper:./zkServer.sh stop
Parameter introduction: Accept commands such as start and stop.
Scenario: Used to manage the starting and stopping of the ZooKeeper service.

zkServer.shScripts are scripts provided by ZooKeeper for starting, stopping, and managing ZooKeeper servers. It provides a series of commands for managing the running status of the ZooKeeper server, such as starting, stopping, restarting, etc.

2-2. zkServer-Usage

zkServer.sh {
    
    start|stop|restart|status|upgrade|print-cmd}

Parameter Description:

start: Start the ZooKeeper server.zkServer.sh start
stop: Stop the ZooKeeper server.zkServer.sh stop
restart: Restart the ZooKeeper server.zkServer.sh restart
status: View the running status of the ZooKeeper server.zkServer.sh status
upgrade: Upgrade ZooKeeper server.
print-cmd: Print the commands of the ZooKeeper server.

2-3. zkServer-Notes

When executing these commands, you need to add zkServer.sh before the command and specify the specific operation after it.
Before performing ZooKeeper related operations, please ensure that the correct environment variables have been configured, such as using the zkEnv.sh script to set the environment.
For start, stop, restart operations, appropriate permissions may be required to perform them.

UsezkServer.sh script to easily manage the start, stop and restart of the ZooKeeper server. You can also view the running status of the ZooKeeper server through the status command. to ensure the service is running properly.

zkServer.shScripts are scripts provided by ZooKeeper for managing ZooKeeper servers, including functions such as starting, stopping, restarting, and status querying. Although the function of this script itself is relatively basic, it can be combined with other advanced usage to improve the operation and maintenance efficiency and management of ZooKeeper.

2-4. zkServer-Advanced usage

Monitoring using JMX

When starting ZooKeeper, you can configure JMX monitoring parameters to manage and monitor the running status of the ZooKeeper server through JMX

./zkServer.sh start -jvmflags "-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=1234 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false"

Multiple instance deployment on the same host

By copying the ZooKeeper installation directory and configuring different data directories and port numbers, you can start multiple ZooKeeper instances to improve system availability and scalability.

./zkServer.sh start /path/to/zkConfig1
./zkServer.sh start /path/to/zkConfig2
./zkServer.sh start /path/to/zkConfig3

Customized logs and output

Configure ZooKeeper’s log level and output path to better monitor and debug ZooKeeper’s running status

./zkServer.sh start 2>&1 > /path/to/zk.log &

Use systemd or other tools to manage

Combine commands such as starting and stopping ZooKeeper with the systemd process management tool to manage ZooKeeper services better.

# 服务启停则可以用如下方式
systemctl start zookeeper
systemctl stop zookeeper

Implementation:

Create file in the /lib/systemd/system/ directoryzookeeper.service

[root@poc01 bin]# vim /lib/systemd/system/zookeeper.service
[Unit]
Description=Apache ZooKeeper Distributed Coordination Service
After=network.target

[Service]
User=root
Group=root
WorkingDirectory=/opt/module/zookeeper-3.4.6
ExecStart=/opt/module/zookeeper-3.4.6/bin/zkServer.sh start
ExecStop=/opt/module/zookeeper-3.4.6/bin/zkServer.sh stop
Restart=always
RestartSec=10

[Install]
WantedBy=multi-user.target

# 重新加载配置信息
[root@poc01 ~]# systemctl daemon-reload

# 启动服务
[root@poc01 ~]# systemctl start zookeeper
# 查看状态
[root@poc01 ~]# systemctl status zookeeper

# 添加至开机自启
[root@poc01 ~]# systemctl enable zookeeper.service
# 关闭开机自启
[root@poc01 ~]# systemctl disable zookeeper.service

Pay attention to modify the corresponding path

3. zkEnv.sh

3-1. zkEnv-Usage Example

source zkEnv.sh, load the configuration into the current Shell environment
Parameter introduction: This script usually does not require additional parameters
Scenario: Before starting the ZooKeeper instance, you can set the Java environment, memory configuration, JVM parameters, etc.

3-2. zkEnv-Usage

source zkEnv.sh

[root@poc01 ~]# source /opt/module/zookeeper-3.4.6/bin/zkEnv.sh

Set the operating environment of ZooKeeper, including JAVA_HOME, ZooKeeper installation path, etc.
Configure some basic parameters of ZooKeeper, such as memory size, log file path, etc.

3-3. zkEnv-Notes

Before running ZooKeeper related commands, source zkEnv.sh will be executed by default to configure environment variables to ensure that ZooKeeper can run correctly.

You can see the following code in zkServer.sh:

if [ -e "$ZOOBIN/../libexec/zkEnv.sh" ]; then
  . "$ZOOBINDIR/../libexec/zkEnv.sh"
else
  . "$ZOOBINDIR/zkEnv.sh"
fi

zkEnv.shThe script is mainly used to configure the environment variables when ZooKeeper is running to ensure that the environment settings required for ZooKeeper's operation are correct, thereby ensuring the normal operation of the ZooKeeper service.

3-4. zkEnv-Advanced usage

Configure JVM parameters

By zkEnv.sh setting JVM-related parameters, such as memory size, GC strategy, etc., you can improve the performance and stability of ZooKeeper.

export SERVER_JVMFLAGS="-Xmx2G -XX:+UseG1GC"

Customizing ZooKeeper logs

Set ZooKeeper's log level, output path, etc. to better monitor and debug ZooKeeper's running status.

export ZOO_LOG4J_PROP="INFO,ROLLINGFILE"
export ZOO_LOG_DIR="/path/to/zookeeper/logs"

Set authentication information

For situations where security authentication is required, you can configure ZooKeeper's authentication information and use usernames and passwords for access control.

export ZOOKEEPER_SERVER_OPTS="$ZOOKEEPER_SERVER_OPTS -Dzookeeper.DigestAuthenticationProvider.superDigest=super:user:password"

Configure network parameters

Adjust network-related parameters, such as port number, connection timeout, etc., to adapt to specific network environments.

export ZOOKEEPER_CLIENT_PORT=2181
export ZOOKEEPER_TICK_TIME=2000

4. zkCleanup.sh

4-1. zkCleanup-Usage Example

./zkCleanup.sh /path/to/zookeeper/data/version-2 -n 5, which will retain the last 5 snapshots and transaction log files.
Parameter introduction:zkCleanup.sh Accepts the data directory path and the number of files to be retained as parameters.
Scenario: When there are too many log files and snapshot files in the ZooKeeper data directory, it is used to clean up old files and free up disk space.

zkCleanup.shThe script is mainly used to clean up snapshots and transaction logs in the ZooKeeper data directory to reduce disk usage and maintain the health of ZooKeeper data. This script allows you to limit the number of snapshots and transaction logs retained, as well as clean up data in specified paths. This script is generally used to maintain the ZooKeeper data directory, clear old data files, and prevent excessive data from affecting performance.

4-2. zkCleanup-Usage

zkCleanup.sh [dataDir] [-n count] [-d days] [-t hours] [-r] [-h]

dataDir: The path to the ZooKeeper data directory, which is ./data by default.
-n count: Number of snapshots and log files to retain, default is 3.
-d days: Delete snapshots and log files older than the specified number of days.
-t hours: Delete snapshots and log files older than the specified number of hours.
-r: After specifying, the data in the specified directory will be deleted recursively.
-h: Display help information.

# 默认情况下保留 3 个快照和日志文件
zkCleanup.sh /path/to/zookeeper/data

# 保留 2 个快照和日志文件，删除超过 5 天的数据文件
zkCleanup.sh /path/to/zookeeper/data -n 2 -d 5

# 删除超过 12 小时的数据文件
zkCleanup.sh /path/to/zookeeper/data -t 12

# 递归删除指定目录下的数据
zkCleanup.sh /path/to/zookeeper/data -r

4-3. zkCleanup-Notes

When running the zkCleanup.sh script, please ensure that the ZooKeeper service has been stopped to avoid affecting the normal operation of ZooKeeper
The script deletes expired snapshots and transaction log files, ensuring important data is backed up before cleaning.
For the -d and -t parameters, if specified at the same time, cleaning will be performed based on both days and hours.
You can execute the zkCleanup.sh script regularly in conjunction with the operating system's scheduled tasks or other scripts to automatically clean up ZooKeeper's data files to maintain the health of the ZooKeeper data directory

4-4. zkCleanup-Usage Example

# 停止 ZooKeeper 服务
[root@poc01 ~]# zkServer.sh stop
[root@poc02 ~]# zkServer.sh stop
[root@poc03 ~]# zkServer.sh stop

# 找到 ZooKeeper 的数据目录，其中包括事务日志文件（version-2 目录下的文件）和快照文件
[root@poc01 ~]# ll /opt/module/zookeeper-3.4.6/zkData/version-2/
total 68
-rw-r--r-- 1 root root        1 Dec 12 16:56 acceptedEpoch
-rw-r--r-- 1 root root        1 Dec 12 16:56 currentEpoch
-rw-r--r-- 1 root root 67108880 Dec 12 10:24 log.100000001
-rw-r--r-- 1 root root 67108880 Dec 12 15:56 log.200000001
-rw-r--r-- 1 root root 67108880 Dec 12 16:49 log.200000026
-rw-r--r-- 1 root root 67108880 Dec 12 16:55 log.300000001
-rw-r--r-- 1 root root 67108880 Dec 12 16:56 log.400000001
-rw-r--r-- 1 root root      296 Dec 11 17:25 snapshot.0
-rw-r--r-- 1 root root      788 Dec 12 10:31 snapshot.100000006
-rw-r--r-- 1 root root     1425 Dec 12 16:43 snapshot.200000025
-rw-r--r-- 1 root root     1895 Dec 12 16:50 snapshot.20000002e
-rw-r--r-- 1 root root     1895 Dec 12 16:56 snapshot.300000002

# 清理操作
[root@poc01 ~]# zkCleanup.sh /opt/module/zookeeper-3.4.6/zkData/version-2 -n 3
[root@poc02 ~]# zkCleanup.sh /opt/module/zookeeper-3.4.6/zkData/version-2 -n 3
[root@poc03 ~]# zkCleanup.sh /opt/module/zookeeper-3.4.6/zkData/version-2 -n 3

# 启动 ZooKeeper 服务
[root@poc01 ~]# zkServer.sh start
[root@poc02 ~]# zkServer.sh start
[root@poc03 ~]# zkServer.sh start