Build PostgreSQL11 cluster based on Pgpool-II4.1.0
This article mainly introduces streaming replication based on pgpool4.1 and PostgreSQL to achieve PostgreSQL read-write separation and high availability. For configuration, refer to the official Pgpool-II document . The version used is Pgpool4.1, PostgreSQL11.6.
The article is slightly longer. . . . . . . . . . . Although I do n’t want to
Introduction to Pgpool
The introduction of pgpool is not in this article. Those who need it can refer to another article of mine. Portal: Introduction to pgpool
Environmental preparation
This time using three CentOS7.3 servers to build a cluster, first deploy the PostgreSQL11.6 version database on the three servers. You can find the deployment tutorial online. The specific plan is as follows:
| Hostname | IP Address |
|---|---|
| Virtual IP | 192.168.111.6 |
| server1 | 192.168.111.1 |
| server2 | 192.168.111.2 |
| server3 | 192.168.111.3 |
Refer to the official website for the architecture diagram:
Build configuration
server configuration
For failover and online recovery, you need to ssh to other servers to execute commands, so you need to configure passwordless ssh login between servers (for example, if ssh is already configured between servers, you can skip this section, but you need to modify scripts such as failover.sh)
[all servers]# cd ~/.ssh
[all servers]# ssh-keygen -t rsa -f id_rsa_pgpool
[all servers]# ssh-copy-id -i id_rsa_pgpool.pub postgres@server1
[all servers]# ssh-copy-id -i id_rsa_pgpool.pub postgres@server2
[all servers]# ssh-copy-id -i id_rsa_pgpool.pub postgres@server3
[all servers]# su postgres
[all servers]$ cd ~/.ssh
[all servers]$ ssh-keygen -t rsa -f id_rsa_pgpool
[all servers]$ ssh-copy-id -i id_rsa_pgpool.pub postgres@server1
[all servers]$ ssh-copy-id -i id_rsa_pgpool.pub postgres@server2
[all servers]$ ssh-copy-id -i id_rsa_pgpool.pub postgres@server3
pgpool installation
The installation of pgpool is not introduced in this article. Those who need it can refer to another article. Portal: pgpool installation .
PostgreSQL database configuration
- WAL archiving
requires WAL archiving to be self-configurable, this example does not currently use WAL archiving
[all servers]# su - postgres
[all servers]$ mkdir /var/lib/pgsql/archivedir
## postgres.conf 配置
archive_mode = on
archive_command = 'cp "%p" "/var/lib/pgsql/archivedir/%f"'
There is an archived script, you can use this script to modify how many days to keep archive pg_archive.sh
- postgres.conf configuration
This configuration is only configured on the master node, and the slave node uses the pgpool online recovery function configuration
listen_addresses = '*'
port = 5432
max_wal_senders = 10
max_replication_slots = 10
wal_level = replica
hot_standby = on
wal_log_hints = on
- Database user and password configuration
| username | password | use |
|---|---|---|
| repl | repl | Used for PostgreSQL streaming replication |
| pgpool | pgpool | Used for pgpool-II heartbeat detection and replication delay detection |
| postgres | postgres | Used for online recovery |
[server1]# psql -U postgres -p 5432
postgres=# CREATE ROLE pgpool WITH LOGIN PASSWORD 'pgpool';
postgres=# CREATE ROLE repl WITH REPLICATION LOGIN PASSWORD 'repl';
## 用于SHOW POOL_NODES 展示 "replication_state" and "replication_sync_state"
postgres=# GRANT pg_monitor TO pgpool;
- pg_hba.con configuration
Add user authentication configuration, add repl copy user to pg_hba. All passwords in this document are verified using md5.
host all all 0.0.0.0/0 md5
host replication repl 0.0.0.0/0 md5
- Configure .pgpass file for passwordless operation
Because the script is used for operation during failover and online recovery, commands such as pg_basebakup and pg_rewind are used in the script, so password-free operation needs to be configured
homeCreate a .pgpassfile in the postgres user's directory , and the file permissions are 600.
[all servers]# su - postgres
[all servers]$ vi ~/.pgpass
## 格式为：hostname:port:database:username:password
server1:5432:replication:repl:<repl user password>
server2:5432:replication:repl:<repl user passowrd>
server3:5432:replication:repl:<repl user passowrd>
server1:5432:postgres:postgres:<postgres user passowrd>
server2:5432:postgres:postgres:<postgres user passowrd>
server3:5432:postgres:postgres:<postgres user passowrd>
[all servers]$ chmod 600 ~/.pgpass
pgpool configuration
# cp -p /usr/local/pgpool/etc/pgpool.conf.sample-stream /usr/local/pgpool/etc/pgpool.conf- Configure PostgreSQL database information
# - Backend Connection Settings -
# 有几台PostgreSQL数据库，配置几个后端信息，使用后缀名0、1、2……
backend_hostname0 = 'server1' # Host name or IP address to connect to for backend 0
backend_port0 = 5432 # Port number for backend 0
backend_weight0 = 1 # Weight for backend 0 (only in load balancing mode)
backend_data_directory0 = '/data/pgsql/sport/' # Data directory for backend 0
# Controls various backend behavior
# ALLOW_TO_FAILOVER or DISALLOW_TO_FAILOVER
backend_flag0 = 'ALLOW_TO_FAILOVER'
backend_hostname1 = 'server2'
backend_port1 = 5432
backend_weight1 = 1
backend_data_directory1 = '/var/lib/pgsql/11/data'
backend_flag1 = 'ALLOW_TO_FAILOVER'
backend_hostname2 = 'server3'
backend_port2 = 5432
backend_weight2 = 1
backend_data_directory2 = '/var/lib/pgsql/11/data'
backend_flag2 = 'ALLOW_TO_FAILOVER'
- Basic configuration
listen_addresses = '*'
pid_file_name = '/var/run/pgpool/pgpool.pid'
# PID file name
# Can be specified as relative to the"
# location of pgpool.conf file or
# as an absolute path
# (change requires restart)
logdir = '/var/run/pgpool'
# Directory of pgPool status file
# (change requires restart)
Create a directory for pgpool operationmkdir -p /var/run/pgpool
- Configure replication delay check
sr_check_user = 'pgpool'
## 自从4.0版本后，如果密码设置为的话，pgpool会首先从`pool_passwd`文件中获取密码，然后在使用空密码
sr_check_password = 'pgpool'
- Configure health check
# Health check period
# Disabled (0) by default
health_check_period = 5
# Health check timeout
# 0 means no timeout
health_check_timeout = 30
health_check_user = 'pgpool'
health_check_password = 'pgpool'
health_check_max_retries = 3
- Configure failover
failover_command = '/usr/locla/pgpool/etc/failover.sh %d %h %p %D %m %H %M %P %r %R %N %S'
follow_master_command = '/usr/locla/pgpool/etc/follow_master.sh %d %h %p %D %m %H %M %P %r %R'
About the script, the download address and script usage instructions will be given later
# 给脚本设置执行权限，注意：该脚本需要在三台pgpool服务器中都需要创建
chmod +x /usr/local/pgpoll/etc/{failover.sh,follow_master.sh}
- Configuring Online Recovery
In order to perform online recovery using Pgpool-II, we need to configure the PostgreSQL username and online recovery commandsrecovery_1st_stage. Since performing online recovery requires superuser privileges in PostgreSQL, we specify the postgres user in recovery_user. Then, we create recovery_1st_stage and pgpool_remote_start in the database directory of the PostgreSQL master server (server1), and add execute permissions.
recovery_user = 'postgres'
# Online recovery user
recovery_password = 'postgres'
# Online recovery password
recovery_1st_stage_command = 'recovery_1st_stage'
## 该脚本只需要在数据库主节点创建，后续使用在线恢复时会复制过去
[server1]# su - postgres
[server1]$ vi /data/pgsql/sport/recovery_1st_stage
[server1]$ vi /data/pgsql/sport/pgpool_remote_start
[server1]$ chmod +x /data/pgsql/sport/{recovery_1st_stage,pgpool_remote_start}
The database creates an extension, the extension is to be able to perform online recovery (if the extension has been created during installation, you can skip this step)
[server1]# su - postgres
[server1]$ psql template1 -c "CREATE EXTENSION pgpool_recovery"
- Configure client authentication
## pgpool.conf中
enable_pool_hba = on
The authentication file is /usr/local/pgpool/etc/pool_hba.confbasically the same as PostgreSQL. (For scram-sha-256 method, please refer to pgpool official website)
host all pgpool 0.0.0.0/0 md5
host all postgres 0.0.0.0/0 md5
cd /usr/local/pgpool/etc
../bin/pg_md5 -p -m -u postgres pool_passwd
../bin/pg_md5 -p -m -u pgpool pool_passwd
cat /etc/pgpool-II/pool_passwd
# pgpool:AESheq2ZMZjynddMWk5sKP/Rw==
# postgres:AESHs/pWL5rtXy2IwuzroHfqg==
- Watchdog configuration
use_watchdog = on
delegate_IP = '192.168.111.6' ##vip配置
## 网卡名字需要正确配置
if_up_cmd = '/usr/bin/sudo /sbin/ip addr add $_IP_$/24 dev enp0s8 label enp0s8:0'
if_down_cmd = '/usr/bin/sudo /sbin/ip addr del $_IP_$/24 dev enp0s8'
arping_cmd = '/usr/bin/sudo /usr/sbin/arping -U $_IP_$ -w 1 -I enp0s8'
配置其它pgpool信息，注意：该配置在三台服务器不一样，只需要配置另外几台即可
[server1配置如下]
# - Other pgpool Connection Settings -
other_pgpool_hostname0 = 'server2'
other_pgpool_port0 = 9999
other_wd_port0 = 9000
other_pgpool_hostname1 = 'server3'
other_pgpool_port1 = 9999
other_wd_port1 = 9000
heartbeat_destination0 = 'server2'
heartbeat_destination_port0 = 9694
heartbeat_device0 = ''
heartbeat_destination1 = 'server3'
heartbeat_destination_port1 = 9694
heartbeat_device1 = ''
[server2配置如下]
# - Other pgpool Connection Settings -
other_pgpool_hostname0 = 'server1'
other_pgpool_port0 = 9999
other_wd_port0 = 9000
other_pgpool_hostname1 = 'server3'
other_pgpool_port1 = 9999
other_wd_port1 = 9000
heartbeat_destination0 = 'server1'
heartbeat_destination_port0 = 9694
heartbeat_device0 = ''
heartbeat_destination1 = 'server3'
heartbeat_destination_port1 = 9694
heartbeat_device1 = ''
[server3配置如下]
# - Other pgpool Connection Settings -
other_pgpool_hostname0 = 'server1'
other_pgpool_port0 = 9999
other_wd_port0 = 9000
other_pgpool_hostname1 = 'server2'
other_pgpool_port1 = 9999
other_wd_port1 = 9000
heartbeat_destination0 = 'server1'
heartbeat_destination_port0 = 9694
heartbeat_device0 = ''
heartbeat_destination1 = 'server2'
heartbeat_destination_port1 = 9694
heartbeat_device1 = ''
- pcp command configuration
cd /usr/local/pgpool/bin
echo 'pgpool:'`pg_md5 pgpool` >> /usr/local/pgpool/etc/pcp.conf
Pgpool above configuration are configured, will eventually generate pgpool.conf, , pool_hba.conf, pool_passwd, pcp.confyou may have configured after a configuration file to the other server copy. Note: The watchdog configuration in pgpool.conf needs to be modified on other servers.
Related scripts
A total of four scripts are required here:
- failover.sh directory:
/usr/local/pgpool/etc/lower - follow_master.sh directory:
/usr/local/pgpool/etc/lower - The recovery_1st_stage directory is: under the PostgreSQL database data directory
- The pgpool_remote_start directory is: under the PostgreSQL database data directory
Script address: https://github.com/MrSmallLiu/pgpool The
script is referenced on the official website, but slightly modified:
- When creating and deleting replication slots, if you use the ip address, you will get an error because the name is not allowed
., modify:${FAILED_NODE_HOST//./_} - A bug in the follow_master.sh script
# drop replication slot
ssh -T -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null postgres@${NEW_MASTER_NODE_HOST} -i ~/.ssh/id_rsa_pgpool "
${PGHOME}/bin/psql -p ${NEW_MASTER_NODE_PORT} -c \"SELECT pg_drop_replication_slot('${FAILED_NODE_HOST//./_}')\"
"
Note :
- Note that all the script
PGHOMEvariables PostgreSQL installation path - Note that in all scripts
ARCHIVEDIR, if they are not used, they need to be commented, and the corresponding use place should be modified - Note the
PGPOOL_PATHvariablepgpoolbin path in all scripts - Note that in recovery_1st_stage script
PRIMARY_NODE_HOSTvariable that takeshostnamethe value, you can look at whether the command line with expectations in their own test server. Non-conformers usehostnamectl set-hostname server1
start up
- Background process
/usr/local/pgpool/bin/pgpool - debug run
/usr/local/pgpool/bin/pgpool -n -d
/usr/local/pgpool/bin/pgpool -n - The standby database starts
-n 表示在pgpool.conf配置文件中PostgreSQL配置的后面的序号
/usr/local/pgpool/bin/pcp_recovery_node -h 192.168.111.6 -p 9898 -U pgpool -n 1
Password:
pcp_recovery_node -- Command Successful
/usr/local/pgpool/bin/pcp_recovery_node -h 192.168.111.6 -p 9898 -U pgpool -n 2
Password:
pcp_recovery_node -- Command Successful
stop
/usr/local/pgpool/bin/pgpool -m fast stop
test
- Use client connection (navicat, pgAdmin, etc.)
psql -h 192.168.111.6 -p 9999 -U passwd: postgres - carried out
show pool_nodes
Load balancing test
You can configure the PostgreSQL log to print sql statements, and use the VIP address to connect to the test output to see that the query will load to the three servers (by default, load balancing is used according to the connection, several new queries are required to test)
Automatic failover test
You can use PostgreSQL command to stop the master node database to test the failover
Write at the end
- The above configuration can be run normally after our deployment test, but this article was written after the subsequent finishing, there may be some omissions. You can contact me if you have any questions during the configuration process, fail to start normally after configuration, or have documentation comments. Please feel free to mention it to me, and then I will modify the document to help more people.
- I will take the time to sort out a pgpool deployment flow chart later so that everyone can understand the deployment process.
- The specific explanation and optimization of the configuration file will be written later in the relevant documents, welcome to continue to pay attention
Regarding this article, you are welcome to criticize and correct
About the author
GISer
Xiao Liu Xiansen
QQ: 1016817543
Email: [email protected]
github: https://github.com/MrSmallLiu (welcome star)
Related Links
The following are some libraries that I participated in the development. Welcome to Star, Issues, PR
