MHA high availability configuration
1. Case knowledge points
1.1 What is MHA
- Developed by youshimaton, a Japanese DeNA company (now working at Facebook)
- A set of excellent high-availability software for failover and master-slave promotion in MySQL high availability environment
- Support failover
- In the MySQL failover process, MHA can automatically complete the database failover operation within 0~30 seconds, and in the process of failover, MHA can ensure data consistency to the greatest extent to achieve true High availability
- MHA also provides the online main library switching function, which can safely switch the currently running main library to a new main library (by upgrading the slave library to the main library), which can be completed in about 0.5-2 seconds
1.2, the composition of MHA
- MHA Manager (Management Node)
- MHA Node (Data Node)
- Note: Manager is hosted on Node, and Node needs to be installed first
1.3. Why use MHA?
- The traditional mysql master-slave architecture has a single point of failure
- In the traditional architecture, there is only one mysql master server, so when a single point of failure occurs, the entire server cluster will be paralyzed
- In order to solve this situation, we need to re-establish a main server when the main server is down, responsible for monitoring and other tasks
When we do MySQL read-write separation and master-slave replication, a master server is connected to multiple slave servers. If the master server hangs, there will be a single point of failure, so we need to use MHA high availability Failover.
1.4 Features of MHA
- During the automatic failover process, MHA tries to save the binary log from the down main server to ensure that data is not lost to the greatest extent
- Using semi-synchronous replication can greatly reduce the risk of data loss
- Currently MHA supports one-master multiple-slave architecture, with at least three services, namely one-master two-slave
2. MHA case
2.1. Experimental topology
2.2. Experimental environment
| Host | IP address | operating system |
|---|---|---|
| mysql1 (master) | 20.0.0.23 | centos-7.6-x86_64 |
| mysql2 (slave/master/backup) | 20.0.0.24 | centos-7.6-x86_64 |
| mysql3 (slave) | 20.0.0.25 | centos-7.6-x86_64 |
| Manger | 20.0.0.22 | centos-7.6-x86_64 |
2.3. Experimental purpose
- Monitor the MySQL database through MHA, automatically switch in case of failure, without affecting business
- When the main library fails, the alternative database library automatically becomes the main library
2.4. Experimental process
note
Because the operating system is centos7, download MHA 0.57 version, MHA has very high requirements for the version!
2.4.1, install MySQL database
The three database servers need to install the MySQL database first. If you can’t install it, you can check my previous blog.
Blog address: Install the database blog address
2.4.2, configure MySQL master-slave replication
- Do network after installing the database
mysql 1：
[root@mysql1 ~]# vi /etc/hosts ## 添加下面三句话
20.0.0.23 mysql1
20.0.0.24 mysql2
20.0.0.25 mysql3
==》》 wq
mysql 2：
[root@mysql2 ~]# vi /etc/hosts ## 添加下面三句话
20.0.0.23 mysql1
20.0.0.24 mysql2
20.0.0.25 mysql3
==》》 wq
mysql 3：
[root@mysql3 ~]# vi /etc/hosts ## 添加下面三句话
20.0.0.23 mysql1
20.0.0.24 mysql2
20.0.0.25 mysql3
==》》 wq
- Ping test for each database
[root@mysql2 ~]# ping mysql1
PING mysql1 (20.0.0.23) 56(84) bytes of data.
64 bytes from mysql1 (20.0.0.23): icmp_seq=1 ttl=64 time=0.293 ms
64 bytes from mysql1 (20.0.0.23): icmp_seq=2 ttl=64 time=0.383 ms
[root@mysql1 cmake-2.8.6]# ping mysql2
PING mysql2 (20.0.0.24) 56(84) bytes of data.
64 bytes from mysql2 (20.0.0.24): icmp_seq=1 ttl=64 time=1.17 ms
64 bytes from mysql2 (20.0.0.24): icmp_seq=2 ttl=64 time=0.392 ms
[root@mysql1 cmake-2.8.6]# ping mysql3
PING mysql3 (20.0.0.25) 56(84) bytes of data.
64 bytes from mysql3 (20.0.0.25): icmp_seq=1 ttl=64 time=0.317 ms
64 bytes from mysql3 (20.0.0.25): icmp_seq=2 ttl=64 time=0.338 ms
- Modify Master's main configuration file /etc/my.cnf file, the server-id of the three servers cannot be the same
mysql 1：
[root@Mysql1 ~]# cat /etc/my.cnf
#default-character-set=utf8 ## 这句话注释掉，不然健康检查过不去
[mysqld]
server-id = 1
log_bin = master-bin
log-slave-updates = true ## 允许从服务器同步
==》》wq
[root@mysql1 ~]# systemctl restart mysqld
mysql 2：
配置从服务器1：
在/etc/my.cnf 中修改或者增加下面内容。
[root@Mysql2 ~]# vim /etc/my.cnf
#default-character-set=utf8 ## 这句话注释掉，不然健康检查过不去
server-id = 2
log_bin = master-bin
relay-log = relay-log-bin
relay-log-index = slave-relay-bin.index
==》》wq
[root@mysql1 ~]# systemctl restart mysqld
mysql 3：
配置从服务器2：
在/etc/my.cnf 中修改或者增加下面内容。
[root@Mysql2 ~]# vim /etc/my.cnf
#default-character-set=utf8 ## 这句话注释掉，不然健康检查过不去
server-id = 3
log_bin = master-bin
relay-log = relay-log-bin
relay-log-index = slave-relay-bin.index
==》》wq
[root@mysql1 ~]# systemctl restart mysqld
- Mysql1, Mysql2, Mysql3 make two soft links respectively
[root@Mysql1 ~]# ln -s /usr/local/mysql/bin/mysql /usr/sbin/ ## 不建立软链接，系统识别不了，健康检查会报错
[root@Mysql1 ~]# ln -s /usr/local/mysql/bin/mysqlbinlog /usr/sbin/
- Configure MySQL one master and two slaves
① The MySQL master-slave configuration is relatively simple. What needs attention is authorization. Proceed as follows:
在所有数据库节点上授权两个用户，一个是从库同步使用，另外一个是 manager 使用。
mysql> grant replication slave on *.* to 'myslave'@'20.0.0.%' identified by '123'; ## 主从复制授权
mysql> grant all privileges on *.* to 'mha'@'20.0.0.%' identified by 'manager'; ## 三个数据库可以被manger所管理
mysql> flush privileges;
② View binary files and synchronization points on the Mysql1 host
mysql> show master status;
+-------------------+----------+--------------+------------------+-------------------+
| File | Position | Binlog_Do_DB | Binlog_Ignore_DB | Executed_Gtid_Set |
+-------------------+----------+--------------+------------------+-------------------+
| master-bin.000001 | 895 | | | |
+-------------------+----------+--------------+------------------+-------------------+
③ Next, perform synchronization in Mysql2 and Mysql3 respectively.
mysql> change master to master_host='20.0.0.23',master_user='myslave',master_password='123',master_log_file='master-bin.000001',master_log_pos=895;
mysql> start slave;
④ Check whether the IO and SQL threads are both yes to indicate whether the synchronization is normal.
View in mysql 2 and mysql 3 respectively
mysql> show slave status\G;
Slave_IO_Running: Yes
Slave_SQL_Running: Yes
⑤ Two slave libraries must be set to read-only mode:
mysql> set global read_only=1;
mysql> flush privileges;
2.4.3, install MHA software
- Create a live network source (if the YUM warehouse installed a local source at that time, you must create a live network source)
[root@mysql1 ~]# cd /etc/yum.repos.d/
[root@mysql1 yum.repos.d]# mv backup/CentOS-Base.repo ./
[root@mysql1 yum.repos.d]# ll
total 8
drwxr-xr-x. 2 root root 163 Oct 21 02:21 backup
-rw-r--r--. 1 root root 1664 Nov 23 2018 CentOS-Base.repo
-rw-r--r--. 1 root root 116 Sep 13 21:35 local.repo
[root@mysql1 yum.repos.d]# yum clean all
[root@mysql1 yum.repos.d]# yum makecache
- The MHA-dependent environment is installed on all 4 servers, first install the epel source.
[root@MHA-manager ~]# yum install epel-release --nogpgcheck -y
yum install -y perl-DBD-MySQL \
perl-Config-Tiny \
perl-Log-Dispatch \
perl-Parallel-ForkManager \
perl-ExtUtils-CBuilder \
perl-ExtUtils-MakeMaker \
perl-CPAN
- The MHA software package is different for each operating system version. Here, CentOS7.4 must choose the 0.57 version, and the
node component must be installed on <Note: All servers>, and finally the manager component is installed on the MHA-manager node,
because the manager depends on node Components, the following are all operations on Mysql1 to demonstrate the installation of node components.
[root@Mysql1 ~]# tar zxvf mha4mysql-node-0.57.tar.gz
[root@Mysql1 ~]# cd mha4mysql-node-0.57
[root@mysql1 mha4mysql-node-0.57]# ls
AUTHORS COPYING inc Makefile.PL META.yml rpm
bin debian lib MANIFEST README t
[root@Mysql1 mha4mysql-node-0.57]# perl Makefile.PL
[root@Mysql1 mha4mysql-node-0.57]# make && make install
- Install the manager component on MHA-manager (note: the node component must be installed before the manager component can be installed)
[root@MHA-manager ~]# tar zxvf mha4MHA-manager-0.57.tar.gz
[root@MHA-manager ~]# cd mha4MHA-manager-0.57
[root@localhost mha4mysql-manager-0.57]# ls
AUTHORS COPYING inc Makefile.PL META.yml rpm t
bin debian lib MANIFEST README samples tests
[root@MHA-manager mha4MHA-manager-0.57]# perl Makefile.PL
[root@MHA-manager mha4MHA-manager-0.57]# make && make install
- Configure passwordless authentication
1. 在 manager 上配置到所有数据库节点的无密码认证
[root@MHA-manager ~]# ssh-keygen -t rsa //一路按回车键
[root@MHA-manager ~]# ssh-copy-id 20.0.0.23 ## 按yes，然后输入 root 密码
[root@MHA-manager ~]# ssh-copy-id 20.0.0.24
[root@MHA-manager ~]# ssh-copy-id 20.0.0.25
2. 在 Mysql1 上配置到数据库节点Mysql2和Mysql3的无密码认证
[root@Mysql1 ~]# ssh-keygen -t rsa //一路按回车键
[root@Mysql1 ~]# ssh-copy-id 20.0.0.24 ## 按yes，然后输入 root 密码
[root@Mysql1 ~]# ssh-copy-id 20.0.0.25
3. 在 Mysql2 上配置到数据库节点Mysql1和Mysql3的无密码认证
[root@Mysql2 ~]# ssh-keygen -t rsa //一路按回车键
[root@Mysql2 ~]# ssh-copy-id 20.0.0.23 ## 按yes，然后输入 root 密码
[root@Mysql2 ~]# ssh-copy-id 20.0.0.25
4. 在 Mysql3 上配置到数据库节点Mysql1和Mysql2的无密码认证
[root@Mysql3 ~]# ssh-keygen -t rsa //一路按回车键
[root@Mysql3 ~]# ssh-copy-id 20.0.0.23 ## 按yes，然后输入 root 密码
[root@Mysql3 ~]# ssh-copy-id 20.0.0.24
- Configure MHA
- Copy the relevant scripts on the manager node to the /usr/local/bin directory.
[root@MHA-manager ~]# cp -ra /root/mha4mysql-manager-0.57/samples/scripts /usr/local/bin
//拷贝后会有四个执行文件
[root@atlas ~]# ll /usr/local/bin/scripts/
总用量 32
-rwxr-xr-x 1 mysql mysql 3648 5 月 31 2015 master_ip_failover #自动切换时 VIP 管理的脚本
-rwxr-xr-x 1 mysql mysql 9872 5 月 25 09:07 master_ip_online_change #在线切换时 vip 的管理
-rwxr-xr-x 1 mysql mysql 11867 5 月 31 2015 power_manager #故障发生后关闭主机的脚本
-rwxr-xr-x 1 mysql mysql 1360 5 月 31 2015 send_report #因故障切换后发送报警的脚本
- Copy the above-mentioned VIP management script during automatic switching to the /usr/local/bin directory, where the script is used to manage VIP,
[root@MHA-manager ~]# cp /usr/local/bin/scripts/master_ip_failover /usr/local/bin
- The revised content is as follows: (delete the original content and copy it directly)
[root@MHA-manager ~]#vim /usr/local/bin/master_ip_failover
#!/usr/bin/env perl
use strict;
use warnings FATAL => 'all';
use Getopt::Long;
my (
$command, $ssh_user, $orig_master_host, $orig_master_ip,
$orig_master_port, $new_master_host, $new_master_ip, $new_master_port
);
#############################添加内容部分#########################################
my $vip = '192.168.195.200';
my $brdc = '192.168.195.255';
my $ifdev = 'ens33';
my $key = '1';
my $ssh_start_vip = "/sbin/ifconfig ens33:$key $vip";
my $ssh_stop_vip = "/sbin/ifconfig ens33:$key down";
my $exit_code = 0;
##################################################################################
GetOptions(
'command=s' => \$command,
'ssh_user=s' => \$ssh_user,
'orig_master_host=s' => \$orig_master_host,
'orig_master_ip=s' => \$orig_master_ip,
'orig_master_port=i' => \$orig_master_port,
'new_master_host=s' => \$new_master_host,
'new_master_ip=s' => \$new_master_ip,
'new_master_port=i' => \$new_master_port,
);
exit &main();
sub main {
print "\n\nIN SCRIPT TEST====$ssh_stop_vip==$ssh_start_vip===\n\n";
if ( $command eq "stop" || $command eq "stopssh" ) {
my $exit_code = 1;
eval {
print "Disabling the VIP on old master: $orig_master_host \n";
&stop_vip();
$exit_code = 0;
};
if ($@) {
warn "Got Error: $@\n";
exit $exit_code;
}
exit $exit_code;
}
elsif ( $command eq "start" ) {
my $exit_code = 10;
eval {
print "Enabling the VIP - $vip on the new master - $new_master_host \n";
&start_vip();
$exit_code = 0;
};
if ($@) {
warn $@;
exit $exit_code;
}
exit $exit_code;
}
elsif ( $command eq "status" ) {
print "Checking the Status of the script.. OK \n";
exit 0;
}
else {
&usage();
exit 1;
}
}
sub start_vip() {
`ssh $ssh_user\@$new_master_host \" $ssh_start_vip \"`;
}
# A simple system call that disable the VIP on the old_master
sub stop_vip() {
`ssh $ssh_user\@$orig_master_host \" $ssh_stop_vip \"`;
}
sub usage {
print
"Usage: master_ip_failover --command=start|stop|stopssh|status --orig_master_host=host --orig_master_ip=ip --orig_master_port=port --new_master_host=host --new_master_ip=ip --new_master_port=port\n";
}
①、：% s/^#// 把全文的开头的#号删除
②、 第一行开头加上#号 #!/usr/bin/env perl
③、my $vip = '20.0.0.200'; ## 漂移地址，改成自己网段的IP
my $brdc = '20.0.0.255'; ## 改成自己网段
- Create MHA software directory and copy configuration files
[root@MHA-manager ~]# mkdir /etc/masterha
[root@MHA-manager ~]# cp /root/mha4mysql-manager-0.57/samples/conf/app1.cnf /etc/masterha/
[root@MHA-manager ~]# vim /etc/masterha/app1.cnf
[server default]
manager_log=/var/log/masterha/app1/manager.log
manager_workdir=/var/log/masterha/app1
master_binlog_dir=/usr/local/mysql/data
master_ip_failover_script=/usr/local/bin/master_ip_failover
master_ip_online_change_script=/usr/local/bin/master_ip_online_change
password=manager
ping_interval=1
remote_workdir=/tmp
repl_password=123
repl_user=myslave
secondary_check_script=/usr/local/bin/masterha_secondary_check -s 20.0.0.24 -s 20.0.0.25
shutdown_script=""
ssh_user=root
user=mha
[server1]
hostname=20.0.0.23
port=3306
[server2]
candidate_master=1
check_repl_delay=0
hostname=20.0.0.24
port=3306
[server3]
hostname=20.0.0.25
port=3306
- Test ssh without password authentication, if it is normal, it will output successfully, as shown below.
[root@MHA-manager ~]# masterha_check_ssh -conf=/etc/masterha/app1.cnf
Tue Nov 26 23:09:46 2019 - [debug] Connecting via SSH from [email protected](192.168.195.131:22) to [email protected](192.168.195.130:22)..
Tue Nov 26 23:09:47 2019 - [debug] ok.
Tue Nov 26 23:09:47 2019 - [info] All SSH connection tests passed successfully.
- Test health status
root@MHA-manager ~]# masterha_check_repl -conf=/etc/masterha/app1.cnf ## 测试健康状态
Checking the Status of the script.. OK
Tue Nov 26 23:10:29 2019 - [info] OK.
Tue Nov 26 23:10:29 2019 - [warning] shutdown_script is not defined.
Tue Nov 26 23:10:29 2019 - [info] Got exit code 0 (Not master dead).
MySQL Replication Health is OK.
- Note: The first configuration needs to go to the master server master to manually open the virtual IP
[root@Mysql1 ~]# /sbin/ifconfig ens33:1 20.0.0.200/24
[root@Mysql1 ~]# ifconfig 查看一下
如果是最小化安装需要先安装网络组件
[root@mysql1 ~]# yum -y install net-tools
[root@Mysql1 ~]# ifconfig
ens33:1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 20.0.0.200 netmask 255.0.0.0 broadcast 20.255.255.255
ether 00:0c:29:a4:3e:ec txqueuelen 1000 (Ethernet)
- Start MHA, on the manager server
[root@MHA-manager ~]# nohup masterha_manager --conf=/etc/masterha/app1.cnf --remove_dead_master_conf --ignore_last_failover < /dev/null > /var/log/masterha/app1/manager.log 2>&1 &
[1] 129929
[root@localhost ~]# jobs ## 查看后台运行的任务
[1]+ Running nohup masterha_manager --conf=/etc/masterha/app1.cnf --remove_dead_master_conf --ignore_last_failover < /dev/null > /var/log/masterha/app1/manager.log 2>&1 &
- Check the MHA status, you can see that the current master is the Mysql1 node
[root@MHA-manager ~]# masterha_check_status --conf=/etc/masterha/app1.cnf
app1 (pid:32493) is running(0:PING_OK), master:20.0.0.23 ## master是20.0.0.23，对的
- Check the MHA log to see that the current master is 20.0.0.23
[root@MHA-manager ~]# cat /var/log/masterha/app1/manager.log
Tue Nov 26 23:12:30 2019 - [info] Checking master_ip_failover_script status:
Tue Nov 26 23:12:30 2019 - [info] /usr/local/bin/master_ip_failover --command=status --ssh_user=root --orig_master_host=20.0.0.23 --orig_master_ip=20.0.0.23 --orig_master_port=3306
2.5, verification
- Let's close the main database first and check whether the drift address drifts
[root@MHA-manager ~]#tailf /var/log/masterha/app1/manager.log ## 启用监控观察日志记录
[root@mysql1 ~]# systemctl stop mysqld ## 关闭主数据库mysql1上的数据库
- Now we go to the Manger to check the log to see if the drifting address is successfully drifted
我们可以在日志里面看到漂移地址已经成功
20.0.0.25(20.0.0.25:3306): OK: Applying all logs succeeded. Slave started, replicating from 20.0.0.24(20.0.0.24:3306)
我们现在去mysql2查看一下网卡配置信息，漂移地址已经漂移成功！
[root@mysql2 ~]# ifconfig
ens33:1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 20.0.0.200 netmask 255.0.0.0 broadcast 20.255.255.255
ether 00:0c:29:a4:3e:ec txqueuelen 1000 (Ethernet)
再查看一下mysql 1的网卡配置信息
[root@mysql2 ~]# ifconfig
已经没有 ens33:1 虚拟接口信息