Linux操作文档——MySQL搭建MHA高可用（5.7.26）

---

一、MHA简述

MHA（Master High Availability）目前在MySQL高可用方面是一个相对成熟的解决方案，它由日本DeNA公司youshimaton（现就职于Facebook公司）开发，是一套优秀的作为MySQL高可用性环境下故障切换和主从提升的高可用软件。在MySQL故障切换过程中，MHA能做到在0~30秒之内自动完成数据库的故障切换操作，并且在进行故障切换的过程中，MHA能在最大程度上保证数据的一致性，以达到真正意义上的高可用。 MHA里有两个角色一个是MHA Node（数据节点）另一个是MHA Manager（管理节点）。 MHA Manager可以单独部署在一台独立的机器上管理多个master-slave集群，也可以部署在一台slave节点上。MHA Node运行在每台MySQL服务器上，MHA Manager会定时探测集群中的master节点，当master出现故障时，它可以自动将最新数据的slave提升为新的master，然后将所有其他的slave重新指向新的master。整个故障转移过程对应用程序完全透明。

一般来说最少需要MySQL一主二从，管理节点可以放在单独的服务器也可以放在一台从库服务器上。
建议使用MySQL5.7以上的较新版本，CentOS7或Debian9以上版本。
需要注意的是MHA版本之间有差异，mha4mysql-node-0.56-0.el6.noarch.rpm需要跟MySQL5.6配合使用。mha4mysql-node-0.58-0.el7.noarch.rpm跟MySQL5.7以上版本配合使用。

manager端	node端
mysql-03（从）	mysql-01（主），mysql-02（从），mysql-03（从）

二、搭建MHA

manager工具包	说明
masterha_manger	启动MHA
masterha_check_ssh	检查MHA的SSH配置状况
masterha_check_repl	检查MySQL复制状况
masterha_master_monitor	检测master是否宕机
masterha_check_status	检测当前MHA运行状态
masterha_master_switch	控制故障转移（自动或者手动）
masterha_conf_host	添加或删除配置的server信息

node工具包	说明
save_binary_logs	保存和复制master的二进制日志
apply_diff_relay_logs	识别差异的中继日志事件并将其差异的事件应用于其他的
purge_relay_logs	清除中继日志（不会阻塞SQL线程）

1、全局设置（三台都需要）

1、创建软连接

[root@mysql-1 ~]# ln -s /usr/local/mysql/bin/mysqlbinlog /usr/bin/mysqlbinlog
[root@mysql-1 ~]# ln -s /usr/local/mysql/bin/mysql  /usr/bin/mysql

2、开启无密码传送

[root@mysql-1 ~]# ssh-keygen -t rsa
[root@mysql-1 ~]# ssh-copy-id 192.168.1.10:
[root@mysql-1 ~]# ssh-copy-id 192.168.1.20:
[root@mysql-1 ~]# ssh-copy-id 192.168.1.30:

3、安装软件包

[root@mysql-1 ~]# yum -y install perl-DBD-MySQL
[root@mysql-1 ~]# wget https://github.com/yoshinorim/mha4mysql-node/releases/download/v0.58/mha4mysql-node-0.58-0.el7.centos.noarch.rpm
[root@mysql-1 ~]# scp mha4mysql-node-0.58-0.el7.centos.noarch.rpm 192.168.1.20:
[root@mysql-1 ~]# scp mha4mysql-node-0.58-0.el7.centos.noarch.rpm 192.168.1.30:
[root@mysql-1 ~]# rpm -ivh mha4mysql-node-0.58-0.el7.centos.noarch.rpm 

2、在主库中创建用户

[root@mysql-1 ~]# mysql
db01 [(none)]>grant all privileges on *.* to mha@'192.168.1.%' identified by 'mha';

3、管理节点安装Manager软件

[root@mysql-3 ~]# wget -O /etc/yum.repos.d/epel.repo http://mirrors.aliyun.com/repo/epel-7.repo
[root@mysql-3 ~]# yum repolist 
[root@mysql-3 ~]# yum makecache fast
[root@mysql-3 ~]# yum -y install perl-Config-Tiny epel-release perl-Log-Dispatch perl-Parallel-ForkManager perl-Time-HiRes
[root@mysql-3 ~]# wget https://github.com/yoshinorim/mha4mysql-manager/releases/download/v0.58/mha4mysql-manager-0.58-0.el7.centos.noarch.rpm
[root@mysql-3 ~]# rpm -ivh mha4mysql-manager-0.58-0.el7.centos.noarch.rpm

4、准备配置文件

[root@mysql-3 ~]# mkdir -p /etc/mha            //创建配置文件目录
[root@mysql-3 ~]# mkdir -p /var/log/mha/app1            //创建日志目录
[root@mysql-3 ~]# cat > /etc/mha/app1.cnf <<EOF            //编辑mha配置文件
[server default]
manager_log=/var/log/mha/app1/manager
manager_workdir=/var/log/mha/app1
#二进制日志目录
master_binlog_dir=/data/binlog
#mha用户信息
user=mha
password=mha
#探测心跳的间隔时间（秒），默认探测3次
ping_interval=2
#主从的用户信息
repl_user=repl
repl_password=123
ssh_user=root
[server1]                                   
hostname=192.168.1.10
port=3306                                  
[server2]            
hostname=192.168.1.20
port=3306
[server3]
hostname=192.168.1.30
port=3306
EOF

5、状态检查并开启MHA

[root@mysql-3 ~]# masterha_check_ssh  --conf=/etc/mha/app1.cnf
Mon Oct 19 21:19:07 2020 - [debug]   ok.
Mon Oct 19 21:19:07 2020 - [info] All SSH connection tests passed successfully.
[root@mysql-3 ~]# masterha_check_repl  --conf=/etc/mha/app1.cnf 
MySQL Replication Health is OK.
[root@mysql-3 ~]# nohup masterha_manager --conf=/etc/mha/app1.cnf --remove_dead_master_conf --ignore_last_failover  < /dev/null> /var/log/mha/app1/manager.log 2>&1 &
[root@mysql-3 ~]# masterha_check_status --conf=/etc/mha/app1.cnf
app1 (pid:69402) is running(0:PING_OK), master:192.168.1.10

参数	说明
–conf	指定配置文件
–remove_dead_master_conf	当主库宕机时自动将故障节点从配置文件中去除
–ignore_last_failover	忽略最后一次的切换

三、应用透明

1、准备所需脚本

[root@mysql-3 ~]# vim master_ip_failover
#!/usr/bin/env perl
use strict;
use warnings FATAL => 'all';

use Getopt::Long;

my (
    $command,          $ssh_user,        $orig_master_host, $orig_master_ip,
    $orig_master_port, $new_master_host, $new_master_ip,    $new_master_port
);

my $vip = '192.168.1.100/24';  # Virtual IP 
my $key = "1";
my $ssh_start_vip = "/sbin/ifconfig ens33:$key $vip";
my $ssh_stop_vip = "/sbin/ifconfig ens33:$key down";

GetOptions(
    'command=s'          => \$command,
    'ssh_user=s'         => \$ssh_user,
    'orig_master_host=s' => \$orig_master_host,
    'orig_master_ip=s'   => \$orig_master_ip,
    'orig_master_port=i' => \$orig_master_port,
    'new_master_host=s'  => \$new_master_host,
    'new_master_ip=s'    => \$new_master_ip,
    'new_master_port=i'  => \$new_master_port,
);

exit &main();

sub main {
    
    

    print "\n\nIN SCRIPT TEST====$ssh_stop_vip==$ssh_start_vip===\n\n"; 

    if ( $command eq "stop" || $command eq "stopssh" ) {
    
    

        # $orig_master_host, $orig_master_ip, $orig_master_port are passed.
        # If you manage master ip address at global catalog database,
        # invalidate orig_master_ip here.
        my $exit_code = 1;
        eval {
    
    
            print "Disabling the VIP on old master: $orig_master_host \n";
            &stop_vip();
            $exit_code = 0;
        };
        if ($@) {
    
    
            warn "Got Error: $@\n";
            exit $exit_code;
        }
        exit $exit_code;
    }
    elsif ( $command eq "start" ) {
    
    

        # all arguments are passed.
        # If you manage master ip address at global catalog database,
        # activate new_master_ip here.
        # You can also grant write access (create user, set read_only=0, etc) here.
        my $exit_code = 10;
        eval {
    
    
            print "Enabling the VIP - $vip on the new master - $new_master_host \n";
            &start_vip();
            $exit_code = 0;
        };
        if ($@) {
    
    
            warn $@;
            exit $exit_code;
        }
        exit $exit_code;
    }
    elsif ( $command eq "status" ) {
    
    
        print "Checking the Status of the script.. OK \n"; 
        `ssh $ssh_user\@$orig_master_host \" $ssh_start_vip \"`;
        exit 0;
    }
    else {
    
    
        &usage();
        exit 1;
    }
}

# A simple system call that enable the VIP on the new master 
sub start_vip() {
    
    
    `ssh $ssh_user\@$new_master_host \" $ssh_start_vip \"`;
}
# A simple system call that disable the VIP on the old_master
sub stop_vip() {
    
    
    `ssh $ssh_user\@$orig_master_host \" $ssh_stop_vip \"`;
}

sub usage {
    
    
    print
    "Usage: master_ip_failover --command=start|stop|stopssh|status --orig_master_host=host --orig_master_ip=ip --orig_master_port=port --new_master_host=host --new_master_ip=ip --new_master_port=port\n";
}
[root@mysql-3 ~]# cp master_ip_failover /usr/local/bin/master_ip_failover
[root@mysql-3 ~]# cd /usr/local/bin/
[root@mysql-3 bin]# vim master_ip_failover
my $vip = '192.168.1.100/24';
my $key = "1";
my $ssh_start_vip = "/sbin/ifconfig ens33:$key $vip";
my $ssh_stop_vip = "/sbin/ifconfig ens33:$key down";
[root@mysql-3 ~]# chmod +x /usr/local/bin/master_ip_failover
[root@mysql-3 ~]# yum -y install dos2unix
[root@mysql-3 ~]# dos2unix /usr/local/bin/master_ip_failover

2、更改配置文件

[root@mysql-3 ~]# vim /etc/mha/app1.cnf 
[server default]
manager_log=/var/log/mha/app1/manager
manager_workdir=/var/log/mha/app1
master_binlog_dir=/data/binlog
user=mha
password=mha
ping_interval=2
repl_password=123
repl_user=repl
ssh_user=root
master_ip_failover_script=/usr/local/bin/master_ip_failover
[server1]
hostname=192.168.1.10
port=3306
[server2]
hostname=192.168.1.20
port=3306
[server3]
hostname=192.168.1.30
port=3306

3、主库手工添加vip

[root@mysql-1 ~]# ifconfig ens33:1 192.168.1.100/24

4、重新启动MHA

[root@mysql-3 ~]# masterha_stop --conf=/etc/mha/app1.cnf
Stopped app1 successfully.
[1]+  退出 1                nohup masterha_manager --conf=/etc/mha/app1.cnf --remove_dead_master_conf --ignore_last_failover < /dev/null > /var/log/mha/app1/manager.log 2>&1
[root@mysql-3 ~]# nohup masterha_manager --conf=/etc/mha/app1.cnf --remove_dead_master_conf --ignore_last_failover  < /dev/null> /var/log/mha/app1/manager.log 2>&1 &
[1] 17601
[root@mysql-3 ~]# masterha_check_status --conf=/etc/mha/app1.cnf
app1 (pid:17601) is running(0:PING_OK), master:192.168.1.10

四、数据补偿

1、准备备份数据库配置文件

[root@mysql-3 ~]# vim /etc/mha/app1.cnf 
[server default]
manager_log=/var/log/mha/app1/manager
manager_workdir=/var/log/mha/app1
master_binlog_dir=/data/binlog
user=mha
password=mha
ping_interval=2
repl_password=123
repl_user=repl
ssh_user=root
master_ip_failover_script=/usr/local/bin/master_ip_failover
[server1]
hostname=192.168.1.10
port=3306
[server2]
hostname=192.168.1.20
port=3306
[server3]
hostname=192.168.1.30
port=3306
[binlog1]
no_master=1
hostname=192.168.1.40
master_binlog_dir=/data/mysql/binlog

2、创建目录

[root@mysql-3 ~]# mkdir -p /data/mysql/binlog
[root@mysql-3 ~]# chown -R mysql.mysql /data/*

3、拉取主库binlog日志

[root@mysql-3 ~]# cd /data/mysql/binlog
[root@mysql-3 binlog]# mysqlbinlog  -R --host=192.168.1.10 --user=mha --password=mha --raw  --stop-never mysql-bin.000001 &
[2] 35693
[root@mysql-3 binlog]# ls
mysql-bin.000001  mysql-bin.000002
[root@mysql-3 binlog]# ll
总用量 8
-rw-r----- 1 root root 177 10月 23 18:51 mysql-bin.000001
-rw-r----- 1 root root 736 10月 23 18:51 mysql-bin.000002

拉取日志的起点，需要按照目前主库正在使用的binlog为起点

mysql01 [(none)]>show master status;
+------------------+----------+--------------+------------------+------------------------------------------+
| File             | Position | Binlog_Do_DB | Binlog_Ignore_DB | Executed_Gtid_Set                        |
+------------------+----------+--------------+------------------+------------------------------------------+
| mysql-bin.000002 |      736 |              |                  | 72f32e7d-150b-11eb-afeb-000c2979853f:1-2 |
+------------------+----------+--------------+------------------+------------------------------------------+

五、MHA故障处理

1、启动故障节点

[root@mysql-1 ~]# systemctl restart mysqld

2、恢复主从

1、查看当前主库信息

[root@mysql-3 ~]# grep "CHANGE MASTER TO"  /var/log/mha/app1/manager
Thu Jul 18 18:31:54 2019 - [info]  All other slaves should start replication from here. Statement should be: CHANGE MASTER TO MASTER_HOST='192.168.1.20', MASTER_PORT=3306, MASTER_AUTO_POSITION=1, MASTER_USER='repl', MASTER_PASSWORD='xxx';

2、故障节点加入主库

mysql01 [(none)]>CHANGE MASTER TO 
	MASTER_HOST='192.168.1.20', 
	MASTER_PORT=3306, 
	MASTER_AUTO_POSITION=1, 
	MASTER_USER='repl', 
	MASTER_PASSWORD='123';

db01 [(none)]>start slave;

3、MHA管理节点恢复配置文件

[root@mysql-3 ~]# vim /etc/mha/app1.cnf 
[server1]
hostname=192.168.1.10
port=3306
[server2]
hostname=192.168.1.20
port=3306
[server3]
hostname=192.168.1.30
port=3306
[root@db03 bin]# nohup masterha_manager --conf=/etc/mha/app1.cnf --remove_dead_master_conf --ignore_last_failover  < /dev/null> /var/log/mha/app1/manager.log 2>&1 &

4、重新启动MHA并恢复数据补偿

[root@mysql-3 ~]# masterha_check_status --conf=/etc/mha/app1.cnf
app1 (pid:16543) is running(0:PING_OK), master:192.168.1.20
[root@mysql-3 ~]# cd /data/mysql/binlog    
[root@mysql-3 ~]# rm -rf /data/mysql/binlog/*
[root@mysql-3 ~]# cd /data/mysql/binlog
[root@mysql-3 binlog]# mysqlbinlog -R --host=192.168.1.20 --user=mha --password=mha --raw  --stop-never mysql-bin.000001 &

六、MHA配合Atlas实现读写分离

Atlas 介绍
Atlas是由 Qihoo 360, Web平台部基础架构团队开发维护的一个基于MySQL协议的数据中间层项目。
它是在mysql-proxy 0.8.2版本的基础上，对其进行了优化，增加了一些新的功能特性。
360内部使用Atlas运行的mysql业务，每天承载的读写请求数达几十亿条。
高并发下不稳定。
下载地址
https://github.com/Qihoo360/Atlas/releases

注意：
1、Atlas只能安装运行在64位的系统上
2、Centos 5.X安装 Atlas-XX.el5.x86_64.rpm，Centos 6.X安装Atlas-XX.el6.x86_64.rpm。
3、后端mysql版本应大于5.1，建议使用Mysql 5.6以上

1、安装配置

[root@mysql-3 ~]# wget https://github.com/Qihoo360/Atlas/releases/download/2.2.1/Atlas-2.2.1.el6.x86_64.rpm
[root@mysql-3 ~]# yum -y install Atlas-2.2.1.el6.x86_64.rpm
[root@mysql-3 ~]# cd /usr/local/mysql-proxy/conf
[root@mysql-3 conf]# mv test.cnf test.cnf.bak
[root@mysql-3 conf]# vim test.cnf
[mysql-proxy]
admin-username = user
admin-password = pwd
proxy-backend-addresses = 192.168.1.100:3306
proxy-read-only-backend-addresses = 192.168.1.20:3306,192.168.1.30:3306
pwds = repl:3yb5jEku5h4=,mha:O2jBXONX098=
daemon = true
keepalive = true
event-threads = 8
log-level = message
log-path = /usr/local/mysql-proxy/log
sql-log=ON
proxy-address = 0.0.0.0:33060
admin-address = 0.0.0.0:2345
charset=utf8
[root@mysql-3 ~]# /usr/local/mysql-proxy/bin/mysql-proxyd test start
OK: MySQL-Proxy of test is started
[root@mysql-3 ~]# ps -ef |grep proxy
root      42818      1  0 22:44 ?        00:00:00 /usr/local/mysql-proxy/bin/mysql-proxy --defaults-file=/usr/local/mysql-proxy/conf/test.cnf
root      42819  42818  0 22:44 ?        00:00:00 /usr/local/mysql-proxy/bin/mysql-proxy --defaults-file=/usr/local/mysql-proxy/conf/test.cnf
root      42832  16804  0 22:44 pts/0    00:00:00 grep --color=auto proxy
[root@mysql-3 ~]# mysql -umha -pmha  -h 192.168.1.30 -P 33060
mysql03 [(none)]>select @@server_id;
+-------------+
| @@server_id |
+-------------+
|          52 |
+-------------+
1 row in set (0.03 sec)

mysql03 [(none)]>select @@server_id;
+-------------+
| @@server_id |
+-------------+
|          53 |
+-------------+
1 row in set (0.01 sec)

mysql03 [(none)]>begin;select @@server_id;commit;
Query OK, 0 rows affected (0.00 sec)

+-------------+
| @@server_id |
+-------------+
|          51 |
+-------------+
1 row in set (0.00 sec)

Query OK, 0 rows affected (0.00 sec)

2、添加生产用户

1、查看主库

[root@mysql-3 ~]# masterha_check_status --conf=/etc/mha/app1.cnf
app1 (pid:35587) is running(0:PING_OK), master:192.168.1.10

2、主库添加生产用户

[root@mysql-1 ~]# mysql
mysql01 [(none)]>grant select ,update,insert on *.* to zhangsan@'192.168.1.%' identified by '123456';

3、Atlas添加用户

[root@mysql-3 ~]# /usr/local/mysql-proxy/bin/encrypt 123456      //加密密码
/iZxz+0GRoA=
[root@mysql-3 ~]# vim /usr/local/mysql-proxy/conf/test.cnf
pwds = repl:3yb5jEku5h4=,mha:O2jBXONX098=,zhangsan:/iZxz+0GRoA=
[root@mysql-3 ~]# /usr/local/mysql-proxy/bin/mysql-proxyd test restart
OK: MySQL-Proxy of test is stopped
OK: MySQL-Proxy of test is started
[root@mysql-3 ~]# mysql -uzhangsan -p123456  -h 192.168.1.30 -P 33060
mysql03 [(none)]>

3、 Atlas基本管理

1、连接管理接口

[root@mysql-3 ~]# mysql -uuser -ppwd -h127.0.0.1 -P2345
mysql03 [(none)]>select * from help;
+----------------------------+---------------------------------------------------------+
| command                    | description                                             |
+----------------------------+---------------------------------------------------------+
| SELECT * FROM help         | shows this help                                         |
| SELECT * FROM backends     | lists the backends and their state                      |
| SET OFFLINE $backend_id    | offline backend server, $backend_id is backend_ndx's id |
| SET ONLINE $backend_id     | online backend server, ...                              |
| ADD MASTER $backend        | example: "add master 127.0.0.1:3306", ...               |
| ADD SLAVE $backend         | example: "add slave 127.0.0.1:3306", ...                |
| REMOVE BACKEND $backend_id | example: "remove backend 1", ...                        |
| SELECT * FROM clients      | lists the clients                                       |
| ADD CLIENT $client         | example: "add client 192.168.1.2", ...                  |
| REMOVE CLIENT $client      | example: "remove client 192.168.1.2", ...               |
| SELECT * FROM pwds         | lists the pwds                                          |
| ADD PWD $pwd               | example: "add pwd user:raw_password", ...               |
| ADD ENPWD $pwd             | example: "add enpwd user:encrypted_password", ...       |
| REMOVE PWD $pwd            | example: "remove pwd user", ...                         |
| SAVE CONFIG                | save the backends to config file                        |
| SELECT VERSION             | display the version of Atlas                            |
+----------------------------+---------------------------------------------------------+
16 rows in set (0.00 sec)

2、常用管理命令

1、查看节点

mysql03 [(none)]>SELECT * FROM backends;
+-------------+--------------------+-------+------+
| backend_ndx | address            | state | type |
+-------------+--------------------+-------+------+
|           1 | 192.168.1.100:3306 | up    | rw   |
|           2 | 192.168.1.20:3306  | up    | ro   |
|           3 | 192.168.1.30:3306  | up    | ro   |
+-------------+--------------------+-------+------+

2、下线节点

mysql03 [(none)]>set offline 2;
+-------------+-------------------+---------+------+
| backend_ndx | address           | state   | type |
+-------------+-------------------+---------+------+
|           2 | 192.168.1.20:3306 | offline | ro   |
+-------------+-------------------+---------+------+

3、上线节点

mysql03 [(none)]>set online 2;
+-------------+-------------------+---------+------+
| backend_ndx | address           | state   | type |
+-------------+-------------------+---------+------+
|           2 | 192.168.1.20:3306 | unknown | ro   |
+-------------+-------------------+---------+------+

4、删除节点

mysql03 [(none)]>REMOVE BACKEND 3;

5、添加节点（从）

mysql03 [(none)]>ADD SLAVE 192.168.1.30:3306;

6、添加生产用户

mysql03 [(none)]>ADD PWD lisi:123456;
Empty set (0.00 sec)

mysql03 [(none)]>select * from pwds;
+----------+--------------+
| username | password     |
+----------+--------------+
| repl     | 3yb5jEku5h4= |
| mha      | O2jBXONX098= |
| zhangsan | /iZxz+0GRoA= |
| lisi     | /iZxz+0GRoA= |
+----------+--------------+
4 rows in set (0.00 sec)

7、保存修改

mysql03 [(none)]>save config;
Empty set (0.00 sec)