Corosync+Pacemaker+NFS+Mysql高可用集群部署（使用资源管理工具crmsh配置）
框架：crmsh(Corosync+pacemaker)+nfs+mysql
集群节点1:192.168.88.132 cen7.field.com
集群节点2:192.168.88.133 node2.field.com
集群节点3:192.168.88.135 node3.field.com
vip: 192.168.88.188 资源代理：ocf:heartbeat:IPaddr
nfs服务器：node3.field.com 资源代理：ocf:heartbeat:Filesystem
mysql服务器：cen7.field.com node1.field.com 资源代理：lsb:mysqld
配置集群的前提
（1）、时间同步；
（2）、基于当前正在使用的主机名互相访问；
（3）、是否会用到仲裁设备；
参考 ：《使用资源管理工具pcs配置Corosync+pacemaker》
一、配置NFS服务器：共享目录作为mysql的datadir
该步骤中：主要通过创建逻辑卷并设置开机自动挂载到nfs共享目录，用于存放mysql数据库文件，其所有者必须是mysql用户和mysql组。
1、创建逻辑卷
1）、fdisk创建物理卷分区
[root@node3 ~]# fdisk /dev/sda
欢迎使用 fdisk (util-linux 2.23.2)。
更改将停留在内存中，直到您决定将更改写入磁盘。
使用写入命令前请三思。
命令(输入 m 获取帮助)：p
磁盘 /dev/sda：32.2 GB, 32212254720 字节，62914560 个扇区
Units = 扇区 of 1 * 512 = 512 bytes
扇区大小(逻辑/物理)：512 字节 / 512 字节
I/O 大小(最小/最佳)：512 字节 / 512 字节
磁盘标签类型：dos
磁盘标识符：0x00057c95
设备 Boot Start End Blocks Id System
/dev/sda1 * 2048 1075199 536576 83 Linux
/dev/sda2 1075200 3174399 1049600 82 Linux swap / Solaris
/dev/sda3 3174400 36728831 16777216 83 Linux
命令(输入 m 获取帮助)：n
Partition type:
p primary (3 primary, 0 extended, 1 free)
e extended
Select (default e): p
已选择分区 4
起始 扇区 (36728832-62914559，默认为 36728832)：
将使用默认值 36728832
Last 扇区, +扇区 or +size{K,M,G} (36728832-62914559，默认为 62914559)：+2G
分区 4 已设置为 Linux 类型，大小设为 2 GiB
命令(输入 m 获取帮助)：t
分区号 (1-4，默认 4)：4
Hex 代码(输入 L 列出所有代码)：8e -->说明：用作物理卷的分区id为8e
已将分区“Linux”的类型更改为“Linux LVM”
命令(输入 m 获取帮助)：w
The partition table has been altered!
Calling ioctl() to re-read partition table.
WARNING: Re-reading the partition table failed with error 16: 设备或资源忙.
The kernel still uses the old table. The new table will be used at
the next reboot or after you run partprobe(8) or kpartx(8)
正在同步磁盘。
2）、使用partprobe让内核识别新建的分区
[root@node3 ~]# partprobe /dev/sda
3）、查看新建立的分区
[root@node3 ~]# fdisk -l
磁盘 /dev/sda：32.2 GB, 32212254720 字节，62914560 个扇区
Units = 扇区 of 1 * 512 = 512 bytes
扇区大小(逻辑/物理)：512 字节 / 512 字节
I/O 大小(最小/最佳)：512 字节 / 512 字节
磁盘标签类型：dos
磁盘标识符：0x00057c95
设备 Boot Start End Blocks Id System
/dev/sda1 * 2048 1075199 536576 83 Linux
/dev/sda2 1075200 3174399 1049600 82 Linux swap / Solaris
/dev/sda3 3174400 36728831 16777216 83 Linux
/dev/sda4 36728832 40923135 2097152 8e Linux LVM -->标记为LVM
4）、创建物理卷
[root@node3 ~]# pvcreate /dev/sda4
Physical volume "/dev/sda4" successfully created.
5）、创建卷组
[root@node3 ~]# vgcreate myvg /dev/sda4
Volume group "myvg" successfully created
6）、创建大小为500M的逻辑卷
[root@node3 ~]# lvcreate -L 500M -n lv_mydata myvg
Logical volume "lv_mydata" created.
7）、格式化逻辑卷
[root@node3 ~]# mke2fs -j /dev/myvg/lv_mydata
mke2fs 1.42.9 (28-Dec-2013)
文件系统标签=
OS type: Linux
块大小=1024 (log=0)
分块大小=1024 (log=0)
Stride=0 blocks, Stripe width=0 blocks
128016 inodes, 512000 blocks
25600 blocks (5.00%) reserved for the super user
第一个数据块=1
Maximum filesystem blocks=67633152
63 block groups
8192 blocks per group, 8192 fragments per group
2032 inodes per group
Superblock backups stored on blocks:
8193, 24577, 40961, 57345, 73729, 204801, 221185, 401409
Allocating group tables: 完成
正在写入inode表: 完成
Creating journal (8192 blocks): 完成
Writing superblocks and filesystem accounting information: 完成
2、挂载共享存储
1）、创建挂载的目录/mydata作为nfs共享存储介质
[root@node3 ~]# mkdir /mydata
2）、设置开机自动挂载
[root@node3 ~]# vim /etc/fstab
/dev/myvg/lv_mydata /mydata ext3 defaults 0 0
3）、挂载测试挂载成功与否
[root@node3 ~]# mount -a
[root@node3 ~]# ls /mydata/
lost+found
3、启动nfs服务并配置共享存储
1）、启动nfs服务
[root@node3 ~]# systemctl start nfs.service
[root@node3 ~]# systemctl status nfs.service
● nfs-server.service - NFS server and services
Loaded: loaded (/usr/lib/systemd/system/nfs-server.service; disabled; vendor preset: disabled)
Active: active (exited) since 六 2018-08-04 03:43:47 CST; 6s ago
Process: 1442 ExecStart=/usr/sbin/rpc.nfsd $RPCNFSDARGS (code=exited, status=0/SUCCESS)
Process: 1438 ExecStartPre=/bin/sh -c /bin/kill -HUP `cat /run/gssproxy.pid` (code=exited, status=0/SUCCESS)
Process: 1436 ExecStartPre=/usr/sbin/exportfs -r (code=exited, status=0/SUCCESS)
Main PID: 1442 (code=exited, status=0/SUCCESS)
CGroup: /system.slice/nfs-server.service
8月 04 03:43:47 node3.field.com systemd[1]: Starting NFS server and services...
8月 04 03:43:47 node3.field.com systemd[1]: Started NFS server and services.
2）、添加共享存储文件，用于存放mysql数据的目录
[root@node3 ~]# vim /etc/exports
#/www/hadocs 192.168.88.0/24(rw)
/mydata 192.168.88.0/24(rw,no_root_squash)
#该网段的主机可读写，不需root用户
3）、编辑测试页
[root@node3 ~]# echo "<h1>Test page on NFS Service</h1>" >/mydata/index.html
4）、重启nfs服务并修改共享目录属组属主为mysql，注意：此步骤为必须操作
[root@node3 ~]# systemctl restart nfs.service
[root@node3 ~]# chown -R mysql.mysql /mydata
chown: 无效的用户: "mysql.mysql"
[root@node3 ~]# groupadd -g 306 -r mysql
[root@node3 ~]# useradd -g mysql -u 306 -s /sbin/nologin mysql
[root@node3 ~]# id mysql
uid=306(mysql) gid=306(mysql) 组=306(mysql)
[root@node3 ~]# chown -R mysql.mysql /mydata
二、编译、安装、配置、测试Mariadb
该步骤主要分两大步：cen7节点安装和node2节点安
（一）、cen7节点上的编译、安装、配置、测试Mariadb
1、安装前配置
1）、创建mysql用户、组和挂载目录
[root@cen7 ~]# groupadd -g 306 -r mysql
[root@cen7 ~]# useradd -u 306 -g mysql -r mysql
[root@cen7 ~]# mkdir /mydata
2）、挂载NFS文件系统并确认
[root@cen7 ~]# mount -t nfs 192.168.88.135:/mydata /mydata
[root@cen7 ~]# mount |grep nfs
sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime)
192.168.88.135:/mydata on /mydata type nfs4 (rw,relatime,vers=4.1,rsize=65536,wsize=65536,namlen=255,hard,proto=tcp,port=0,timeo=600,retrans=2,sec=sys,clientaddr=192.168.88.132,local_lock=none,addr=192.168.88.135)
[root@cen7 ~]# ls /mydata
index.html lost+found
3）、以mysql用户身份创建data目录，作为初始化时的数据存放目录（datadir）
注意：mysql初始化只需在集群节点中的一个进行一次，后续集群会自动挂载配置，另一台机mariadb不需初始化，所以不用以mysql用户创建data目录
[root@cen7 ~]# su - mysql
su: 警告：无法更改到 /home/mysql 目录: 没有那个文件或目录
-bash-4.2$ cd /mydata
-bash-4.2$ mkdir data
-bash-4.2$ exit
登出
2、源码编译安装Mariadb
说明：此处采用10.0以上版本可能会因为c++版本低编译无法通过，笔者采用mariadb-5.5.61.tar.gz安装包
[root@cen7 ~]# ls
anaconda-ks.cfg mariadb-5.5.61.tar.gz python-parallax-1.0.0a1-7.1.noarch.rpm
crmsh-3.0.0-6.2.noarch.rpm ~None python-pssh-2.3.1-7.3.noarch.rpm
crmsh-scripts-3.0.0-6.2.noarch.rpm pssh-2.3.1-7.3.noarch.rpm
1）、编译安装mariadb
[root@cen7 ~]# tar xf mariadb-5.5.61.tar.gz -C /usr/local
[root@cen7 ~]# cd /usr/local/
[root@cen7 local]# ln -sv mariadb-5.5.61 mysql
"mysql" -> "mariadb-5.5.61"
[root@cen7 local]# cd mysql
注意：修改文件属主属组为mysql
[root@cen7 mysql]# chown -R mysql:mysql .
[root@cen7 mysql]# cmake . -DCMAKE_INSTALL_PREFIX=/usr/local/mysql -DMYSQL_DATADIR=/mydata/data/ -DSYSCONFDIR=/etc -DWITH_INNOBASE_STORAGE_ENGINE=1 -DWITH_ARCHIVE_STORAGE_ENGINE=1 -DWITH_BLACKHOLE_STORAGE_ENGINE=1 -DWITH_READLINE=1 -DWITH_SSL=system -DWITH_ZLIB=system -DWITH_LIBWRAP=0 -DMYSQL_UNIX_ADDR=/tmp/mysql.sock -DDEFAULT_CHARSET=utf8 -DDEFAULT_COLLATION=utf8_general_ci
[root@cen7 mysql]# make && make install
[root@cen7 mysql]# chown -R mysql:mysql .
2）、初始化mysql数据库：只需在该节点进行初始化，后续会自动挂载。
[root@cen7 mysql]# scripts/mysql_install_db --user=mysql --datadir=/mydata/data
Installing MariaDB/MySQL system tables in '/mydata/data' ...
180803 6:02:53 [Note] ./bin/mysqld (mysqld 5.5.61-MariaDB) starting as process 60647 ...
OK
Filling help tables...
180803 6:02:55 [Note] ./bin/mysqld (mysqld 5.5.61-MariaDB) starting as process 60656 ...
OK
To start mysqld at boot time you have to copy
support-files/mysql.server to the right place for your system
PLEASE REMEMBER TO SET A PASSWORD FOR THE MariaDB root USER !
To do so, start the server, then issue the following commands:
'./bin/mysqladmin' -u root password 'new-password'
'./bin/mysqladmin' -u root -h cen7.field.com password 'new-password'
Alternatively you can run:
'./bin/mysql_secure_installation'
which will also give you the option of removing the test
databases and anonymous user created by default. This is
strongly recommended for production servers.
See the MariaDB Knowledgebase at http://mariadb.com/kb or the
MySQL manual for more instructions.
You can start the MariaDB daemon with:
cd '.' ; ./bin/mysqld_safe --datadir='/mydata/data'
You can test the MariaDB daemon with mysql-test-run.pl
cd './mysql-test' ; perl mysql-test-run.pl
Please report any problems at http://mariadb.org/jira
The latest information about MariaDB is available at http://mariadb.org/.
You can find additional information about the MySQL part at:
http://dev.mysql.com
Consider joining MariaDB's strong and vibrant community:
https://mariadb.org/get-involved/
[root@cen7 mysql]# chown -R root .
3）、复制my-large.cnf为mysql主配置文件并修改
[root@cen7 mysql]# cp support-files/my-large.cnf /etc/my.cnf
cp：是否覆盖"/etc/my.cnf"？ y
[root@cen7 mysql]# vim /etc/my.cnf
#thread_concurrency=CPU*2
thread_concurrency = 6
#指定mysql数据文件的存放位置：
datadir = /mydata/data
4）、配置使用service管理mysqld服务并关闭开机自启动，用CRM控制mysql启停
[root@cen7 mysql]# cp support-files/mysql.server /etc/rc.d/init.d/mysqld
[root@cen7 mysql]# chkconfig --add mysqld
[root@cen7 mysql]# chkconfig mysqld off
[root@cen7 mysql]# service mysqld start
env: /etc/init.d/mysqld: 权限不够
[root@cen7 mysql]# chmod +x /etc/rc.d/init.d/mysqld
3、启动mysql服务并确认状态
[root@cen7 mysql]# service mysqld start
Starting MariaDB.180803 06:08:41 mysqld_safe Logging to '/mydata/data/cen7.field.com.err'.
180803 06:08:41 mysqld_safe Starting mysqld daemon with databases from /mydata/data
. SUCCESS!
[root@cen7 mysql]# ps -ef | grep 3306
mysql 61180 60874 1 06:08 pts/0 00:00:00 /usr/local/mysql/bin/mysqld --basedir=/usr/local/mysql --datadir=/mydata/data --plugin-dir=/usr/local/mysql/lib/plugin --user=mysql --log-error=/mydata/data/cen7.field.com.err --pid-file=/mydata/data/cen7.field.com.pid --socket=/tmp/mysql.sock --port=3306
root 61233 3454 2 06:09 pts/0 00:00:00 grep --color=auto 3306
[root@cen7 mysql]# service mysqld stop
Shutting down MariaDB..... SUCCESS!
4、针对用户操作习惯的更改
使用源码包编译安装mariadb使用方法与用户习惯有所出入，通过创建链接，配置环境变量可以使mysql符合用户操作习惯，此步可以不进行。
1）、输出mysql的man手册至man命令的查找路径
[root@cen7 mysql]# vim /etc/man.config
MANPATH /usr/local/mysql/man
2）、将mysql的头文件链接至系统头文件路径/usr/include：
[root@cen7 mysql]# ln -sv /usr/local/mysql/include /usr/include/mysql
"/usr/include/mysql" -> "/usr/local/mysql/include"
3）、输出mysql的库文件给系统库查找路径
[root@cen7 mysql]# echo '/usr/local/mysql/lib' > /etc/ld.so.conf.d/mysql.conf
4）、让系统重新载入系统库
[root@cen7 mysql]# ldconfig
5）、编辑/etc/profile添加mysql环境，让系统所有用户可以直接使用mysql的相关命令:
[root@cen7 mysql]# vim /etc/profile
MYSQL_HOME=/usr/local/mysql
PATH=$PATH:$MYSQL_HOME/bin
6）、使配置生效
[root@cen7 mysql]# . /etc/profile
5、验证mysql是否安装成功：启动mysql，连接mysql客户端。
[root@cen7 mysql]# service mysqld start
Starting MariaDB.180803 06:12:02 mysqld_safe Logging to '/mydata/data/cen7.field.com.err'.
180803 06:12:02 mysqld_safe Starting mysqld daemon with databases from /mydata/data
.. SUCCESS!
[root@cen7 mysql]# mysql
Welcome to the MariaDB monitor. Commands end with ; or \g.
Your MariaDB connection id is 2
Server version: 5.5.61-MariaDB Source distribution
Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
MariaDB [(none)]> quit
Bye
6、复制配置文件到node2节点：mysql配置文件各节点相同
[root@cen7 corosync]# scp /etc/my.cnf node2:/etc/my.cnf
[root@cen7 corosync]# service mysqld stop
Shutting down MariaDB.... SUCCESS!
7、卸载共享目录：最后必须卸载共享目录
[root@cen7 corosync]# umount /mydata
（二）、node2.field.com上的配置Mariadb
因node2节点上mariadb安装配置同cen7节点，只是不需要再次初始化数据库，此处不再针对操作做出说明。
以下是操作步骤及代码：
[root@node2 ~]# tar xf mariadb-5.5.61.tar.gz -C /usr/local/
[root@node2 ~]# groupadd -g 306 -r mysql
[root@node2 ~]# useradd -u 306 -g mysql -r mysql
[root@node2 ~]# mkdir /mydata
[root@node2 ~]# cd /usr/local/
[root@node2 local]# ln -sv mariadb-5.5.61 mysql
"mysql" -> "mariadb-5.5.61"
[root@node2 local]# cd mysql
[root@node2 mysql]# chown -R mysql:mysql .
[root@node2 mysql]# yum install cmake openssl-devel zlib-devel ncurses-devel bision gcc gcc-c++ -y
[root@node2 mysql]# cmake . -DCMAKE_INSTALL_PREFIX=/usr/local/mysql -DMYSQL_DATADIR=/mydata/data/ -DSYSCONFDIR=/etc -DWITH_INNOBASE_STORAGE_ENGINE=1 -DWITH_ARCHIVE_STORAGE_ENGINE=1 -DWITH_BLACKHOLE_STORAGE_ENGINE=1 -DWITH_READLINE=1 -DWITH_SSL=system -DWITH_ZLIB=system -DWITH_LIBWRAP=0 -DMYSQL_UNIX_ADDR=/tmp/mysql.sock -DDEFAULT_CHARSET=utf8 -DDEFAULT_COLLATION=utf8_general_ci
[root@node2 mysql]# make && make install
[root@node2 mysql]# chown -R root:mysql .
[root@node2 mysql]# cp support-files/mysql.server /etc/rc.d/init.d/mysqld
[root@node2 mysql]# chkconfig --add mysqld
[root@node2 mysql]# chkconfig mysqld off
[root@node2 mysql]# vim /etc/man.config
[root@node2 mysql]# ln -sv /usr/local/mysql/include /usr/include/mysql
"/usr/include/mysql" -> "/usr/local/mysql/include"
[root@node2 mysql]# echo '/usr/local/mysql/lib' > /etc/ld.so.conf.d/mysql.conf
[root@node2 mysql]# ldconfig
[root@node2 mysql]# vim /etc/profile
MYSQL_HOME=/usr/local/mysql
PATH=$PATH:$MYSQL_HOME/bin
[root@node2 mysql]# vim /etc/profile
[root@node2 mysql]# . /etc/profile
[root@node2 mysql]# systemctl start mysqld.service
Failed to start mysqld.service: Unit not found.
[root@node2 mysql]# systemctl start mysql.service
Failed to start mysql.service: Unit not found.
[root@node2 mysql]# chmod +x /etc/rc.d/init.d/mysqld
[root@node2 mysql]# service mysqld start
Starting MariaDB.180803 06:49:01 mysqld_safe Logging to '/mydata/data/node2.field.com.err'.
180803 06:49:01 mysqld_safe Starting mysqld daemon with databases from /mydata/data
/usr/local/mysql/bin/mysqld_safe_helper: Can't create/write to file '/mydata/data/node2.field.com.err' (Errcode: 2)
ERROR!
#此报错是因为未挂载mysql数据存放目录
挂载/mydata目录，验证Mariadb是否安装成功
[root@node2 corosync]# mount -t nfs 192.168.88.135:/mydata /mydata
[root@node2 corosync]# service mysqld start
Starting MariaDB.180803 07:04:16 mysqld_safe Logging to '/mydata/data/node2.field.com.err'.
180803 07:04:16 mysqld_safe Starting mysqld daemon with databases from /mydata/data
SUCCESS!
[root@node2 corosync]# mysql
Welcome to the MariaDB monitor. Commands end with ; or \g.
Your MariaDB connection id is 2
Server version: 5.5.61-MariaDB Source distribution
Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
MariaDB [(none)]>
MariaDB [(none)]> quit
Bye
#可以看到：挂载/mydata目录后，Mariadb客户端接入成功。
注意：测试成功后关闭mariadb并卸载共享目录
[root@node2 corosync]# service mysqld stop
[root@node2 corosync]# umount /mydata
三、使用crmsh配置Corosync+Pacemaker
1、将集群节点加入/etc/ansible/hosts中，便于管理
#或使用systemctl start|stop|restart xxxx
[root@cen7 corosync]# vim /etc/ansible/hosts
[hamysql]
192.168.88.132
192.168.88.133
2、启动corosync和pacemaker，删除此前配置
[root@cen7 mysql]# ansible hamysql -m service -a 'name=corosync state=started'
[root@cen7 mysql]# ansible hamysql -m service -a 'name=pacemaker state=started'
[root@cen7 mysql]# crm status
Stack: corosync
Current DC: cen7.field.com (version 1.1.18-11.el7_5.3-2b07d5c5a9) - partition with quorum
Last updated: Sat Aug 4 04:33:13 2018
Last change: Sat Aug 4 01:03:41 2018 by root via cibadmin on cen7.field.com
2 nodes configured
3 resources configured
Online: [ cen7.field.com node2.field.com ]
Full list of resources:
Resource Group: mycluster
mysqlip (ocf::heartbeat:IPaddr): Started node2.field.com
mynfs (ocf::heartbeat:Filesystem): Stopped
mysqld (lsb:mysqld): Stopped
使用crm交互模式：删除此前定义的资源
[root@cen7 mysql]# crm
crm(live)# resource
crm(live)resource# help
#删除资源之前需要先停止资源，各命令用法可以使用“help [command]”查看
crm(live)resource# help stop
crm(live)resource# show
Resource Group: mycluster
mysqlip (ocf::heartbeat:IPaddr): Started
mynfs (ocf::heartbeat:Filesystem): Stopped
mysqld (lsb:mysqld): Stopped
crm(live)resource# stop mysqlip
crm(live)resource# show
Resource Group: mycluster
mysqlip (ocf::heartbeat:IPaddr): Stopped (disabled)
mynfs (ocf::heartbeat:Filesystem): Stopped
mysqld (lsb:mysqld): Stopped
crm(live)resource# cd
crm(live)# configure
crm(live)configure# help delete
crm(live)configure# delete mycluster
crm(live)configure# delete mysqlip
crm(live)configure# delete mynfs
crm(live)configure# delete mysqld
crm(live)configure# verify
ERROR: Warnings found during check: config may not be valid
#注意：如果有无意义的配置会报如上error，使用show查看配置，如此处的cluster-name。
#直接使用edit编辑删除无效配置:wq保存退出即可。
crm(live)configure# show
node 1: cen7.field.com \
attributes standby=off
node 2: node2.field.com \
attributes standby=off
property cib-bootstrap-options: \
have-watchdog=false \
dc-version=1.1.18-11.el7_5.3-2b07d5c5a9 \
cluster-infrastructure=corosync \
cluster-name=mycluster \
stonith-enabled=false \
default-resource-stickiness=0 \
no-quorum-policy=ignore
crm(live)configure# edit
crm(live)configure# verify
crm(live)configure# verify
#可以看到，删除无效配置后verify不会报ERROR
crm(live)configure# show
node 1: cen7.field.com \
attributes standby=off
node 2: node2.field.com \
attributes standby=off
property cib-bootstrap-options: \
have-watchdog=false \
dc-version=1.1.18-11.el7_5.3-2b07d5c5a9 \
cluster-infrastructure=corosync \
stonith-enabled=false \
no-quorum-policy=ignore
crm(live)configure# commit
crm(live)configure# quit
bye
3、编辑配置文件corosync.conf
[root@cen7 corosync]# cp corosync.conf corosync.conf.bak080222
[root@cen7 corosync]# cat corosync.conf.bak080215 >corosync.conf
[root@cen7 corosync]# vim corosync.conf
[root@cen7 corosync]# grep -v '^[[:space:]]*#' corosync.conf
totem {
version: 2
crypto_cipher: aes128
crypto_hash: sha1
secauth: on
interface {
ringnumber: 0
bindnetaddr: 192.168.88.0
mcastaddr: 239.188.1.188
mcastport: 5405
ttl: 1
}
}
nodelist {
node {
ring0_addr: 192.168.88.132
nodeid: 1
}
node {
ring0_addr: 192.168.88.133
nodeid: 2
}
}
logging {
fileline: off
to_stderr: no
to_logfile: yes
logfile: /var/log/cluster/corosync.log
to_syslog: no
debug: off
timestamp: on
logger_subsys {
subsys: QUORUM
debug: off
}
}
quorum {
provider: corosync_votequorum
}
4、生成节点间通信时用到的认证密钥文件，并复制到node2节点
[root@cen7 corosync]# corosync-keygen
Corosync Cluster Engine Authentication key generator.
Gathering 1024 bits for key from /dev/random.
Press keys on your keyboard to generate entropy.
Press keys on your keyboard to generate entropy (bits = 920).
Press keys on your keyboard to generate entropy (bits = 1000).
Writing corosync key to /etc/corosync/authkey.
[root@cen7 corosync]# ls
authkey corosync.conf corosync.conf.bak080215 corosync.conf.bak080222 corosync.conf.example corosync.conf.example.udpu corosync.xml.example uidgid.d
[root@cen7 corosync]# scp -p authkey corosync.conf node2:/etc/corosync/
authkey 100% 128 44.1KB/s 00:00
corosync.conf
5、启动corosync和pacemaker，并确认状态是否正常
[root@cen7 corosync]# ansible hamysql -m service -a 'name=corosync state=started enabled=yes'
[root@cen7 corosync]# ansible hamysql -m service -a 'name=pacemaker state=started enabled=yes'
1）、查看是否能够正常发出初始化成员节点通知
[root@cen7 corosync]# grep TOTEM /var/log/messages|tail
Aug 3 06:25:24 cen7 corosync[62060]: [TOTEM ] A new membership (192.168.88.132:348) was formed. Members
Aug 3 06:25:27 cen7 corosync[62060]: [TOTEM ] A new membership (192.168.88.132:352) was formed. Members
Aug 3 06:25:29 cen7 corosync[62060]: [TOTEM ] A new membership (192.168.88.132:356) was formed. Members joined: 3
Aug 3 06:26:16 cen7 corosync[62060]: [TOTEM ] A processor failed, forming new configuration.
Aug 3 06:26:16 cen7 corosync[62060]: [TOTEM ] A new membership (192.168.88.132:368) was formed. Members joined: 3 2 left: 3 2
Aug 3 06:26:16 cen7 corosync[62060]: [TOTEM ] Failed to receive the leave message. failed: 3 2
Aug 3 06:28:21 cen7 corosync[62060]: [TOTEM ] A new membership (192.168.88.132:372) was formed. Members
Aug 3 06:30:14 cen7 corosync[62060]: [TOTEM ] A processor failed, forming new configuration.
Aug 3 06:30:14 cen7 corosync[62060]: [TOTEM ] A new membership (192.168.88.132:376) was formed. Members
Aug 3 06:34:54 cen7 corosync[62060]: [TOTEM ] A new membership (192.168.88.132:380) was formed. Members left: 2
2）、查看corosync引擎状态
[root@cen7 corosync]# grep -e "Corosync Cluster Engine" -e "configuration file" /var/log/messages|tail -n 3
Aug 4 04:16:16 cen7 systemd: Starting Corosync Cluster Engine...
Aug 4 04:16:23 cen7 corosync: Starting Corosync Cluster Engine (corosync): [ 确定 ]
Aug 4 04:16:23 cen7 systemd: Started Corosync Cluster Engine.
3）、确认启动过程无ERROR产生
[root@cen7 corosync]# grep ERROR: /var/log/messages | grep -v unpack_resources
6、查看集群节点状态
1）、查看cen7节点状态：正常
[root@cen7 corosync]# systemctl status corosync.service
● corosync.service - Corosync Cluster Engine
Loaded: loaded (/usr/lib/systemd/system/corosync.service; enabled; vendor preset: disabled)
Active: active (running) since 六 2018-08-04 04:16:23 CST; 41min ago
Docs: man:corosync
man:corosync.conf
man:corosync_overview
Process: 765 ExecStart=/usr/share/corosync/corosync start (code=exited, status=0/SUCCESS)
Main PID: 795 (corosync)
CGroup: /system.slice/corosync.service
└─795 corosync
8月 04 04:16:16 cen7.field.com systemd[1]: Starting Corosync Cluster Engine...
8月 04 04:16:23 cen7.field.com corosync[765]: Starting Corosync Cluster Engine (corosync): [ 确定 ]
8月 04 04:16:23 cen7.field.com systemd[1]: Started Corosync Cluster Engine.
[root@cen7 corosync]# corosync-cfgtool -s
Printing ring status.
Local node ID 1
RING ID 0
id = 192.168.88.132 -->说明：此处如果为127.0.0.1则说明启动错误
#编辑/etc/hosts文件，注销127.0.0.1行，只保留ip和域名后重启Corosync即可
status = ring 0 active with no faults
2）、关闭stonith
[root@cen7 corosync]# crm configure property stonith-enabled=false
使用crm_verify检查配置有没问题
[root@cen7 corosync]# crm_verify -L -V
3）、查看node2节点状态：正常
[root@node2 ~]# service corosync status
Redirecting to /bin/systemctl status corosync.service
● corosync.service - Corosync Cluster Engine
Loaded: loaded (/usr/lib/systemd/system/corosync.service; enabled; vendor preset: disabled)
Active: active (running) since 六 2018-08-04 04:16:23 CST; 34min ago
Docs: man:corosync
man:corosync.conf
man:corosync_overview
Process: 839 ExecStart=/usr/share/corosync/corosync start (code=exited, status=0/SUCCESS)
Main PID: 858 (corosync)
CGroup: /system.slice/corosync.service
└─858 corosync
8月 04 04:16:17 node2.field.com systemd[1]: Starting Corosync Cluster Engine...
8月 04 04:16:23 node2.field.com corosync[839]: Starting Corosync Cluster Engine (corosync): [ 确定 ]
8月 04 04:16:23 node2.field.com systemd[1]: Started Corosync Cluster Engine.
[root@node2 ~]# corosync-cfgtool -s
Printing ring status.
Local node ID 2
RING ID 0
id = 192.168.88.133
status = ring 0 active with no faults
4）、查看集群状态信息
[root@cen7 mysql]# crm status
Stack: corosync
Current DC: cen7.field.com (version 1.1.18-11.el7_5.3-2b07d5c5a9) - partition with quorum
Last updated: Sat Aug 4 04:45:27 2018
Last change: Sat Aug 4 04:45:18 2018 by root via cibadmin on cen7.field.com
2 nodes configured
0 resources configured
Online: [ cen7.field.com node2.field.com ]
No resources
#可以看到集群节点的启动状态：已经Online
四、使用crmsh定义资源，配置Corosync+Pacemaker+nfs+Mysql高可用集群
说明：笔者使用crm交互式模式定义
1、定义vip资源ocf:heartbeat:IPaddr
部分资源定义命令说明：
primitive定义资源：
monitor：定义资源监控选项：interval（检测间隔时间）timeout（监控超时时间）
[root@cen7 corosync]# crm
crm(live)# configure
#以下命令表示定义vip资源mysqlip:虚拟IP地址“192.168.88.188”，检测间隔时间为30s，超时时长为20s
crm(live)configure# primitive mysqlip ocf:heartbeat:IPaddr params ip="192.168.88.188" op monitor interval=30s timeout=20s
crm(live)configure# verify
#verify检查语法
crm(live)configure# commit
#commit提交配置
crm(live)configure# show
node 1: cen7.field.com \
attributes standby=off
node 2: node2.field.com \
attributes standby=off
primitive mysqlip IPaddr \
params ip=192.168.88.188 \
op monitor interval=30s timeout=20s
property cib-bootstrap-options: \
have-watchdog=false \
dc-version=1.1.18-11.el7_5.3-2b07d5c5a9 \
cluster-infrastructure=corosync \
stonith-enabled=false \
no-quorum-policy=ignore
crm(live)configure# cd
crm(live)# status
Stack: corosync
Current DC: cen7.field.com (version 1.1.18-11.el7_5.3-2b07d5c5a9) - partition with quorum
Last updated: Sat Aug 4 05:07:38 2018
Last change: Sat Aug 4 05:04:36 2018 by root via cibadmin on cen7.field.com
2 nodes configured
1 resource configured
Online: [ cen7.field.com node2.field.com ]
Full list of resources:
mysqlip (ocf::heartbeat:IPaddr): Started cen7.field.com
#可以看到：资源mysqlip已经启动在cen7上
crm(live)# quit
bye
[root@cen7 corosync]# ip addr list | grep ens
2: ens32: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
inet 192.168.88.132/24 brd 192.168.88.255 scope global noprefixroute ens32
inet 192.168.88.188/24 brd 192.168.88.255 scope global secondary ens32
#可以看到：vip已经启动在cen7上
2、定义nfs文件系统资源：文件系统资源属于ocf
部分命令及参数说明：
param资源管理参数：文件系统资源中，必须指明的参数有device（设备）、directory（挂载目录）、fstype（文件系统类型）
start启动资源选项：interval（间隔时间）、timeout（超时时间）
stop停止资源选项：interval（间隔时间）、timeout（超时时间）
monitor资源监控选项：interval（间隔时间）timeout（监控超时时间）
[root@cen7 corosync]# crm
crm(live)# configure
#以下命令表示创建nfs文件系统资源：
#将"192.168.88.135:/mydata"nfs共享目录挂载到/mydata目录下，启动超时时间60s，关闭超时时间60s，监控检测间隔时间为20s，超时时长为40s
crm(live)configure# primitive mynfs ocf:heartbeat:Filesystem params device="192.168.88.135:/mydata" directory="/mydata" fstype="nfs" op start timeout=60s op stop timeout=60s op monitor interval=20s timeout=40s
crm(live)configure# verify
crm(live)configure# commit
crm(live)configure# cd
crm(live)# status
Stack: corosync
Current DC: cen7.field.com (version 1.1.18-11.el7_5.3-2b07d5c5a9) - partition with quorum
Last updated: Sat Aug 4 05:10:30 2018
Last change: Sat Aug 4 05:10:26 2018 by root via cibadmin on cen7.field.com
2 nodes configured
2 resources configured
Online: [ cen7.field.com node2.field.com ]
Full list of resources:
mysqlip (ocf::heartbeat:IPaddr): Started cen7.field.com
mynfs (ocf::heartbeat:Filesystem): Started node2.field.com
可以发现：资源mynfs已经定义并成功启动在node2节点上
#未配置约束时资源会顺序启动在不同节点
crm(live)# quit
bye
3、定义mysqld资源：编译安装的mysql属于lsb资源代理
#以下命令表示定义mysqld资源，检测间隔时间为30s，超时时长为30s
crm(live)configure# primitive mysql ocf:heartbeat:mysql op monitor interval=30s timeout=30s
crm(live)configure# verify
crm(live)configure# commit
crm(live)configure# cd
crm(live)# status
Stack: corosync
Current DC: cen7.field.com (version 1.1.18-11.el7_5.3-2b07d5c5a9) - partition with quorum
Last updated: Sat Aug 4 05:25:07 2018
Last change: Sat Aug 4 05:24:57 2018 by root via cibadmin on cen7.field.com
2 nodes configured
3 resources configured
Online: [ cen7.field.com node2.field.com ]
Full list of resources:
mysqlip (ocf::heartbeat:IPaddr): Started cen7.field.com
mynfs (ocf::heartbeat:Filesystem): Started node2.field.com
mysqlsrv (ocf::heartbeat:mysql): Stopped
Failed Actions:
* mysqlsrv_start_0 on node2.field.com 'not installed' (5): call=37, status=complete, exitreason='Setup problem: couldn't find command: /usr/bin/safe_mysqld',
last-rc-change='Sat Aug 4 05:24:59 2018', queued=0ms, exec=63ms
* mysqlsrv_start_0 on cen7.field.com 'not installed' (5): call=37, status=complete, exitreason='Setup problem: couldn't find command: /usr/bin/safe_mysqld',
last-rc-change='Sat Aug 4 05:24:58 2018', queued=0ms, exec=64ms
crm(live)# configure
crm(live)configure# edit
#因为是编译安装的mysql，不能用ocf设置，删除mysql资源定义并:wq保存退出
改用lsb下的mysqld定义mysql资源
crm(live)configure# primitive mysqld lsb:mysqld
crm(live)configure# verify
crm(live)configure# commit
crm(live)configure# cd
crm(live)# status
Stack: corosync
Current DC: cen7.field.com (version 1.1.18-11.el7_5.3-2b07d5c5a9) - partition with quorum
Last updated: Sat Aug 4 05:26:50 2018
Last change: Sat Aug 4 05:26:44 2018 by root via cibadmin on cen7.field.com
2 nodes configured
3 resources configured
Online: [ cen7.field.com node2.field.com ]
Full list of resources:
mysqlip (ocf::heartbeat:IPaddr): Started cen7.field.com
mynfs (ocf::heartbeat:Filesystem): Started node2.field.com
mysqld (lsb:mysqld): Starting node2.field.com
Failed Actions:
* mysqld_start_0 on cen7.field.com 'unknown error' (1): call=44, status=complete, exitreason='',
last-rc-change='Sat Aug 4 05:26:45 2018', queued=0ms, exec=1376ms
#此错误是因为没找到datadir中的初始化文件，重启Corosync可以解决
4、定义约束：资源必须启动在同一个节点
约束说明：
colocation #排列约束，指定哪些资源捆绑一起，在同一节点上运行
order #指定排列约束中的资源启动顺序，该顺序和colocation顺序相反
location #位置约束，指定资源首选在哪些节点上运行
定义约束的顺序：排列约束-->顺序约束-->位置约束
“inf:”表示黏性值无穷大，定义inf:的两个资源始终在同一节点运行。
“-inf:”表示黏性值负无穷，定义-inf:的两个资源始终不在同一节点运行。
#以下配置表示mysqlip和(mynfs mysqld)间的黏性值无穷大，即三个资源始终在同一节点上运行。
crm(live)configure# colocation mysqlip_with_mynfs_and_mysqld inf: mysqlip ( mynfs mysqld)
crm(live)configure# verify
crm(live)configure# commit
#mandatory默认值，表示强制约束，强制约束mynfs和mysqld的状态必须相同且均为start
crm(live)configure# order mysqld-after-mynfs mandatory: mynfs mysqld:start
crm(live)configure# verify
crm(live)configure# commit
#强制约束mysqld资源必须在mysqlip资源启动后再启动
crm(live)configure# order mysqld-after-mysqlip mandatory: mysqlip mysqld
crm(live)configure# verify
crm(live)configure# commit
crm(live)configure# show
node 1: cen7.field.com \
attributes standby=off
node 2: node2.field.com \
attributes standby=off
primitive mynfs Filesystem \
params device="192.168.88.135:/mydata" directory="/mydata" fstype=nfs \
op start timeout=60s interval=0 \
op stop timeout=60s interval=0 \
op monitor interval=20s timeout=40s
primitive mysqld lsb:mysqld
primitive mysqlip IPaddr \
params ip=192.168.88.188 \
op monitor interval=30s timeout=20s
order mysqld-after-mynfs Mandatory: mynfs mysqld:start
order mysqld-after-mysqlip Mandatory: mysqlip mysqld
colocation mysqlip_with_mynfs_and_mysqld inf: mysqlip ( mynfs mysqld )
property cib-bootstrap-options: \
have-watchdog=false \
dc-version=1.1.18-11.el7_5.3-2b07d5c5a9 \
cluster-infrastructure=corosync \
stonith-enabled=false \
no-quorum-policy=ignore
crm(live)configure# quit
bye
五、测试高可用集群配置成功与否，并检测高可用性
1、重启corosync和pacemaker
[root@cen7 corosync]# ansible hamysql -m service -a 'name=corosync state=restarted'
[root@cen7 corosync]# ansible hamysql -m service -a 'name=pacemaker state=restarted'
2、查看集群状态信息
[root@cen7 corosync]# crm status
Stack: corosync
Current DC: node2.field.com (version 1.1.18-11.el7_5.3-2b07d5c5a9) - partition with quorum
Last updated: Sat Aug 4 05:34:30 2018
Last change: Sat Aug 4 05:30:59 2018 by root via cibadmin on cen7.field.com
2 nodes configured
3 resources configured
Online: [ cen7.field.com node2.field.com ]
Full list of resources:
mysqlip (ocf::heartbeat:IPaddr): Started node2.field.com
mynfs (ocf::heartbeat:Filesystem): Started node2.field.com
mysqld (lsb:mysqld): Started node2.field.com
3、在node2节点测试
1）、连接mysql客户端：成功
[root@node2 ~]# mysql
Welcome to the MariaDB monitor. Commands end with ; or \g.
Your MariaDB connection id is 2
Server version: 5.5.61-MariaDB Source distribution
Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
MariaDB [(none)]> quit
Bye
2）、VIP流转：成功
[root@node2 ~]# ip addr list | grep ens
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
inet 192.168.88.133/24 brd 192.168.88.255 scope global noprefixroute dynamic ens33
inet 192.168.88.188/24 brd 192.168.88.255 scope global secondary ens33
4、集群高可用性测试
1）、置node2.field.com状态为备用状态，检查能否自动切换
[root@cen7 corosync]# crm
crm(live)# node
crm(live)node# standby node2.field.com
crm(live)node# cd
crm(live)# status
Stack: corosync
Current DC: node2.field.com (version 1.1.18-11.el7_5.3-2b07d5c5a9) - partition with quorum
Last updated: Sat Aug 4 05:38:52 2018
Last change: Sat Aug 4 05:37:51 2018 by root via crm_attribute on cen7.field.com
2 nodes configured
3 resources configured
Node node2.field.com: standby
Online: [ cen7.field.com ]
Full list of resources:
mysqlip (ocf::heartbeat:IPaddr): Started cen7.field.com
mynfs (ocf::heartbeat:Filesystem): Started cen7.field.com
mysqld (lsb:mysqld): Started cen7.field.com
crm(live)# quit
bye
2）、连接mysql测试：成功
[root@cen7 corosync]# mysql
Welcome to the MariaDB monitor. Commands end with ; or \g.
Your MariaDB connection id is 2
Server version: 5.5.61-MariaDB Source distribution
Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
MariaDB [(none)]> show databases;
+--------------------+
| Database |
+--------------------+
| information_schema |
| mysql |
| performance_schema |
| test |
+--------------------+
4 rows in set (0.21 sec)
MariaDB [(none)]> quit
Bye
3）、查看VIP是否流转成功：成功
[root@cen7 corosync]# ip addr list | grep ens
2: ens32: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
inet 192.168.88.132/24 brd 192.168.88.255 scope global noprefixroute ens32
inet 192.168.88.188/24 brd 192.168.88.255 scope global secondary ens32
附：crmsh整体配置
[root@cen7 corosync]# crm
crm(live)# node
crm(live)node# online node2.field.com
crm(live)node# cd
crm(live)# configure
crm(live)configure# group mycluster mysqlip mynfs mysqld
INFO: modified order:mysqld-after-mysqlip from mysqlip to mycluster
INFO: modified colocation:mysqlip_with_mynfs_and_mysqld from mysqlip to mycluster
INFO: modified order:mysqld-after-mynfs from mynfs to mycluster
INFO: modified colocation:mysqlip_with_mynfs_and_mysqld from mynfs to mycluster
INFO: modified order:mysqld-after-mynfs from mysqld to mycluster
INFO: modified order:mysqld-after-mysqlip from mysqld to mycluster
INFO: modified colocation:mysqlip_with_mynfs_and_mysqld from mysqld to mycluster
crm(live)configure# verify
crm(live)configure# commit
crm(live)configure# show
node 1: cen7.field.com \
attributes standby=off
node 2: node2.field.com \
attributes standby=off
primitive mynfs Filesystem \
params device="192.168.88.135:/mydata" directory="/mydata" fstype=nfs \
op start timeout=60s interval=0 \
op stop timeout=60s interval=0 \
op monitor interval=20s timeout=40s
primitive mysqld lsb:mysqld
primitive mysqlip IPaddr \
params ip=192.168.88.188 \
op monitor interval=30s timeout=20s
group mycluster mysqlip mynfs mysqld
property cib-bootstrap-options: \
have-watchdog=false \
dc-version=1.1.18-11.el7_5.3-2b07d5c5a9 \
cluster-infrastructure=corosync \
stonith-enabled=false \
no-quorum-policy=ignore
crm(live)configure# cd
crm(live)# status
Stack: corosync
Current DC: node2.field.com (version 1.1.18-11.el7_5.3-2b07d5c5a9) - partition with quorum
Last updated: Sat Aug 4 05:42:51 2018
Last change: Sat Aug 4 05:42:45 2018 by root via cibadmin on cen7.field.com
2 nodes configured
3 resources configured
Online: [ cen7.field.com node2.field.com ]
Full list of resources:
Resource Group: mycluster
mysqlip (ocf::heartbeat:IPaddr): Started cen7.field.com
mynfs (ocf::heartbeat:Filesystem): Started cen7.field.com
mysqld (lsb:mysqld): Started cen7.field.com
crm(live)# quit
bye
[root@cen7 corosync]#