Margo SRE Week 6 Coursework

1. Briefly describe DDL, DML, DCL, and DQL, and explain the sequence of each keyword query in mysql

1.1 Brief description of DDL, DML, DCL, DQL

• DDL: Data Definition Language Data Definition Language

CREATE，DROP，ALTER

• DML: Data Manipulation Language Data Manipulation Language

INSERT, DELETE, UPDATE
software development: CRUD

• DQL: Data Query Language Data Query Language

SELECT

• DCL: Data Control Language Data Control Language

GRANT，REVOKE

• TCL: Transaction Control Language transaction control language

COMMIT，ROLLBACK，SAVEPOINT

1.2 The order of each keyword query in mysql

The key word Keyword forms a clause clause, and multiple clauses form a statement
Example:

SELECT * #SELECT child clause
FROM products #FROM child clause
WHERE price>666 #WHERE child clause

Explanation: A set of SQL statements consists of three clauses, SELECT, FROM and WHERE are keywords
Get SQL command usage help:
Official help:
https://dev.mysql.com/doc/refman/8.0/en/sql-statements .html

Example: View SQL Help

MariaDB [mysql]> help contents
You asked for help about help category: "Contents"
For more information, type 'help <item>', where <item> is one of the following
categories:
   Account Management
   Administration
   Compound Statements
   Data Definition
   Data Manipulation
   Data Types
   Functions
   Functions and Modifiers for Use with GROUP BY
   Geographic Features
   Help Metadata
   Language Structure
   Plugins
   Procedures
   Table Maintenance
   Transactions
   User-Defined Functions
   Utility
MySQL [(none)]> help Data Types
You asked for help about help category: "Data Types"
For more information, type 'help <item>', where <item> is one of the following
topics:
   AUTO_INCREMENT
   BIGINT
   BINARY
   BIT
   BLOB
   BLOB DATA TYPE
   
MySQL [(none)]> help bit
Name: 'BIT'
Description:
BIT[(M)]
A bit-value type. M indicates the number of bits per value, from 1 to
64. The default is 1 if M is omitted.
URL: https://dev.mysql.com/doc/refman/5.7/en/numeric-type-syntax.html

view SQL help

mysql> HELP KEYWORD

The execution sequence of mysql keywords is as follows:

from
on/using
join
where
group by
having
select
distinct
union
order by
limit

Explain the keyword execution order
Mysql keyword execution order - in-depth analysis

Mysql will first execute from, and then filter the target table according to the on keyword, and then join or use the filtered results, so that a temporary table will be formed. Then execute the where condition to filter the temporary table, so that the required data is basically filtered out. Then you can group the data by groupby, and if necessary, you will execute having to further filter the data. Here, the data is basically finalized after execution. Next, you need to select to filter the target column. After you are done, you need to use distinct The query of such a table is basically over. If you need multi-table query, you also need to use union or unionall to connect the results of multiple tables. Then there is the order by for sorting the data. After sorting the order, it is natural to take pagination. In this way, a complete sql will be executed.

There will be some special situations that make us unable to hit the index (even if the index is created), which also requires everyone to pay attention to during development

• Type inconsistency

  select * from big where name = 123;		-- 未命中
  select * from big where email = 123;	-- 未命中
  
  特殊的主键：
  	select * from big where id = "123";	-- 命中
  

• use not equal to

  select * from big where name != "武沛齐";				-- 未命中
  select * from big where email != "[email protected]";  -- 未命中
  
  特殊的主键：
  	select * from big where id != 123;	-- 命中
  

• or, when there are unindexed columns in the or condition, it will fail.

  select * from big where id = 123 or password="xx";			-- 未命中
  select * from big where name = "wupeiqi" or password="xx";	-- 未命中
  特别的：
  	select * from big where id = 10 or password="xx" and name="xx"; -- 命中
  

• Sorting, when sorting according to the index, if the selected mapping is not an index, the index will not be used.

  select * from big order by name asc;     -- 未命中
  select * from big order by name desc;    -- 未命中
  
  特别的主键：
  	select * from big order by id desc;  -- 命中
  

• like, when fuzzy matching.

  select * from big where name like "%u-12-19999";	-- 未命中
  select * from big where name like "_u-12-19999";	-- 未命中
  select * from big where name like "wu-%-10";		-- 未命中
  
  特别的：
  	select * from big where name like "wu-1111-%";	-- 命中
  	select * from big where name like "wuw-%";		-- 命中
  

• use function

  select * from big where reverse(name) = "wupeiqi";  -- 未命中
  
  特别的：
  	select * from big where name = reverse("wupeiqi");  -- 命中
  

• The leftmost prefix, if it is a joint index, follow the leftmost prefix principle.

  如果联合索引为：(name,password)
      name and password       -- 命中
      name                 	-- 命中
      password                -- 未命中
      name or password       	-- 未命中
  

2. Design 10 sql query statements by yourself, you need to use keywords [GROUP BY/HAVING/ORDER BY/LIMIT], at least two of them are used at the same time.

2.1 Create a database & table structure and enter data according to the above figure

create database day27db default charset utf8 collate utf8_general_ci;
use day27db;
#导出
# 结构+数据
mysqldump -u root -p  day27db > /Users/wupeiqi/day27db2.sql

# 结构
mysqldump -u root -p -d day27db > /Users/wupeiqi/day27db3.sql
create table class(
	cid int not null auto_increment primary key,
    caption varchar(16) not null
)default charset=utf8;

INSERT INTO class VALUES ('1', '三年二班'), ('2', '三年三班'), ('3', '一年二班'), ('4', '二年九班');


create table student(
	 sid int not null auto_increment primary key,
    gender char(1) not null,
    class_id int not null,
    sname varchar(16) not null,
    constraint fk_student_class foreign key (class_id) references class(cid)
)default charset=utf8;

INSERT INTO student VALUES ('1', '男', '1', '理解'), ('2', '女', '1', '钢蛋'), ('3', '男', '1', '张三'), ('4', '男', '1', '张一'), ('5', '女', '1', '张二'), ('6', '男', '1', '张四'), ('7', '女', '2', '铁锤'), ('8', '男', '2', '李三'), ('9', '男', '2', '李一'), ('10', '女', '2', '李二'), ('11', '男', '2', '李四'), ('12', '女', '3', '如花'), ('13', '男', '3', '刘三'), ('14', '男', '3', '刘一'), ('15', '女', '3', '刘二'), ('16', '男', '3', '刘四');


create table teacher(
	 tid int not null auto_increment primary key,
    tname varchar(16) not null
)default charset=utf8;

INSERT INTO `teacher` VALUES ('1', '张磊老师'), ('2', '李平老师'), ('3', '刘海燕老师'), ('4', '朱云海老师'), ('5', '李杰老师');


create table course(
	   cid int not null auto_increment primary key,
    cname varchar(16) not null,
    teacher_id int not null,
    constraint fk_course_teacher foreign key (teacher_id) references teacher(tid)
)default charset=utf8;

INSERT INTO `course` VALUES ('1', '生物', '1'), ('2', '物理', '2'), ('3', '体育', '3'), ('4', '美术', '2');


CREATE TABLE `score` (
  `sid` int NOT NULL AUTO_INCREMENT PRIMARY KEY,
  `student_id` int NOT NULL,
  `course_id` int NOT NULL,
  `num` int NOT NULL,
  CONSTRAINT `fk_score_course` FOREIGN KEY (`course_id`) REFERENCES `course` (`cid`),
  CONSTRAINT `fk_score_student` FOREIGN KEY (`student_id`) REFERENCES `student` (`sid`)
) DEFAULT CHARSET=utf8;


INSERT INTO `score` VALUES ('1', '1', '1', '10'), ('2', '1', '2', '9'), ('5', '1', '4', '66'), ('6', '2', '1', '8'), ('8', '2', '3', '68'), ('9', '2', '4', '99'), ('10', '3', '1', '77'), ('11', '3', '2', '66'), ('12', '3', '3', '87'), ('13', '3', '4', '99'), ('14', '4', '1', '79'), ('15', '4', '2', '11'), ('16', '4', '3', '67'), ('17', '4', '4', '100'), ('18', '5', '1', '79'), ('19', '5', '2', '11'), ('20', '5', '3', '67'), ('21', '5', '4', '100'), ('22', '6', '1', '9'), ('23', '6', '2', '100'), ('24', '6', '3', '67'), ('25', '6', '4', '100'), ('26', '7', '1', '9'), ('27', '7', '2', '100'), ('28', '7', '3', '67'), ('29', '7', '4', '88'), ('30', '8', '1', '9'), ('31', '8', '2', '100'), ('32', '8', '3', '67'), ('33', '8', '4', '88'), ('34', '9', '1', '91'), ('35', '9', '2', '88'), ('36', '9', '3', '67'), ('37', '9', '4', '22'), ('38', '10', '1', '90'), ('39', '10', '2', '77'), ('40', '10', '3', '43'), ('41', '10', '4', '87'), ('42', '11', '1', '90'), ('43', '11', '2', '77'), ('44', '11', '3', '43'), ('45', '11', '4', '87'), ('46', '12', '1', '90'), ('47', '12', '2', '77'), ('48', '12', '3', '43'), ('49', '12', '4', '87'), ('52', '13', '3', '87');

2.2 Create user luffy and grant all permissions to this database.

create user 'luffy'@'%' identified by 'root123';
grant all privileges on day27db.* TO 'luffy'@'%';
flush privileges;

Query the list of students with the same name and surname, and count the number of students with the same name. (group by +having)

select sname,count(1) from student group by sname having count(1) > 1;

Query all students in "Class Two, Three Years"

select * from student left join class on student.class_id = class.cid where class.caption="三年二班";

Query the student ID, name, grade, and course name of students whose scores are less than 60 points

select 
	student.sid,
	student.sname,
	score.num,
	course.cname 
from 
	score 
	left join student on score.student_id=student.sid 
	left join course on score.course_id =course.cid 
where num <60;

Query the student number, name, number of courses, and total grades of all students

select student_id,student.sname,count(1),sum(num) from score left join student on score.student_id=student.sid group by student_id;

Query the average score of each subject, display: course ID, course name, average score (sort the first three courses according to the average score from large to small)

select course_id,course.cname,avg(num) as A from score left join course on score.course_id =course.cid group by course_id order by A desc limit 3;

Query the student ID and name of students who have taken all courses

SELECT
	student.sid,
	student.sname
FROM
	score
	LEFT JOIN student ON score.student_id = student.sid 
GROUP BY
	student_id 
HAVING
	count( 1 ) = ( SELECT count( 1 ) FROM course );

Three, xtrabackup backup and restore database practice

3.1 xtrabackup installation

In the EPEL source

yum install percona-xtrabackup
#centos8没有提供
[root@centos7 ~]#yum info percona-xtrabackup 
Available Packages
Name       : percona-xtrabackup
Arch       : x86_64
Version     : 2.3.6
Release     : 1.el7
Size       : 4.6 M
Repo       : epel/7/x86_64
Summary     : Online backup for InnoDB/XtraDB in MySQL, Percona Server and 
MariaDB
URL         : http://www.percona.com/software/percona-xtrabackup/
License     : GPLv2
Description : Online backup for InnoDB/XtraDB in MySQL, MariaDB and Percona 
Server
范例: 最新版本下载安装：
https://www.percona.com/downloads/XtraBackup/LATEST/

3.2 xtrabackup usage

The xtrabackup tool backup and restore requires three steps to achieve

1. Backup: make a full or incremental backup of the database
2. Pre-preparation: Before restoring, organize the backup data into a temporary directory
3. Restore: Copy the organized data back to the database directory

xtrabackup options reference
Backup

innobackupex [option] BACKUP-ROOT-DIR

Option description:

–user: #This option indicates the backup account
–password: #This option indicates the backup password
–host: #This option indicates the address of the backup database
–databases: #The parameter accepted by this option is the database name. If you want to specify multiple databases, They need to be separated by spaces; for example: "xtra_test dba_test". At the same time, when specifying a database, you can also specify only one of the tables.
Such as: "mydatabase.mytable". This option is invalid for the innodb engine table, or all innodb tables will be backed up
–defaults-file: #This option specifies which file to read MySQL configuration from, must be placed in the first option position of the command line
–incremental: #This option means to create a Incremental backup, you need to specify –incremental-basedir
–incremental-basedir: #This option specifies the directory of the previous full backup or incremental backup, and use
–incremental-dir at the same time as –incremental: #This option indicates incremental backup when restoring Directory
–include=name: #Specify the table name, format: databasename.tablename

Prepare:

innobackupex --apply-log [option] BACKUP-DIR

Option description:

–apply-log: #In general, after the backup is completed, the data cannot be used for recovery operations, because the backup data may contain transactions that have not yet been committed or transactions that have been committed but have not yet been synchronized to the data file. Therefore, at this point the datafile is still processing an inconsistent state. The function of this option
is to make the data file in a consistent state by rolling back the uncommitted transaction and synchronizing the committed transaction to the data file –use
-memory: #Used together with the –apply-log option, when prepare backup, do crash The size of the memory allocated by recovery, in bytes, can also be 1MB, 1M, 1G, 1GB, etc. 1G is recommended
–export: #Indicates that you can export a separate table and then import it into other Mysql
–redo-only: #This option is in prepare base full backup, which is used when merging incremental backups, but does not include the merging of the last incremental backup

reduction:

innobackupex --copy-back [options] BACKUP-DIR
innobackupex --move-back [options] [–defaults-group=GROUP-NAME] BACKUP-DIR

Option description:

–copy-back: #Copy the backup data file to the datadir of the MySQL server when doing data recovery
–move-back: #This option is similar to –copy-back, the only difference is that it does not copy the file, but moves the file to the destination land. This option removes the backup file, so be careful when using it. Usage scenario: There is not enough disk space to keep data files and Backup copies at the same time
–force-non-empty-directories #When specifying this parameter, make the innobackupex --copy-back or --move back option transfer files to non-empty directories, which already exist files will not be overwritten. If --copy-back and --move-back files need to copy a file that already exists in datadir from the backup directory, an error will be reported and failed

Restoration Notes:

1. The datadir directory must be empty.
   The --copy-back option does not override unless specified with the innobackupex --force-non-empty-directorires option
2. Before restoring, the MySQL instance must be shut down, and a running instance cannot be restored to the datadir directory
3. Since the file attributes will be preserved, in most cases it is necessary to change the owner of the file to mysql before starting the instance, these files will
   belong to the user who created the backup, execute chown -R mysql:mysql /data/mysql, the above needs to be in the user Completed before calling
   innobackupex

Practical case: using xtrabackup to achieve full backup and restore

Note: currently percona-xtrabackup-24-2.4.18-1.el8.x86_64.rpm does not support mariadb-10.3 version on CentOS 8
Case: Use xtrabackup8.0 to fully backup and restore MySQL8.0

1 安装xtrabackup包
wget https://downloads.percona.com/downloads/Percona-XtraBackup-LATEST/Percona-XtraBackup-8.0.29-22/binary/redhat/8/x86_64/percona-xtrabackup-80-8.0.29-22.1.el8.x86_64.rpm
[root@centos8 ~]#yum -y install yum -y install percona-xtrabackup-80-8.0.29-22.1.el8.x86_64.rpm
2 在原主机做完全备份到/backup
[root@centos8 ~]#mkdir /backup
[root@centos8 ~]#xtrabackup -uroot -pmagedu --backup --target-dir=/backup/base
#目标主机无需创建/backup目录,直接复制目录本身
grant BACKUP_ADMIN on *.* to 'root'@'%'; #处理账号没有BACKUP_ADMIN权限
[root@centos8 ~]#scp -r /backup/   目标主机:/
3 在目标主机上还原
注意：恢复主机MySQL服务停止，并且数据目录为空
1）预准备：确保数据一致，提交完成的事务，回滚未完成的事务
[root@centos8 ~]#yum -y install percona-xtrabackup-80-8.0.23-16.1.el8.x86_64.rpm
[root@centos8 ~]#xtrabackup --prepare --target-dir=/backup/base
2）复制到数据库目录
注意：数据库目录必须为空，MySQL服务不能启动
[root@centos8 ~]#xtrabackup --copy-back --target-dir=/backup/base
3）还原属性
[root@centos8 ~]#chown -R mysql:mysql /var/lib/mysql
4）启动服务
[root@centos8 ~]#service mysqld start 

Case: full backup and restore of the new version of xtrabackup
This case is based on the MySQL5.7 implementation of CentOS 8, and also supports MySQL5.5 and Mariadb5.5, and the steps are the same as the above case

1 安装xtrabackup包 
#先安装MySQL5.7和xtrabackup包
[root@centos8 ~]#yum -y install percona-xtrabackup-24-2.4.20-1.el8.x86_64.rpm
2 在原主机做完全备份到/backup
[root@centos8 ~]#mkdir /backup
[root@centos8 ~]#xtrabackup -uroot -pmagedu --backup --target-dir=/backup/base
#目标主机无需创建/backup目录,直接复制目录本身
[root@centos8 ~]#scp -r /backup/   目标主机:/
3 在目标主机上还原
1）预准备：确保数据一致，提交完成的事务，回滚未完成的事务
[root@centos8 ~]#yum -y install percona-xtrabackup-24-2.4.20-1.el8.x86_64.rpm
[root@centos8 ~]#xtrabackup --prepare --target-dir=/backup/base
2）复制到数据库目录
注意：数据库目录必须为空，MySQL服务不能启动
[root@centos8 ~]#xtrabackup --copy-back --target-dir=/backup/base
3）还原属性
[root@centos8 ~]#chown -R mysql:mysql /var/lib/mysql
4）启动服务
[root@centos8 ~]#service mysqld start 

Practical case: Using xtrabackup to fully and incrementally backup and restore
Case: Using xtrabackup8.0 to fully and incrementally backup and restore MySQL8.0

1 备份过程
1）完全备份：
[root@centos8 ~]#yum -y install percona-xtrabackup-24-2.4.20-1.el8.x86_64.rpm
[root@centos8 ~]#mkdir /backup/
[root@centos8 ~]#xtrabackup -uroot -pmagedu --backup --target-dir=/backup/base
2）第一次修改数据
3）第一次增量备份
[root@centos8 ~]#xtrabackup -uroot -pmagedu --backup --target-dir=/backup/inc1 --
incremental-basedir=/backup/base
4）第二次修改数据
5）第二次增量
[root@centos8 ~]#xtrabackup -uroot -pmagedu --backup --target-dir=/backup/inc2 --
incremental-basedir=/backup/inc1
6）[root@centos8 ~]#scp -r /backup/* 目标主机:/backup/
#备份过程生成三个备份目录
/backup/{
    
    base，inc1，inc2}
2还原过程
1）预准备完成备份，此选项--apply-log-only 阻止回滚未完成的事务
[root@centos8 ~]#yum -y install percona-xtrabackup-24-2.4.20-1.el8.x86_64.rpm
[root@centos8 ~]#xtrabackup --prepare --apply-log-only --target-dir=/backup/base 
2）合并第1次增量备份到完全备份
[root@centos8 ~]#xtrabackup --prepare --apply-log-only --target-dir=/backup/base 
--incremental-dir=/backup/inc1
3）合并第2次增量备份到完全备份：最后一次还原不需要加选项--apply-log-only
[root@centos8 ~]#xtrabackup --prepare --target-dir=/backup/base --incremental
dir=/backup/inc2
4）复制到数据库目录，注意数据库目录必须为空，MySQL服务不能启动
[root@centos8 ~]#xtrabackup --copy-back --target-dir=/backup/base
5）还原属性：
[root@centos8 ~]#chown -R mysql:mysql /var/lib/mysql
6）启动服务：
[root@centos8 ~]#service mysqld start

4. Realize mysql master-slave replication, master-master replication and semi-synchronous replication

4.1 mysql master-slave replication steps

主节点配置：
(1) 启用二进制日志
[mysqld]
log_bin
(2) 为当前节点设置一个全局惟一的ID号
[mysqld]
server-id=#
log-basename=master  #可选项，设置datadir中日志名称，确保不依赖主机名
server-id的取值范围
1 to 4294967295 (>= MariaDB 10.2.2)，默认值为1，MySQL8.0默认值为1
0 to 4294967295 (<= MariaDB 10.2.1)，默认值为0，如果从节点为0，所有master都将拒绝此
slave的连接
(3) 查看从二进制日志的文件和位置开始进行复制
SHOW MASTER STATUS;
(4) 创建有复制权限的用户账号
GRANT REPLICATION SLAVE  ON *.* TO 'repluser'@'HOST' IDENTIFIED BY 'replpass';
#MySQL8.0 分成两步实现
mysql> create user repluser@'10.0.0.%' identified by '123456';
mysql> grant replication slave on *.* to repluser@'10.0.0.%'; 
从节点配置：
从节点配置：
(1) 启动中继日志
[mysqld]
server_id=# #为当前节点设置一个全局惟的ID号
log-bin
read_only=ON #设置数据库只读，针对supper user无效
relay_log=relay-log #relay log的文件路径，默认值hostname-relay-bin
relay_log_index=relay-log.index  #默认值hostname-relay-bin.index
(2) 使用有复制权限的用户账号连接至主服务器，并启动复制线程
官方说明
https://dev.mysql.com/doc/refman/8.0/en/change-master-to.html
CHANGE MASTER TO MASTER_HOST='masterhost', 
MASTER_USER='repluser', 
MASTER_PASSWORD='replpass', 
MASTER_LOG_FILE='mariadb-bin.xxxxxx', 
MASTER_LOG_POS=#
MASTER_DELAY = interval; #可指定延迟复制实现访问误操作,单位秒
START SLAVE [IO_THREAD|SQL_THREAD];
SHOW SLAVE STATUS; 
#查看 relaylog 事件
SHOW RELAYLOG EVENTS in 'relay-bin.00000x';

范例：主服务器非新建时,主服务器运行一段时间后，新增从节点服务器
如果主节点已经运行了一段时间，且有大量数据时，如何配置并启动slave节点

 - 通过备份恢复数据至从服务器 
 - 复制起始位置为备份时，二进制日志文件及其POS
#在主服务器完全备份
[root@master ~]#mysqldump -A -F --single-transaction --master-data=1 > 
/backup/fullbackup_`date +%F_%T`.sql
[root@master ~]#ll /backup/
total 2988
-rw-r--r-- 1 root root 3055918 Nov 27 17:41 fullbackup_2019-11-27_17:41:17.sql
[root@master ~]#scp /backup/fullbackup_2019-11-27_17\:41\:17.sql 
192.168.8.11:/data/
#建议优化主和从节点服务器的性能
MariaDB [hellodb]> set global innodb_flush_log_at_trx_commit=2
MariaDB [hellodb]> set global sync_binlog=0
MariaDB [hellodb]> set global innodb_flush_log_at_trx_commit=2;
Query OK, 0 rows affected (0.001 sec)
MariaDB [hellodb]> show variables like 'sync_binlog';
| Variable_name       | Value |
+---------------------+-------+
| sync_binlog         | 0     |
|---------------------+-------+
5 rows in set (0.001 sec) 
#将完全备份还原到新的从节点
[root@slave ~]#dnf -y install mariadb-server
[root@slave ~]#vim /etc/my.cnf.d/mariadb-server.cnf 
[mysqld]
server-id=11
read-only
[root@slave ~]#systemctl restart mariadb
#配置从节点，从完全备份的位置之后开始复制
[root@slave ~]#grep '^CHANGE MASTER' /data/fullbackup_2019-11-27_17\:41\:17.sql 
CHANGE MASTER TO MASTER_LOG_FILE='mariadb-bin.000003', MASTER_LOG_POS=389; 
[root@slave ~]#vim /data/fullbackup_2019-11-27_17\:41\:17.sql
CHANGE MASTER TO
MASTER_HOST='10.0.0.8',
MASTER_USER='repluser',
MASTER_PASSWORD='magedu',
MASTER_PORT=3306,                                                             
MASTER_LOG_FILE='mariadb-bin.000003', MASTER_LOG_POS=389; 
[root@slave ~]#mysql < /data/fullbackup_2019-11-27_17\:41\:17.sql
[root@slave ~]#mysql 
Welcome to the MariaDB monitor. Commands end with ; or \g.
Your MariaDB connection id is 9
Server version: 10.3.11-MariaDB MariaDB Server
Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
MariaDB [(none)]> show slave status\G;
*************************** 1. row ***************************
               Slave_IO_State: 
                   Master_Host: 10.0.0.8
                   Master_User: repluser
                   Master_Port: 3306
                 Connect_Retry: 60
               Master_Log_File: mariadb-bin.000003
           Read_Master_Log_Pos: 389
               Relay_Log_File: mariadb-relay-bin.000001
                 Relay_Log_Pos: 4
         Relay_Master_Log_File: mariadb-bin.000003
             Slave_IO_Running: No
             Slave_SQL_Running: No

4.1.1 Notes related to master-slave replication

限制从服务器为只读
read_only=ON
#注意：此限制对拥有SUPER权限的用户均无效
以下命令会阻止所有用户, 包括主服务器复制的更新
FLUSH TABLES WITH READ LOCK;
在从节点清除信息
以下都需要先 STOP SLAVE
RESET SLAVE #从服务器清除master.info ，relay-log.info, relay log ，开始新的relay log
RESET SLAVE  ALL #清除所有从服务器上设置的主服务器同步信息，如HOST，PORT, USER和 PASSWORD 等

复制错误解决方法
可以在从服务器忽略几个主服务器的复制事件，此为global变量，或指定跳过事件的ID
注意: Centos 8.1以上版本上的MariaDB10.3主从节点同时建同名的库和表不会冲突，建主键记录会产生
冲突
#系统变量，指定跳过复制事件的个数
SET GLOBAL sql_slave_skip_counter = N
#服务器选项，只读系统变量，指定跳过事件的ID
[mysqld]
slave_skip_errors=1007|ALL  

4.2 mysql master-slave replication steps

主主复制：两个节点，都可以更新数据，并且互为主从
容易产生的问题：数据不一致；因此慎用
考虑要点：自动增长id
配置一个节点使用奇数id
auto_increment_offset=1   #开始点
auto_increment_increment=2 #增长幅度
另一个节点使用偶数id
auto_increment_offset=2
auto_increment_increment=2
操作与mysql主从类同就不详细记录了
主主复制的配置步骤：
 (1) 各节点使用一个惟一server_id
 (2) 都启动binary log和relay log
 (3) 创建拥有复制权限的用户账号
 (4) 定义自动增长id字段的数值范围各为奇偶
 (5) 均把对方指定为主节点，并启动复制线程

4.3 Semi-synchronous replication

(If one of the two slave nodes 2 chooses to synchronize successfully, it will return to the client that the synchronization is successful. If there is a problem with the network of the master node and the synchronization cannot be performed, it will not wait for a timeout mechanism of 10 seconds, and directly return the result successfully)
By default, the replication function of MySQL is asynchronous. Asynchronous replication can provide the best performance. The master library sends the binlog log to the slave library and
ends immediately, and does not verify whether the slave library has received it. This means that when the master server or the slave server fails, the slave server may
not receive the binlog logs sent by the master server, which will cause data inconsistencies between the master server and the slave server, and even cause
data loss during recovery
MySQL5.5 version is to ensure the consistency of master-slave data. The component (plug-in) of semi-synchronous replication is added, which can control whether the slave library IO thread
transfers the relaylog to the disk, and once the disk is placed, the plug-in returns ACK to the master library ACK_REC. After receiving the ACK, the transaction of the main library can be submitted
successfully. By default, if no ACK is returned for more than 10 seconds, the replication behavior will switch to asynchronous replication.
MySQL5.6 and 5.7 have also added some better features, but the data consistency cannot be fully guaranteed. If the production business is more concerned about
the master-slave eventual consistency (for example: finance, etc.). Recommend an architecture that can use MGR, or a consistent architecture such as PXC

半同步复制默认设置
rpl_semi_sync_master_wait_point=after_commit
缺点：
缺点1： 幻读
当用户提交一个事务，该事务已经写入redo日志和binlog日志，但该事务还没写入从库，此时处在waiting 
slave dump处，此时另一个用户可以读取到这条数据，而他自己却不能；
缺点2：数据丢失
一个提交的事务在waiting slave dump处crash后，主库将比从库多一条数据
增强半同步复制(MySQL5.7新增功能)
rpl_semi_rsync_master_wait_point=after_sync
优点
改善1：解决幻读
当用户发起一个事务，该事务先写入二进制后，再向从库进行同步，由于还没有完成提交，此时其他用户无法
读取到该数据，解决了幻读
改善2：解决数据丢失
一个事务在waiting slave dump处crash掉后，可以通过观察从库上是否存在主库的last gtid值，如果
存在，这条数据正常恢复，如果不存在则删除主库的那条多余的GTID值，然后恢复，保证了数据的完整性

半同步复制实现：
官方文档：
https://dev.mysql.com/doc/refman/8.0/en/replication-semisync.html
https://dev.mysql.com/doc/refman/5.7/en/replication-semisync.html
https://mariadb.com/kb/en/library/semisynchronous-replication/

Example: CentOS8 implements semi-synchronous replication in MySQL8.0

#查看插件文件
[root@centos8 ~]#rpm -ql mysql-server |grep semisync
/usr/lib64/mysql/plugin/semisync_master.so
/usr/lib64/mysql/plugin/semisync_slave.so
#master服务器配置
[root@master ~]#vim /etc/my.cnf.d/mysql-server.cnf
[mysqld]
server-id=8
log-bin
rpl_semi_sync_master_enabled=ON     #修改此行,需要先安装semisync_master.so插件后,再重
启,否则无法启动
rpl_semi_sync_master_timeout=3000   #设置3s内无法同步，也将返回成功信息给客户端
#slave服务器配置
[root@slave1 ~]#vim /etc/my.cnf.d/mysql-server.cnf
[mysqld]
server-id=18
rpl_semi_sync_slave_enabled=ON #修改此行,需要先安装semisync_slave.so插件后,再重启,否则
无法启动
[root@slave2 ~]#vim /etc/my.cnf.d/mysql-server.cnf
[mysqld]
server-id=28
rpl_semi_sync_slave_enabled=ON #修改此行,需要先安装semisync_slave.so插件后,再重启,否则
无法启动
#主服务器配置:
mysql>INSTALL PLUGIN rpl_semi_sync_master SONAME 'semisync_master.so'; #永久安装插
件
mysql>UNINSTALL PLUGIN rpl_semi_sync_master ;
mysql>SHOW PLUGINS; #查看插件
mysql>SET GLOBAL rpl_semi_sync_master_enabled=1; #临时修改变量
mysql>SET GLOBAL rpl_semi_sync_master_timeout = 3000;  #超时长1s,默认值为10s
mysql>SHOW GLOBAL VARIABLES LIKE '%semi%'; +-------------------------------------------+------------+
| Variable_name                             | Value     |
+-------------------------------------------+------------+
| rpl_semi_sync_master_enabled             | ON         |
| rpl_semi_sync_master_timeout             | 10000     |
| rpl_semi_sync_master_trace_level         | 32         |
| rpl_semi_sync_master_wait_for_slave_count | 1         |
| rpl_semi_sync_master_wait_no_slave       | ON         |
| rpl_semi_sync_master_wait_point           | AFTER_SYNC |
+-------------------------------------------+------------+ 6 rows in set (0.00 sec)
mysql> SHOW GLOBAL STATUS LIKE '%semi%'; +--------------------------------------------+-------+
| Variable_name                             | Value |
+--------------------------------------------+-------+
| Rpl_semi_sync_master_clients               | 2     |
| Rpl_semi_sync_master_net_avg_wait_time     | 0     |
| Rpl_semi_sync_master_net_wait_time         | 0     |
| Rpl_semi_sync_master_net_waits             | 0     |
| Rpl_semi_sync_master_no_times             | 1     |
| Rpl_semi_sync_master_no_tx                 | 2     |
| Rpl_semi_sync_master_status               | ON   |
| Rpl_semi_sync_master_timefunc_failures     | 0     |
| Rpl_semi_sync_master_tx_avg_wait_time     | 0     |
| Rpl_semi_sync_master_tx_wait_time         | 0     |
| Rpl_semi_sync_master_tx_waits             | 0     |
| Rpl_semi_sync_master_wait_pos_backtraverse | 0     |
| Rpl_semi_sync_master_wait_sessions         | 0     |
| Rpl_semi_sync_master_yes_tx               | 0     |
+--------------------------------------------+-------+
14 rows in set (0.00 sec)
#从服务器配置:
mysql>INSTALL PLUGIN rpl_semi_sync_slave SONAME 'semisync_slave.so';
mysql>SET GLOBAL rpl_semi_sync_slave_enabled=1; #临时修改变量
mysql> SHOW GLOBAL VARIABLES LIKE '%semi%'; +---------------------------------+-------+
| Variable_name                   | Value |
+---------------------------------+-------+
| rpl_semi_sync_slave_enabled     | ON   |
| rpl_semi_sync_slave_trace_level | 32   |
+---------------------------------+-------+ 2 rows in set (0.00 sec)
#注意:如果已经实现主从复制,需要stop slave;start slave;
mysql> stop slave;
mysql> start slave;
mysql> SHOW GLOBAL STATUS LIKE '%semi%'; +----------------------------+-------+
| Variable_name             | Value |
+----------------------------+-------+
| Rpl_semi_sync_slave_status | ON   |
+----------------------------+-------+ 1 row in set (0.00 sec)

Example: CentOS 8 implements semi-synchronous replication on Mariadb-10.3.11

#在master实现，启用半同步功能
[root@master ~]#vim /etc/my.cnf.d/mariadb-server.cnf
[mysqld]
server-id=8
log-bin
plugin-load-add = semisync_master
rpl_semi_sync_master_enabled=ON
rpl_semi_sync_master_timeout=3000   #设置3s内无法同步，也将返回成功信息给客户端
[root@centos8 ~]#systemctl restart mariadb
MariaDB [(none)]> SHOW GLOBAL VARIABLES LIKE '%semi%';
+---------------------------------------+--------------+
| Variable_name                         | Value        |
+---------------------------------------+--------------+
| rpl_semi_sync_master_enabled          | ON           | | rpl_semi_sync_master_timeout          | 3000         | | rpl_semi_sync_master_trace_level      | 32           | | rpl_semi_sync_master_wait_no_slave    | ON           | | rpl_semi_sync_master_wait_point       | AFTER_COMMIT | | rpl_semi_sync_slave_delay_master      | OFF          | | rpl_semi_sync_slave_enabled           | OFF          | | rpl_semi_sync_slave_kill_conn_timeout | 5            | | rpl_semi_sync_slave_trace_level       | 32           |
+---------------------------------------+--------------+
9 rows in set (0.002 sec)
MariaDB [(none)]> SHOW GLOBAL STATUS LIKE '%semi%';
+--------------------------------------------+-------+
| Variable_name                              | Value |
+--------------------------------------------+-------+
| Rpl_semi_sync_master_clients               | 0     | | Rpl_semi_sync_master_get_ack               | 0     | | Rpl_semi_sync_master_net_avg_wait_time     | 0     | | Rpl_semi_sync_master_net_wait_time         | 0     | | Rpl_semi_sync_master_net_waits             | 0     | | Rpl_semi_sync_master_no_times              | 0     | | Rpl_semi_sync_master_no_tx                 | 0     | | Rpl_semi_sync_master_request_ack           | 0     | | Rpl_semi_sync_master_status                | ON    | | Rpl_semi_sync_master_timefunc_failures     | 0     | | Rpl_semi_sync_master_tx_avg_wait_time      | 0     |
| Rpl_semi_sync_master_tx_wait_time          | 0     | | Rpl_semi_sync_master_tx_waits              | 0     | | Rpl_semi_sync_master_wait_pos_backtraverse | 0     | | Rpl_semi_sync_master_wait_sessions         | 0     | | Rpl_semi_sync_master_yes_tx                | 0     | | Rpl_semi_sync_slave_send_ack               | 0     | | Rpl_semi_sync_slave_status                 | OFF   |
+--------------------------------------------+-------+
18 rows in set (0.001 sec) #在其它所有slave节点上都实现，启用半同步功能
[root@slave ~]#vim /etc/my.cnf.d/mariadb-server.cnf 
[mysqld]
server-id=18
plugin_load_add = semisync_slave
rpl_semi_sync_slave_enabled=ON  
[root@slave ~]#systemctl restart mariadb
[root@slave ~]#mysql 
MariaDB [(none)]>  SHOW GLOBAL VARIABLES  LIKE '%semi%';
+---------------------------------------+--------------+
| Variable_name                         | Value        |
+---------------------------------------+--------------+
| rpl_semi_sync_master_enabled          | OFF          | | rpl_semi_sync_master_timeout          | 10000        | | rpl_semi_sync_master_trace_level      | 32           | | rpl_semi_sync_master_wait_no_slave    | ON           | | rpl_semi_sync_master_wait_point       | AFTER_COMMIT | | rpl_semi_sync_slave_delay_master      | OFF          | | rpl_semi_sync_slave_enabled           | ON           | | rpl_semi_sync_slave_kill_conn_timeout | 5            | | rpl_semi_sync_slave_trace_level       | 32           |
+---------------------------------------+--------------+
9 rows in set (0.001 sec)
MariaDB [(none)]>  SHOW GLOBAL STATUS  LIKE '%semi%';
+--------------------------------------------+-------+
| Variable_name                              | Value |
+--------------------------------------------+-------+
| Rpl_semi_sync_master_clients               | 0     | | Rpl_semi_sync_master_get_ack               | 0     | | Rpl_semi_sync_master_net_avg_wait_time     | 0     | | Rpl_semi_sync_master_net_wait_time         | 0     | | Rpl_semi_sync_master_net_waits             | 0     | | Rpl_semi_sync_master_no_times              | 0     | | Rpl_semi_sync_master_no_tx                 | 0     | | Rpl_semi_sync_master_request_ack           | 0     | | Rpl_semi_sync_master_status                | OFF   | | Rpl_semi_sync_master_timefunc_failures     | 0     | | Rpl_semi_sync_master_tx_avg_wait_time      | 0     | | Rpl_semi_sync_master_tx_wait_time          | 0     | | Rpl_semi_sync_master_tx_waits              | 0     | | Rpl_semi_sync_master_wait_pos_backtraverse | 0     | | Rpl_semi_sync_master_wait_sessions         | 0     | | Rpl_semi_sync_master_yes_tx                | 0     | | Rpl_semi_sync_slave_send_ack               | 0     | | Rpl_semi_sync_slave_status                 | ON    |
+--------------------------------------------+-------+
18 rows in set (0.001 sec)
MariaDB [(none)]> #在master上实现
MariaDB [db1]> SHOW GLOBAL STATUS LIKE '%semi%';
+--------------------------------------------+-------+
| Variable_name                              | Value |
+--------------------------------------------+-------+
| Rpl_semi_sync_master_clients               | 2     |  #两个从节点
| Rpl_semi_sync_master_get_ack               | 4     | | Rpl_semi_sync_master_net_avg_wait_time     | 0     | | Rpl_semi_sync_master_net_wait_time         | 0     | | Rpl_semi_sync_master_net_waits             | 4     | | Rpl_semi_sync_master_no_times              | 1     | | Rpl_semi_sync_master_no_tx                 | 1     | | Rpl_semi_sync_master_request_ack           | 3     | | Rpl_semi_sync_master_status                | ON    | | Rpl_semi_sync_master_timefunc_failures     | 0     | | Rpl_semi_sync_master_tx_avg_wait_time      | 1177  | | Rpl_semi_sync_master_tx_wait_time          | 2355  | | Rpl_semi_sync_master_tx_waits              | 2     | | Rpl_semi_sync_master_wait_pos_backtraverse | 0     | | Rpl_semi_sync_master_wait_sessions         | 0     | | Rpl_semi_sync_master_yes_tx                | 2     | | Rpl_semi_sync_slave_send_ack               | 0     | | Rpl_semi_sync_slave_status                 | OFF   |
+--------------------------------------------+-------+
18 rows in set (0.001 sec) #测试
#在master实现，创建数据库，立即成功
MariaDB [db1]> create database db2;
Query OK, 1 row affected (0.004 sec) #在所有slave节点实现，停止复制线程
MariaDB [(none)]> stop slave;
Query OK, 0 rows affected (0.011 sec) #在master实现，创建数据库，等待3s才能成功
MariaDB [db1]> create database db3;
Query OK, 1 row affected (3.003 sec) #在任意一个slave节点实现，恢复复制线程
MariaDB [(none)]> start slave;
Query OK, 0 rows affected (0.006 sec) #在master实现，创建数据库，立即成功
MariaDB [db1]> create database db4;
Query OK, 1 row affected (0.002 sec) #在所有从节点停止同步线程,在主节点可以看到以下日志信息
MariaDB [db1]> stop slave; [root@centos8 ~]#tail /var/log/mariadb/mariadb.log
2020-08-29 10:11:19 15 [Warning] IP address '10.0.0.28' could not be resolved: 
Name or service not known
2020-08-29 10:11:19 15 [Note] Start binlog_dump to slave_server(28), 
pos(mariadb-bin.000001, 330)
2020-08-29 10:11:19 15 [Note] Start semi-sync binlog_dump to slave (server_id: 
28), pos(mariadb-bin.000001, 330)
2020-08-29 10:12:34 15 [Note] Stop semi-sync binlog_dump to slave (server_id: 
28)
2020-08-29 10:16:05 17 [Note] Start binlog_dump to slave_server(28), 
pos(mariadb-bin.000002, 27378670)
2020-08-29 10:16:05 17 [Note] Start semi-sync binlog_dump to slave (server_id: 
28), pos(mariadb-bin.000002, 27378670)
2020-08-29 10:16:31 12 [Note] Stop semi-sync binlog_dump to slave (server_id: 
18)
2020-08-29 10:16:37 17 [Note] Stop semi-sync binlog_dump to slave (server_id: 
28)
2020-08-29 10:17:19 14 [Warning] Timeout waiting for reply of binlog (file: 
mariadb-bin.000002, pos: 27378922), semi-sync up to file mariadb-bin.000002, 
position 27378795.
2020-08-29 10:17:19 14 [Note] Semi-sync replication switched OFF.

Example: CentOS 7 implements semi-synchronous replication of Mariadb 5.5.65

#主服务器配置:
MariaDB [(none)]>INSTALL PLUGIN rpl_semi_sync_master SONAME
'semisync_master.so';
MariaDB [(none)]>UNINSTALL PLUGIN rpl_semi_sync_master ;
MariaDB [(none)]>SHOW PLUGINS; #查看插件
MariaDB [(none)]>SET GLOBAL rpl_semi_sync_master_enabled=1;
MariaDB [(none)]>SET GLOBAL rpl_semi_sync_master_timeout = 1000;  #超时长1s,默认值
为10s
MariaDB [(none)]>SHOW GLOBAL VARIABLES LIKE '%semi%';
MariaDB [(none)]>SHOW GLOBAL STATUS LIKE '%semi%'; #从服务器配置:
MariaDB [(none)]>INSTALL PLUGIN rpl_semi_sync_slave SONAME 'semisync_slave.so';
MariaDB [(none)]>SET GLOBAL rpl_semi_sync_slave_enabled=1;

Five, use mycat to realize the separation of reading and writing of mysql

Mycat installation

下载安装JDK
yum -y install java
#确认安装成功
java -version
openjdk version "1.8.0_201"
OpenJDK Runtime Environment (build 1.8.0_201-b09)
OpenJDK 64-Bit Server VM (build 25.201-b09, mixed mode)

Download and install mycat

wget http://dl.mycat.org.cn/1.6.7.4/Mycat-server-1.6.7.4-release/Mycat-server-
1.6.7.4-release-20200105164103-linux.tar.gz
mkdir /apps
tar xvf Mycat-server-1.6.7.4-release-20200105164103-linux.tar.gz  -C /apps
ls /apps/mycat/bin
 catlet conf lib logs version.txt

Mycat installation directory structure:

bin mycat命令，启动、重启、停止等
catlet catlet为Mycat的一个扩展功能
conf Mycat 配置信息,重点关注
lib Mycat引用的jar包，Mycat是java开发的
logs 日志文件，包括Mycat启动的日志和运行的日志
version.txt mycat版本说明
logs目录:
wrapper.log mycat启动日志
mycat.log mycat详细工作日志
Mycat的配置文件都在conf目录里面，这里介绍几个常用的文件：
server.xml Mycat软件本身相关的配置文件，设置账号、参数等
schema.xml Mycat对应的物理数据库和数据库表的配置,读写分离、高可用、分布式策略定制、
节点控制
rule.xml Mycat分片（分库分表）规则配置文件,记录分片规则列表、使用方法等

start and connect

#配置环境变量
vim /etc/profile.d/mycat.sh
PATH=/apps/mycat/bin:$PATH
source /etc/profile.d/mycat.sh
#启动
mycat start
#查看日志，确定成功
cat /app/mycat/logs/wrapper.log 
...省略...
INFO   | jvm 1   | 2019/11/01 21:41:02 | MyCAT Server startup successfully. see 
logs in logs/mycat.log
#连接mycat：
mysql -uroot -p123456 -h 127.0.0.1 -P8066

Mycat main configuration file description

server.xml
存放Mycat软件本身相关的配置文件，比如：连接Mycat的用户，密码，数据库名称等
server.xml文件中配置的参数解释说明：
参数 说明
user 用户配置节点
name 客户端登录MyCAT的用户名，也就是客户端用来连接Mycat的用户名。
password 客户端登录MyCAT的密码
schemas 数据库名，这里会和schema.xml中的配置关联，多个用逗号分开，例如:db1,db2
privileges 配置用户针对表的增删改查的权限
readOnly mycat逻辑库所具有的权限。true为只读，false为读写都有，默认为false
注意：
server.xml文件里登录mycat的用户名和密码可以任意定义，这个账号和密码是为客户机登录
mycat时使用的账号信息
逻辑库名(如上面的TESTDB，也就是登录mycat后显示的库名，切换这个库之后，显示的就是代理
的真实mysql数据库的表)要在schema.xml里面也定义，否则会导致mycat服务启动失败！
这里只定义了一个标签，所以把多余的都注释了。如果定义多个标签，即设置多个连接mycat的用
户名和密码，那么就需要在schema.xml文件中定义多个对应的库！
schema.xml
是最主要的配置项，此文件关联mysql读写分离策略，读写分离、分库分表策略、分片节点都是在此文
件中配置的.MyCat作为中间件，它只是一个代理，本身并不进行数据存储，需要连接后端的MySQL物理
服务器，此文件就是用来连接MySQL服务器的
schema.xml文件中配置的参数解释说明：
参数 说明
schema 数据库设置，此数据库为逻辑数据库，name与server.xml中schema对应
dataNode 分片信息，也就是分库相关配置
dataHost 物理数据库，真正存储数据的数据库
配置说明
name属性唯一标识dataHost标签，供上层的标签使用。
maxCon属性指定每个读写实例连接池的最大连接。也就是说，标签内嵌套的writeHost、readHost标
签都会使用这个属性的值来实例化出连接池的最大连接数
minCon属性指定每个读写实例连接池的最小连接，初始化连接池的大小
每个节点的属性逐一说明
schema:
属性 说明
name 逻辑数据库名，与server.xml中的schema对应
checkSQLschema 数据库前缀相关设置，这里为false
sqlMaxLimit select 时默认的limit，避免查询全表
table
属性 说明
name 表名，物理数据库中表名
dataNode 表存储到哪些节点，多个节点用逗号分隔。节点为下文dataNode设置的name
primaryKey 主键字段名，自动生成主键时需要设置
autoIncrement 是否自增
rule 分片规则名，具体规则下文rule详细介绍
dataNode
属性 说明
name 节点名，与table中dataNode对应
datahost 物理数据库名，与datahost中name对应
database 物理数据库中数据库名
dataHost
属性 说明
name 物理数据库名，与dataNode中dataHost对应
balance 均衡负载的方式
writeType 写入方式
dbType 数据库类型
heartbeat 心跳检测语句，注意语句结尾的分号要加
schema.xml文件中有三点需要注意：balance="1"，writeType="0" ,switchType="1" 
schema.xml中的balance的取值决定了负载均衡对非事务内的读操作的处理。balance 属性负载均衡类
型，目前的取值有 4 种：
balance="0"：不开启读写分离机制，所有读操作都发送到当前可用的writeHost上,即读请求仅发送到
writeHost上
balance="1"：一般用此模式，读请求随机分发到当前writeHost对应的readHost和standby的
writeHost上。即全部的readHost与stand by writeHost 参与 select 语句的负载均衡，简单的说，当双
主双从模式(M1 ->S1 ， M2->S2，并且 M1 与 M2 互为主备)，正常情况下， M2,S1, S2 都参与 select 语句的负载均衡
balance="2"：读请求随机分发到当前dataHost内所有的writeHost和readHost上。即所有读操作都随
机的在writeHost、 readhost 上分发
balance="3"：读请求随机分发到当前writeHost对应的readHost上。即所有读请求随机的分发到
wiriterHost 对应的 readhost 执行, writerHost 不负担读压力，注意 balance=3 只在 1.4 及其以后版本
有，1.3 没有
writeHost和readHost 标签
这两个标签都指定后端数据库的相关配置给mycat，用于实例化后端连接池。
唯一不同的是：writeHost指定写实例、readHost指定读实例，组着这些读写实例来满足系统的要求。
在一个dataHost内可以定义多个writeHost和readHost。但是，如果writeHost指定的后端数据库宕机，
那么这个writeHost绑定的所有readHost都将不可用。另一方面，由于这个writeHost宕机系统会自动的
检测到，并切换到备用的writeHost上去
注意：
Mycat主从分离只是在读的时候做了处理，写入数据的时候，只会写入到writehost，需要通过mycat的
主从复制将数据复制到readhost

5.1 Practical case: use Mycat to realize the separation of reading and writing of MySQL

System environment of all hosts

cat /etc/centos-release
CentOS Linux release 8.0.1905 (Core) 
mycat-server 10.0.0.8 #内存建议2G以上
mysql-master 10.0.0.18 MySQL 8.0 或者Mariadb 10.3.17
mysql-slave  10.0.0.28 MySQL 8.0 或者Mariadb 10.3.17

Turn off SELinux and Firewall

systemctl stop firewalld
setenforce 0
时间同步

1. Create a MySQL master-slave database

[root@centos8 ~]#yum -y install mysql-server
#或者
[root@centos8 ~]#yum -y install mariadb-server 

1. Modify configuration files on master and slave

#master上的my.cnf
[root@centos8 ~]#vim /etc/my.cnf.d/mariadb-server.cnf
[mysqld]
server-id = 1
log-bin
#slave上的my.cnf
[mysqld]
server-id = 2 
[root@centos8 ~]#systemctl start mariadb

2. Create a replication user on the master

[root@centos8 ~]#mysql -uroot -p
MariaDB [(none)]>GRANT REPLICATION SLAVE ON *.* TO 'repluser'@'10.0.0.%' 
IDENTIFIED BY 'replpass';
mysql> FLUSH PRIVILEGES;    
mysql> show master status;
+------------------+----------+--------------+------------------+----------------
---+
| File             | Position | Binlog_Do_DB | Binlog_Ignore_DB |
Executed_Gtid_Set |
+------------------+----------+--------------+------------------+----------------
---+
|mariadb-bin.000001|      403 |              |                  |               
    |
+------------------+----------+--------------+------------------+----------------
---+
1 row in set (0.00 sec)

3. Execute on Slave

[root@centos8 ~]#mysql -uroot -p
mysql> CHANGE MASTER TO
->     MASTER_HOST='10.0.0.%',
->     MASTER_USER='repluser',
->     MASTER_PASSWORD='replpass',
->     MASTER_LOG_FILE='mariadb-bin.000001',
->     MASTER_LOG_POS=403;
mysql> start slave;

2. Install mycat on the MySQL proxy server 10.0.0.8 and start it

root@centos8 ~]#yum -y install java 
#确认安装成功
[root@centos8 ~]#java -version
openjdk version "1.8.0_201"
OpenJDK Runtime Environment (build 1.8.0_201-b09)
OpenJDK 64-Bit Server VM (build 25.201-b09, mixed mode)
#下载并安装
[root@centos8 ~]#wget http://dl.mycat.org.cn/1.6.7.6/20210303094759/Mycat-server-
1.6.7.6-release-20210303094759-linux.tar.gz
#wget http://dl.mycat.org.cn/1.6.7.4/Mycat-server-1.6.7.4-release/Mycat-server-
1.6.7.4-release-20200105164103-linux.tar.gz
[root@centos8 ~]#mkdir /apps
[root@centos8 ~]#tar xvf Mycat-server-1.6.7.6-release-20210303094759-linux.tar.gz
-C /apps/
#tar xvf Mycat-server-1.6.7.4-release-20200105164103-linux.tar.gz -C /apps
#配置环境变量
[root@centos8 ~]#echo 'PATH=/apps/mycat/bin:$PATH' > /etc/profile.d/mycat.sh
[root@centos8 ~]#source /etc/profile.d/mycat.sh
#查看端口
[root@centos8 ~]#ss -ntl
State         Recv-Q       Send-Q     Local Address:Port     Peer Address:Port   
              
LISTEN        0             128              0.0.0.0:22            0.0.0.0:*     
               
LISTEN        0             128                 [::]:22               [::]:*  
#启动mycat
[root@mycat ~]#file /apps/mycat/bin/mycat 
/apps/mycat/bin/mycat: POSIX shell script, ASCII text executable
[root@mycat ~]#mycat
Usage: /apps/mycat/bin/mycat {
    
     console | start | stop | restart | status | dump 
}
#注意: 此步启动较慢,需要等一会儿,另外如果内存太小,会导致无法启动
[root@centos8 ~]#mycat start
Starting Mycat-server...
#可以看到打开多个端口,其中8066端口用于连接MyCAT
[root@centos8 ~]#ss -ntlp
State           Recv-Q           Send-Q Local Address:Port   Peer Address:Port   
                            
LISTEN          0                128          0.0.0.0:22          0.0.0.0:*     
          users:(("sshd",pid=791,fd=5))             
LISTEN          0                1          127.0.0.1:32000       0.0.0.0:*     
          users:(("java",pid=4640,fd=4))            
LISTEN          0                128             [::]:22             [::]:*     
          users:(("sshd",pid=791,fd=7))             
LISTEN          0                50                 *:1984             *:*     
          users:(("java",pid=4640,fd=57))           
LISTEN          0                100               *:8066             *:*     
          users:(("java",pid=4640,fd=87))           
LISTEN          0                50                 *:43465             *:*     
          users:(("java",pid=4640,fd=58))           
LISTEN          0                100               *:9066             *:*     
          users:(("java",pid=4640,fd=83))           
LISTEN          0                50                 *:45259             *:*     
          users:(("java",pid=4640,fd=56))
#查看日志，确定成功，可能需要等一会儿才能看到成功的提示
[root@centos8 ~]#tail /apps/mycat/logs/wrapper.log 
ERROR | wrapper | 2020/02/28 15:21:48 | Startup failed: Timed out waiting for
a signal from the JVM.
ERROR | wrapper | 2020/02/28 15:21:48 | JVM did not exit on request, 
terminated
INFO   | wrapper | 2020/02/28 15:21:48 | JVM exited on its own while waiting to 
kill the application.
STATUS | wrapper | 2020/02/28 15:21:48 | JVM exited in response to signal 
SIGKILL (9).
STATUS | wrapper | 2020/02/28 15:21:52 | Launching a JVM...
INFO   | jvm 2   | 2020/02/28 15:21:52 | OpenJDK 64-Bit Server VM warning: 
ignoring option MaxPermSize=64M; support was removed in 8.0
INFO   | jvm 2   | 2020/02/28 15:22:13 | Wrapper (Version 3.2.3) 
http://wrapper.tanukisoftware.org
INFO   | jvm 2   | 2020/02/28 15:22:13 |   Copyright 1999-2006 Tanuki Software, 
Inc. All Rights Reserved.
INFO   | jvm 2   | 2020/02/28 15:22:13 | 
INFO   | jvm 2   | 2020/02/28 15:22:31 | MyCAT Server startup successfully. see 
logs in logs/mycat.log
#用默认密码123456来连接mycat
[root@centos8 ~]#mysql -uroot -p123456 -h 10.0.0.8 -P8066
Welcome to the MariaDB monitor. Commands end with ; or \g.
Your MySQL connection id is 1
Server version: 5.6.29-mycat-1.6-RELEASE-20161028204710 MyCat Server 
(OpenCloundDB)
Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
MySQL [(none)]> show databases;
+----------+
| DATABASE |
+----------+
| TESTDB   |
+----------+ 
1 row in set (0.01 sec)
MySQL [TESTDB]> show tables;
+------------------+
| Tables in TESTDB |
+------------------+
| address         |
| travelrecord     |
+------------------+ 
2 rows in set (0.01 sec)
MySQL [TESTDB]> select * from travelrecord ;
ERROR 1105 (HY000): backend connect: java.lang.IllegalArgumentException: Invalid 
DataSource:0
MySQL [TESTDB]>

4. Modify the server.xml file on the mycat server to configure the connection information of Mycat

[root@centos8 ~]#vim /apps/mycat/conf/server.xml
...省略...
#修改下面行的8066改为3306复制到到独立非注释行
<property name="serverPort">3306</property>
<property name="handlelDistributedTransactions">0</property> #将上面行放在此行前面
#或者删除注释,并修改下面行的8066改为3306
<property name="serverPort">3306</property>
<property name="managerPort">9066</property>
<property name="idleTimeout">300000</property>
<property name="authTimeout">15000</property>
<property name="bindIp">0.0.0.0</property>
<property name="dataNodeIdleCheckPeriod">300000</property> #5 * 60 * 1000L; //连
接空闲检查 删除#后面此部分
<property name="frontWriteQueueSize">4096</property> <property 
name="processors">32</property> #--> 删除#后面此部分
 .....
<user name="root">                                       #连接Mycat的用户名
   <property name="password">magedu</property>          #连接Mycat的密码
   <property name="schemas">TESTDB</property>           #数据库名要和schema.xml相
对应
</user>
</mycat:server>

The root is used here, the password is magedu, and the logical database is TESTDB. These information can be freely defined, and both read and write permissions
are available. There is no special permission for the table. Focus on the above configuration, and leave the rest as default.
5. Modify schema.xml to implement read-write separation strategy

[root@centos8 ~]#vim /apps/mycat/conf/schema.xml                                 
      
<!DOCTYPE mycat:schema SYSTEM "schema.dtd">
<mycat:schema xmlns:mycat="http://io.mycat/">
<schema name="TESTDB" checkSQLschema="***false***" sqlMaxLimit="100"
dataNode="***dn1***"></schema>
<dataNode name="dn1" dataHost="localhost1" database="***mycat***" />  #其中mycat表
示后端服务器实际的数据库名称
<dataHost name="localhost1" maxCon="1000" minCon="10" balance="***1***"
writeType="0" dbType="mysql" dbDriver="native" switchType="1"
slaveThreshold="100">
<heartbeat>select user()</heartbeat>
***<writeHost host="host1" url="10.0.0.18:3306" user="root"
password="123456">***
***<readHost host="host2" url="10.0.0.28:3306" user="root" password="123456"
/>***
</writeHost>
</dataHost>
</mycat:schema>
#以上***部分表示原配置文件中需要修改的内容
#注意大小写
#最终文件内容
[root@mycat ~]#cat /apps/mycat/conf/schema.xml
<?xml version="1.0"?>
<!DOCTYPE mycat:schema SYSTEM "schema.dtd">
<mycat:schema xmlns:mycat="http://io.mycat/">
 <schema name="TESTDB" checkSQLschema="false" sqlMaxLimit="100"
dataNode="dn1">
 </schema>
 <dataNode name="dn1" dataHost="localhost1" database="hellodb" />
 <dataHost name="localhost1" maxCon="1000" minCon="10" balance="1"
  writeType="0" dbType="mysql" dbDriver="native" switchType="1"
slaveThreshold="100">
 <heartbeat>select user()</heartbeat>
 <writeHost host="host1" url="10.0.0.18:3306" user="root"
   password="123456">
         <readHost host="host2" url="10.0.0.28:3306" user="root"
password="123456" />
 </writeHost>
   </dataHost>
</mycat:schema>
#重新启动mycat
[root@centos8 ~]#mycat restart

In the above configuration, the balance is changed to 1, which means read-write separation. The effect achieved by the above configuration is that 10.0.0.18 is the master database, and 10.0.0.28 is
the slave database
. Note: Make sure that you can successfully log in to the mysql database on the 10.0.0.18 and 10.0.0.28 machines with the root/123456 authority. At the same time,
you must also authorize the mycat machine to successfully log in to the mysql database of these two machines with root/123456 permissions! ! This is very important, otherwise
it will cause failure of library and table operations after logging in to mycat!
Example: schema.xml

6. Create a user on the backend main server and authorize mycat

[root@centos8 ~]#mysql -uroot -p
mysql> create database mycat;
mysql>GRANT ALL ON *.* TO 'root'@'10.0.0.%' IDENTIFIED BY '123456' ;
mysql> flush privileges; 

7. Connect and test on the Mycat server

[root@centos8 ~]#mysql -uroot -pmagedu -h127.0.0.1 TESTDB
mysql> show databases;
+----------+
| DATABASE |
+----------+
| TESTDB   |   //只能看一个虚拟数据库
+----------+
mysql> use TESTDB;
mysql> create table t1(id int);
MySQL> select @@server_id;
MySQL> select @@hostname; 

8. Realize read-write separation through general log confirmation

在mysql中查看通用日志
show variables like 'general_log';  #查看日志是否开启
set global general_log=on;    #开启日志功能
show variables like 'general_log_file'; #查看日志文件保存位置
set global general_log_file='tmp/general.log'; #设置日志文件保存位置
在主和从服务器分别启用通用日志，查看读写分离
[root@centos8 ~]#vim /etc/my.cnf.d/mariadb-server.cnf
[mysqld]
general_log=ON
[root@centos8 ~]#systemctl restart mariadb
[root@centos8 ~]#tail -f /var/lib/mysql/centos8.log 

9. Stop the slave node, MyCAT automatically dispatches the read request to the master node

[root@slave ~]#systemctl stop mariadb
[root@client ~]#mysql -uroot -pmagedu -h10.0.0.8 -P8066
MySQL [(none)]> select @@server_id;
+-------------+
| @@server_id |
+-------------+
|          1 |
+-------------+ 
1 row in set (0.00 sec)
MySQL [(none)]> 
#停止主节点，MyCAT不会自动调度写请求至从节点
MySQL [TESTDB]> insert teachers values(5,'wang',30,'M');
ERROR 1184 (HY000): java.net.ConnectException: Connection refused

10. MyCAT's health check method for the back-end server select user()

#开启通用日志
[root@master ~]#mysql
mysql> set global  general_log=1;
[root@slave ~]#mysql
mysql> set global  general_log=1; 
#查看通用日志
[root@master ~]#tail -f /var/lib/mysql/master.log 
/usr/libexec/mysqld, Version: 8.0.17 (Source distribution). started with:
Tcp port: 3306 Unix socket: /var/lib/mysql/mysql.sock
Time                 Id Command   Argument
2021-02-22T08:52:57.086198Z   17 Query select user()
2021-02-22T08:53:07.086340Z   24 Query select user()
2021-02-22T08:53:17.086095Z   16 Query select user()
2021-02-22T08:53:27.086629Z   18 Query select user()
[root@slave ~]#tail -f /var/lib/mysql/slave.log 
/usr/libexec/mysqld, Version: 8.0.17 (Source distribution). started with:
Tcp port: 3306 Unix socket: /var/lib/mysql/mysql.sock
Time                 Id Command   Argument
2021-02-22T08:46:01.437376Z   10 Query select user()
2021-02-22T08:46:11.438172Z   11 Query select user()
2021-02-22T08:46:21.437458Z   12 Query select user()
2021-02-22T08:46:31.437742Z   13 Query select user()

Sixth, implement openvpn deployment, and pass the test, output blog or own document archive.