Preface
Yesterday's I implemented a three-nodes PXC,but there were some errors when proceeding it.So I decide to research what's wrong with it.The operations and evidences shows below.
Procedure
Fisrt of all,I find out that yesterday's error is due to lack of "socat" package on first node zlm2.
1 //Here's the error log of it. 2 2018-08-08T08:48:18.198487+01:00 0 [Note] WSREP: Initiating SST/IST transfer on DONOR side (wsrep_sst_xtrabackup-v2 --role 'donor' --address '192.168.1.102:4444/xtrabackup_sst//1' --socket '/tmp/mysql3308.sock' --datadir '/data/mysql/mysql3308/data/' --defaults-file '/data/mysql/mysql3308/my3308.cnf' --defaults-group-suffix '' --binlog '/data/mysql/mysql3308/logs/mysql-bin' --gtid 'bd5525ab-9a15-11e8-aa0f-4b830c783fc7:21') 3 2018-08-08T08:48:18.200272+01:00 2 [Note] WSREP: DONOR thread signaled with 0 4 which: no socat in (/usr/sbin:/sbin:/opt/mysql/Percona-XtraDB-Cluster-5.7.22-rel22-29.26.1.Linux.x86_64.ssl102/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin:/usr/local/mysql/bin:/usr/local/zabbix/bin:/usr/local/zabbix/sbin) 5 2018-08-08T06:48:18.782344Z WSREP_SST: [ERROR] ******************* FATAL ERROR ********************** 6 2018-08-08T06:48:18.784043Z WSREP_SST: [ERROR] socat not found in path: /usr/sbin:/sbin:/opt/mysql/Percona-XtraDB-Cluster-5.7.22-rel22-29.26.1.Linux.x86_64.ssl102/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin:/usr/local/mysql/bin:/usr/local/zabbix/bin:/usr/local/zabbix/sbin 7 2018-08-08T06:48:18.785926Z WSREP_SST: [ERROR] ****************************************************** 8 2018-08-08T08:48:18.786496+01:00 0 [ERROR] WSREP: Process completed with error: wsrep_sst_xtrabackup-v2 --role 'donor' --address '192.168.1.102:4444/xtrabackup_sst//1' --socket '/tmp/mysql3308.sock' --datadir '/data/mysql/mysql3308/data/' --defaults-file '/data/mysql/mysql3308/my3308.cnf' --defaults-group-suffix '' --binlog '/data/mysql/mysql3308/logs/mysql-bin' --gtid 'bd5525ab-9a15-11e8-aa0f-4b830c783fc7:21': 2 (No such file or directory) 9 2018-08-08T08:48:18.786587+01:00 0 [ERROR] WSREP: Command did not run: wsrep_sst_xtrabackup-v2 --role 'donor' --address '192.168.1.102:4444/xtrabackup_sst//1' --socket '/tmp/mysql3308.sock' --datadir '/data/mysql/mysql3308/data/' --defaults-file '/data/mysql/mysql3308/my3308.cnf' --defaults-group-suffix '' --binlog '/data/mysql/mysql3308/logs/mysql-bin' --gtid 'bd5525ab-9a15-11e8-aa0f-4b830c783fc7:21' 10 2018-08-08T08:48:18.787610+01:00 0 [Warning] WSREP: 1.0 (zlm2): State transfer to 0.0 (zlm3) failed: -2 (No such file or directory) 11 12 [root@zlm2 08:53:31 /data/mysql/mysql3308/data] 13 #rpm -qa|grep socat 14 15 [root@zlm2 09:11:08 /data/mysql/mysql3308/data] 16 #yum install socat 17 ... //Omitted. 18 19 Installed: 20 socat.x86_64 0:1.7.3.2-2.el7 21 22 Complete!
Restart the PXC mysqld on zlm2.
1 [root@zlm2 09:21:38 /data/mysql/mysql3308/data] 2 #mysqladmin -P3308 shutdown 3 4 [root@zlm2 09:22:10 /data/mysql/mysql3308/data] 5 #ps aux|grep mysqld 6 mysql 6295 0.1 22.4 1565212 227920 pts/0 Sl 08:47 0:02 mysqld --defaults-file=/data/mysql/mysql3308/my3308.cnf --wsrep-new-cluster 7 root 6835 0.0 0.0 112640 956 pts/3 R+ 09:22 0:00 grep --color=auto mysqld 8 9 [root@zlm2 09:22:15 /data/mysql/mysql3308/data] 10 #ps aux|grep mysqld 11 root 6841 0.0 0.0 112640 960 pts/3 R+ 09:22 0:00 grep --color=auto mysqld 12 13 [root@zlm2 09:30:04 /data/mysql/mysql3308/data] 14 #mysqld --defaults-file=/data/mysql/mysql3308/my3308.cnf --wsrep-new-cluster & 15 [1] 7055 16 17 [root@zlm2 09:30:20 /data/mysql/mysql3308/data] 18 #ps aux|grep mysqld 19 mysql 7055 0.4 22.1 1564952 224900 pts/3 Sl 09:30 0:02 mysqld --defaults-file=/data/mysql/mysql3308/my3308.cnf --wsrep-new-cluster 20 root 7583 0.0 0.0 112640 956 pts/3 R+ 09:39 0:00 grep --color=auto mysqld
It still show the ERROR output when node zlm3 startup.
1 2018-08-08T09:32:14.789691+01:00 0 [Note] WSREP: (e783b5a9, 'tcp://0.0.0.0:4567') connection established to 2bcb8dba tcp://192.168.1.102:4567 2 2018-08-08T09:32:14.790669+01:00 0 [Note] WSREP: (e783b5a9, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: 3 2018-08-08T09:32:15.289238+01:00 0 [Note] WSREP: declaring 2bcb8dba at tcp://192.168.1.102:4567 stable 4 2018-08-08T09:32:15.290161+01:00 0 [Note] WSREP: Node e783b5a9 state primary 5 2018-08-08T09:32:15.291214+01:00 0 [Note] WSREP: Current view of cluster as seen by this node 6 view (view_id(PRIM,2bcb8dba,2) 7 memb { 8 2bcb8dba,0 9 e783b5a9,0 10 } 11 joined { 12 } 13 left { 14 } 15 partitioned { 16 } 17 ) 18 2018-08-08T09:32:15.291254+01:00 0 [Note] WSREP: Save the discovered primary-component to disk 19 2018-08-08T09:32:15.291707+01:00 0 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 1, memb_num = 2 20 2018-08-08T09:32:15.291726+01:00 0 [Note] WSREP: STATE EXCHANGE: Waiting for state UUID. 21 2018-08-08T09:32:15.791844+01:00 0 [Note] WSREP: STATE EXCHANGE: sent state msg: 2c64cc25-9add-11e8-bdeb-67bbb141911b 22 2018-08-08T09:32:15.793169+01:00 0 [Note] WSREP: STATE EXCHANGE: got state msg: 2c64cc25-9add-11e8-bdeb-67bbb141911b from 0 (zlm3) 23 2018-08-08T09:32:15.793198+01:00 0 [Note] WSREP: STATE EXCHANGE: got state msg: 2c64cc25-9add-11e8-bdeb-67bbb141911b from 1 (zlm2) 24 2018-08-08T09:32:15.793205+01:00 0 [Note] WSREP: Quorum results: 25 version = 4, 26 component = PRIMARY, 27 conf_id = 1, 28 members = 1/2 (primary/total), 29 act_id = 21, 30 last_appl. = 0, 31 protocols = 0/8/3 (gcs/repl/appl), 32 group UUID = bd5525ab-9a15-11e8-aa0f-4b830c783fc7 33 2018-08-08T09:32:15.793211+01:00 0 [Note] WSREP: Flow-control interval: [141, 141] 34 2018-08-08T09:32:15.793213+01:00 0 [Note] WSREP: Trying to continue unpaused monitor 35 2018-08-08T09:32:15.793314+01:00 2 [Note] WSREP: REPL Protocols: 8 (3, 2) 36 2018-08-08T09:32:15.793324+01:00 2 [Note] WSREP: New cluster view: global state: bd5525ab-9a15-11e8-aa0f-4b830c783fc7:21, view# 2: Primary, number of nodes: 2, my index: 1, protocol version 3 37 2018-08-08T09:32:15.793327+01:00 2 [Note] WSREP: Setting wsrep_ready to true 38 2018-08-08T09:32:15.793330+01:00 2 [Note] WSREP: Auto Increment Offset/Increment re-align with cluster membership change (Offset: 1 -> 2) (Increment: 1 -> 2) 39 2018-08-08T09:32:15.793332+01:00 2 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification. 40 2018-08-08T09:32:15.793337+01:00 2 [Note] WSREP: Assign initial position for certification: 21, protocol version: 3 41 2018-08-08T09:32:15.793351+01:00 0 [Note] WSREP: Service thread queue flushed. 42 2018-08-08T09:32:16.392574+01:00 0 [Note] WSREP: Member 0.0 (zlm3) requested state transfer from '*any*'. Selected 1.0 (zlm2)(SYNCED) as donor. 43 2018-08-08T09:32:16.392599+01:00 0 [Note] WSREP: Shifting SYNCED -> DONOR/DESYNCED (TO: 21) 44 2018-08-08T09:32:16.392659+01:00 2 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification. 45 2018-08-08T09:32:16.392879+01:00 0 [Note] WSREP: Initiating SST/IST transfer on DONOR side (wsrep_sst_xtrabackup-v2 --role 'donor' --address '192.168.1.102:4444/xtrabackup_sst//1' --socket '/tmp/mysql3308.sock' --datadir '/data/mysql/mysql3308/data/' --defaults-file '/data/mysql/mysql3308/my3308.cnf' --defaults-group-suffix '' --binlog '/data/mysql/mysql3308/logs/mysql-bin' --gtid 'bd5525ab-9a15-11e8-aa0f-4b830c783fc7:21') 46 2018-08-08T09:32:16.396838+01:00 2 [Note] WSREP: DONOR thread signaled with 0 47 2018-08-08T09:32:17.917352+01:00 0 [Note] WSREP: (e783b5a9, 'tcp://0.0.0.0:4567') turning message relay requesting off 48 2018-08-08T07:32:27.003790Z WSREP_SST: [INFO] Streaming the backup to joiner at 192.168.1.102 4444 49 2018-08-08T09:32:27.124206+01:00 4 [Note] Access denied for user 'sst'@'192.168.1.101' (using password: YES) 50 2018-08-08T09:32:27.138281+01:00 5 [Note] Aborted connection 5 to db: 'unconnected' user: 'sst' host: 'localhost' (Got an error reading communication packets) 51 2018-08-08T07:32:27.139336Z WSREP_SST: [ERROR] ******************* FATAL ERROR ********************** 52 2018-08-08T07:32:27.140923Z WSREP_SST: [ERROR] xtrabackup finished with error: 1. Check /data/mysql/mysql3308/data//innobackup.backup.log 53 --------------- innobackup.backup.log (START) -------------------- 54 xtrabackup: recognized server arguments: --datadir=/data/mysql/mysql3308/data --server-id=1013308 --open_files_limit=65536 --log_bin=/data/mysql/mysql3308/logs/mysql-bin --innodb_data_file_path=ibdata1:100M:autoextend --innodb_buffer_pool_filename=ib_buffer_pool --innodb_log_group_home_dir=./ --innodb_log_files_in_group=3 --innodb_log_file_size=100M --innodb_file_per_table=1 --innodb_open_files=65535 --innodb_page_size=16k --innodb_read_io_threads=4 --innodb_write_io_threads=4 --innodb_io_capacity=200 --innodb_doublewrite=1 --innodb_log_buffer_size=128M --innodb_flush_log_at_trx_commit=1 --innodb_buffer_pool_size=100M --innodb_flush_method=O_DIRECT --defaults_group=mysqld --parallel=4 55 xtrabackup: recognized client arguments: --datadir=/data/mysql/mysql3308/data --server-id=1013308 --open_files_limit=65536 --log_bin=/data/mysql/mysql3308/logs/mysql-bin --innodb_data_file_path=ibdata1:100M:autoextend --innodb_buffer_pool_filename=ib_buffer_pool --innodb_log_group_home_dir=./ --innodb_log_files_in_group=3 --innodb_log_file_size=100M --innodb_file_per_table=1 --innodb_open_files=65535 --innodb_page_size=16k --innodb_read_io_threads=4 --innodb_write_io_threads=4 --innodb_io_capacity=200 --innodb_doublewrite=1 --innodb_log_buffer_size=128M --innodb_flush_log_at_trx_commit=1 --innodb_buffer_pool_size=100M --innodb_flush_method=O_DIRECT --defaults_group=mysqld --parallel=4 --port=3308 --user=sst --password=* --socket=/tmp/mysql3308.sock --lock-ddl=1 --backup=1 --galera-info=1 --binlog-info=ON --stream=xbstream --target-dir=/tmp/pxc_sst_sn2p/donor_xb_qh5x 56 180808 09:32:27 version_check Connecting to MySQL server with DSN 'dbi:mysql:;mysql_read_default_group=xtrabackup;port=3308;mysql_socket=/tmp/mysql3308.sock' as 'sst' (using password: YES). 57 Failed to connect to MySQL server: DBI connect(';mysql_read_default_group=xtrabackup;port=3308;mysql_socket=/tmp/mysql3308.sock','sst',...) failed: Access denied for user 'sst'@'192.168.1.101' (using password: YES) at - line 1314. 58 180808 09:32:27 Connecting to MySQL server host: localhost, user: sst, password: set, port: 3308, socket: /tmp/mysql3308.sock 59 Using server version 5.7.22-22-29.26-log 60 180808 09:32:27 Executing LOCK TABLES FOR BACKUP... 61 Error: failed to execute query SHOW ENGINE INNODB STATUS: Access denied; you need (at least one of) the PROCESS privilege(s) for this operation 62 --------------- innobackup.backup.log (END) ---------------------- 63 2018-08-08T07:32:27.143691Z WSREP_SST: [ERROR] ****************************************************** 64 2018-08-08T07:32:27.145779Z WSREP_SST: [ERROR] Cleanup after exit with status:22 65 2018-08-08T09:32:27.158932+01:00 0 [ERROR] WSREP: Process completed with error: wsrep_sst_xtrabackup-v2 --role 'donor' --address '192.168.1.102:4444/xtrabackup_sst//1' --socket '/tmp/mysql3308.sock' --datadir '/data/mysql/mysql3308/data/' --defaults-file '/data/mysql/mysql3308/my3308.cnf' --defaults-group-suffix '' --binlog '/data/mysql/mysql3308/logs/mysql-bin' --gtid 'bd5525ab-9a15-11e8-aa0f-4b830c783fc7:21': 22 (Invalid argument) 66 2018-08-08T09:32:27.159041+01:00 0 [ERROR] WSREP: Command did not run: wsrep_sst_xtrabackup-v2 --role 'donor' --address '192.168.1.102:4444/xtrabackup_sst//1' --socket '/tmp/mysql3308.sock' --datadir '/data/mysql/mysql3308/data/' --defaults-file '/data/mysql/mysql3308/my3308.cnf' --defaults-group-suffix '' --binlog '/data/mysql/mysql3308/logs/mysql-bin' --gtid 'bd5525ab-9a15-11e8-aa0f-4b830c783fc7:21' 67 2018-08-08T09:32:27.160430+01:00 0 [Warning] WSREP: 1.0 (zlm2): State transfer to 0.0 (zlm3) failed: -22 (Invalid argument) 68 69 //Error: failed to execute query SHOW ENGINE INNODB STATUS: Access denied; you need (at least one of) the PROCESS privilege(s) for this operation. 70 //It seams the user sst@localhost is lack of privilege of "PROCESS".
Modify the privilege of user sst@localhost.
1 (zlm@192.168.1.101 3308)[(none)]>select user,host from mysql.user; 2 +---------------+-------------+ 3 | user | host | 4 +---------------+-------------+ 5 | zlm | 192.168.1.% | 6 | mysql.session | localhost | 7 | mysql.sys | localhost | 8 | root | localhost | 9 | sst | localhost | 10 +---------------+-------------+ 11 5 rows in set (0.02 sec) 12 13 (zlm@192.168.1.101 3308)[(none)]>show grants for sst@localhost; 14 +---------------------------------------------------------------------------+ 15 | Grants for sst@localhost | 16 +---------------------------------------------------------------------------+ 17 | GRANT RELOAD, LOCK TABLES, REPLICATION CLIENT ON *.* TO 'sst'@'localhost' | 18 +---------------------------------------------------------------------------+ 19 1 row in set (0.00 sec) 20 21 (zlm@192.168.1.101 3308)[(none)]>grant process on *.* to 'sst'@'localhost'; 22 ERROR 1045 (28000): Access denied for user 'zlm'@'192.168.1.%' (using password: YES) 23 (zlm@192.168.1.101 3308)[(none)]>exit 24 Bye 25 26 [root@zlm2 10:07:32 ~] 27 #mysql -uroot -pPassw0rd -hlocalhost -S /tmp/mysql3308.sock 28 mysql: [Warning] Using a password on the command line interface can be insecure. 29 Welcome to the MySQL monitor. Commands end with ; or \g. 30 Your MySQL connection id is 9 31 Server version: 5.7.22-22-29.26-log Percona XtraDB Cluster binary (GPL) 5.7.22-29.26, Revision da86071, wsrep_29.26 32 33 Copyright (c) 2009-2018 Percona LLC and/or its affiliates 34 Copyright (c) 2000, 2018, Oracle and/or its affiliates. All rights reserved. 35 36 Oracle is a registered trademark of Oracle Corporation and/or its 37 affiliates. Other names may be trademarks of their respective 38 owners. 39 40 Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. 41 42 (root@localhost mysql3308.sock)[(none)]>grant process on *.* to sst@localhost; 43 Query OK, 0 rows affected, 1 warning (0.01 sec) 44 45 (root@localhost mysql3308.sock)[(none)]>show grants for sst@localhost; 46 +------------------------------------------------------------------------------------+ 47 | Grants for sst@localhost | 48 +------------------------------------------------------------------------------------+ 49 | GRANT RELOAD, PROCESS, LOCK TABLES, REPLICATION CLIENT ON *.* TO 'sst'@'localhost' | 50 +------------------------------------------------------------------------------------+ 51 1 row in set (0.00 sec)
Restart PXC again on zlm2.
[root@zlm2 10:11:10 ~] #ps aux|grep mysqld mysql 7055 0.1 22.2 1564952 225952 pts/3 Sl 09:30 0:04 mysqld --defaults-file=/data/mysql/mysql3308/my3308.cnf --wsrep-new-cluster root 7661 0.0 0.0 112640 960 pts/0 R+ 10:11 0:00 grep --color=auto mysqld [root@zlm2 10:11:28 ~] #mysqladmin -uzlm -pzlmzlm -h192.168.1.101 -P3308 shutdown mysqladmin: [Warning] Using a password on the command line interface can be insecure. [root@zlm2 10:11:50 ~] #ps aux|grep mysqld root 7672 0.0 0.0 112640 960 pts/0 R+ 10:12 0:00 grep --color=auto mysqld [root@zlm2 09:39:30 /data/mysql/mysql3308/data] #mysqld --defaults-file=/data/mysql/mysql3308/my3308.cnf --wsrep-new-cluster & [2] 7690 [1] Done mysqld --defaults-file=/data/mysql/mysql3308/my3308.cnf --wsrep-new-cluster [root@zlm2 10:12:44 /data/mysql/mysql3308/data] #ps aux|grep mysqld mysql 7690 5.8 20.6 1556488 210296 pts/3 Sl 10:12 0:01 mysqld --defaults-file=/data/mysql/mysql3308/my3308.cnf --wsrep-new-cluster root 7729 0.0 0.0 112640 960 pts/3 R+ 10:13 0:00 grep --color=auto mysqld //No ERROR messages were found in error log.
Restart PXC again on zlm3 and check the output of error log.
1 [root@zlm3 10:27:10 ~] 2 #ps aux|grep mysqld 3 root 6694 0.0 0.0 112640 960 pts/1 R+ 10:43 0:00 grep --color=auto mysqld 4 [2]+ Exit 1 mysqld --defaults-file=/data/mysql/mysql3308/my3308.cnf 5 6 [root@zlm3 10:30:01 /data/mysql/mysql3308/data] 7 #cat error.log |grep ERROR 8 2018-08-08T08:27:19.819918Z WSREP_SST: [ERROR] Cleanup after exit with status:1 9 2018-08-08T10:27:19.826527+01:00 0 [ERROR] WSREP: Process completed with error: wsrep_sst_xtrabackup-v2 --role 'joiner' --address '192.168.1.102' --datadir '/data/mysql/mysql3308/data/' --defaults-file '/data/mysql/mysql3308/my3308.cnf' --defaults-group-suffix '' --parent '6021' --binlog '/data/mysql/mysql3308/logs/mysql-bin' : 1 (Operation not permitted) 10 2018-08-08T10:27:19.826557+01:00 0 [ERROR] WSREP: Failed to read uuid:seqno from joiner script. 11 2018-08-08T10:27:19.826565+01:00 0 [ERROR] WSREP: SST script aborted with error 1 (Operation not permitted) 12 2018-08-08T10:27:19.826592+01:00 0 [ERROR] WSREP: SST failed: 1 (Operation not permitted) 13 2018-08-08T10:27:19.826595+01:00 0 [ERROR] Aborting
Check the error log on donor node zlm2 begin with the time of starting mysqld on joiner zlm3.
1 2018-08-08T10:54:47.539174+01:00 0 [Note] WSREP: (d4006db4, 'tcp://0.0.0.0:4567') connection established to b3de093b tcp://192.168.1.102:4567 2 2018-08-08T10:54:47.539620+01:00 0 [Note] WSREP: (d4006db4, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: 3 2018-08-08T10:54:48.038167+01:00 0 [Note] WSREP: declaring b3de093b at tcp://192.168.1.102:4567 stable 4 2018-08-08T10:54:48.038656+01:00 0 [Note] WSREP: Node d4006db4 state primary 5 2018-08-08T10:54:48.039842+01:00 0 [Note] WSREP: Current view of cluster as seen by this node 6 view (view_id(PRIM,b3de093b,8) 7 memb { 8 b3de093b,0 9 d4006db4,0 10 } 11 joined { 12 } 13 left { 14 } 15 partitioned { 16 } 17 ) 18 2018-08-08T10:54:48.039890+01:00 0 [Note] WSREP: Save the discovered primary-component to disk 19 2018-08-08T10:54:48.040138+01:00 0 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 1, memb_num = 2 20 2018-08-08T10:54:48.040154+01:00 0 [Note] WSREP: STATE EXCHANGE: Waiting for state UUID. 21 2018-08-08T10:54:48.540351+01:00 0 [Note] WSREP: STATE EXCHANGE: sent state msg: b47737d3-9ae8-11e8-81e0-7bfd4d57e067 22 2018-08-08T10:54:48.540644+01:00 0 [Note] WSREP: STATE EXCHANGE: got state msg: b47737d3-9ae8-11e8-81e0-7bfd4d57e067 from 0 (zlm3) 23 2018-08-08T10:54:48.541101+01:00 0 [Note] WSREP: STATE EXCHANGE: got state msg: b47737d3-9ae8-11e8-81e0-7bfd4d57e067 from 1 (zlm2) 24 2018-08-08T10:54:48.541140+01:00 0 [Note] WSREP: Quorum results: 25 version = 4, 26 component = PRIMARY, 27 conf_id = 7, 28 members = 1/2 (primary/total), 29 act_id = 22, 30 last_appl. = 0, 31 protocols = 0/8/3 (gcs/repl/appl), 32 group UUID = bd5525ab-9a15-11e8-aa0f-4b830c783fc7 33 2018-08-08T10:54:48.541150+01:00 0 [Note] WSREP: Flow-control interval: [141, 141] 34 2018-08-08T10:54:48.541155+01:00 0 [Note] WSREP: Trying to continue unpaused monitor 35 2018-08-08T10:54:48.541224+01:00 2 [Note] WSREP: REPL Protocols: 8 (3, 2) 36 2018-08-08T10:54:48.541234+01:00 2 [Note] WSREP: New cluster view: global state: bd5525ab-9a15-11e8-aa0f-4b830c783fc7:22, view# 8: Primary, number of nodes: 2, my index: 1, protocol version 3 37 2018-08-08T10:54:48.541238+01:00 2 [Note] WSREP: Setting wsrep_ready to true 38 2018-08-08T10:54:48.541241+01:00 2 [Note] WSREP: Auto Increment Offset/Increment re-align with cluster membership change (Offset: 1 -> 2) (Increment: 1 -> 2) 39 2018-08-08T10:54:48.541245+01:00 2 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification. 40 2018-08-08T10:54:48.541252+01:00 2 [Note] WSREP: Assign initial position for certification: 22, protocol version: 3 41 2018-08-08T10:54:48.541264+01:00 0 [Note] WSREP: Service thread queue flushed. 42 2018-08-08T10:54:49.007421+01:00 0 [Note] WSREP: Member 0.0 (zlm3) requested state transfer from '*any*'. Selected 1.0 (zlm2)(SYNCED) as donor. 43 2018-08-08T10:54:49.007441+01:00 0 [Note] WSREP: Shifting SYNCED -> DONOR/DESYNCED (TO: 22) 44 2018-08-08T10:54:49.007480+01:00 2 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification. 45 2018-08-08T10:54:49.007626+01:00 0 [Note] WSREP: Initiating SST/IST transfer on DONOR side (wsrep_sst_xtrabackup-v2 --role 'donor' --address '192.168.1.102:4444/xtrabackup_sst//1' --socket '/tmp/mysql3308.sock' --datadir '/data/mysql/mysql3308/data/' --defaults-file '/data/mysql/mysql3308/my3308.cnf' --defaults-group-suffix '' --binlog '/data/mysql/mysql3308/logs/mysql-bin' --gtid 'bd5525ab-9a15-11e8-aa0f-4b830c783fc7:22') 46 2018-08-08T10:54:49.012384+01:00 2 [Note] WSREP: DONOR thread signaled with 0 47 2018-08-08T10:54:50.911071+01:00 0 [Note] WSREP: (d4006db4, 'tcp://0.0.0.0:4567') turning message relay requesting off 48 2018-08-08T08:54:59.535108Z WSREP_SST: [INFO] Streaming the backup to joiner at 192.168.1.102 4444 49 2018-08-08T10:54:59.626864+01:00 8 [Note] Access denied for user 'sst'@'192.168.1.101' (using password: YES) 50 2018-08-08T10:55:05.537365+01:00 0 [Note] WSREP: 1.0 (zlm2): State transfer to 0.0 (zlm3) complete. 51 2018-08-08T10:55:05.537399+01:00 0 [Note] WSREP: Shifting DONOR/DESYNCED -> JOINED (TO: 22) 52 2018-08-08T10:55:05.542864+01:00 0 [Note] WSREP: Member 1.0 (zlm2) synced with group. 53 2018-08-08T10:55:05.542949+01:00 0 [Note] WSREP: Shifting JOINED -> SYNCED (TO: 22) 54 2018-08-08T10:55:05.543015+01:00 2 [Note] WSREP: Synchronized with group, ready for connections 55 2018-08-08T10:55:05.543022+01:00 2 [Note] WSREP: Setting wsrep_ready to true 56 2018-08-08T10:55:05.543027+01:00 2 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification. 57 2018-08-08T10:55:22.477724+01:00 0 [Note] WSREP: forgetting b3de093b (tcp://192.168.1.102:4567) 58 2018-08-08T10:55:22.479542+01:00 0 [Note] WSREP: Node d4006db4 state primary 59 2018-08-08T10:55:22.479612+01:00 0 [Note] WSREP: Current view of cluster as seen by this node 60 view (view_id(PRIM,d4006db4,9) 61 memb { 62 d4006db4,0 63 } 64 joined { 65 } 66 left { 67 } 68 partitioned { 69 b3de093b,0 70 } 71 ) 72 2018-08-08T10:55:22.479621+01:00 0 [Note] WSREP: Save the discovered primary-component to disk 73 2018-08-08T10:55:22.479763+01:00 0 [Note] WSREP: forgetting b3de093b (tcp://192.168.1.102:4567) 74 2018-08-08T10:55:22.479875+01:00 0 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 1 75 2018-08-08T10:55:22.479976+01:00 0 [Note] WSREP: STATE_EXCHANGE: sent state UUID: c8aef280-9ae8-11e8-80c9-d74e514aa266 76 2018-08-08T10:55:22.479976+01:00 0 [Note] WSREP: STATE EXCHANGE: sent state msg: c8aef280-9ae8-11e8-80c9-d74e514aa266 77 2018-08-08T10:55:22.479976+01:00 0 [Note] WSREP: STATE EXCHANGE: got state msg: c8aef280-9ae8-11e8-80c9-d74e514aa266 from 0 (zlm2) 78 2018-08-08T10:55:22.479978+01:00 0 [Note] WSREP: Quorum results: 79 version = 4, 80 component = PRIMARY, 81 conf_id = 8, 82 members = 1/1 (primary/total), 83 act_id = 22, 84 last_appl. = 0, 85 protocols = 0/8/3 (gcs/repl/appl), 86 group UUID = bd5525ab-9a15-11e8-aa0f-4b830c783fc7 87 2018-08-08T10:55:22.479983+01:00 0 [Note] WSREP: Flow-control interval: [100, 100] 88 2018-08-08T10:55:22.480000+01:00 0 [Note] WSREP: Trying to continue unpaused monitor 89 2018-08-08T10:55:22.480032+01:00 2 [Note] WSREP: REPL Protocols: 8 (3, 2) 90 2018-08-08T10:55:22.480037+01:00 2 [Note] WSREP: New cluster view: global state: bd5525ab-9a15-11e8-aa0f-4b830c783fc7:22, view# 9: Primary, number of nodes: 1, my index: 0, protocol version 3 91 2018-08-08T10:55:22.480040+01:00 2 [Note] WSREP: Setting wsrep_ready to true 92 2018-08-08T10:55:22.480043+01:00 2 [Note] WSREP: Auto Increment Offset/Increment re-align with cluster membership change (Offset: 2 -> 1) (Increment: 2 -> 1) 93 2018-08-08T10:55:22.480046+01:00 2 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification. 94 2018-08-08T10:55:22.480051+01:00 2 [Note] WSREP: Assign initial position for certification: 22, protocol version: 3 95 2018-08-08T10:55:22.480060+01:00 0 [Note] WSREP: Service thread queue flushed. 96 2018-08-08T10:55:26.085271+01:00 0 [Note] WSREP: (d4006db4, 'tcp://0.0.0.0:4567') connection established to b3de093b tcp://192.168.1.102:4567 97 2018-08-08T10:55:26.085306+01:00 0 [Warning] WSREP: discarding established (time wait) b3de093b (tcp://192.168.1.102:4567) 98 2018-08-08T10:55:27.948973+01:00 0 [Note] WSREP: cleaning up b3de093b (tcp://192.168.1.102:4567) 99 100 //There's no ERROR messages at all this time.