mysql数据恢复 insert\update\delete 工具MyFlash

一、简介
MyFlash是由美团点评公司技术工程部开发维护的一个回滚DML操作的工具。该工具通过解析v4版本的binlog，完成回滚操作。相对已有的回滚工具，其增加了更多的过滤选项，让回滚更加容易。该工具已经在美团点评内部使用
代码地址：https://github.com/Meituan-Dianping/MyFlash
二、详细说明
安装
使用
测试用例
三、限制
binlog格式必须为row,且binlog_row_image=full
仅支持5.6与5.7
只能回滚DML（增、删、改）
四、FAQ
实现的原理是什么？
答：参考文章http://url.cn/5yVTfLY
支持gtid吗？
答：支持。请参考使用
在开启gtid的MySQL server上，应用flashback报错，错误为：ERROR 1782 (HY000) at line 16: @@SESSION.GTID_NEXT cannot be set to ANONYMOUS when @@GLOBAL.GTID_MODE = ON. ?
答：在导入时加入--skip-gtids mysqlbinlog --skip-gtids | mysql -uxxx -pxxx
如果回滚后的binlog日志尺寸超过20M，在导入时，很耗时。如何处理?
答：参考使用 ,搜索maxSplitSize。使用该参数可以对文件进行切片
五、联系方式
QQ群:645702809

安装
1.How to build
该工具推荐用户在下载源码之后，进行动态编译链接安装

动态编译链接
gcc -w `pkg-config --cflags --libs glib-2.0` source/binlogParseGlib.c -o binary/flashback
然而用户不想每次去重新编译，可以选择使用静态链接，但是该方法需要知道glib库的版本和位置，因此对于每台机器略有不同，请谨慎使用

静态编译链接
为了保证在一台机器上编译后，可以在其它机器上使用，需要满足以下两个条件 a) 将glib做成静态链接 b）在编译的那台机器的glibc版本
（查看方法为ldd --version）要小于等于要运行该软件的那台机器glibc版本因此需要你在一台glibc版本较低的机器上运行如下命令即可。

gcc -w -g `pkg-config --cflags glib-2.0` source/binlogParseGlib.c -o binary/flashback /usr/lib64/libglib-2.0.a -lrt

使用
1.How to use
cd binary
./flashback --help

Usage:
flashback [OPTION...]

Help Options:
-?, --help Show help options

Application Options:
--databaseNames databaseName to apply. if multiple, seperate by comma(,)
--tableNames tableName to apply. if multiple, seperate by comma(,)
--start-position start position
--stop-position stop position
--start-datetime start time (format %Y-%m-%d %H:%M:%S)
--stop-datetime stop time (format %Y-%m-%d %H:%M:%S)
--sqlTypes sql type to filter . support INSERT, UPDATE ,DELETE. if multiple, seperate by comma(,)
--maxSplitSize max file size after split, the uint is M
--binlogFileNames binlog files to process. if multiple, seperate by comma(,)
--outBinlogFileNameBase output binlog file name base
--logLevel log level, available option is debug,warning,error
--include-gtids gtids to process
--exclude-gtids gtids to skip

2.Parameter explantion
下面的这些参数是可以任意组合的。

1.databaseNames 指定需要回滚的数据库名。多个数据库可以用“,”隔开。如果不指定该参数，相当于指定了所有数据库。
2.tableNames 指定需要回滚的表名。多个表可以用“,”隔开。如果不指定该参数，相当于指定了所有表。
3.start-position 指定回滚开始的位置。如不指定，从文件的开始处回滚。请指定正确的有效的位置，否则无法回滚
4.stop-position 指定回滚结束的位置。如不指定，回滚到文件结尾。请指定正确的有效的位置，否则无法回滚
5.start-datetime 指定回滚的开始时间。注意格式必须是 %Y-%m-%d %H:%M:%S。如不指定，则不限定时间
6.stop-datetime 指定回滚的结束时间。注意格式必须是 %Y-%m-%d %H:%M:%S。如不指定，则不限定时间
7.sqlTypes 指定需要回滚的sql类型。目前支持的过滤类型是INSERT, UPDATE ,DELETE。多个类型可以用“,”隔开。
8.maxSplitSize 一旦指定该参数，对文件进行固定尺寸的分割（单位为M），过滤条件有效，但不进行回滚操作。该参数主要用来将大的binlog文件切割，防止单次应用的binlog尺寸过大，对线上造成压力
9.binlogFileNames 指定需要回滚的binlog文件，目前只支持单个文件，后续会增加多个文件支持
10.outBinlogFileNameBase 指定输出的binlog文件前缀，如不指定，则默认为binlog_output_base.flashback
11.logLevel 仅供开发者使用，默认级别为error级别。在生产环境中不要修改这个级别，否则输出过多
12.include-gtids 指定需要回滚的gtid,支持gtid的单个和范围两种形式。
13.exclude-gtids 指定不需要回滚的gtid，用法同include-gtids

3.example
1.回滚整个文件

./flashback --binlogFileNames=haha.000041
mysqlbinlog binlog_output_base.flashback | mysql -h<host> -u<user> -p

2.回滚该文件中的所有insert语句
./flashback --sqlTypes='INSERT' --binlogFileNames=haha.000041
mysqlbinlog binlog_output_base.flashback | mysql -h<host> -u<user> -p

3.回滚大文件

回滚
./flashback --binlogFileNames=haha.000042
切割大文件
./flashback --maxSplitSize=1 --binlogFileNames=binlog_output_base.flashback
应用
mysqlbinlog binlog_output_base.flashback.000001 | mysql -h<host> -u<user> -p
...
mysqlbinlog binlog_output_base.flashback.<N> | mysql -h<host> -u<user> -p

测试用例
###测试表结构

CREATE TABLE `testFlashback2` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`nameShort` varchar(20) DEFAULT NULL,
`nameLong` varchar(260) DEFAULT NULL,
`amount` decimal(19,9) DEFAULT NULL,
`amountFloat` float DEFAULT NULL,
`amountDouble` double DEFAULT NULL,
`createDatetime6` datetime(6) DEFAULT NULL,
`createDatetime` datetime DEFAULT NULL,
`createTimestamp` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
`nameText` text,
`nameBlob` blob,
`nameMedium` mediumtext,
PRIMARY KEY (`id`)
) ENGINE=InnoDB
####插入回滚

flush logs
insert into testFlashback2(nameShort,nameLong,amount,amountFloat,amountDouble,createDatetime6,createDatetime,createTimestamp,nameText,nameBlob,nameMedium)
values('aaa','bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb',10.5,10.6,10.7,'2017-10-26 10:00:00','2017-10-26 10:00:00','2017-10-26 10:00:00','cccc','dddd','eee');
insert into testFlashback2(nameShort,nameLong,amount,amountFloat,amountDouble,createDatetime6,createDatetime,createTimestamp,nameText,nameBlob,nameMedium)
values('aaa','bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb',10.5,10.6,10.7,'2017-10-26 10:00:00','2017-10-26 10:00:00','2017-10-26 10:00:00','cccc','dddd','eee');
flush logs;
./binary/flashback --binlogFileNames=/var/lib/mysql/haha.000048
在当前运行目录下产生binlog_output_base.flashback文件
mysqlbinlog --skip-gtids binlog_output_base.flashback | mysql --socket=/var/lib/mysql/mysql.sock test
执行结果
mysql> select * from testFlashback2;
Empty set (0.00 sec)
插入回滚成功
####删除回滚

delete from testFlashback2;
insert into testFlashback2(nameShort,nameLong,amount,amountFloat,amountDouble,createDatetime6,createDatetime,createTimestamp,nameText,nameBlob,nameMedium)
values('aaa','bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb',10.5,10.6,10.7,'2017-10-26 10:00:00','2017-10-26 10:00:00','2017-10-26 10:00:00','cccc','dddd','eee');
flush logs;
delete from testFlashback2;
./binary/flashback --binlogFileNames=/var/lib/mysql/haha.000050
在当前运行目录下产生binlog_output_base.flashback文件
mysqlbinlog --skip-gtids binlog_output_base.flashback | mysql --socket=/var/lib/mysql/mysql.sock test
执行结果
+----+-----------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------------+-------------+--------------+----------------------------+---------------------+---------------------+----------+----------+------------+
| id | nameShort | nameLong | amount | amountFloat | amountDouble | createDatetime6 | createDatetime | createTimestamp | nameText | nameBlob | nameMedium |
+----+-----------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------------+-------------+--------------+----------------------------+---------------------+---------------------+----------+----------+------------+
| 4 | aaa | bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb | 10.500000000 | 10.6 | 10.7 | 2017-10-26 10:00:00.000000 | 2017-10-26 10:00:00 | 2017-10-26 10:00:00 | cccc | dddd | eee |
+----+-----------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------------+-------------+--------------+----------------------------+---------------------+---------------------+----------+----------+------------+
####更新回滚

CREATE TABLE `DP_FeedBackForPOI_For_Flashback` (
`FeedID` int(11) NOT NULL AUTO_INCREMENT,
`UserName` varchar(50) DEFAULT NULL,
`UserEmail` varchar(100) DEFAULT NULL,
`UserPhone` varchar(20) DEFAULT NULL,
`UserID` int(11) DEFAULT '0',
`FeedTitle` varchar(200) DEFAULT NULL,
`FeedComments` text,
`FeedAdddate` datetime DEFAULT NULL,
`FeedStatus` tinyint(3) unsigned DEFAULT '0',
`SendTitle` varchar(200) DEFAULT NULL,
`SendComments` text,
`SendDate` datetime DEFAULT NULL,
`MailTO` varchar(100) DEFAULT NULL,
`FeedType` tinyint(3) unsigned DEFAULT '0',
`LastAdminID` smallint(6) DEFAULT NULL,
`ReferID` int(11) DEFAULT '0',
`ReferUserID` int(11) DEFAULT '0',
`ReferShopID` int(11) DEFAULT '0',
`FeedGroupID` int(11) DEFAULT '0',
`CauseType` tinyint(4) DEFAULT '0',
`ClientType` tinyint(4) DEFAULT '0',
`LastAdminName` varchar(50) DEFAULT NULL,
PRIMARY KEY (`FeedID`),
KEY `LastAdminID` (`LastAdminID`) USING BTREE,
KEY `FeedStatus` (`FeedStatus`,`FeedAdddate`,`FeedType`) USING BTREE,
KEY `MailTO` (`MailTO`,`FeedAdddate`,`FeedStatus`) USING BTREE,
KEY `FeedType` (`FeedType`,`FeedAdddate`,`UserEmail`) USING BTREE,
KEY `FeedGroupID` (`FeedGroupID`) USING BTREE,
KEY `FeedID_FeedStatus` (`FeedStatus`,`FeedGroupID`) USING BTREE,
KEY `UserName` (`UserName`,`MailTO`,`FeedType`,`FeedGroupID`) USING BTREE,
KEY `IX_UserID` (`UserID`) USING BTREE,
KEY `IX_ReferUserID` (`ReferUserID`) USING BTREE,
KEY `IX_ReferShopID` (`ReferShopID`) USING BTREE,
KEY `FeedType_2` (`FeedType`,`ReferID`)
) ENGINE=InnoDB AUTO_INCREMENT=2078564 DEFAULT CHARSET=utf8 ROW_FORMAT=COMPACT;
mysql> checksum table DP_FeedBackForPOI_For_Flashback;
+---------------------------------+-----------+
| Table | Checksum |
+---------------------------------+-----------+
| DP_FeedBackForPOI_For_Flashback | 717087411 |
+---------------------------------+-----------+
update DP_FeedBackForPOI_For_Flashback set UserName='wangguangyou' limit 1000;
mysql> checksum table DP_FeedBackForPOI_For_Flashback;
+---------------------------------+-----------+
| Table | Checksum |
+---------------------------------+-----------+
| DP_FeedBackForPOI_For_Flashback | 3532811761|
+---------------------------------+-----------+
./binary/flashback --binlogFileNames=/var/lib/mysql/haha.000058
mysqlbinlog --skip-gtids binlog_output_base.flashback | mysql --socket=/var/lib/mysql/mysql.sock test
mysql> checksum table DP_FeedBackForPOI_For_Flashback;
+---------------------------------+-----------+
| Table | Checksum |
+---------------------------------+-----------+
| DP_FeedBackForPOI_For_Flashback | 717087411 |
+---------------------------------+-----------+

yum install git -y
git clone https://github.com/Meituan-Dianping/MyFlash.git
cd MyFlash/
yum install glib2 -y
gcc -w `pkg-config --cflags --libs glib-2.0` source/binlogParseGlib.c -o binary/flashback
cd binary
./flashback --help

[root@mysqlhq binary]# ./flashback --help
Usage:
flashback [OPTION?]

Help Options:
-h, --help Show help options

([email protected]:3306) [test]> flush logs;
([email protected]:3306) [test]> show binary logs;
| mysql-bin.000009 | 120 |
###测试表结构

CREATE TABLE `testFlashback2` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`nameShort` varchar(20) DEFAULT NULL,
`nameLong` varchar(260) DEFAULT NULL,
`amount` decimal(19,9) DEFAULT NULL,
`amountFloat` float DEFAULT NULL,
`amountDouble` double DEFAULT NULL,
`createDatetime6` datetime(6) DEFAULT NULL,
`createDatetime` datetime DEFAULT NULL,
`createTimestamp` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
`nameText` text,
`nameBlob` blob,
`nameMedium` mediumtext,
PRIMARY KEY (`id`)
) ENGINE=InnoDB

####插入回滚

insert into testFlashback2(nameShort,nameLong,amount,amountFloat,amountDouble,createDatetime6,createDatetime,createTimestamp,nameText,nameBlob,nameMedium) 
values('aaa','bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb',10.5,10.6,10.7,'2017-10-26 10:00:00','2017-10-26 10:00:00','2017-10-26 10:00:00','cccc','dddd','eee');
insert into testFlashback2(nameShort,nameLong,amount,amountFloat,amountDouble,createDatetime6,createDatetime,createTimestamp,nameText,nameBlob,nameMedium)
values('aaa','bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb',10.5,10.6,10.7,'2017-10-26 10:00:00','2017-10-26 10:00:00','2017-10-26 10:00:00','cccc','dddd','eee');

([email protected]:3306) [test]> show binary logs;
| mysql-bin.000009 | 1748 |
====
##这里正好是一个新binlog文件，并且只有这一个操作，但实际情况并不适用，实际你先找到binlog，转换成sql文件，确认操作的sql和位置点
mysqlbinlog mysql-bin.000097 --base64-output=decode-rows -vv >bin97.sql
cat bin97.sql |grep testFlashback2
###然后在后面加 --start-position等，能选择具体回复的数据项目
./flashback --start-position=6088478 --stop-position=6088641
====
[root@mysqlhq binary]# ./flashback --binlogFileNames=/home/data/mysqldata/3306/binlog/mysql-bin.000009
[root@mysqlhq binary]# ll
total 7344
-rw-r--r-- 1 root root 904 Dec 26 11:27 binlog_output_base.flashback
-rwxr-xr-x 1 root root 80046 Dec 26 11:15 flashback
-rwxr-xr-x 1 root root 7463125 Dec 26 11:15 mysqlbinlog20160408
[root@mysqlhq binary]# chown mysql.mysql binlog_output_base.flashback

[root@mysqlhq ~]# su - mysql
Last login: Tue Dec 25 10:55:11 CST 2018 on pts/0
[mysql@mysqlhq ~]$ cd /data/soft/MyFlash/binary/
[mysql@mysqlhq binary]$ ll
total 7344
-rw-r--r-- 1 mysql mysql 904 Dec 26 11:27 binlog_output_base.flashback
-rwxr-xr-x 1 root root 80046 Dec 26 11:15 flashback
-rwxr-xr-x 1 root root 7463125 Dec 26 11:15 mysqlbinlog20160408
[mysql@mysqlhq binary]$ mysqlbinlog binlog_output_base.flashback | mysql -usystem -p -h127.0.0.1 -P3306 test
Enter password:

([email protected]:3306) [test]> select * from testFlashback2;

####删除回滚

delete from testFlashback2;
insert into testFlashback2(nameShort,nameLong,amount,amountFloat,amountDouble,createDatetime6,createDatetime,createTimestamp,nameText,nameBlob,nameMedium) 
values('aaa','bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb',10.5,10.6,10.7,'2017-10-26 10:00:00','2017-10-26 10:00:00','2017-10-26 10:00:00','cccc','dddd','eee');
flush logs;
delete from testFlashback2;

([email protected]:3306) [test]> show binary logs;
| mysql-bin.000010 | 615 |
[root@mysqlhq binary]# ./flashback --binlogFileNames=/home/data/mysqldata/3306/binlog/mysql-bin.000010
[root@mysqlhq binary]# chown mysql.mysql binlog_output_base.flashback
[mysql@mysqlhq binary]$ mysqlbinlog binlog_output_base.flashback | mysql -usystem -p -h127.0.0.1 -P3306 test
([email protected]:3306) [test]> select *from testFlashback2;

####更新回滚

([email protected]:3306) [test]> checksum table testFlashback2;
+---------------------+------------+
| Table | Checksum |
+---------------------+------------+
| test.testFlashback2 | 3574042806 |
+---------------------+------------+
([email protected]:3306) [test]> show binary logs;
| mysql-bin.000010 | 2155 |
| mysql-bin.000011 | 890 |
[root@mysqlhq binary]# ./flashback --binlogFileNames=/home/data/mysqldata/3306/binlog/mysql-bin.000011
[root@mysqlhq binary]# chown mysql.mysql binlog_output_base.flashback
[mysql@mysqlhq binary]$ mysqlbinlog binlog_output_base.flashback | mysql -usystem -p -h127.0.0.1 -P3306 test
Enter password:
([email protected]:3306) [test]> checksum table testFlashback2;
+---------------------+------------+
| Table | Checksum |
+---------------------+------------+
| test.testFlashback2 | 2649253452 |
+---------------------+------------+

mysql数据恢复 insert\update\delete 工具MyFlash

猜你喜欢