Data consistency check (pt-table-checksum)

introduce

• pt-table-checksum and pt-table-sync are tools released by percona to check the data consistency of MySQL master-slave databases.
• pt-table-checksum uses the MySQL replication principle to perform checksum calculations in the master database, and compares the checksums of the master and slave databases to determine whether the data in the master and slave databases are consistent.
• If data inconsistency is found, or a warning or error is reported during command execution , the return status value is non- zero .
• pt-table-checksum is used to find data differences, and pt-table-sync is used to repair data differences.
• Even if the database has hundreds of thousands of tables and trillions of rows of data, pt-table-checksum works very well.

working principle

• When pt-table-checksum is running, divide each table into multiple chunks , and use replace ... select query to calculate checksum for each chunk .
• It can vary the chunk size so that the checksum query runs in the desired time, with a target time of 0.5 seconds per chunk by default .
• It tracks how fast the server is executing queries and automatically adjusts the chunk size when server performance is overloaded, such as during traffic spikes or background tasks .
• The table to perform verification must have a primary key or a unique index, so that the speed of verification will be fast.
• The chunk size can use --chunk-size-limit to set the maximum value of the chunk .
• When running, it will set the session- level innodb_lock_wait_time_out to 1. If there is a lock waiting, it will release the lock to ensure that the database can be read and written normally.
• By default, when a table has 25 queries at the same time, it will pause. The number of queries can be set with --max-load .
• If its query is killed , it will try again, and if it fails again, it will skip to the next chunk . It performs the same behavior if the lock wait times out. If such an error occurs, warning will be printed , but only once per table. If the network is interrupted, it will try to reconnect and continue execution.
• If pt-table-checksum terminates abnormally, you can resume it with the --resume option, and it will start from the last chunk processed last time . Can execute CTRL+C  to stop manually at any time, and resume again.
• After checking the checksums of all table blocks, it will pause and wait for all slaves to complete the checksum query. Then check whether all slave databases have the same data as the master database, and print a line of results.
• You can manually execute the following query to query the checksum information

SELECT db, tbl, SUM(this_cnt) AS total_rows, COUNT(*) AS chunks
FROM percona.checksums
WHERE (
master_cnt <> this_cnt
OR master_crc <> this_crc
OR ISNULL(master_crc) <> ISNULL(this_crc))
GROUP BY db, tbl;

limit

• pt-table-checksum requires statement -based replication, and binlog_format=statement needs to be set on the master during runtime , and the slave library does not accept this change.
• When the tool runs, it will check the binlog format of all libraries.
• If the replication slave library does not have the checksum schema on the master library, or the table structure of the master library is inconsistent, the replication will be interrupted.

exit status

• 0 is normal, no warnings , errors , checksum differences, or skipped tables or blocks.
• 255 is serious error,
• Other values ​​greater than 0 and less than 255 indicate an abnormality, you can check the manual.

options

option name	illustrate	Defaults	type	group
--ask-pass	Manually enter the password when executing
--[no]check-binlog-format	Check binlog_format value for all libraries	yes
--binary-index	Change the table type created by --create-replicate-table to BLOB type, which is used when the character set is non-standard character set.
--check-interval		1		threshold
--[no]check-plan	The tool uses several heuristic methods to judge whether the execution plan is wrong. If the execution plan is wrong, the table block will be skipped, and the execution time will be increased if it is turned on.	yes
--[no]check-replication-filters	If the filter option of binlog_ignore_db or replicate_do_db is set for the master database or the slave database, the execution tool will terminate with an error because it cannot guarantee that the query and checksum will not cause replication interruption. Use the tool to ensure that there are no filter parameters before enabling this option.	yes		Safety
--check-slave-lag	The tool will monitor the latency of all slave libraries. If the latency of a slave library exceeds the value of --max-lag, the execution of the checksum will be suspended until the latency of the slave library is less than the value of --max-lag.		string	threshold
--[no]check-slave-tables	Check that all slave database tables exist, and the columns of the tables are the same as those of the master database, so as to avoid replication interruption due to inconsistent table structures of the master and slave databases. Also enable this option when ensuring that the master-slave database table structure is consistent.	yes		Safety
--chunk-index	Force the specified index when setting the chunk for the tool, and the specified index will be written into the SQL statement by the tool with the FORCE INDEX clause. Note Manual selection of indexes may cause performance degradation. When the default index is not suitable, you can manually specify the index, but if the manually specified index does not exist, the tool will return to the default method to select the index.		string
--chunk-index-columns	Sets the column number of the leftmost index column of the composite index. When the composite index contains too many columns, it may cause the mysql scan range to be too large or the execution plan to be wrong. This option can improve the efficiency of index selection. Typically used when 4 or more columns are involved.		int
--chunk-size	Set rows for the SQL that executes each checksum, the suffix can be k, m, g. Usually do not need to set this, setting --chunk-time is more effective. Setting this item will invalidate the dynamic adjustment chunk, and make each chunk equal to the set value. In addition, if you set 5000, but a where clause can only match 1000, then this chunk may be skipped. Smaller settings will result in slower checks because --[no]check-plan needs to be set	1000	size
--chunk-size-limit	Block size limit multiple. Because the number of rows obtained by the tool comes from the estimated number of rows in the execution plan, not the exact value. If the chunk exceeds the size of 1000 * limit, the tool will skip the chunk. If limit is 1, it means that no chunk can be larger than --chunk-size; if limit is 0, it means there is no limit.	2.0	float	Safety
--chunk-time	The block size is dynamically adjusted so that each checksum query takes this long to execute. If you adjust this parameter to 0, the block size will not be automatically adjusted.	0.5	float
--columns	Abbreviation - c. If the table being validated does not contain the specified column, the table is skipped. This option applies to all tables, so it really only makes sense when checksumming a table, unless those tables have a common set of columns.		array	filter
--config	Read a comma-separated list of configuration files, if configured, this option must be the first parameter.		array	configuration
--[no]create-replicate-table	如果不存在 replicate 库和表，则创建它们。	yes
--databases	缩写 -d。仅对指定的库列表进行校验和。		hash	过滤器
--databases-regex	只校验 Perl 正则匹配的数据库。		string	过滤器
--defaults-file	从指定的文件读取 mysql 参数。该文件必须包含绝对路径。		string	连接
--[no]empty-replicate-table	对表进行校验和之前，删除复制表之前的校验和结果。删除不是执行 truncate，而是检查到对应表时再删除对应结果。 表不会自动清空，为了下次执行时可以使用之前的结果。可以手动清空。	yes
--engines	缩写 -e。只检查指定引擎的表。		hash	过滤器
--explain	显示计划，但不执行查询、校验和。如果指定2次，则打印每个 chunk 的上下边界，但不执行查询、校验和。	0	累计值	输出
--float-precision	浮点和双数到字符串转换的精度。使用MySQL中的ROUND() 函数将浮点值和双精度值舍入到小数点后指定的位数。		int
--function	校验和的 hash 函数（FNV1A_64、murtur_Hash、SHA1、MD5、CRC32等）默认是 crc32，也可以自己定义，但是函数必须是 mysql 内部支持的。mysql 内部没有很好的哈希函数，crc32 可能会有冲突，但是 md5 和 sha1 会非常占用 cpu。		string
--host	缩写 -h。默认 localhost。		string	连接
--ignore-columns	计算校验和时忽略此以逗号分隔的列列表。如果表的所有列都按–忽略列筛选，则将跳过该表。		hash	过滤器
--ignore-databases	忽略此以逗号分隔的数据库列表。		hash	过滤器
--ignore-databases-regex	忽略名称与此Perl正则表达式匹配的数据库。		string	过滤器
--ignore-engines	忽略此以逗号分隔的存储引擎列表。	FEDERATED,MRG_MyISAM	hash	过滤器
--ignore-tables	忽略这个用逗号分隔的表列表。		hash	过滤器
--ignore-tables-regex	忽略名称与Perl regex匹配的表。		string	过滤器
--max-lag	暂停校验和，直到所有副本的延迟小于此值。工具使用 Seconds_Behind_Master 来比较。如果任何从库的延迟时间超过这个值，则工具将休眠 --check-interval 秒。可以指定 --check-slave-lag，则只检查指定服务器的 lag，而不是所有从库。	1s	time	阈值
--max-load	工具检查每个块之后，执行 show global status，如果任何状态变量高于阈值，则暂停。该选项接受以逗号分隔的MySQL状态变量列表来检查阈值。每个变量后面都可以有一个可选的=MAX_VALUE。如果没有给出，工具通过检查当前值并将其增加20%来确定阈值。如 Threads_running=100，高于120时暂停，小于120时恢复运行。	Threads_running=25	array	阈值
--password	缩写 -p。连接时使用的密码。		string	连接
--pause-file	当此参数指定的文件存在时，将暂停执行。		string
--pid	创建指定的 pid 文件。如果pid 文件已经存在，并且它包含的 pid 与当前 pid 不同，则该工具不会启动。 但是，如果 pid 文件存在并且它包含的 pid 不再运行，则该工具将用当前 pid 覆盖该 pid 文件。当工具退出时，将自动删除 pid 文件。		string
--plugin	定义一个 pt_table_checksum_plugin 类的模块。perl 模块可以挂载到工具的很多部分中。		string
--port	缩写 -P。连接的端口号。		int	连接
--progress	将进度报告打印到STDERR。该值是一个逗号分隔的列表，由两部分组成。第一部分可以是百分比、时间或迭代；第二部分指定更新的打印频率（以百分比、秒或迭代次数为单位）。	time,30	array
--quiet	缩写 -q。只打印重要信息。指定此选项一次将导致工具仅打印具有校验和差异的错误、警告和表。	0	累计值
--recurse	发现从库时在层次结构中递归的级别数。默认值是无限的。		int
--recursion-method	发现从库的递归方法。尽管运行工具不需要检查从库，但是工具不能检查未发现从库中的差异。 可选值还有 cluster、dsn=DSN 、none。 dsn 比较特殊，需要创建特定表并写入数据库信息。	processlist,hosts	array
--replicate	将校验和结果写入指定表。 默认情况下，--[no]create replicate table为true，因此 如果选项不存在，则会自动创建它们。	percona.checksums	string
--[no]replicate-check	完成每个表后，检查副本中的数据差异。	yes
--replicate-check-only	在不执行校验和查询的情况下检查从库的一致性。
--replicate-check-retries	遇到差异时，重试校验和比较的次数。只有当此检查次数之后差异仍然存在时，才认为它有效。将此选项的值设置为2或更大可以缓解使用 –resume 选项时出现的虚假差异。	1	int
--replicate-database	只使用指定数据库存放校验结果表。		string
--resume	从最后完成的块恢复校验和。
--retries	当出现非致命错误时，请多次重试块。非致命错误是诸如锁等待超时或查询被终止等问题。	2	int
--run-time	设置执行校验和的总时间。默认执行完所有库表的校验和后终止。单位后缀有：s、m、h、d。 通常与 --resume 结合使用，可以在分配时间内对尽可能多的表做校验和。		time
--separator	用于 CONCAT_WS() 的分隔符。此字符用于在校验和时联接列的值。	#	string
--skip-check-slave-lag	跳过检查某个从库的延迟，后面填写从库的 DSN。 可以重复，每个从库前面要加一次该选项。		DSN
--slave-user	设置用于连接到从库的用户。		string
--slave-password	设置用于连接从属服务器的密码。它要与–slave user一起使用，并且所有slave上的用户密码必须相同。		string
--set-vars	在 var=value 对的逗号分割列表中设置 mysql 变量。如果无法设置变量，工具将打印警告并继续。		array	连接
--socket	缩写 -S。socket 文件。		string	连接
--slave-skip-tolerance	当主表被标记为仅在一个块中进行校验sum，但从表超过了该块的最大可接受大小时，将跳过该表。	1.0	float
--tables	缩写 -t。仅校验以逗号分隔的表列表的校验和。表名可以用数据库名限定。		hash	过滤器
--tables-regex	仅对名称与此Perl正则表达式匹配的表进行校验和。		string	过滤器
--trim	将 trim() 添加到 varchar 列。
--user	缩写 -u。登陆用户		string	连接
--[no]version-check	检查工具的版本号。		yes
--where	匹配指定的行，只校验部分数据

python 自动一致性校验脚本

db_consistent_check.py 折叠源码

#!/usr/bin/env python3
# -*- coding: UTF-8 -*-
'''=================================================
@Project -> File   ：db_consistent_check -> db_consistent_check
@IDE    ：PyCharm
@Author ：DingLu
@Date   ：2020-02-24 12:27
@Desc   ：mysql databases data consistent check
=================================================='''
"""
检查 ip port 是否是从库
    如果是从库，则获取主库 ip port
检查 ip port 不是从库
    如果是单实例
        break
    如果是主库
        pass
"""
"""
检查主从库 port 是否一致
    一致
        --recursion-method="processlist"
    不一致
        --recursion-method="hosts"
        或
        --recursion-method=dsn=D=xxx,t=xxx
"""
"""
检查数据库参数
    binlog_format 是否是 row
        是
            添加 --nocheck-binlog-format 选项
        不是
            pass

    replication_do_db 空
        --nocheck-replication-filters
"""
"""
是否指定校验和存储表
    是
        --replicate=xxx
    否
        不添加
"""
"""
是否指定数据库
    是
        --databases=xxx
    否
        不添加
"""
"""
是否指定表
    是
        --tables
        或
        --tables-regex
    否
        不添加
"""
"""
拼接命令
    h
    u
    p
    P

"""

import os
import sys
from lib.ssh_server import *
from lib.connect_MySQL import *
from lib.get_local_ip_addr import *

project_dir = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
sys.path.append(os.getcwd().split(project_dir)[0] + project_dir)


class Db_Consis_Check(object):
    """
    数据一致性校验
    """

    def __init__(self, dic):
        self.dic = dic
        self.host = dic['host']
        self.port = dic['port']
        self.user = dic['myuser']
        self.pwd  = dic['mypwd']

    def con_db(self):
        """
        连接 db
        :return: conn 对象
        """
        db_conn = ConnectMySQL(self.dic)
        return db_conn

    def is_slave(self):
        """
        是否为从库
        :return: host_dic
        """
        sql = "show slave status;"
        slave_info_dic_list = self.con_db().execute_sql(sql)
        if slave_info_dic_list:
            slave_info_dic = slave_info_dic_list[0]
            if slave_info_dic:
                master_ip = slave_info_dic['Master_Host']
                master_port = slave_info_dic['Master_Port']
                self.dic['host'] = master_ip
                self.dic['port'] = master_port
            print(self.dic, "是从库")
        return self.dic

    def is_master(self):
        """
        是否为主库
        :return: host_dic
        """
        sql = "show slave hosts;"
        slave_hosts_dic_list = self.con_db().execute_sql(sql)
        if not slave_hosts_dic_list:
            print(self.host, "没有从库, 不能运行 pt-table-checksum")
            return None
        if slave_hosts_dic_list:
            print(self.host, "是主库")
        # 判断从库端口和主库是否一致，如果不一致需要添加 --recursion-method="hosts"
        for dic in slave_hosts_dic_list:
            if int(dic['Port']) != int(self.port):
                print("从库", dic['Server_id'] , '与主库端口不一致，需要添加 --recursion-method="hosts"')
                break
            else:
                pass
        print("从库与主库端口一致")
        return self.dic

    def is_row_format(self):
        sql = "show variables like 'binlog_format';"
        format_dic_list = self.con_db().execute_sql(sql)
        format_dic = format_dic_list[0]
        if format_dic['Variable_name'] == 'binlog_format' and format_dic['Value'] == 'ROW':
            print("行格式是 ROW，需要添加 --nocheck-binlog-format 选项")
            return "--nocheck-binlog-format"
        else:
            return None

    def is_repl_filter(self, repl_filter_flag):
        """
        是否有复制过滤
        :param repl_filter_flag: 过滤标志
        :return: --nocheck-replication-filters
        """
        if repl_filter_flag:
            opt = "--check-replication-filters"
        if not repl_filter_flag:
            opt = "--nocheck-replication-filters"
        print("没有过滤参数，可以添加 ", opt, "选项")
        return opt

    def allocate_dbs(self, dbs_name_string):
        """
        指定校验db
        :param dbs_name_string: db1,db2,dbn
        :return: --databases=db1,db2,dbn
        """
        opt = "--databases=" + dbs_name_string
        print(opt)
        return opt

    def allocate_table(self, db_name_string, tables_name_string):
        """
        指定表，一个或多个表
        :param db_name_string: db1
        :param tables_name_string: tb1,tb2,tbn
        :return: --databases=db1 --tables=tb1,tb2,tbn
        """
        if "," not in db_name_string:
            # 指定了单个db
            opt = "--databases=" + db_name_string + ' ' + "--tables=" + tables_name_string
            return opt
        elif "," in db_name_string:
            print("检查指定表时，只能指定单个DB")
            return None

    def merge_cmd(self, *args, **kwargs):
        # 拼接命令
        opts = opt1 + " " + opt2 + " " + opt3 + " " + opt4

        cmd_string = "pt-table_checksum " + opts + "h=" + self.host + ",u=" + self.user + ",p=" + self.pwd + ",P=" + str(self.port)
        print(cmd_string)
        return cmd_string

    def run_cmd(self,cmd):
        # 运行命令
        local_ip = get_local_ip()
        if self.host == local_ip:
            # 本地执行
            stdout = os.system(cmd)
            return stdout
        elif self.host != local_ip:
            # 远程执行
            obj2 = SshServer(self.dic, cmd)
            remote_stdout = obj2.ssh_to_server()
            return remote_stdout


if __name__ == '__main__':
    host_dic = {'host': '172.16.250.29',
                'port': 3306,
                'uname': 'xxx',
                'pwd': 'xxx',
                'myuser': 'xxx',
                'mypwd': 'xxx'}
    obj = Db_Consis_Check(host_dic)
    host_dic = obj.is_slave()
    host_dic2 = obj.is_master()
    if not host_dic2:
        opt1 = obj.is_row_format()
        opt2 = obj.is_repl_filter('')
        opt3 = opt4 = ''
        cmd = obj.merge_cmd(opt1, opt2, opt3, opt4)
        obj.run_cmd(cmd)