mysql-replication is a pure Python implementation of the MySQL replication protocol built on PyMYSQL. Events such as insert, update, delete data and raw SQL queries can be received.
effect:
MySQL to NoSQL Database Replication
MySQL to Search Engine Replication
Cache Invalidation
Auditing
Real-time Analysis of Changes in the Database
Use environment:
MySQL 5.5, 5.6 and 5.7
Python >= 2.7
Python 3.3, 3.4, 3.5 and 3.6 (3.2 is not supported)
PyPy (which is indeed faster than the standard Python interpreter)
This project is used for business-critical production in some medium-sized internet companies. But all use cases are not fully tested in the real world.
prerequisites:
MySQL server configuration:
[mysqld] server-id = 1 log_bin = /var/log/mysql/mysql-bin.log expire_logs_days = 10 max_binlog_size = 100M binlog-format = row #Very important if you want to receive write, update and delete row events
Document address:
https://github.com/noplay/python-mysql-replication
https://python-mysql-replication.readthedocs.io/en/latest/
Install:
pip install mysql-replication
Example of use:
#!/usr/bin/env python3 # _*_ coding:utf8 _*_ import json import sys,os import pymysqlreplication from pymysqlreplication import BinLogStreamReader from pymysqlreplication.row_event import (DeleteRowsEvent,UpdateRowsEvent,WriteRowsEvent,TableMapEvent) MYSQL_SETTINGS = {"host": "192.168.1.238","port": 3306,"user": "root","passwd": "123456"} ##配置链接地址 def main(): stream = BinLogStreamReader(connection_settings=MYSQL_SETTINGS,server_id=5,blocking=False,only_schemas = ["masterdb"],freeze_schema=True) for binlogevent in stream: try: if binlogevent.schema == "masterdb": if isinstance(binlogevent, WriteRowsEvent): print("类型:insert;库表:`%s`.`%s`;%s"%(binlogevent.schema,binlogevent.table,binlogevent.rows[0]["values"])) elif isinstance(binlogevent, DeleteRowsEvent): if binlogevent.primary_key : ##如果有主键 print("类型:delete;库表:`%s`.`%s`;主键:%s;%s"%(binlogevent.schema,binlogevent.table,binlogevent.primary_key,binlogevent.rows[0]["values"])) elif isinstance(binlogevent, UpdateRowsEvent): if binlogevent.primary_key: print("类型:update;库表:`%s`.`%s`;主键:%s;%s"%(binlogevent.schema,binlogevent.table,binlogevent.primary_key,binlogevent.rows[0])) elif isinstance(binlogevent,TableMapEvent): pass else: binlogevent.dump() except Exception: pass stream.close() if __name__ == "__main__": main()
BinLogStreamReader parameter description:
ctl_connection_settings: connection settings for cluster save mode information
resume_stream: start from latest event or old available event from location or binlog
log_file: set replication start log file
log_pos: set replication start log pos (resume_stream should be true)
auto_position: use master_auto_position gtid setting position
blocking: reading blocked on stream
only_events: array of allowed events
ignored_events: array of ignored events
only_tables: array containing tables to watch (only for binlog_format ROW)
ignored_tables: array containing tables to skip
only_schemas : Array containing schemas to watch
ignored_schemas : Array containing schemas to skip
freeze_schema : If true, ALTER TABLE is not supported. faster.
skip_to_timestamp: Skip all events until the specified timestamp is reached.
report_slave: Report slaves in SHOW SLAVE HOSTS.
slave_uuid: Report slave_uuid in SHOW SLAVE HOSTS.
fail_on_table_metadata_unavailable: should throw an exception if we can't get table information about row_events
slave_heartbeat: (seconds) The master should actively send heartbeat connections. This also reduces GTID replication traffic when replication resumes (in the case where many events are skipped in the binlog). See MASTER_HEARTBEAT_PERIOD in the mysql documentation for semantics