DM Source Read series (eight) Online Schema Change synchronization support

Author: lan

This article is a series of articles to read DM Source eighth chapter, the articles of DM in the customized data synchronization functions in detail, including libraries routing table (Table routing), black and white lists (Black & white table lists), column values conversion (Column mapping), to achieve the four main functions binlog filter (binlog event filter).

This article will to gh-ost, for example, detail how DM is supported by some third-party online schema change synchronization program on MySQL, including a brief online schema change programs, online schema change synchronization scheme, as well as to achieve synchronization detail.

MySQL's Online Schema Change Program

There are some third-party tools support Online Schema Change in MySQL above, more mainstream, including pt-online-schema-change , and GH-OST .

The principle of these tools are relatively similar, it will gh-ost example to explain the analysis.

It may be generally understood from FIG logic flow gh-ost of:

1. Use in the operation target database create table ghost table like origin tableto create the ghost table;
2. In accordance with the requirements change table structure, such as add column/index;
3. ghost itself into a MySQL replica slave, the whole amount of the original data tables and data synchronization binlog incremental changes to the ghost table;
4. Data synchronization is performed after the completion of rename origin table to table_del, table_gho to origin tablehandover is completed and the original table ghost table

pt-online-schema-change data synchronization is achieved by way of the trigger, the remaining process is similar.

In the task configuration by setting the DM's online-ddl-schemeonline schema change to configure the program, currently only supports gh-ost / pt two configuration options.

DM Online Schema Change synchronization scheme

According to the process described in the previous chapters, pt, and gh-ost replicate data addition mode is different, other processes are similar, and this may cause the native mode modification binlog replication almost no data can be synchronized. However, in order to reduce the amount of DM data synchronization, a simplified process flow in some scenarios (e.g., shard tables merge), and make additional optimization, i.e., not synchronized ghost data table.

Continue to analyze the process online schema change, and data synchronization from the point of view of these need to focus on the following points:

• Incremental data synchronization pattern has not changed the original table
• ghost table with the original table will have almost the same redundancy binlog events
• By rename origin table to table_del, table_gho to origin tableswitching the original ghost table and the table is completed

If a ghost table alter DDLreplace rename origin table to table_del, table_gho to origin table, then we can achieve our aim of sync ghost table data.

DM Online Schema Change synchronization implementation details

Online schema change module code to achieve the following:

• gh-ost achieve synchronization code
• pt-online-schema-change code to achieve synchronization

DM will be synchronized table is divided into three categories :

• real table - the original table
• trash table - non-critical online schema change table data generated in the process, such as in _ghc, _delsuffix table
• ghost table - table corresponding to the original data table through DDL changes, such as in _ghothe suffix table

When the DM encounter DDL, will the calling code online schema change processing module , first determine the type of table, then make a different treatment for different types:

• real table - to rename table statement schema checking, execution returns directly
• trash table - do some checking mode to rename table statement, simply ignore sync
• ghost table
  • If a DDL Create / Statement Table drop , then the residual information memory continues to empty ignore the synchronization DDL
  • If the DDL is the rename the Table of Statement , the preservation of the memory of the ghost table DDLs return
  • If other types of DDL, put these DDL saved in memory

The following is an example of execution, to facilitate understanding of the above code logic shining:

1. Section 1: Use create table like statement creation ghost table, DM will clear the memory online_ddl. _t2_ghoCorresponding DDL information
2. Section 2: execute alter table statement, DM DDL saved in memory
3. Section 3: trash table of DDLs will be ignored
4. Section 4: encounters the ghost table rename table statement will be replaced Section DDL 2 and DDL the table name is replaced with the corresponding table name to perform Real

Note: rename table statement pattern inspection primarily to ensure that in addition to the online schema change process change rename origin table to table_del, table_gho to origin tablethan any other rename table statement, to avoid complicating the synchronization status.

summary

This article describes in detail DM synchronization program supports online schema change, the content includes a brief online schema change programs, online schema change synchronization scheme, synchronization and implementation details. The next chapter will DM the shard DDL merge programs in detail, so stay tuned.

Read the original : www.pingcap.com/blog-cn/dm-...

Reproduced in: https: //juejin.im/post/5d0af079f265da1b95705c97