CSDN Topic Challenge Phase 2
Participating Topic: Big Data Technology Sharing

insert image description here

1. Why do you want to do this?

Based on the company's previous system, the three-party ordering and membership system was used. After the company's technical staff supplemented it, they wanted to do more operations for members, so they decided to replace the membership system with a self-developed system. Therefore, you need to migrate member-related data to your company's database.

Hereinafter, "three-party system" will be used to refer to the software and systems of the three parties.

My responsibility is very simple, which is to migrate the data related to the membership module in the tripartite system to our database, without involving business logic.

2. Environment

Three-party system database: pgsql;
our database system: mongo;
our script environment: node+Angular

3. Ideas

Due to the large amount of data, and no third-party system should not be stopped for data migration, that is to say, there may be new information when migrating data, and because the third-party system involves many other system modifications, there is a need to continue Possibility of cutover.

Therefore, two migration methods are required:

Migration method 1: initial migration. This migration solution is mainly used for the first migration of data, and the amount of data will be relatively large;
Migration means two: continuous migration. After the migration based on the beginning of the period, the data migration from the previous day to the present (within a certain period of time) is regularly performed every day;

Due to the large amount of data and many pgsql tables involved, data migration is performed according to functional modules, such as: membership basic data migration, membership card migration, consumption record migration...

Since there is an auto-increment primary key in pgsql, the migration method 1, the main migration logic is, the script triggers the module data migration,

According to the auto-increment primary key, data paging cycle migration is performed;
First obtain the paged pgsql data, and then judge whether it is equal to the number of paged data, if it is equal, continue to cycle the mark, if it is not equal, return the not in cycle mark;
Convert pgsql data to the data format required by mongo;
Write mongo data in batches through bulkWrite;
cycle continues;

Migration method 2, continuous migration, the main migration logic is, the script triggers module data migration

According to the time, data paging cycle migration is performed;
First obtain the paged pgsql data, and then judge whether it is equal to the number of paged data, if it is equal, continue to cycle the mark, if it is not equal, return the not in cycle mark;
Convert pgsql data to the data format required by mongo;
Compare mongo data to determine whether to add or modify;
Write/modify mongo data in batches through bulkWrite;
cycle continues;

4. Preparations

Since it is a three-party project, and the other party does not have a database document, so one step at a time, let's take a look at what preparations I have made!

Communicate with the three parties, and first obtain the relevant database documents of the relevant modules;
Understand the data stored in the third-party database and compare it with your own system according to the visualization tool (PC), to see if there are missing database documents that have not been received;
The three-party database fields required for internal communication and consensus database migration and the corresponding database fields of our company, and recorded;
Verify the data of the three-party database and make records, such as the value of the necessary fields, whether the field has a value of null, "", pay attention to whether there are duplicate values in the only field (don't underestimate this, it has been backstabbed many times), special field format For example: mobile phone number, ID card, birthday, etc. At the same time, it is particularly important to count the abnormal data and export the corresponding data for backup according to the importance of the abnormality;
Coordinate the product manager of our company and the third-party system personnel to answer and record abnormal information, which is particularly important and can be used in subsequent battles;
Please refer to the following for specific reasons for determining the database corresponding to each module and the cutover sequence of each module;
Start coding according to the planning logic Yes, the time for coding is not long;
After completing the migration code, do a data test migration and verify the data;

When the above work is being carried out, there is a very important issue, that is, it must be recorded, put into writing, and shared with relevant internal members in a timely manner.
Because, we have to be mentally prepared, a tripartite system without database documents, there must be a certain amount of confusion in the data, if it is not well recorded, there will be many problems after migration. Don't have unrealistic fantasies, "How can such a mature system have data confusion?"
As a programmer, I once shared a law with the team, "Murphy's Law", the bad things you can think of, it It's bound to come, sooner or later. What I explained to the team at the time was that there may be code bugs that you can think of. If you don’t fix them, then some users will trigger this bug, sooner or later. Of course, this has also been verified countless times in the days that followed in our team.

5. Verify

After the data migration is completed, data verification must be done. Because the data volume of the three-party system involved is close to 9000W, data verification is very necessary. Therefore, we design a data validation scheme for two systems.
The main verification data are as follows:

Whether the number of data items is equal, such as: the number of members, the number of WeChat followers, the number of members and fans bound
Randomly take the pgsql data value of the third-party database, and compare it with the corresponding data in our company's mongo database, and whether it is correct;
Verify whether the total of the fields is correct, such as the total of membership points and the total amount of membership cards;

…

6. Questions

Based on this data migration, I stepped on some pitfalls and learned a lot of experience. I hope the following experience can help friends who are preparing to cut over data.

The primary key of the three-party database is an auto-incrementing primary key, but not all field changes will trigger the last_update_time field of some databases.

Solution: Paging query migration by auto-incrementing the primary key

There is a master-slave library in the three-party database, and the link time set by the slave library is 30s. After a large number of pages, the query speed will slow down, which will cause the link database to time out

Solution: link the main library, pay attention to the operation, only query, not modify

Some tables in the third-party database do not have indexes, but they are not suitable for views, because the amount of data is too large, and views will waste resources

Solution: Use the auto-increment primary key pagination query, and obtain the maximum primary key for each pagination, and query the partial quantity data after this primary key

When mongoose's aggregate aggregate operation query, the memory exceeds 32M, causing the query to fail

Solution: aggregate aggregation cooperates with allowDiskUse: true operation to write data into temporary files

A large amount of data with abnormal data, including but not limited to: non-standard value, data binding error, too many operations that a specific person knows (a certain operation causes the value to be different from the normal specification, and it is not commented);

Solution: Record and export relevant data in a timely manner, coordinate relevant personnel to make data judgments (retain/discard), data cleaning

In the three-party system database, there are test data in the formal environment, and the test data is not generated through the formal process, and there are cognitive level bugs in the system operation, for example, the accumulated membership points are negative, the recharge amount of the membership card is negative, etc. …

Solution: 1. Record and export the relevant data, make a note of the problem, 2. Contact the relevant personnel, whether to delete the data, and communicate with the previous party on the leveling experience, whether to keep this kind of data with the other party

There are the same fields among the tables of the three-party database but the meanings are different. Here, a certain field expresses the same meaning in many tables, but suddenly it represents another meaning in another table. It is related to the fact that the three-party system itself does not have a database description document, resulting in The field annotations of the documents given at the beginning of the period are the same, resulting in data mismatch when verifying the data in the third-party database

Solution: Send your own query statement to the group, and ask the third-party developers if there is any problem with the query statement?

The same IP frequently requests a large number of third-party databases, causing the database to mistake the IP as a malicious attack

Solution: When triggering the loop mechanism, increase the delay setting, such as delaying 200 milliseconds to trigger the next loop

7. Precautions

One: In the new system, the unique identification of the data in the redundant tripartite system is convenient for later data query. In order to cause a lot of trouble in the later maintenance stage of this operation, some people always think that their data will be damaged by migrating data. At this time, it is very convenient to query the differences between the old and new systems.

Two: Some fields with special meaning must be considered carefully before making a choice. For example, some of the members are associated with the enterprise WeChat id, but they may not be used in the new system. After a game, they still keep it. Because it is a unique identifier for other systems, it may not be used anywhere.

Three: Record abnormal data, abnormal problem description, quantity, original data content, and solution. Be sure to keep it well. This record has saved me many times in the six months after the data migration. One of the most memorable times is that the accountant made a statistics on the balance of the membership card and found that there were several thousand dollars more. After a gradual investigation, it was found that the money was extra after the migration, and this money happened to be the test data that existed in the three-party system at that time. There are two test membership card data, and the amount is negative.

Four: After migrating the data, do a good job of data statistics, each table, how much original data, how much has been migrated, and how much in total. Used for subsequent review reports.

8. Summary

In fact, the overall code development time for this data migration accounted for about 30% of the overall time. The main proportion of the time was to verify the third-party database data and find solutions for how to deal with these dirty data.
Because we have a very serious obstacle, and the developers of the third-party system are not sure how many pits there are in the database, so we can only step on them one by one.
Therefore, in the stage of verifying the three-party data, I often get angry and need some extra time to calm down my turbulent mood!
The amount of data is too large and there are many tables involved. Therefore, it is very necessary to formulate a comprehensive migration plan. Migrate each module first, and record each module one by one after executing a module, so that the overall operation will be More handy, so as not to lose everything.

The amount of data is too large and involves a lot of business. Therefore, a perfect plan, clear time nodes, and continuous synchronization of progress are the cornerstones for the team to be full of confidence.

message

Bold assumptions, careful verification

Some things about cutting over 9000W data