1. What is the status frame changes resulting from springmvc service springcloud micro framework?
Poor scalability, performance difficulties
The basic web application performance bottlenecks are in the database. This system uses a mysql database. Application of three corresponding to the three databases. No separate read and write. Both read and write operations on a database. The largest amount of data in the table at that time 50 million data. QPS database operations peak at about 1000, can support the pressure measurement result of QPS 2000. This indicator makes me surprised. Why have such a good performance? First of all, there is no logic complex queries, all queries in a table operation, no cross-table transaction processing, complex processing, broken down into multiple statements to execute. The most complex action, perform nearly 20 times the data query. Second, and most important factor here is the SSD disk. From the current situation, it should be able to hang on until the end of the year, which also enough time for our technical innovation. Despite this, for mysql still not sure. Every time operations to engage in activities, we are playing scared staring, prayer should not be too effective.
From the application layer, the current literacy ratio of 10: 1, the interface was visited 10 billion. Peak period traffic in 300QPS. The company's business is growing rapidly, doubling the amount of data half a year, 10 times the projected traffic growth. There is also a serious challenge, the students threatened to engage in product spike, spike ... the amount of hundreds of thousands per second must be to support. This exceeds the withstand pressure range MYSQL, need to be cut to a read operation on memory database, but in SSH architecture, have separate read and write to achieve a beating. In addition, as Hibernate encapsulates the operation of the database without writing SQL, and fine optimization can also get it working. Every time the system slows down, you have to ask DBA, SQL help see those stuck. Every so often, so please DBA Export SQL statements, studies how to build the index.
System bloated, long learning cycle
More than 100 interfaces, is divided into three large projects. The maximum project has more than 1300 class, followed by more than 600 and more than 300 classes. SSH architecture, SVN version control, resin as a container, Nginx pre-routing. Reassuring this route, it is a strong support throughout the reconstruction work. Pure back-end of the project, the mobile terminal app, PCWEB applications interface. It also makes more difficult the reconstruction work is greatly reduced. If the front end also coupled in, cool it more acidic.
Large-scale system for team members to take over the difficult. After payment service independent developers from the original five people, within two months expanded to 10 people. At the same time, the students exciting product also related to play with blood, like all kinds of ideas have become products, development pressure surge. But the new students, watching hundreds of classes, often at a loss, unable to start. I do not know what function to achieve, and which functions are to be improved. Until three months later, new employees only gradually into the role. Nevertheless, there are still many dinosaur-level code, no one dared challenge. The largest of a class is over 2000 lines, the core of the method more than 500 lines, a large number of duplicate code, each adjustment ended in failure.
High costs of cooperation
With the increase of the project team, the development of each new version requires people to work together to modify the same item code. Although the use of version control tools to manage the branch, but inevitably, a lot of time spent on the code conflict management. New features, enhancements, bug fixes, support for a variety of clients, are carried out on a project, you need to establish different branches, the peak of the five or six branches at the same time are common. In this case, the frequency of code violations is very high. One week a small version of the development, 1 day in conflict resolution are normal.
Difficult test
Test work has gradually deteriorated.
Test environment to build a high degree of difficulty. With the increase of branches, each branch into the test, you need to prepare a separate test environment. Construction of the high environmental costs.
Just finished testing the function, due to branch consolidation conflict management, we have to re-run it again. Seriously affect the project schedule.
On line high risk
With the increasing complexity of the systems, on-line risks are also increasingly large. A small functional changes, print a log, fix a bug, requires a whole line. Once a modified wrong place, the system collapsed. On-line for a long time, once on the line, half at night is a must
The advantages of micro-services reform
Before you begin to pay the transformation project, we have just completed a micro-service architecture firm data warehouse project improvements. The project implementation process in detail, in the community did dockone share details, see here. We believe that adjusted to the micro-service architecture can solve these problems.
Performance issues
For high performance requirements of the interface can be optimized by way of establishing a data cache.
Learning cycle
Only a few interfaces item contains a tightly coupled single business logic, read the code developer interface 1-2 hours, you can get started quickly.
Cooperation costs
Each project is relatively independent, interact only through the interface between projects. After determining the interface, development, testing, on-line, are carried out independently, thus reducing communication costs.
version control
Because the project is the interface between the dependent and not dependent on the code, each project can create a separate code base. At the same time the project segmentation is relatively small, each project development, there will be only one of its developers to make changes. This is the basic code that does not exist merger work, but also to avoid problems in the process of merging the code. In fact, based on the development of micro-services architecture, we did not adopt a branch policy, but directly with the trunk development.
Test difficulty
Independent deployment of each project, independent testing. By eliminating the code branches, no code merger problems, repeat the test workload is reduced.
On-line risk
Each item on a separate line, even if the question arises, it affects only a small number of interfaces.
new technology
After a quarter in micro-services reform, new technologies are introduced into the system, development is no longer confined to SSH architecture. Spark, Hadoop, Hbase and other large data processing related technologies, Couchbase, Redis and other caching systems, are beginning to use in the project, and effectively solve the problem of the existence of the business.
Of course, good foundation, micro-service problems brought about by a lot, including projects and more difficult to troubleshoot when things go wrong, etc.
2. How to ensure that the user login information and other large data traffic data consistency problem?
Redis used as a distributed cache, cache coherency also relates to cache penetration / breakdown, avalanche cache, problems such as the failure hot dataset.
Delete the data in the cache, and then go to update the database, last updated data in the cache
write requests come, we delete the data in the cache,
then deleted successfully, we'll update the data in the database, update the database at this time if data fails, the entire write request fails, direct return data is not changed, this time over a read request, the cache does not discover corresponding data will read data from the database, while data is written to the cache, this data and cache data in the database are the same, there is no data consistency problems
after updating data in the database, updating data in the cache again, this time to update data in the cache fails, return directly in the database the data is the latest data, starts up the read request, the cache does not discover corresponding data will read data from the database, while the data is written to the cache, the data cache and the data in the database at this time are is the same, there is no data consistency problems
updating the cache successful, then the cache According to data in the database and is consistent, there is no data consistency problems
At first glance, this solution perfect solution to the problem of data consistency, we might come to more complex business scenarios, then bigger quantity of concurrent read QPS per second for example, 1w +, which is let us analyze the above scheme business logic:
Users write requests come, we delete the cache, and then update the database
in the process of updating the database, the update has not been completed at this time, the value of the original database is still the old value, then a read request came
found the cache has no value will go to the database query data, and then writes to the cache, then the database has not been updated over, read requests to obtain the original data is still old data
then the database update is complete, but failed to update the cache, then the cache is used before the old data with new data in the database will be inconsistent data occurs, data consistency problems have emerged
Thus, the above scheme is also problematic, especially in a large amount of concurrent case, the probability of this kind of phenomenon is very substantial; this case how do we handle it?
We carefully analyze the above situation can be found, read and write requests in parallel, which is the root cause of data consistency, concurrent requests can lead to data consistency problems, then the idea to solve such problems there - request the serial!
Specific business logic is as follows:
Write requests over the write request to the cache buffer queue and begin the write request specific operations (delete data in the cache, update the database, updating the cache)
If in the process of updating the database, again a read request to read request again stored in the cache queue, waiting queue before the write request execution is complete, will perform a read request
written request before deleting cache failure, direct return data in the database at this time is the old value and the data cache is the same, cache coherency problem does not occur
after deleting the cache write request is successful, update the database, if the database update fails, returns to direct, write request ends, this time value in the database is still the old value, read requests come and found no data in the cache will request directly to the database, while data is written to the cache, then the data consistency problem does not occur
after a successful update the data, and then update the cache, the cache fails if the update at this time , then there is no data in the cache, the database is the new value, the end of the write request or read request at this time Like, there is no data found in the cache, it will read data from the database and stored in the cache, the cache is updated no matter where in fact, success or failure, all data consistency problem does not occur
