Introduction: Main content: 1. IO model and problems 2. Resource competition and distributed lock 3. Redis panic buying system example
main content:
1. IO model and problems
Second, resource competition and distributed locks
Three, Redis panic buying system instance
1. IO model and problems
1）Run-to-Completion in a solo thread
The IO model of the Redis Community Edition is relatively simple, usually one IO thread implements the analysis and processing of all commands.
The problem is that if there is a slow query command, other queries will be queued. That is, when a client executes a command execution very slowly, the subsequent commands will be blocked. Using Sentinel to judge live will cause the ping command to be delayed. The ping command is also affected by slow query. If the engine is stuck, the ping fails, which makes it impossible to judge whether the service is available at this time, because this is a misjudgment.
If it is found that the service is not responding at this time, we switch from Master to Slave, and it turns out that slow query slows down Slave. In this case, ping will misjudge and make it difficult to monitor whether the service is reliable.
summary of a problem:
1. All user requests from different clients are actually executed in a single thread after each event arrives. Wait for each event to be processed before processing the next one;
2. Single-threaded run-to-completion means no dispatcher and no back-end multi-worker;
If slow queries such as keys, lrange, hgetall, etc. slow down a query, then subsequent requests will be slowed down.
Defects of using Sentinel to judge live:
• The ping command is alive: the ping command is also affected by the slow query. If the engine is stuck, the ping fails;
• Duplex Failure: Sentinel cannot continue to work if it encounters a slow query due to slow query switching (standby becomes master).
2）Make it a cluster
The same problem applies when multiple shards are used to form a cluster. If one of the shards is slowed down by a slow query, for example, a user calls a cross-shard command, such as mget, and accesses the broken shard, it will still be stuck, which will cause all subsequent commands to be blocked.
summary of a problem:
1. Similarly, the cluster version cannot solve the problem of a single DB being stuck;
2. Query holes: If the user invokes a cross-shard command, such as mget, and accesses the problematic shard, it will still be stuck.
3）“Could not get a resource from the pool”
Common Redis clients such as Jedis will be equipped with a connection pool. When a business thread accesses Redis, each query will take a long connection to access it. If the query is slow and the connection does not return, it will wait a long time because the connection cannot be used by other threads before the request returns.
If the queries are relatively slow, each business thread will take a new long connection. In this case, all long connections will be gradually consumed, resulting in an exception eventually being thrown-there are no new resources in the connection pool. Because the Redis server is a single thread, when a long connection of the client is blocked by a slow query, subsequent requests on the connection cannot be processed in time, because the current connection cannot be released to the connection pool.
The connection pool is used because the Redis protocol does not support connection convergence, and Message does not have an ID, so Request and Response cannot be associated. If you want to achieve asynchronous, you can put the callback into a queue (one queue for each connection) when each request is sent, and take it out of the queue for callback execution after the request returns, that is, the FIFO model. But the server connection cannot make the server return out of order, because there is no way to correspond to out of order on the client side. The general client implementation is relatively simple to use BIO. Take a connection and block it, wait for it to return, and then let it be used by other threads.
But in fact, asynchronous does not improve efficiency, because the server actually has only one thread. Even if the client modifies the access method, which makes many connections to send requests, it still needs to be queued on the server because it is a single thread. Slow queries will still block other long connections.
Another very serious problem is that Redis's threading model has poor performance when the number of IO threads exceeds 10,000. If there are 20,000 to 30,000 long connections, the performance will be slow to the extent that the business cannot afford it. And business machines, such as 300 to 500 machines, each with 50 long connections, can easily reach the bottleneck.
to sum up:
The connection pool is used because the Redis protocol does not support connection convergence
• Message has no ID, so Request and Response cannot be associated;
• Very similar to HTTP 1.x messages.
When a slow query appears in the Engine layer, the request will be returned slowly
• It is easy for users to use up the connection pool;
• When there are a lot of application machines, it is easy to hit the limit of 10K links, and the callback speed is slow according to 50 max_conn per client connection pool;
1. For each query, you must first take out a connection from the connection pool, and then put it back into the connection pool when the request returns;
2. If the user returns in a timely manner, the number of connections that the connection pool has always kept is not high
• But once it fails to return and there is a new request, you can only checkout one more connection;
• When the connection pool is checked out, there will be an exception of no connection: "Could not get a resource from the pool".
Add one more method to implement asynchronous interface on the current Redis protocol:
1. Similar to the above, a connection is allocated a callback queue, before the asynchronous request is sent, the processing callback is put into the queue, and the callback is taken out for execution after the response comes back. This method is relatively common, and mainstream clients that support asynchronous requests generally implement this way.
2. There are some tricks, such as using Multi-Exec and ping commands to wrap the request. For example, to call the set kv command, wrap it in the following form:
multi
ping {id}
set k v
exec
The return from the server is:
{id}
OK
This is a way to implement the ID of the message "entrained" in the protocol using the characteristics of Multi-Exec's atomic execution and the return of the ping parameters. 4) Redis 2.x/4.x/5.x version thread model
In the well-known version of Redis5.X before, the model has not changed. All command processing is single-threaded, and all reading, processing, and writing are run in one main IO. There are several BIO threads in the background, whose tasks are mainly to close files, refresh files, and so on.
After 4.0, LAZY_FREE was added, and some large KEYs can be released asynchronously to avoid blocking synchronous task processing. However, on 2.8, the service will often be stuck when a large key is deleted or expired. Therefore, it is recommended that users use a server with 4.0 or higher to avoid such a large key deletion problem.
5 ) Flame diagram of Redis 5.x version
Performance analysis is shown in the following figure: the first two parts are command processing, the middle is "read", and the rightmost "write" accounts for 61.16%. It can be seen that the performance ratio is basically consumed on network IO .
6) Redis 6.x version of threading model
The improved model of Redis 6.x version can entrust the "read" task to the IO thread for processing after the readable event is triggered in the main thread. After the full read is completed, the result is returned, processed again, and then "write" can also be distributed Writing to IO threads can obviously improve performance.
This kind of performance improvement, there is only one running thread. If there are some O(1) commands, such as simple "read" and "write" commands, the improvement effect is very high. But if the command itself is very complicated, because the DB still has only one running thread, the improvement effect will be poor.
There is another problem. After entrusting the "read" task, you need to wait for the return, and the "write" also needs to wait for the return. Therefore, the main thread has been waiting for a long time, and the service cannot be provided during this period, so the Redis 6.x model still There is room for improvement.
7) Thread model of Alibaba Cloud Redis Enterprise Edition (Tair enhanced performance)
The Alibaba Cloud Redis Enterprise Edition model goes a step further, splitting the entire event, the main thread is only responsible for command processing, and all read and write processing is solely responsible for the IO thread, no longer the connection always belongs to the main thread. After the event starts, read it. When the client connects in, it is directly handed over to other IO threads. From then on, the main thread no longer cares about all events that are readable and writable by the client.
When a command arrives, the IO thread will forward the command to the main thread for processing. After the processing is completed, the processing result will be transferred to the IO thread through notification, and the IO thread will write to the greatest extent to remove the waiting time of the main thread, so that the performance is improved. Improve further.
The disadvantage is that there is only one thread processing commands. The effect of O(1) command promotion is very ideal, but the effect is not very ideal for commands that consume more CPU.
8) Performance comparison test
As shown in the figure below, the gray on the left is: redis community 5.0.7, and the orange on the right is: redis enhanced performance, redis6.X's multi-threaded performance is between these two. The command in the figure below tests the "read" command, which does not consume CPU itself, and the bottleneck is IO, so the effect is very ideal. In the worst case, assuming that the command itself is very CPU intensive, the two versions will approach infinitely until they are even.
It is worth mentioning that the plan for redis community edition 7 has been released. According to the current plan, redis community edition 7 will adopt a modification scheme similar to the one currently adopted by Alibaba Cloud, gradually approaching the performance bottleneck of a single main thread.
Here I add that performance is only one aspect. Another benefit of handing over the connection to other IOs is that the linear increase in the number of connections can be obtained, and the processing capacity of a larger number of connections can be continuously improved by increasing the number of IO threads. Alibaba Cloud's enterprise version of Redis provides tens of thousands of connections by default, and higher connections, such as 50,000 to 60,000 long connections, can also be supported by a work order to solve the problem of insufficient connections when the user's business layer machines are massively expanded. .
Second, resource competition and distributed locks
1 ) CAS/CAD high-performance distributed lock
The Redis string write command has a parameter called NX, which means that it can be written when the string does not exist, which is a natural locking scenario. With this feature, it is very easy to lock, the value takes a random value, and the NX parameter can be used to ensure atomicity when set.
With EX is for the business machine to add the lock, if for some reason it is offline (or suspended animation or the like), resulting in the lock not being released normally, the lock will never be unlocked. Therefore, an expiration time is required to ensure that the lock will be released after the business machine fails.
The parameter "5" here is just an example, and it does not necessarily have to be 5 seconds. It depends on what the business machine needs to do.
It is troublesome to delete distributed locks. For example, after the machine is locked, it suddenly encounters a situation, freezes or loses connection for some reason. After losing connection, 5 seconds have passed, this lock has been invalidated, other machines have been locked, and then the previous machine is available again, but after processing, such as deleting the Key, the original is deleted. Locks that do not belong to it. Therefore, deletion requires a judgment. When the value is equal to the previously written value, it can be deleted. Redis currently does not have such a command, which is generally implemented through Lua.
To delete the key when the value is equal to the value in the engine, you can use the "Compare And Delete" CAD command. The CAS/CAD command and the TairString mentioned later are open source in the form of Module: https://github.com/alibaba/TairString . No matter which Redis version the user uses (need to support the Module mechanism), they can directly load the Module and use these APIs.
When renewing CAS, we give an expiration time, such as "5 seconds", if the business is not processed within this time, there needs to be a mechanism to renew the contract. For example, the transaction has not been executed, and 3 seconds have passed, so the running time needs to be extended in time. Renewing the contract is the same as deleting. We cannot directly renew the contract. You must renew the contract when the value is equal to the value in the engine. The contract can only be renewed if it is proved that the lock is held by the current thread, so this is a CAS operation. In the same way, if there is no API, you need to write a piece of Lua to realize the renewal of the lock.
In fact, distributed is not particularly reliable. For example, as mentioned above, although the lock is lost after adding the lock and the lock is held by others, it is suddenly available again. At this time, the code will not judge whether the lock is held by the current thread. Yes, it may re-enter. Therefore, Redis distributed locks, including other distributed locks, are not 100% reliable.
This section summarizes:
• CAS/CAD is an extension of Redis String;
• The implementation of distributed locks;
• Renew (using CAS)
• Detailed documentation: https://help.aliyun.com/document_detail/146758.html ;
CAS/CAD and TairString mentioned later are open sourced as modules: https://github.com/alibaba/TairString .
2) Lua implementation of CAS/CAD
If there is no CAS/CAD command, you need to write a piece of Lua. The first is to read the Key. If the value is equal to my value, then you can delete it; the second is to renew the contract, and the value is equal to my value. Update the time.
It should be noted that the value that will change every time the script is called must be passed as a parameter, because as long as the script is different, Redis will cache the script. As of now, the community version 6.2 still does not limit the configuration of the cache size, and there is no step-by-step configuration. According to the strategy, the execution of the script flush command to clean up the cache is also a synchronous operation. It is necessary to avoid the script cache being too large (the ability to delete the cache asynchronously has been added to the community version by Alibaba Cloud engineers, and Redis 6.2 starts to support script flush async).
The method of use is to first execute the script load command to load Lua into Redis, and then use the evalsha command to call the script with parameters, first to reduce network bandwidth, and second to avoid loading different scripts each time. It should be noted that evalsha may return that the script does not exist. You need to deal with this error and re-script load to solve it.
The Lua implementation of CAS/CAD also requires attention:
• In fact, due to Redis's own data consistency guarantees and crash recovery capabilities, distributed locks are not particularly reliable;
• The Redis author proposed the Redlock algorithm, but there are many controversies: Reference 1 , Reference 2 , Reference 3 ;
• If you have higher requirements for reliability, you can consider other solutions such as Zookeeper (reliability++, performance--);
• Or, use the message queue to serialize this mutually exclusive operation. Of course, this should be designed according to the business system.
3) Redis LUA
Generally speaking, it is not recommended to use LUA in Redis. The execution of LUA requires parsing, translation, and then the entire process.
First: because Redis LUA is equivalent to adjusting LUA in C, and then adjusting C in LUA, the return value will be converted twice, first from the Redis protocol return value to the LUA object, and then from the LUA object to C The data is returned.
Second: There are a lot of LUA parsing, VM processing, including lua.vm memory usage, will be slower than normal command time. It is recommended to use LUA to write only relatively simple ones, such as if judgment. Try to avoid loops, try to avoid heavy operations, and try to avoid big data access and acquisition. Because the engine has only one thread, when the CPU is consumed in LUA, there are fewer CPUs to process business commands, so use it with caution.
to sum up:
• “The LUA Iceberg inside Redis”
The script's compile-load-run-unload is very CPU intensive. The entire Lua is equivalent to pushing complex transactions to Redis for execution. If you are not careful, the memory will burst and the engine will run out of computing power and Redis will hang.
• “Script + EVALSHA”
You can pre-compile and load the script in Redis (without unload and clean), use EVALSHA to execute it, it will save CPU than pure EVAL, but Redis restart/switch/change the code cache will fail and require reload, which is still a defective solution . It is recommended to use complex data structures or modules instead of Lua.
• For JIT technology in the storage engine, "EVAL is evil", try to avoid using Lua to consume memory and computing resources (save trouble and worry);
• Some SDKs (such as Redisson) have built-in Lua in many advanced implementations. Developers may step into the storm of CPU computing inexplicably and must be cautious.
Three, Redis panic buying system instance
1 ) Features of panic buying/seckilling scenes
• The spike campaign carries out regular and quantitative sales of scarce or special prices, attracting a large number of consumers to snap up, but only a small number of consumers can successfully place an order. Therefore, the spike activity will generate tens of times and hundreds of times larger than usual page visit traffic and order request traffic in a short period of time.
• The spike activity can be divided into 3 stages:
• Before the spike: the user keeps refreshing the product detail page, and the page request reaches an instantaneous peak;
• The spike start: the user clicks the spike button, and the order request reaches the instantaneous peak value;
• After the spike: A small number of users who have successfully placed an order continue to refresh their orders or return orders, and most users continue to refresh the product details page to wait for the opportunity.
2 ) General method of panic buying/seckilling scene
• Panic buying/seckilling actually mainly solves the problem of high concurrent reading and writing of hot data .
• The process of panic buying/seckilling is a process of constantly "pruning" requests:
1. Minimize the user's read and write requests to the application server as much as possible (the client intercepts part of it);
2. The request from the application to the server should reduce access to the back-end storage system (part of the server LocalCache intercepts);
3. Requests that require the storage system to minimize access to the database (use Redis to intercept the vast majority);
4. The final request arrives at the database (you can also queue up in the message queue again, in case the back-end storage system does not respond, the application server must have a solution).
• Basic principles
1. Less data (static, CDN, front-end resource merging, page dynamic and static separation, LocalCache) do everything possible to reduce the demand for the dynamic part of the page. If most of the front-end page is static, it can be all through CDN or other mechanisms Blocked, the server request will be much less in terms of volume and bytes.
2. The path is short (the path from the front end to the end is as short as possible, minimize the dependence on different systems, and support current limiting and downgrading); after the user initiates the path to the final spike, there are fewer business systems that depend on it. The road system also has less competition, and each layer must support current limiting and downgrading. After the current limiting is downgraded, the front-end prompts are optimized.
3. Order prohibition (application service stateless horizontal expansion, storage service to avoid hot spots). Any part of the service must support stateless horizontal expansion. For the state that is stored, avoid hotspots, generally avoiding some read and write hotspots.
• When to deduct inventory
1. Place an order to reduce inventory (to avoid malicious orders without payment, to ensure that inventory data cannot be negative when large concurrent requests);
2. Reduce inventory by payment (the experience is affected if the payment is not successfully paid after the order is placed);
3. Overtime release of withholding inventory (can be combined with Quartz and other frameworks, and security and anti-cheating must be done).
Generally, the third option is chosen. The first two have defects. The first one is difficult to avoid maliciously placing an order without payment. The second one succeeds in placing an order, but the payment cannot be made because there is no inventory. Both experiences are very bad. Generally, the inventory is withheld first, and the inventory will be released if the order expires. Combined with the TV framework, security and anti-cheating mechanisms will be implemented at the same time.
• The general implementation of Redis
1. String structure
• Use incr/decr/incrby/decrby directly, note that Redis does not currently support upper and lower bounds;
• If you want to avoid negative or related inventory sku deductions, you can only use Lua.
2. List structure
• Each product is a List, and each Node is an inventory unit;
• Use lpop/rpop commands to deduct inventory until nil (key not exist) is returned.
List shortcomings are more obvious, such as: the occupied memory becomes larger, and if you deduct more than one at a time, lpop will have to be adjusted many times, which is very bad for performance.
3. Set/Hash structure
• Generally used to remove duplication and limit users to only purchase a specified number (hincrby counts, hget judges the purchased quantity);
• Note that the user UID must be mapped to multiple keys for reading and writing, and must not be placed in a certain key (hot spot); because the read and write bottleneck of a typical hot key will directly cause a business bottleneck.
4. If the business scenario permits, hot commodities can use multiple keys: key_1, key_2, key_3...
• random selection;
• User UID mapping (different user levels can also set different inventory levels).
3 ) TairString: String that supports high-concurrency CAS
Another structure in the module, TairString, modifies Redis String, supports high-concurrency CAS String, carries Version String, and has a Version value. When reading and writing, bring the Version value to achieve optimism. Note that the data structure corresponding to this String is different. One type, cannot be mixed with Redis ordinary String.
The role of TairString, as shown in the figure above, first give an exGet value, it will return (value, version), and then based on the value operation, update with the previous version, if it is consistent, then update, otherwise read again, and then change and update , To achieve CAS operation, it is optimistic lock on the server side.
For the above scenarios, the exCAS interface is further optimized. ExCAS is the same as exSet, but after encountering a version conflict, it not only returns an inconsistent version error, but also returns a new value and a new version. In this case, the API call is reduced again, first exSet and then use exCAS to operate, if it fails, then "exSet -> exCAS" to reduce network interaction and reduce the amount of access to Redis.
This section summarizes:
TairString : String that supports high-concurrency CAS.
• String carrying Version
• Ensure the atomicity of concurrent updates;
• Realize updates through Version, optimistic locking;
• Cannot be mixed with Redis ordinary String.
• More semantics
• exIncr/exIncrBy: snapped up/sec (with upper and lower bounds);
• exSet -> exCAS: Reduce network interaction.
• Detailed documentation: https://help.aliyun.com/document_detail/147094.html .
• Open source as Module: https://github.com/alibaba/TairString .
4 ) Comparison of atomic count between String and exString
The String method INCRBY has no upper and lower bounds; the exString method is EXINCRBY, which provides various parameters and upper and lower bounds. For example, you can directly specify the minimum value to be 0. When it is equal to 0, it cannot be reduced. In addition, it also supports expiration. For example, a certain product can only be snapped up during a certain period of time, and it will be invalidated after this time point. The business system will also impose some restrictions, the cache can be restricted, and the cache will be cleared after a time point. If the inventory is limited, for example, if no one buys, the product will be eliminated after 10 seconds; if someone has been buying, the cache will always be renewed, you can put a parameter in EXINCRBY, and it will be renewed every time INCRBY or API is called. Period to increase the hit rate.
What can I do with the counter expiration time?
1 A product is designated to be snapped up in a certain period of time, and the inventory needs to expire after a certain period of time.
2. If the cached inventory is limited, the items that no one bought will expire and be deleted, and if someone purchases it, it will be automatically renewed for a period of time (increasing the cache hit rate).
As shown in the figure below, using Redis String, you can write the above paragraph of Lua. When "get" KEY[1] is greater than "0", "decrby" is subtracted by "1", otherwise an "overflow" error is returned, and it has been reduced to "0". Can not be reduced. The following is an example of execution, ex_item is set to "3", and then subtracted, subtracted, and subtracted. When it is greater than "0", it returns an "overflow" error.
Using exString is very simple, directly exset a value, and then "exincrby k -1". Note that String and TairString are of different types, and APIs cannot be mixed.
