Advanced Redis (master-slave replication, Redis cluster, cache penetration, cache breakdown, cache avalanche)

Master-slave replication is used to establish a database environment exactly the same as the master database, called the slave database; the master database is generally a real-time business database, and there are generally several functions and usage occasions of the slave database: one is as a backup database, After the master database server fails, it can switch to the slave database to continue working; second, it can perform backup and data statistics in the slave database, which will not affect the performance of the master database.

1.2, the role of master-slave replication

Do hot backup of data, as a backup database, after the primary database server fails, it can switch to the secondary database to continue working to avoid data loss.

Primary server: Generally used for write operations.

Slave server: generally used for read operations.

1.3. Environment construction

First, copy the redis.conf configuration file under redis to a new folder, here I create a myredis folder

[root@localhost myredis]# cp /usr/local/redis/redis-6.2.6/redis.conf myredis/redis.conf

Copy port 6379

[root@localhost myredis]# vim redis6379.conf

Edit the following code to the configuration file
include usr/local/redis/myredis/redis.conf
pidfile /var/run/redis_6379.conf
port 6379
dbfilename dump6379.rdb

Copy port 6380

[root@localhost myredis]# vim redis6380.conf

Edit the following code to the configuration file
include usr/local/redis/myredis/redis.conf
pidfile /var/run/redis_6380.conf
port 6380
dbfilename dump6380.rdb

Copy port 6381

[root@localhost myredis]# vim redis6381.conf

Edit the following code to the configuration file
include usr/local/redis/myredis/redis.conf
pidfile /var/run/redis_6381.conf
port 6381
dbfilename dump6381.rdb

Start redis on each port

[root@localhost myredis]# redis-server redis6379.conf
[root@localhost myredis]# redis-server redis6380.conf
[root@localhost myredis]# redis-server redis6381.conf

View redis process

[root@localhost myredis]# ps -ef | grep redis
root 5209 1 0 06:10 ? 00:00:01 redis-server 127.0.0.1:6380
root 5243 1 0 06:10 ? 00:00:01 redis-server 127.0.0.1:6381
root 5611 1 0 06:24 ? 00:00:00 redis-server 127.0.0.1:6379
root 5620 3992 0 06:24 pts/0 00:00:00 grep --color=auto redis

Open three windows to connect to three servers

redis-cli -p 6379
redis-cli -p 6380
redis-cli -p 6381

1.4. One master and two servants

Here, 6379 is regarded as the master server, and 6380 and 6381 are regarded as the slave servers, then 6380 and 6381 need to bind the master server through commands, and enter the following commands on the 6380 and 6381 servers respectively, and the 6379 server is regarded as the master server.

127.0.0.1:6380> slaveof 127.0.0.1 6379
OK

127.0.0.1:6381> slaveof 127.0.0.1 6379
OK

-- Test master server write operation
127.0.0.1:6379> set a 1
OK
-- Slave server read operation
127.0.0.1:6380> get a
"1"
127.0.0.1:6381> get a
"1"

1.5. Precautions

1. When one of the slave servers shuts down and hangs up, will the slave server be reconnected to the master server when it restarts?

No, it will automatically become the primary server.

2. If the failed slave server is reconnected to the master server, will the data of the master server be copied?

meeting.

3. When the master server hangs up, the slave server will not do anything. Check the info replication server status of the slave server, and confirm that the master server is just hung up. Restart the master server and return to the master service position.

1.6, anti-customer-oriented

hang up the main server

-- The main server is down --
127.0.0.1:6379> shutdown

Make the slave server the master server

-- Set the slave server as the master server --
127.0.0.1:6380> slaveof no one
OK

-- 6381 connects to the 6380 server from the server --
127.0.0.1:6381> slaveof 127.0.0.1 6380
OK

1.7. Sentinel mode (sentinel)

It is a disaster recovery solution. The above master server is down. Although the slave server has become the master server, there is a problem, which is to manually become the server

Sentinel: It is actually a Redis server in a special mode, which stores its own information, the information of the master server, and the information of the slave server.

Use one or more sentinels to monitor whether the master server (that is, the server performing write operations) is performing tasks normally. Once the sentry finds that the master server is unavailable, it will find a suitable slave server to become the master server.

Restore to the state of 6379 master servers, 6380, 6381 slave servers

Create a sentinel.conf file in the current myredis directory, do not write the wrong name, the configuration is as follows

sentinel monitor mymaster 127.0.0.1 6379 1

mymaster is the server name for the monitoring object. 1 As long as one sentinel thinks that the master is down, it can switch, and at the same time, one sentinel will be elected to migrate the data

Start sentry mode

[root@localhost myredis]# redis-sentinel sentinel.conf

Shut down the main server 6379 server, and check whether the sentinel will automatically turn the slave server into the main server

-- Main server shutdown downtime --
127.0.0.1:6379> SHUTDOWN

-- 只剩下6380和6381--
[root@localhost myredis]# ps -ef|grep redis
root 5209 1 0 06:10 ? 00:00:10 redis-server 127.0.0.1:6380
root 5243 1 0 06:10 ? 00:00:10 redis-server 127.0.0.1:6381
yanqi 6597 5400 0 07:36 pts/2 00:00:00 redis-cli -p 6380
yanqi 6598 5327 0 07:36 pts/1 00:00:00 redis-cli -p 6381
root 7279 7080 0 08:07 pts/3 00:00:00 redis-sentinel *:26379 [sentinel]
root 7333 3992 0 08:10 pts/0 00:00:00 grep --color=auto redis

At this time, Sentinel will detect that the master server is down, and will choose another slave server to be the master server

2. Redis cluster

2.1. What is a cluster

If the master-slave replication is to perform their duties, then the cluster is a group of the same individuals doing the same thing. In Redis, when the number of data write operations is large, it is not efficient to use only a single server for write operations. If you use a cluster solution and use multiple Redis servers for write operations, a large number of For data, you write a little, I write a little, and everyone shares a little, so the efficiency will be much higher. Just like moving bricks on a construction site, the efficiency of one person moving is definitely very slow, but if more than a dozen people are called to move together, it will be emptied in a short time. The more people there are, the higher the efficiency is, which means that our data capacity has been greatly improved. !

The idea of cluster mode can be used in many places. In short, this idea can be used in scenarios where an individual cannot complete or the efficiency is very low.

2.2. What is a redis cluster

The concept of Redis-Cluster Redis-Cluster adopts a centerless structure. Each node in the cluster is in an equal relationship and is peer-to-peer. Each node saves its own data and the state of the entire cluster. Each node is connected to all other nodes, and these connections are kept active, which ensures that we only need to connect to any node in the cluster to obtain data from other nodes.
Data decentralized storage Redis cluster does not use traditional consistent hashing to allocate data, but uses another method called hash slot (hash slot) to allocate data. Redis cluster allocates 16384 slots by default. When we set a key, we will use the CRC16 algorithm to take the modulus to obtain the slot to which it belongs, and then assign the key to the nodes in the hash slot interval. The specific algorithm is: CRC16(key) %16384.
Redis saves data in memory, and our computers generally do not have a large memory, which means that Redis is not suitable for storing large data. Hbase or MogoDB of the Hadoop ecosystem are suitable for storing large data. Redis is more suitable for handling high concurrency. The storage capacity of one device is very limited, but the cooperation of multiple devices can increase the memory many times, which requires the use of clusters.

There are many ways to build a Redis cluster, such as using client-side sharding, Twemproxy developed by Twitter, Codis developed by Pea Pod, etc., but redis-cluster clusters are supported from redis 3.0 onwards, which is the official solution proposed by Redis, Redis -Cluster adopts a non-central structure, each node saves data and the entire cluster state, and each node is connected to all other nodes.

2.3. Hash slots in Redis cluster

The concept of hash slots is introduced in the Redis cluster. There are 16384 hash slots in the Redis cluster. When performing a set operation, each key will pass the CRC16 check and then take the modulus of 16384 to determine which slot to place in. When building a Redis cluster A part of hash slots will be allocated to each master node in the cluster first. For example, the current cluster has 3 master nodes,

The master1 node contains hash slots 0~5500, the master2 node contains hash slots 5501~11000, and the master3 node contains hash slots 11001~16384.

When we execute "set key value", if CRC16(key) % 16384 = 777, then this key will be assigned to the master1 node, as shown in the figure below

2.4. Communication of nodes in Redis cluster

Since the data in the Redis cluster is stored separately through hash slots, each node in the cluster needs to know the status information of all other nodes, including the current cluster status, the hash slots that each node in the cluster is responsible for, and the status information in the cluster. The master-slave status of each node, the survival status of each node in the cluster, etc. In the Redis cluster, TCP connections are established between nodes and the gossip protocol is used to propagate cluster information. As shown below:

2.5. How to judge that a node is down

The first thing to say is that each node stores information about all the master nodes and slave nodes of the cluster. They judge whether the node can be connected by mutual ping-pong. If more than half of the nodes do not respond when they ping a node, the cluster will think that the node is down, and then connect to its standby node.

3. Cache penetration

3.1. Overview of cache penetration

Cache penetration refers to data that is neither in the cache nor in the database . In this way, each request will check the database instead of the cache. If there are a large number of (highly concurrent) requests coming in at the same time, it will cause huge query pressure on the database. , or even crush the database.

3.2. Solutions

1. Cache empty values: If the data returned by a query is empty, no matter whether the data exists or not, we still cache the empty value and set the expiration time of the empty value to be very short.

2. Set the whitelist for access: use the bitmaps type to define an accessible list. The list id is used as the offset of the bitmaps. Each visit is compared with the id of the bitmaps. If it does not exist, it will be intercepted and access is not allowed.

3. Bloom filter: The bottom layer of the Bloom filter uses a bit array to store data, and the default value of the elements in the array is 0. When the Bloom filter is initialized for the first time, all existing keys in the database will be calculated through a series of hash algorithms (for example: three hash algorithms), each key will calculate multiple positions, and then these positions The element value on is set to 1. Afterwards, when a user key request comes over, the same hash algorithm is used to calculate the location.

If the element values in multiple positions are all 1, it means that the key already exists in the database. At this time, it is allowed to continue to operate backward.
If the element value in more than one position is 0, it means that the key does not exist in the database. At this time, the request can be rejected and returned directly.

In fact, the most fatal problem with Bloom filters is that if the data in the database is updated, the Bloom filters need to be updated synchronously. But it and the database are two data sources, so there may be data inconsistencies.

For example: a new user is added to the database, and the user data needs to be synchronized to the Bloom filter in real time. But the sync failed due to a network exception.

At this time, the user just requested. Since the Bloom filter does not have the data of the key, the request is directly rejected. But this is a normal user, also blocked.

Obviously, if such normal users are intercepted, some businesses cannot tolerate it. Therefore, the Bloom filter depends on the actual business scenario before deciding whether to use it. It helps us solve the problem of cache penetration, but at the same time it brings new problems.

4. Cache breakdown

4.1. Overview of cache breakdown

Sometimes, when we access hot data. For example: we buy a popular product in a certain mall. In order to ensure access speed, under normal circumstances, the mall system will store product information in the cache. But if at a certain moment, the product expires when it expires. At this time, if a large number of users request the same product, but the product is invalid in the cache, all of these user requests will go directly to the database at once, which may cause excessive pressure on the database in an instant and directly hang up.

4.2. Solutions

1. Set hotspot data to never expire. (It is also a solution, but it is not recommended in the actual process)

2. The way of locking: the object of the lock is the key. In this way, when a large number of requests for the same key are sent in concurrently, only one request can acquire the lock, and then the thread that acquires the lock queries the database, and then puts the result into to the cache, and then release the lock. At this time, other requests that are waiting for the lock can continue to execute. Since there is already data in the cache at this time, the data is directly obtained from the cache and returned without querying the database. That is, locking allows one person to query data and store the data in the cache, and other people will directly obtain it from the cache when they query again.

5. Cache Avalanche

5.1. Overview of Cache Avalanche

Cache avalanche refers to when a large number of keys in the cache expire at the same time, or Redis directly crashes, causing a large number of query requests to reach the database, resulting in a sudden increase in the query pressure of the database, or even directly hanging up.

5.2. Solutions

Add random values to the TTL of different keys to avoid a large number of keys expiring at the same time
Use Redis clusters to improve service availability, avoid data loss after Redis service downtime and direct requests to the database
Add a degraded current limiting policy to the cache business
Add multi-level cache to business