Redis high availability-data persistence

The high availability of Redis is guaranteed from the following aspects:

Data persistence
Master-slave replication
Automatic failure recovery
Cluster

The author will introduce the above aspects one by one. Today's article will first introduce Redis data persistence.

1. Why do we need persistence?

Redis is an in-memory database. All data is stored in memory. If the instance goes down, all data will be lost. Redis provides a complete data persistence mechanism, which can persist data to disk, which is convenient for us to back up and quickly restore data. .

Two, Redis data persistence method

Redis provides two data persistence methods:

RDB: Save the data in the memory to the hard disk in the form of a snapshot, and generate the dump.rdb file, which is a binary file. Because the RDB file is stored in the disk, even if the Redis server process exits, the RDB file is still there. The Redis server detects the existence of the RDB file when it starts, and it automatically loads the RDB file and uses it to restore the database state.
AOF: AOF persistence records the database status by saving the write commands executed by the Redis server. The AOF file saves the Redis write command request protocol , which is in plain text format. You can directly open the AOF file to view the file content.

Note:
1. Redis turns on the RDB persistence function by
default ; 2. AOF is off by default. If the AOF persistence function is manually turned on, the AOF file will be used first to restore the database state, because the update frequency of the AOF file is usually higher than the update frequency of the RDB file high.
Redis starts the data loading process

1. RDB endurance

1.1 When will RDB persistence be triggered?

The situations that trigger RDB persistence are divided into manual triggers and automatic triggers.

1.1.1 Manual trigger

Manual trigger command:

save: Blocking. An instance with a large memory will cause a long time block during execution, during which the main process cannot process any command requests.
bgsave: fork child process, the child process is responsible for the creation of RDB files, and the parent process can continue to process command requests. Blocking occurs in the fork phase, and the time is short.

1.1.2 Automatic trigger

The conditions for automatically triggering RDB are:

Use save configuration;
configuration format: save [seconds] [changes]
means that bgsave is automatically triggered when there are [changes] changes in the data set within [seconds] seconds.
The default configuration is:

#只要以下3个条件中的任意一个满足，bgsave就会执行。
#服务器在900秒内，对数据库进行了至少1次修改
save 900 1
#服务器在300秒内，对数据库进行了至少10次修改
save 300 10
#服务器在60秒内，对数据库进行了至少10000次修改
save 60 10000

Implementation principle:

#serverCron服务定时器每100ms执行一次检查
if(now()-rdb_last_save_time < m(指定秒数) and rdb_changes_since_last_save>n(修改次数)){
    
    
   bgsave();
}

Perform full copy operation from the node;
When the shutdown command is executed, bgsave will be executed automatically if AOF is not turned on;
debug reload command;

Note :如果开启了自动RDB，flushall因涉及的操作较多，可能会触发自动RDB，新产生的RDB文件将为空。

1.2 RDB related configuration in redis.conf

#RDB自动触发条件(满足任意一个就可以触发RDB)
save 900 1
save 300 10
save 60 10000
#bgsave创建快照的时候出错了（比如，fork子进程内存不足或rdb文件所在的文件夹没有写入权限），redis主进程将不再接受新的写入命令
stop-writes-on-bgsave-error yes
#对RDB文件进行压缩
rdbcompression yes 
#RDB文件名称
dbfilename dump.rdb
#对RDB进行校验
rdbchecksum yes
 #（保存目录，这里是相对目录，和redis.conf是同一个目录）
dir ./

1.3 RDB persistence steps

Execute the bgsave command, the Redis parent process judges whether there are currently executing child processes, such as RDB/AOF child processes, if there is a bgsave command, it returns directly;
The parent process executes the fork operation to create a child process. The parent process will be blocked during the fork operation. You can view the latest_fork_usec option through the info stats command to get the time of the last fork operation in microseconds;
After the parent process fork is completed, the bgsave command returns the "Background saving started" message and no longer blocks the parent process, and can continue to respond to other commands;
The child process creates an RDB file, generates a temporary snapshot file based on the memory of the parent process, and atomically replaces the original file after completion. Execute the lastsave command to get the time of the last RDB generation, corresponding to the rdb_last_save_time option of info statistics.
The process sends a signal to the parent process to indicate completion, and the parent process updates statistics. For details, see the rdb_* related options under info Persistence.

Note : fork子进程时，需要生成另外一份内存，如果原来内存占用比较大，会存在内存膨胀的问题。
For the wrong format RDB file, you can use a redis-check-rdbtool to repair it.

2. AOF persistence

2.1 AOF related configuration in redis.conf

#开启AOF，默认关闭
appendonly yes
#AOF文件名称
appendfilename appendonly.aof
 #（保存目录，这里是相对目录，和redis.conf是同一个目录）
dir ./ 

#AOF文件写入磁盘频率
appendfsync always  #每次收到写命令就立即强制写入磁盘，是最安全的。但该模式下速度也是最慢的，一般不推荐使用。
appendfsync everysec #默认方式，每秒钟强制写入磁盘一次，在性能和持久化方面做平衡，推荐该方式。
appendfsync no  #完全依赖操作系统的写入，一般为30秒左右一次，性能最好但是持久化最没有保证，不推荐。

/*AOF重写相关命令*/
#在日志重写时，不进行命令追加操作，而只是将其放在缓冲区里，避免与命令的追加造成DISK IO上的冲突。设置为yes表示rewrite期间对新写操作不fsync,暂时存在内存中,等rewrite完成后再写入，默认为no，建议yes
no-appendfsync-on-rewrite yes
#AOF重写触发条件
auto-aof-rewrite-percentage 100 #当前AOF文件大小是上次日志重写得到AOF文件大小的二倍时，自动启动新的日志重写过程。
auto-aof-rewrite-min-size 64mb #当前AOF文件启动新的日志重写过程的最小值，避免刚刚启动Reids时由于文件尺寸较小导致频繁的重写。

2.2 AOF persistence steps

The implementation of AOF persistence can be divided into three steps: command appending, file writing, and file synchronization.

Name append: After the server has executed a write command, it will append the executed write command to the end of the aof_buf buffer in the server state in the protocol format;
File writing and synchronization: The server regularly checks whether the content in the aof_buf buffer area needs to be written and saved to the AOF file. Whether file writing is required is set according to the appendfsync option. You can set the appendfsync option value based on the efficiency and safety of AOF persistence.

Note : For the AOF file in the wrong format, first make a backup, and then use the redis-check-aof --fixcommand to repair it. After the repair, use diff-u to compare the data differences to find out the missing data, and some can be manually modified and completed.

2.3 AOF rewrite

2.3.1 Why is AOF rewritten?

When the write command is appended to the AOF file, the AOF will become larger and larger, occupying too much hard disk space, and the data loading after restarting will be very slow, so AOF rewriting is required.

2.3.2 AOF rewrite trigger mechanism

There are two ways to trigger AOF rewriting: manual trigger and automatic trigger.

Manual trigger: manually execute the command BGREWRITEAOF;
Auto: by configuring redis.conf in auto-aof-rewrite-percentageand auto-aof-rewrite-min-sizeto realize that these two conditions are 同时满足triggered when the AOF.

2.3.3 AOF rewrite rules

AOF rewrite does not read and analyze the existing AOF file, it is achieved by reading the current database status of the server.
Therefore, expired data will not be written to the new AOF file, and multiple commands for the same key will be replaced by one command because of directly reading the database status. In addition, in order to avoid the client input buffer overflow caused by too much data contained in a key, the rewrite program will first determine the key contains when processing keys that may contain multiple elements such as lists, sets, ordered sets, and hashes. If the number of elements exceeds the configured constant value (the default is 64), it will be split into multiple commands.

2.3.4 AOF rewrite steps

Execute AOF rewrite request. If the current process is performing AOF rewriting, the rewriting operation is not performed. If the process is executing bgsave, it will wait for the execution to complete before executing.
The parent process executes fork to create a child process, and the overhead is equivalent to the bgsave process.
(1) After the fork operation of the main process is completed, continue to respond to other commands. All modification commands are still written into the AOF buffer and synchronized to the hard disk according to the appendfsync strategy to ensure the correctness of the original AOF mechanism.
(2) Since the fork operation uses copy-on-write technology, the child process can only share the memory data during the fork operation. Since the parent process still responds to commands, Redis uses the "AOF rewrite buffer" to save this part of the new data to prevent this part of data from being lost during the generation of the new AOF file.
The child process writes to the new AOF file according to the memory snapshot and the command merging rules. The amount of data written to the hard disk in batches each time is controlled by the configuration aof-rewrite-incremental-fsync, the default is 32MB, to prevent the hard disk from being blocked by too much data in a single flash.
(1) After the new AOF file is written, the child process sends a signal to the parent process, and the parent process updates statistical information. For details, see aof_* related statistics under info persistence.
(2) The parent process writes the data in the AOF rewrite buffer to the new AOF file.
(3) Replace the old file with the new AOF file to complete the AOF rewrite.

3. Comparison of RDB and AOF

The performance of RDB is higher than that of AOF:
● The data is stored in binary mode, the data file is relatively small, and the loading is fast;
● When storing, it is stored in accordance with the save strategy in the configuration, each time a lot of data is aggregated and stored in batches, and the writing efficiency is very Good, and AOF generally works in real-time storage or quasi-real-time mode. Relatively speaking, the storage frequency is high, but the efficiency is low.
The security of AOF data is higher than that of RDB:
● Storage is based on the idea of cumulative batches, that is to say, if allowed, the more accumulated data, the higher the writing efficiency, but the accumulation of data is completed by the accumulation of time Yes, if the data is not written to RDB for a long time, but Redis encounters a crash, then the data that is not written cannot be recovered, but the AOF method is on the contrary, according to the strategy of storage frequency configured by AOF, the least can be achieved Data loss and high data recovery capability.
The choice between the two is a trade-off between performance and data security.

3. References

https://zhuanlan.zhihu.com/p/111306444
"Redis Design and Implementation"