Redis of NoSQL (3) (RDB persistence and AOF persistence and their advantages and disadvantages)

---

1. RDB endurance

• RDB persistence refers to the generation of a snapshot of the data in the current process in the memory and saves it to the hard disk (so it is also called snapshot persistence) within a specified time interval, using binary compression for storage, and the saved file suffix is ​​rdb
• When Redis restarts, you can read the snapshot file to restore the data

1. Trigger conditions

RDB persistence triggers are divided into manual triggers and automatic triggers

1.1 Manual trigger

• Both the save command and the bgsave command can generate RDB files
• The save command will block the Redis server process until the RDB file is created. During the blocking period of the Redis server, the server cannot process any command requests
• The bgsave command will create a child process, the child process is responsible for creating the RDB file, the parent process (that is, the Redis main process) will continue to process the request
• During the execution of the bgsave command, only the fork child process will block the server. For the save command, the whole process will block the server. Therefore, save has been basically abandoned, and the use of save must be eliminated in the online environment.

1.2 Automatic trigger

• When RDB persistence is automatically triggered, Redis will also choose bgsave instead of save for persistence

save m n
#自动触发最常见的情况是在配置文件中通过 savemn，指定当 m 秒内发生 n 次变化时，会触发 bgsave

vim /etc/redis/6379.conf
#219行，以下三个save条件满足任意一个时，都会引起bgsave的调用
save 900 1      :当时间到900秒时，如果redis数据发生了至少1次变化，则执行bgsave
save 300 10     :当时间到300秒时， 如果redis数据发生了至少10次变化，则执行bgsave
save 60 10000   :当时间到60秒时，如果redis数据发生了至少10000次变化， 则执行bgsave
#254行，指定 RDB 文件名
dbfilename dump.rdb
#264行，指定 RDB 文件和 AOF 文件所在目录
dir /var/lib/redis/6379
#242行，是否开启 RDB 文件压缩
rdbcompression yes

In addition to savemn, there are other automatic trigger mechanisms that will trigger bgsave:

• In the master-slave replication scenario, if the slave node performs a full copy operation, the master node will execute the bgsave command and send the rdb file to the slave node
• When the shutdown command is executed, the rdb persistence is automatically executed

2. Execution process

1. The Redis parent process first judges whether it is currently executing save or the child process of bgsave/bgrewriteaof. If it is executing, the bgsave command returns directly, and the child processes of bgsave/bgrewriteaof cannot be executed at the same time; mainly based on performance considerations: two Concurrent child processes perform a large number of disk write operations at the same time, which may cause serious performance problems
2. The parent process executes a fork operation to create a child process. During this process, the parent process is blocked, and Redis cannot execute any commands from the client
3. After the parent process forks, the bgsave command returns the "Background saving started" message and no longer blocks the parent process, and can respond to other commands
4. The child process creates an RDB file, generates a temporary snapshot file based on the memory snapshot of the parent process, and atomically replaces the original file after completion
5. The child process sends a signal to the parent process to indicate completion, and the parent process updates statistics

3. Load at startup

• The loading of RDB files is automatically executed when the server starts, and there is no special command
• But because AOF has a higher priority, when AOF is turned on, Redis will load AOF files first to restore data
• Only when AOF is closed, the RDB file will be detected when the Redis server is started and automatically loaded
• The server is blocked while loading the RDB file until the loading is complete
• When Redis loads the RDB file, it will verify the RDB file. If the file is damaged, an error will be printed in the log (Redis startup failed)

---

2. AOF persistence

• RDB persistence is to write process data to a file, while AOF persistence is to record every write and delete command executed by Redis in a separate log file, and query operations will not be recorded
• When Redis restarts, execute the commands in the AOF file again to restore the data
• Compared with RDB, AOF has better real-time performance, so it has become the mainstream persistence solution

1. Turn on AOF

• The Redis server turns on RDB by default and turns off AOF
• To enable AOF, you need to configure in the configuration file:

vim /etc/redis/6379.conf
#700行修改， 开启AOF
appendonly yes
#704行,指定AOF文件名称
appendfilename "appendonly.aof"
#796行,是否忽略最后一条可能存在问题的指令
aof-load-truncated yes

/etc/init.d/redis_6379 restart
#重启以使配置生效，开启 AOF

2. Execution process

• Since each write command of Redis needs to be recorded, AOF does not need to be triggered. The following describes the execution process of AOF
• The implementation process of AOF includes:
  • Command append (append): append Redis write commands to the buffer aof_buf;
  • File writing (write) and file synchronization (sync): Synchronize the content in aof_buf to the hard disk according to different synchronization strategies;
  • File rewrite (rewrite): Periodically rewrite AOF files to achieve compression

2.1 Command Append (append)

• Redis first appends the write command to the buffer instead of directly writing to the file, mainly to avoid writing directly to the hard disk every time a write command is issued, causing hard disk IO to become the bottleneck of the Redis load
• The format of the command addition is the protocol format requested by the Redis command. It is a plain text format, which has the advantages of good compatibility, strong readability, easy processing, simple operation and avoiding secondary overhead.
• In the AOF file, except for the select command used to specify the database (for example, select0 to select the database No. 0) is added by Redis, the rest are write commands sent by the client

2.2 File writing (write) and file synchronization (sync)

2.2.1 Redis provides a variety of AOF cache synchronization file strategies

The strategy involves the write function and fsync function of the operating system, which are described as follows:

• In order to improve the efficiency of file writing, in modern operating systems, when the user calls the write function to write data to the file, the operating system usually temporarily stores the data in a memory buffer. When the buffer is full or exceeds the specified value After the time limit, the data in the buffer is actually written to the hard disk
• Although this operation improves efficiency, it also brings security issues: if the computer is shut down, the data in the memory buffer will be lost
• Therefore, the system also provides synchronization functions such as fsync and fdatasync, which can force the operating system to write the data in the buffer to the hard disk immediately, thereby ensuring data security

2.2.2 There are three synchronization methods for the synchronization file strategy of the AOF cache area

they are, respectively:

vim /etc/redis/6379.conf
#729行
appendfsync always：
#命令写入 aof_ buf 后立即调用系统 fsync 操作同步到 AOF 文件，fsync 完成后线程返回
#这种情况下，每次有写命令都要同步到 AOF 文件，硬盘 IO 成为性能瓶颈，Redis只能支持大约几百TPS写入，严重降低了 Redis 的性能
#即便是使用固态硬盘(SSD)，每秒大约也只能处理几万个命令，而且会大大降低 SSD 的寿命

appendfsync no：
#命令写入 aof_ buf 后调用系统 write 操作，不对 AOF 文件做 fsync 同步
#同步由操作系统负责，通常同步周期为 30 秒
#这种情况下，文件同步的时间不可控，且缓冲区中堆积的数据会很多，数据安全性无法保证

appendfsynceverysec：
#命令写入 aof_ buf 后调用系统 write 操作，write 完成后线程返回
#fsync 同步文件操作由专门的线程每秒调用一次
#everysec 是前述两种策略的折中，是性能和数据安全性的平衡，因此是 Redis 的默认配置，也是我们推荐的配置

2.3 File rewrite (rewrite)

2.3.1 Overview

• As time goes by, the Redis server executes more and more write commands, and the AOF file will become larger and larger. An overly large AOF file will not only affect the normal operation of the server, but will also take too long for data recovery.
• File rewriting is to rewrite the AOF file in a specified period to reduce the volume of the AOF file. It should be noted that the AOF rewrite is to convert the data in the Redis process into write commands and synchronize to the new AOF file without affecting the old AOF file for any read or write operation!
• Another point to note about file rewriting is: for AOF persistence, although file rewriting is strongly recommended, it is not necessary
• Even if there is no file rewriting, the data can be persisted and imported when Redis starts. Therefore, in some implementations, automatic file rewriting will be turned off and then executed at a certain time every day through a scheduled task

2.3.2 Reasons why file rewriting can compress AOF files

• Expired data is no longer written to the file
• Invalid commands are no longer written to the file: For example, some data is set repeatedly (set mykey v1, set mykey v2), some data is deleted (sadd myset v1, del myset), etc.
• Multiple commands can be combined into one: such as sadd myset v1, sadd myset v2, sadd myset v3 can be combined into sad myset v1 v2 v3
• As can be seen from the above content, since the commands executed by AOF are reduced after rewriting, file rewriting can not only reduce the space occupied by the file, but also speed up the recovery speed.

2.3.3 File rewriting trigger

Divided into manual trigger and automatic trigger

Manual trigger:

• Invoke the bgrewriteaof command directly, the execution of the command is somewhat similar to bgsave
• All fork child processes perform specific work, and they are only blocked when fork

Automatic trigger:

• Automatically execute BGREWRITEAOF by setting the auto-aof-rewrite-min-size option and auto-aof-rewrite-percentage option
• Only when the two options of auto-aof-rewrite-min-size and auto-aof-rewrite-percentage are met at the same time, will the AOF rewrite be automatically triggered, that is, the bgrewriteaof operation

vim /etc/redis/ 6379. conf
#771行
auto-aof- rewrite-percentage 100
#当前 AOF 文件大小(即 aof_current_size)是上次日志重写时AOF文件大小(aof_base_size)两倍时，发生 BGREWRITEAOF操作
auto-aof -rewrite-min-size 64mb
#当前 AOF文件执行 BGREWRITEAOF 命令的最小值
#避免刚开始启动 Reids 时由于文件尺寸较小导致频繁的 BGREWRITEAOF

2.3.4 Two points that need special attention in the process of file rewriting

• Rewriting is performed by the parent process fork and the child process
• The write commands executed by Redis during the rewriting process need to be appended to the new AOF file. For this reason, Redis introduces the aof_rewrite_buf cache

2.3.5 Process of file rewriting

(1): The Redis parent process first judges whether there is a child process currently executing bgsave/bgrewriteaof, if it exists, the bgrewriteaof command returns directly, if there is a bgsave command, it waits for bgsave execution to complete before executing

(2): The parent process executes a fork operation to create a child process, the parent process is blocked in this process

(3.1): After the parent process forks, the bgrewriteaof command returns the "Background append only file rewrite started" message and no longer blocks the parent process, and can respond to other commands; all Redis write commands are still written to the AOF buffer, and follow the appendfsync strategy Synchronize to the hard disk to ensure that the original AOF mechanism is correct
(3.2): Since the fork operation uses copy-on-write technology, the child process can only share the memory data during the fork operation; because the parent process is still responding to commands, Redis uses AOF to rewrite The buffer (aof_rewrite_buf) saves this part of the data to prevent loss of this part of the data during the generation of the new AOF file; that is, during the execution of bgrewriteaof, the Redis write command is simultaneously appended to the aof_buf and aof_rewirte_buf buffers

(4): According to the memory snapshot, the child process writes to the new AOF file according to the command merging rules

(5.1): After the child process finishes writing the new AOF file, it sends a signal to the parent process, and the parent process updates the statistical information, which can be viewed through info persistence.
(5.2): The parent process writes the data in the AOF rewrite buffer to the new AOF file, which ensures that the database state saved in the new AOF file is consistent with the current state of the server
(5.3): Replace the old file with the new AOF file to complete the AOF rewrite

3. Load at startup

• When AOF is turned on, Redis will load AOF files first to restore data when starting
• Only when AOF is closed, the RDB file will be loaded to restore the data
• When AOF is turned on, but the AOF file does not exist, it will not be loaded even if the RDB file exists
• When Redis loads the AOF file, it will verify the AOF file. If the file is damaged, an error will be printed in the log and Redis will fail to start
• But if the end of the AOF file is incomplete (sudden machine downtime, etc.), and the aof-load-truncated parameter is enabled, a warning will be output in the log, Redis ignores the end of the AOF file, and the startup is successful
• The aof-load-truncated parameter is enabled by default

---

Three, the advantages and disadvantages of RDB and AOF

1. RDB endurance

advantage:

• RDB file is compact, small in size, fast network transmission, suitable for full copy
• Recovery speed is much faster than AOF
• Of course, compared with AOF, one of the most important advantages of RDB is that the impact on performance is relatively small

Disadvantages:

• The fatal disadvantage of RDB files is that the persistence of data snapshots determines that real-time persistence cannot be achieved. Today, when data is becoming more and more important, a large amount of data loss is often unacceptable, so AOF persistence has become the mainstream
• In addition, RDB files need to meet a specific format, which has poor compatibility (for example, the old version of Redis is not compatible with the new version of RDB files)
• For RDB persistence, on the one hand, the main Redis process will be blocked when bgsave forks. On the other hand, writing data to the hard disk by the child process will also bring IO pressure.

2. AOF persistence

• Corresponding to RDB persistence, the advantage of AOF lies in supporting second-level persistence and good compatibility
• The disadvantage is that the file size is large, the recovery speed is slow, and the performance impact is large
• For AOF persistence, the frequency of writing data to the hard disk is greatly increased (second level under the everysec strategy), IO pressure is greater, and may even cause AOF additional blocking problems
• The rewriting of the AOF file is similar to the bgsave of RDB, there will be blocking when fork and the IO pressure of the child process.
• Relatively speaking, because AOF writes data to the hard disk more frequently, it will have a greater impact on the performance of the Redis main process.