In order to solve the data security problem caused by single point of failure and improve the high concurrency of read data operations, Redis implements a master-slave master-slave synchronization mechanism, that is, deploying a master node to be responsible for reading and writing operations and synchronizing data with slave nodes Deploy multiple slave nodes to be responsible for reading operations and receiving synchronization operation commands from the master node.
Master-slave synchronization configuration:
The configuration of the master-slave synchronization mechanism is relatively simple: configure the slaveof masterip masterport option in the redis.conf configuration file of the slave node or directly use the slaveof masterip masterport command (after the Redis 5.0 version, use the replicaof command), we can also configure the slave The read-only nature of the node: slave-read-only yes.
It should be noted that in the master-slave synchronization mode, even if the master node is down, the slave node will not become the master node for write operations. It must cooperate with the sentry mechanism or cluster mode to achieve high availability . Master-slave synchronization is only highly available. The basics.
Advantages of master-slave synchronization:
- Achieve hot backup of data in a redundant manner of multiple data copies.
- Avoid data security problems with single points of failure.
- Separation of read and write, load balancing, improve performance and concurrency.
- The slave node belongs to only one master node, and data flows in one direction.
Master-slave synchronization strategy:
1. Full synchronization: also called snapshot synchronization, which generally occurs during the initialization phase of the slave node, or the slave node is disconnected from the master node. After reconnecting, it may be necessary to copy all the data of the master node.
- The slave sends a sync command to the master.
- The master executes the bgsave command to generate the RDB file, and writes the subsequent commands to the cache area.
- After bgsave is executed, the master sends the rdb file to the slave, and the slave loads the rdb file.
- The master sends the commands in the buffer to the slave for execution.
We can see that the overhead of full synchronization is very large. Master generates RDB files, sends RDB files to slaves, and Slave loads RDB files will consume a lot of resources. Therefore, do not execute the sync command unless it is necessary or necessary.
2. Command propagation: After the full synchronization is completed, the state of the slave node and the master node are consistent, but when the master node receives the write command, the state becomes inconsistent, and the master node needs to send the command that causes the inconsistency to the slave node Go to execute and restore the status to consistency.
3. Incremental synchronization: Before the 2.8 version, after the slave node and the master node were disconnected and reconnected, full synchronization was required. This approach is very inefficient. To solve this problem, Redis 2.8 version implements the psync command, psync The command has two modes:
- Full incremental synchronization (full resynchronization): The process is basically the same as full synchronization, which is mainly used to complete the initial data synchronization of the slave node.
- Partial resynchronization (partial resynchronization): mainly used when slave and master are disconnected and reconnected, the master sends the write command during the disconnection to the slave node for execution, keeping the state consistent.
(1) Components:
- Copy offset offset: A copy offset is maintained on the master and the slave, and the master and slave states can be known by comparing the offsets.
- Copy buffer: A fixed-length FIFO queue maintained by the master to store write commands. Because the length is fixed, only the most recent commands can be stored. The commands that are enqueued later will pop the commands that are enqueued first.
- Instance ID: Both master and slave have a running ID.
(2) Implementation details:
- When the master propagates the command, in addition to sending the write command to the slave, it also puts the write command into the copy buffer.
- Each time the master executes a write command, it sets its own offset + N, N is the number of bytes occupied by the write command, and the copy buffer will record the corresponding offset for each byte on the queue.
- Each time a slave receives a write command, it also sets its own offset + N, where N is the number of bytes in the write command. If the offsets on master and slave are the same, it means that the two states are consistent, otherwise, they are inconsistent.
- When the slave is disconnected and reconnected, send the psync command to report its current offset to the master. The master determines whether the reported offset is still in the copy buffer. If it does, it initiates a partial incremental synchronization and sends the write command after the offset. However, if it is not, it means that the command is lost and a complete incremental synchronization is initiated.
- The instance ID is generated when the node is started, and is composed of 40 random hexadecimals. When the slave first copies data from the master, the master sends its ID to the slave, which is maintained by the slave. When the slave is disconnected and reconnected, it will be The ID saved by yourself is sent to the reconnected master. The master judges whether the ID is consistent with its own ID. If they match, it means that the master has not changed. Try partial incremental synchronization. If they do not match, it means that the master has changed and requires full incremental synchronization.
(3) The size of the copy buffer:
- The size of the copy buffer is 1M by default. When the master executes a large number of write commands or the slave is disconnected for a long time, the queue will overflow, resulting in invalid incremental synchronization. Therefore, it is necessary to predict and set the size of the copy buffer to increase The amount is synchronized.
Master-slave synchronization is the basis for Redis to achieve high availability. Understanding the principles and methods of master-slave synchronization will be of great help in learning Redis high availability.
PS: If you have any questions or questions, please leave me a message.
Friends who like this article, please pay attention to the public number, and receive the updated content as soon as possible.
