Known as Redis Plus, let's see how explosive the performance of KeyDB is!

Source: developer.aliyun.com/article/705239

The KeyDB project is a branch from redis fork. As we all know, redis is a single-threaded kv memory storage system, and KeyDB transforms redis into multi-threading when it is 100% compatible with redis API.

Project address: https://github.com/JohnSully/KeyDB

There are relatively few technical details published on the Internet. This article is basically summarized by reading the source code. Please correct me if there are any mistakes or omissions.

multi-threaded architecture

threading model

KeyDB splits the original main thread of redis into main thread and worker thread. Each worker thread is an io thread, responsible for listening to ports, accepting requests, reading data and parsing protocols. as the picture shows:

KeyDB uses the SO_REUSEPORT feature, and multiple threads can bind and listen to the same port.

Each worker thread binds the core to the cpu, reads the data and uses SO_INCOMING_CPUthe feature, and specifies the cpu to receive the data.

After parsing the protocol, each thread will operate the data in memory, and a global lock is used to control multi-thread access to memory data.

The main thread is actually a worker thread, including the work content of the worker thread, and also includes the work content that only the main thread can complete. The subscript 0 in the worker thread array is the main thread.

The main work of the main thread is to implement serverCron, including:

Processing statistics
Client Link Management
Resize and reshard of db data
deal with aof
replication master and backup synchronization
Tasks in cluster mode

link management

All connection management in redis is done in one thread. In the design of KeyDB, each worker thread is responsible for a set of links, and all links are inserted into the link list of this thread for maintenance. The generation, work, and destruction of links must be in the same thread. Add a new field for each link

int iel; /* the event loop index we're registered with */

Used to indicate which thread the link belongs to take over.

KeyDB maintains three key data structures for link management:

clients_pending_write: A thread-specific linked list, maintaining a queue for synchronously sending data to client links
clients_pending_asyncwrite: A thread-specific linked list that maintains a queue that asynchronously sends data to client links
clients_to_close: Global linked list, maintaining client links that need to be closed asynchronously

It is divided into two queues, synchronous and asynchronous, because redis has some linkage APIs, such as pub/sub. After pub, it needs to send messages to the client of sub. The thread executed by pub and the thread of the client of sub are not the same thread. In order to process this In this case, KeyDB will maintain the data sent to clients other than this thread in the asynchronous queue. The logic of synchronous sending is relatively simple, and it is all completed in this thread. The following figure illustrates how to send data to the client synchronously:

As mentioned above, the creation of a connection, receiving data, sending data, and releasing the connection must all be executed in the same thread. An asynchronous send involves an interaction between two threads. KeyDB pipes messages in two threads:

int fdCmdWrite; //写管道
int fdCmdRead; //读管道

AE_ASYNC_OP::CreateFileEventWhen the local thread needs to send data asynchronously, first check whether the client belongs to the local thread, and the non-local thread obtains the client-specific thread ID, and then manages the sending operation for the dedicated thread, and requests to add a write socket event. The dedicated thread adds the corresponding request to the write event when processing the pipeline message, as shown in the figure:

Some redis requests to close the client are not completely closed on the thread where the link is located, so a global asynchronous close list is maintained here.

lock mechanism

KeyDB implements a lock mechanism similar to spinlock, called fastlock.

The main data structures of fastlock are:

struct ticket
{
    uint16_t m_active;  //解锁+1
    uint16_t m_avail;  //加锁+1
};
struct fastlock
{
    volatile struct ticket m_ticket;

    volatile int m_pidOwner; //当前解锁的线程id
    volatile int m_depth; //当前线程重复加锁的次数
};

Use the atomic operation __atomic_load_2, __atomic_fetch_add, __atomic_compare_exchangeto determine whether the lock can be acquired by comparison m_active=m_avail.

Fastlock provides two ways to acquire locks:

try_lock: An acquisition fails and returns directly
lock: Busy waiting 1024 * 1024, use sched_yield the initiative to hand over the cpu after each busy waiting, and move to the end of the task of the cpu to wait for execution.

In KeyDB try_lock, it will be combined with events to avoid busy waiting. Each client has an exclusive lock. Before reading the client data, it will try to lock it first. If it fails, it will exit. Because the data has not been read, it can be processed again in the next processing event loop epoll_wait.

Active-Replica

KeyDB implements a multi-active mechanism. Each replica can be set to be writable but not read-only, and the replicas can synchronize data with each other. The main features are:

Each replica has a uuid flag to remove ring replication
Newly added rreplay API, package incremental command into rreplay command, bring local uuid
The key and value plus the timestamp version number are used as a conflict check. If the same key exists locally and the timestamp version number is greater than the synchronized data, the new write will fail. The current timestamp is shifted 20 bits to the left, and the last 44 bits are incremented to obtain the timestamp version number of the key.

------

We have created a high-quality technical exchange group. When you are with excellent people, you will become excellent yourself. Hurry up and click to join the group and enjoy the joy of growing together. In addition, if you want to change jobs recently, I spent 2 weeks a year ago collecting a wave of face-to-face experience from big factories. If you plan to change jobs after the festival, you can click here to claim it !