Introduction
In redis version 3.0 and before, compressed linked lists (ziplist) and doubly linked lists (linkedlist) were used as the underlying implementation of list. Use ziplist when there are few elements, use linkedlist when there are many elements
After redis version 3.0, quicklist is used as the underlying implementation of list
quicklist is a doubly linked list, each node in the linked list is a ziplist
If a quicklist contains 4 quickListNodes, and the ziplist of each node contains 3 elements, there are 12 values stored in the list.
Why is quicklist designed like this? Probably based on a compromise between space and efficiency
- Doubly linked lists facilitate push and pop operations at both ends of the table. But the memory overhead is relatively large, in addition to saving data, but also to save the pointers of the nodes before and after. And each node is a separate memory block, easy to cause memory fragmentation
- The ziplist is a contiguous memory, no pointers to the preceding and following items are needed to save memory. But when the modification operation is performed, cascading updates will occur, reducing performance
So the quicklist that combines the advantages of the two was born, but this will bring new problems, how many elements are appropriate for each ziplist?
- The shorter the ziplist, the more memory fragmentation will affect storage efficiency. When a ziplist contains only one element, the quicklist degenerates into a doubly linked list
- The longer the ziplist, the more difficult it is to allocate large continuous memory space for the ziplist, which will cause a lot of small pieces of memory space to be wasted. When the quicklist has only one node and all elements exist on the same ziplist, the quicklist degenerates into a ziplist again
We can control the node elements that ziplist can store through the following parameters in the redis.conf configuration file
list-max-ziplist-size -2
- When the value is positive, it indicates the length of the ziplist on the quicklistNode node. For example, when the value is 5, the ziplist of each quicklistNode node contains at most 5 data items
- When the value is negative, it means that the length of the ziplist on the quicklistNode node is limited according to the number of bytes. The optional values are -1 to -5. The meaning of each value is as follows
| value | meaning |
|---|---|
| -1 | The maximum size of ziplist node is 4kb |
| -2 | The maximum size of ziplist node is 8kb |
| -3 | The maximum size of the ziplist node is 16kb |
| -4 | The maximum size of the ziplist node is 32kb |
| -5 | The maximum size of ziplist node is 64kb |
Because the two ends of the list are frequently accessed data, while the data in the middle is rarely accessed. Based on this feature, redis provides the following configuration items for the list, which can compress the data in the node to further save memory
list-compress-depth 0
The meaning of the value is as follows
| value | meaning |
|---|---|
| 0 | Special value, which means no compression |
| 1 | There are 1 nodes at each end of the quicklist that are not compressed, and the middle node is compressed |
| 2 | There are 2 nodes at each end of the quicklist that are not compressed, and the middle node is compressed |
| n | There are n nodes at both ends of the quicklist that are not compressed, and the nodes in the middle are compressed |
This parameter represents the number of nodes that are not compressed at both ends of the quicklist (ie the number of quicklistNodes)
Quicklist data structure definition
typedef struct quicklist {
// 指向quicklist的首节点
quicklistNode *head;
// 指向quicklist的尾节点
quicklistNode *tail;
// quicklist中元素总数
unsigned long count; /* total count of all entries in all ziplists */
// quicklistNode节点个数
unsigned long len; /* number of quicklistNodes */
// ziplist大小设置，存放list-max-ziplist-size参数的值
int fill : 16; /* fill factor for individual nodes */
// 节点压缩深度设置，存放list-compress-depth参数的值
unsigned int compress : 16; /* depth of end nodes not to compress;0=off */
} quicklist;
quicklistNode is defined as follows
typedef struct quicklistNode {
struct quicklistNode *prev;
struct quicklistNode *next;
unsigned char *zl;
unsigned int sz; /* ziplist size in bytes */
unsigned int count : 16; /* count of items in ziplist */
unsigned int encoding : 2; /* RAW==1 or LZF==2 */
unsigned int container : 2; /* NONE==1 or ZIPLIST==2 */
unsigned int recompress : 1; /* was this node previous compressed? */
unsigned int attempted_compress : 1; /* node can't compress; too small */
unsigned int extra : 10; /* more bits to steal for future usage */
} quicklistNode;
prev: pointer to the previous node
next: pointer to the next node
zl: data pointer. If the data of the current node is not compressed, then it points to a ziplist structure; otherwise, it points to a quicklistLZF structure
sz: the total size of the ziplist pointed to by zl (including zlbytes, zltail, zllen, zlend and various data items). It should be noted that if the ziplist is compressed, then the value of this sz is still the size of the ziplist before compression.
count: the number of elements contained in the
ziplist encoding: whether the ziplist is compressed, 1 is not compressed, 2 is compressed with lzf
container : Is a reserved field. The original design is to indicate whether a quicklist node is to store data directly, or use ziplist to store data, or use other structures to store data (used as a data container, so it is called a container). However, in the current implementation, this value is a fixed value of 2, which means that ziplist is used as the data container to
recompress: When we use commands like lindex to view a certain item of originally compressed data, we need to temporarily decompress the data. At this time, set recompress=1 to make a mark, and then recompress the data when there is a chance.
attempted_compress: This value is only useful for Redis automated test programs.
Extra: Extended field, currently not used
Reference blog
[1]http://zhangtielei.com/posts/blog-redis-quicklist.html