5 minutes to understand the internal implementation of Redis (quicklist)

Introduction to Quick Lists

Before Redis 3.2, the storage list (list) data structure used compressed list (ziplist) and linked list (linkedlist), when the number of list elements is relatively small and each element occupies a small space, use compressed list. When the number of list elements is large or an element occupies a large space, a linked list is used.

Considering that the additional space of the linked list is relatively high, the memory of the node is also allocated separately, which affects the efficiency of memory management. In the Redis3.2 version, the list data structure has been transformed, using a quick list (quicklist) instead of a compressed list (ziplist) and a linked list (linkedlist).

The quicklist is a linked list with the compressed list (ziplist) as the node. The linked list is divided into segments. Each segment uses the compressed list for continuous memory storage. Multiple compressed lists are bidirectionally composed of prev and next pointers. linked list. It combines the advantages of compressed lists and linked lists, further compressing memory usage and further improving efficiency.

Let's take a look at the specific implementation of the quick list.

Implementation of a quick list

A quicklist in Redis is quicklistrepresented by a structure that quicklistcontains a doubly linked list of multiple quicklist nodes, each of which holds a compressed list. Let's take a look at each one in detail.

quicklist structure

A quick list is represented by a quickliststructure, which contains the following properties:

• headAttributes: Pointer to the head quicklist node.
• tailAttributes: Pointer to the trailing quicklist node.
• countAttribute: The sum of the number of elements in all compressed lists.
• lenAttribute: The number of quicklist nodes.
• fillAttribute: The maximum size of the compressed list, which stores list-max-ziplist-sizethe value of the parameter. When this configuration is exceeded, a new compression list will be created.
• compressAttribute: node compression depth, storing list-compress-depththe value of the parameter.
• bookmarksAttributes: Arrays used for quick list reallocation of memory space, do not occupy space when not in use.
• bookmark_countAttribute: The size of the bookmarks array.

quick list node

A quick list node is quicklistNoderepresented by a structure, which contains the following properties:

• prevAttributes: Pointer to the previous quicklist node.
• nextAttributes: Pointer to the next quicklist node.
• zlAttribute: A pointer to a compressed list, which points to a quicklistLZFstructure if the current node's data is compressed.
• szAttribute: The total number of bytes occupied by the compressed list.
• countProperties: The number of elements in the compressed list.
• encodingAttribute: storage form, native byte array or LZF compressed storage.
• recompressAttribute: When viewing a certain compressed data, you need to decompress the data temporarily, then set recompress = 1 to make a mark, and recompress the data when there is a chance.

quicklistLZF structure

When the quicklist node data is compressed, the data will be stored in the quicklistLZF structure, which contains the following attributes:

• szAttribute: Indicates the compressed size.
• compressedAttribute: Store the compressed byte array.

Compression mechanism for quick lists

In the quick list, the possibility of accessing the data of the two end nodes is relatively high, and the possibility of accessing the data of the intermediate node is relatively low. If our application scenario conforms to this feature, the data of the intermediate nodes can be compressed using the LZF algorithm, thereby further saving memory space. We can list-compress-depthconfigure parameters.

By default, the list-compress-depthparameter is 0, that is, the data is not compressed; when the parameter is set to 1, all nodes except the head and tail will be compressed; when the parameter is set to 2, except for the head, Nodes other than the next head, tail, and the previous one of the tail will be compressed; when this parameter is set to 2, all nodes except the head, the next one of the head, the next one of the next head, the tail , the previous node of the tail, and the nodes other than the previous one of the tail will be compressed; and so on.

Finally, thank you for being so handsome, and for giving me likes and attention .

{{o.name}}

{{m.name}}