You tend to

1 Introduction

Tendis storage version is an open source distributed high-performance KV storage independently designed and developed by Tencent Interactive Entertainment's CROS DBA team & Tencent Cloud database team. Fully compatible with the redis protocol and uses rocksdb as the storage engine. Users can access the Tendis storage version through redis client with almost no need to modify the code. At the same time, Tendis storage version supports disk capacity that far exceeds memory, which can greatly reduce users' storage costs.
GitHub: https://github.com/Tencent/Tendis
Official documentation: http://tendis.cn/#/Download
address: https://github.com/Tencent/Tendis/releases/tag/2.4.3-rocksdb -v5.13.4

2.Characteristics

1.Redis compatibility: fully compatible with the redis protocol and supports most redis instructions
2. Persistent storage: Use rocksdb as the storage engine. All data is stored in rocksdb in a specific format, supporting up to PB-level storage.
3. Decentralized architecture: Similar to the distributed implementation of redis cluster, all nodes communicate through gossip protocol, and hashtags can be specified to control data distribution and access. The usage and operation and maintenance costs are extremely low.
4. Horizontal expansion: The cluster supports adding and deleting nodes, and data can be migrated between any two nodes according to the slot. The expansion and contraction process is transparent to application operation and maintenance personnel, and supports expansion to 1,000 nodes.
5. Fault high availability: Automatically detect faulty nodes. When a fault occurs, the slave will be promoted to master to continue external services.

3.Performance report

Official performance report, stand-alone hardware configuration

CPU:2.50 GHz,48 core
DISK:NVMe SSD
MEM:192GB

Insert image description here

The following is a tendisplus.conf configuration reference for an NVME SSD for reference.

bind 127.0.0.1
port 51002
loglevel notice
cluster-enabled on
storage rocks
pidfile ./home/tendisplus.pid

# 数据和日志目录要提前创建
logdir ./home/log
dir ./home/db
dumpdir ./home/dump

# tendis的密码，按需指定
#requirepass 123
#masterauth 123

# binlog清理相关参数
truncateBinlogIntervalMs 100
truncateBinlogNum 500000
binlogDelRange 500000

# 网络线程池数量，建议cpu数量/4，
netiothreadnum 8
# 工作线程池数量，建议略多于cpu数量
executorthreadnum 56
# 其他线程池数量配置，根据实际情况配置，想看参数配置
# incrPushThreadnum 4
# migrateSenderThreadnum 2
# migrateReceiveThreadnum 4
# migrateClearThreadnum 2

# rocksdb的blockcache大小，建议使用30%系统内存(64GB)
rocks.blockcachemb 65536

# rocksdb相关配置
rocks.cache_index_and_filter_blocks 0
rocks.max_open_files -1
rocks.compress_type lz4
#rocks.level0_compress_enabled no
#rocks.level1_compress_enabled no
rocks.level_compaction_dynamic_level_bytes 1
# [n >= 4, n <= 64, n ~= cpu_cores]
rocks.max_background_compactions 64
#rocks.max_write_buffer_number 16
#rocks.min_write_buffer_number_to_merge 14
rocks.write_buffer_size 67108864
rocks.target_file_size_base 67108864
rocks.max_bytes_for_level_base 536870912

# mapping-command dbsize emptyint
# mapping-command keys emptymultibulk

# 数据迁移相关配置
cluster-migration-slots-num-per-task 10
cluster-migration-rate-limit 500
cluster-migration-binlog-iters 20
cluster-migration-distance 10000

4. Tendis storage version vs Redis cluster comparison

4.1. Advantages:

All data is stored on disk, providing greater capacity, lower cost, and higher data reliability
Multi-threaded architecture, the performance throughput of a single process is higher than that of a redis single process (30wQPS vs 13wQPS)
Independent gossip network thread supports more node communications and larger clusters
More powerful data relocation capabilities, the original redis cluster key relocation implementation, if a large key is encountered, it will lead to severe global blocking
Based on rocksdb mirroring and complete binlog implementation, it supports rollback at any point in time. The community version of redis currently does not have this capability.
Supports incremental copying and replication disconnection breakpoint resuming. Redis replication disconnection requires full replication.

4.2. Disadvantages:

Compared with pure memory redis, the Tendis storage version has greater latency.
Some commands are not supported yet (pubsub, geo, etc.) and are being improved.
For single-key updates, due to the internal concurrency control of the Tendis storage version, the advantages of multi-threading cannot be used, and it will degrade to single-threading with low performance.

5. Quick trial

1. Download tendis and upload it to the server (the demonstration machine is VMware virtual machine centos7.5)
2. Unzip: tar -xvzf tendisplus-2.3.4-rocksdb-v5.13.4.tgz
3. Enter the scripts directory of tendis: cd tendisplus-2.3.4-rocksdb-v5.13.4/scripts
4. Start tendis: sh start.sh

At this time, a basic tendis process is started. Since the default bind 127.0.0.1 can only be accessed on the server using the redis-cli command, cd tendisplus-2.3.4-rocksdb-v5.13.4/bin, redis-cli -p 51002 Enter the redis client

5. Modify tendisplus.conf so that it can be accessed from the external network

# tendisplus configuration for testing
# 绑定本机IIP
bind 192.168.148.140 
#设置访问密码
requirepass 123456
port 51002
daemon yes
#设置单机模式，非集群模式，默认值no
cluster-enabled no
loglevel notice
logdir ./home/log
dumpdir ./home/dump
dir ./home/db
pidfile ./home/tendisplus.pid
slowlog ./home/log/slowlog
rocks.blockcachemb 4096
executorThreadNum 48

Insert image description here
Connect with the client again: redis-cli -h 192.168.148.140 -p 51002 -a 123456

Connect with redis desktop manager

Insert image description here

6.tendis cluster

Please refer to: http://tendis.cn/#/Tendisplus/%E8%BF%90%E7%BB%B4/new_cluster

1. The configuration file must ensure that cluster-enabled is yes
2. Start multiple tendis instances and launch 6 tendis processes. The port numbers are 30000~30005: 30000, 30001, 30002, 30003, 30004, 30005
3. Node handshake: After starting the tendis service process, you can use similar Redis commands to establish the tendis cluster.

# 连接30000节点meet其他所有节点
shell> redis-cli -p 30000 
127.0.0.1:30000> cluster meet 127.0.0.1 30001
127.0.0.1:30000> cluster meet 127.0.0.1 30002
127.0.0.1:30000> cluster meet 127.0.0.1 30003
127.0.0.1:30000> cluster meet 127.0.0.1 30004
127.0.0.1:30000> cluster meet 127.0.0.1 30005

4. Allocate slots. By default, the Tendis storage version, like Redis, maps all data to 16384 slots, and each key corresponds to a slot. Only when a slot is assigned to a Tendis node can the Tendis node respond to slot-related commands, otherwise it will -MOVE to other nodes.

#分配slots可使用cluster add slots指令
#将16384个slots平均的分配给3个节点
shell> redis-cli -p 30000 cluster addslots {0..5461}
OK
shell> redis-cli -p 30002 cluster addslots {5462..10922}
OK
shell> redis-cli -p 30004 cluster addslots {10923..16383}
OK

#等待若干秒后，可以通过cluster info指令查看集群状态
127.0.0.1:30000> cluster info
cluster_state:ok
cluster_slots_assigned:16384
cluster_slots_ok:16384

5. Master-slave replication, here 3 instances are assigned slots, the remaining three instances can be used as their slaves, use the cluster replicate command

shell> redis-cli -p 30001 cluster replicate 4233f87b9899774feb71bcef1cfeec964de0535b
OK
shell> redis-cli -p 30003 cluster replicate b072bdd0f3d6cf75edcee1345da0d33f59177da0
OK
shell> redis-cli -p 30005 cluster replicate c980e1e23d20e9169aca0cbe7751d8a76b73859f
OK

You can build a cluster with three masters and three slaves.

127.0.0.1:30000> cluster nodes
20f8df612b19c879c92149458856ec08af50d459 127.0.0.1:30001@16374 slave 4233f87b9899774feb71bcef1cfeec964de0535b 0 1606964904000 5 connected
4233f87b9899774feb71bcef1cfeec964de0535b 127.0.0.1:30000@16379 myself,master - 0 1606964903000 0 connected 0-5461
f9d8c3f6679badc6366087d498f005849ad3b14d 127.0.0.1:30003@16375 slave b072bdd0f3d6cf75edcee1345da0d33f59177da0 0 1606964901000 4 connected
b072bdd0f3d6cf75edcee1345da0d33f59177da0 127.0.0.1:30002@16378 master - 0 1606964902996 1 connected 5462-10922
c980e1e23d20e9169aca0cbe7751d8a76b73859f 127.0.0.1:30004@16377 master - 0 1606964904598 2 connected 10923-16383
cd3c5da6415d5c1d125961e0f78badd162c7f7d3 127.0.0.1:30005@16376 slave c980e1e23d20e9169aca0cbe7751d8a76b73859f 0 1606964894000 3 connected