1. Overview of NoSQL
1. Why use NoSQL?
1. The good old days of stand-alone MySQL
In the 1990s, the number of visits to a website was generally not large, and a single database could easily handle it!
At that time, most of them were static web pages, and there were not many dynamic interactive websites.
Under the above architecture, let’s take a look at what are the bottlenecks of data storage?
- The total size of the data, when it cannot fit on one machine
- Data index (B+ Tree) cannot be stored in the memory of a machine
- The amount of access (mixed reading and writing) cannot be tolerated by one instance
If 1 or 3 of the above are met, evolve...
DAL: Database Access Layer
2. Memcached (cache) + MySQL + vertical split
Later, as the number of visits increased, almost most websites using the MySQL architecture began to have performance problems on the database. Web programs no longer only focused on functions, but also pursued performance. Programmers began to use caching technology extensively to relieve the pressure on the database and optimize the structure and index of the database. It became more popular to use file caching to relieve the pressure on the database. However, when the number of visits continued to increase, multiple web machines began to use file caching to reduce the pressure on the database. The cache cannot be shared, and a large number of small file caches also bring relatively high IO pressure. At this time, Memcached has naturally become a very fashionable technology product.
3. MySQL master-slave reading and writing separation
As the writing pressure on the database increases, Memcached can only relieve the reading pressure on the database. The concentration of reading and writing on one database makes the database overwhelmed. Most websites have begun to use master-slave replication technology to separate reading and writing to improve reading and writing performance. And the scalability of the read database, MySQL's master-slave mode has become the standard for websites at this time.
4. Sub-table and sub-database + horizontal split + Mysql cluster
Based on Memcached's cache, MySQL's master-slave replication, and read-write separation, the write pressure of the MySQL main database began to appear bottlenecks, and the amount of data continued to surge. Since MyISAM uses table locks, under high concurrency Serious lock problems will occur, and a large number of high-concurrency MySQL applications begin to use the InnoDB engine instead of MyISAM.
At the same time, it became popular to use sub-tables and sub-databases to alleviate write pressure and expansion problems of data growth. At this time, sub-tables and sub-databases became a popular technology, a popular interview question, and a hot technical issue discussed in the industry. It was at this time that MySQL launched table partitions that were not yet stable, which also brought hope to companies with average technical strength. Although MySQL has launched the MySQL Cluster cluster, its performance cannot meet the needs of the Internet very well, but it only provides a very large guarantee in terms of high reliability.
5. MySQL scalability bottleneck
MySQL database also often stores some large text fields, resulting in very large database tables, which makes database recovery very slow and difficult to quickly restore the database. For example, 10 million 4KB text is close to 40GB in size. If you can eliminate these data from MySQL, MySQL will become very small. The relational database is very powerful, but it cannot cope with all application scenarios well. MySQL has poor scalability (requires complex technology to implement), and big data High IO pressure and difficulty in changing the table structure are the problems faced by developers currently using MySQL.
6. What is it like today? ?
7. Why use NoSQL?
Today we can easily access and capture data through third-party platforms (such as Google, FaceBook, etc.). Users' personal information, social networks, geographical locations, user-generated data and user operation logs have increased exponentially. If we want to mine these user data, SQL databases are no longer suitable for these applications, and the development of NoSQL databases But it can handle these large data very well!
2. What is NoSQL?
NoSQL
NoSQL = Not Only SQL, meaning: not just SQL;
Generally refers to non-relational databases. With the rise of Internet Web2.0 websites, traditional relational databases have become increasingly incapable of coping with web2.0 websites, especially ultra-large and highly concurrent social network service type Web2.0 purely dynamic websites. Inadequate capabilities have exposed many insurmountable problems. Non-relational databases have developed very rapidly due to their own characteristics. NoSQL databases were created to solve the challenges brought by large-scale data collections and multiple data types, especially It is a big data application problem, including the storage of very large-scale data.
(Google or Facebook, for example, collect trillions of bits of data on their users every day). These types of data stores do not require a fixed schema and can scale out without redundant operations.
Features of NoSQL
1. Easy to expand
There are many types of NoSQL databases, but a common feature is that they remove the relational characteristics of relational databases.
There is no relationship between data, so it is very easy to expand, and it also brings scalability at the architectural level.
2. High performance for large data volumes
NoSQL databases have very high read and write performance, especially when dealing with large amounts of data. This is due to its non-relational nature and the simple structure of the database.
Generally, MySQL uses Query Cache, and the cache becomes invalid every time the table is updated. It is a very powerful Cache. In the frequent interactive applications of Web2.0, the Cache performance is not high, while the NoSQL Cache is record-level, which is a kind of Cache. Fine-grained Cache, so NoSQL has much higher performance at this level.
Official record: Redis can write 80,000 times and read 110,000 times per second!
3. Diverse and flexible data models
NoSQL does not need to create fields for the data to be stored in advance and can store customized data formats at any time. In relational databases, adding and deleting fields is a very troublesome matter. If it is a table with a very large amount of data, adding fields is simply a nightmare.
4. Traditional RDBMS VS NoSQL
传统的关系型数据库 RDBMS
- 高度组织化结构化数据
- 结构化查询语言（SQL）
- 数据和关系都存储在单独的表中
- 数据操纵语言，数据定义语言
- 严格的一致性
- 基础事务
NoSQL
- 代表着不仅仅是SQL
- 没有声明性查询语言
- 没有预定义的模式
- 键值对存储，列存储，文档存储，图形数据库
- 最终一致性，而非ACID属性
- 非结构化和不可预知的数据
- CAP定理
- 高性能，高可用性 和 可伸缩性
Expansion: 3V+3 high
3V in the era of big data: mainly the description of the problem
- Massive Volume
- Variety
- Real-timeVelocity
Top 3 Internet requirements: Mainly requirements for programs
- High concurrency
- High availability
- high performance
Today's applications use SQL and NoSQL together. There is no difference in technology, it just depends on how you use it, right?
3. Classic application analysis
Let’s talk about how to store product information on Alibaba’s Chinese website, taking women’s clothing and bags as examples:
Let’s talk about the development history of architecture: Recommended book "Ten Years of Taobao Technology"
1. Evolution process: Source of the following pictures: Alibaba Chinese website architecture design practice
2. Fifth generation
3. Mission of the 5th generation architecture
Related to us, storage issues of multiple data sources and multiple data types
1. Basic information of the product
名称、价格、出厂日期、生产厂商等
关系型数据库：mysql、oracle目前淘宝在去O化（也即，拿掉Oracle）
注意，淘宝内部用的MySQL是里面的大牛自己改造过的。
为什么去IOE：
2008年,王坚博士加入阿里巴巴，成为首席架构师。把云计算植入阿里IT基因。
2013年5月17日，阿里集团最后一台IBM小机在支付宝下线。这是自2009年“去IOE”战略透露以来，“去
IOE”非常重要的一个节点。“去 IOE”指的是摆脱掉IT部署中原有的IBM小型机、Oracle数据库以及EMC
存储的过度依赖。告别最后一台小机，意味着整个阿里集团尽管还有一些Oracle数据库和EMC存储，但是
IBM小型机已全部被替换。2013年7月10日，淘宝重中之重的广告系统使用的Oracle数据库下线，也是整
个淘宝最后一个 Oracle数据库。这两件事合在一起是阿里巴巴技术发展过程中的一个重要里程碑。
2. Product description, details, and evaluation information (multi-text)
多文字信息描述类，IO读写性能变差
存在文档数据库MongDB中
3. Pictures of the product
商品图片展现类
分布式文件系统中
- 淘宝自己的 TFS
- Google的 GFS
- Hadoop的 HDFS
4. Product keywords
搜索引擎，淘宝内用
ISearch：多隆一高兴一个人开发的
所有牛逼的人在牛逼之前,肯定有一段苦逼的岁月,但只要像傻逼一样的坚持,一定终将牛逼
5. Hotspot and high-frequency information about commodities
内存数据库
Tair、Redis、Memcache等
6. Commodity transactions, price calculation, and points accumulation!
外部系统，外部第三方支付接口
支付宝
Difficulties and solutions for large-scale Internet applications (big data, high concurrency, diverse data types)
difficulty:
- Diversity of data types
- Data source diversity and change reconstruction
- Data sources are transformed and the data service platform does not require extensive reconstruction.
Solution:
4. Introduction to NoSQL data model
Case design
Use an e-commerce customer, order, order, and address model to compare relational databases and non-relational databases
How do you design a traditional relational database?
ER diagram (1:1/1:N/N:N, primary and foreign keys, etc. are common)
- User corresponds to multiple orders and multiple addresses
- Each order corresponds to each product, price, and address
- Products corresponding to each item
Chat: User portrait analysis, women’s hearts are hard to figure out. I looked at men’s clothing and razors, and based on her information, I found out that her boyfriend’s birthday is very recently. The background portrait has been analyzed and ready to push ads, but she bought I grabbed a snack and left~
Programmers born in the 1990s are really changing every bit of life bit by bit. If you are lucky enough to enter a big factory, you will find that the friends around you are working hard. They are really the kind of people who can eat in Haidilao. People who suddenly take out their notebooks and write code while eating are considered crazy by others, but only they themselves understand. This is the obsession with technology
How do you design NoSQL
You can try using BSON .
BSON is a binary storage format similar to json, referred to as Binary JSON. Like JSON, it supports embedded document objects and array objects.
Use BSON to draw the constructed data model
{
"customer":{
"id":1000,
"name":"Z3",
"billingAddress":[{
"city":"beijing"}],
"orders":[
{
"id":17,
"customerId":1000,
"orderItems":[{
"productId":27,"price":77.5,"productName":"thinking in
java"}],
"shippingAddress":[{
"city":"beijing"}]
"orderPayment":[{
"ccinfo":"111-222-
333","txnid":"asdfadcd334","billingAddress":{"city":"beijing"}}],
}
]
}
}
Think about the relational model database, how do you check it? If we follow our newly designed BSON, it will be very simple to query.
- Related queries are not recommended for highly concurrent operations. Internet companies use redundant data to avoid related queries.
- Distributed transactions cannot support too much concurrency.
5. Four major categories of NoSQL
KV key value:
- Sina: BerkeleyDB+redis
- Meituan: redis+tair
- Alibaba, Baidu: memcache+redis
Document database (more bson format):
- CouchDB
- MongoDB
- MongoDB is a database based on distributed file storage. Written in C++ language. Designed to provide scalable, high-performance data storage solutions for WEB applications.
- MongoDB is a product between a relational database and a non-relational database. It is the most feature-rich among non-relational databases and is most like a relational database.
Column store database:
- Cassandra, HBase
- Distributed file system
graph relational database
- It is not for graphics, but for relationships, such as: friend circle social network, advertising recommendation system
- Social networks, recommendation systems, etc. Focus on building relationship graphs
- Neo4J, InfoGrid
Comparison of the four
6、CAP + BASE
What are the traditional ACIDs?
Relational databases follow ACID rules. Transaction in English is very similar to transactions in the real world. It has the following four characteristics:
- A (Atomicity) atomicity
原子性很容易理解，也就是说事务里的所有操作要么全部做完，要么都不做，事务成功的条件是事务
里的所有操作都成功，只要有一个操作失败，整个事务就失败，需要回滚。
比如银行转账，从A账户转100元至B账户，分为两个步骤：
1）从A账户取100元；
2）存入100元至B账户。
这两步要么一起完成，要么一起不完成，如果只完成第一步，第二步失败，钱会莫名其妙少了100
元。
- C (Consistency) Consistency
事务前后数据的完整性必须保持一致。
- I (Isolation) isolation
所谓的独立性是指并发的事务之间不会互相影响，如果一个事务要访问的数据正在被另外一个事务修
改，只要另外一个事务未提交，它所访问的数据就不受未提交事务的影响。比如现有有个交易是从A
账户转100元至B账户，在这个交易还未完成的情况下，如果此时B查询自己的账户，是看不到新增加
的100元的
- D (Durability) Durability
持久性是指一旦事务提交后，它所做的修改将会永久的保存在数据库上，即使出现宕机也不会丢失。
CAP (three into two)
- C: Consistency (strong consistency)
- A: Availability
- P: Partition tolerance (partition fault tolerance)
The CAP theory means that in a distributed storage system, at most the above two points can only be achieved.
Since current network hardware will definitely have problems such as delayed packet loss, partition fault tolerance is what we must achieve .
So we can only make a trade-off between consistency and availability. No NoSQL system can guarantee these three points at the same time.
Note: Trade-offs must be made when designing a distributed architecture.
Strike a balance between consistency and availability. Most web applications do not actually require strong consistency.
Therefore, C is sacrificed for P. This is the current direction of distributed database products.
The choice between consistency and availability
For web2.0 websites, many of the main features of relational databases are often useless.
Database transaction consistency requirements
Many web real-time systems do not require strict database transactions and have very low requirements for read consistency. In some cases, the requirements for write consistency are not high. Allows for eventual consistency.
Database real-time writing and reading requirements
For relational databases, if you query a piece of data immediately after inserting it, you can definitely read the data. However, for many web applications, such high real-time performance is not required. For example, after sending a message, after a few seconds, It is completely acceptable for my subscribers to see this update only ten seconds later.
Demand for complex SQL queries, especially multi-table related queries
Any web system with a large amount of data is very taboo about correlation queries of multiple large tables, as well as complex data analysis type report queries, especially SNS type websites. This situation can be avoided from the perspective of requirements and product design. produce. Often there are only primary key queries of a single table and simple conditional paging queries of a single table. The function of SQL is greatly weakened.
The core of CAP theory is: a distributed system cannot satisfy the three requirements of consistency, availability and partition fault tolerance at the same time. It can only satisfy two of them well at the same time. Therefore, according to the CAP principle, NoSQL databases are divided into three categories: satisfying the CA principle, satisfying the CP principle and satisfying the AP principle:
- CA - Single-point cluster, system that meets consistency and availability, usually not very powerful in scalability.
- CP - A system that satisfies consistency and must tolerate partitions, but usually does not perform particularly well.
- AP - A system that meets availability, partition tolerance, and may generally have lower consistency requirements.
BASE theory
BASE theory was proposed by eBay architects. BASE is the result of the trade-off between consistency and availability in CAP. It is derived from the summary of the practice of large-scale Internet distributed systems and is gradually evolved based on the CAP law. The core idea is that even if strong consistency cannot be achieved, each application can adopt appropriate methods according to its own business characteristics to make the system achieve final consistency.
BASE is a solution proposed to solve the problems caused by the strong consistency of relational databases and the reduced availability.
BASE is actually the abbreviation of the following three terms:
- 基本可用(Basically Available)： 基本可用是指分布式系统在出现故障的时候，允许损失部分可用性，即保证核心可用。电商大促时，为了应对访问量激增，部分用户可能会被引导到降级页面，服务层也可能只提供降级服务。这就是损失部分可用性的体现。
- 软状态(Soft State)： 软状态是指允许系统存在中间状态，而该中间状态不会影响系统整体可用性。分布式存储中一般一份数据至少会有三个副本，允许不同节点间副本同步的延时就是软状态的体现。MySQL Replication 的异步复制也是一种体现。
- 最终一致性(Eventual Consistency)： 最终一致性是指系统中的所有数据副本经过一定时间后，最终能够达到一致的状态。弱一致性和强一致性相反，最终一致性是弱一致性的一种特殊情况。
它的思想是通过让系统放松对某一时刻数据一致性的要求来换取系统整体伸缩性和性能上改观。为什么这么说呢，缘由就在于大型系统往往由于地域分布和极高性能的要求，不可能采用分布式事务来完成这些指标，要想获得这些指标，我们必须采用另外一种方式来完成，这里BASE就是解决这个问题的办法！
解释：
1、分布式：不同的多台服务器上面部署不同的服务模块（工程），他们之间通过Rpc通信和调用，对外提供服务和组内协作。
2、集群：不同的多台服务器上面部署相同的服务模块，通过分布式调度软件进行统一的调度，对外提供
服务和访问。
二、Redis入门
1、概述
Redis是什么
Redis：REmote DIctionary Server（远程字典服务器）
是完全开源免费的，用C语言编写的，遵守BSD协议，是一个高性能的（Key/Value）分布式内存数据库，基于内存运行，并支持持久化的NoSQL数据库，是当前最热门的NoSQL数据库之一，也被人们称为数据结构服务器
Redis与其他key-value缓存产品有以下三个特点
- Redis支持数据的持久化，可以将内存中的数据保持在磁盘中，重启的时候可以再次加载进行使用。
- Redis not only supports simple key-value type data, but also provides storage of data structures such as list, set, zset, and hash.
- Redis supports data backup, that is, data backup in master-slave mode.
What can Redis do?
Memory storage and persistence: redis supports asynchronously writing data in memory to the hard disk without affecting the continued service
The operation of taking the latest N data, for example: you can put the IDs of the latest 10 comments in the Redis List collection
Publish and subscribe message system
Map information analysis
timer, counter
…
characteristic
Data types, basic operations and configuration
Persistence and replication, RDB, AOF
transaction control
…
Frequently used websites
https://redis.io/ official website
http://www.redis.cn Chinese website
2. Windows installation
Download address: https://github.com/dmajkic/redis/downloads (material provided)
Unzip it to the environment directory of your computer
Double-click redis-server.exe to start
Access redis-cli through the client
# 基本的set设值
127.0.0.1:6379> set key kuangshen
OK
# 取出存储的值
127.0.0.1:6379> get key
"kuangshen"
important hint
Since enterprises do Redis development, 99% of them are using and installing the Linux version, and the Windows version is almost never involved. The previous explanation is only for the completeness of knowledge. The Windows version is not the focus. You can play it yourself and practice it in the enterprise. Just recognize one version: Linux version
http://www.redis.cn/topics/introduction
3. Linux installation
Download address http://download.redis.io/releases/redis-5.0.7.tar.gz
installation steps
- Download redis-5.0.7.tar.gz and put it in our Linux directory/opt
- In the /opt directory, decompress the command: tar -zxvf redis-5.0.7.tar.gz
- After decompression is completed, the folder appears: redis-5.0.7
- Enter the directory: cd redis-5.0.7
- Execute the make command in the redis-5.0.7 directory
运行make命令时故意出现的错误解析：
1、安装gcc (gcc是linux下的一个编译程序，是c程序的编译工具)
能上网: yum install gcc-c++
版本测试: gcc-v
2、二次make
3、Jemalloc/jemalloc.h: 没有那个文件或目录
运行 make distclean 之后再make
4、Redis Test（可以不用执行）
- If you continue to execute make install after make is completed
- Check the default installation directory: usr/local/bin
/usr 这是一个非常重要的目录，类似于windows下的Program Files,存放用户的程序
- Copy configuration file (backup)
cd /usr/local/bin
ls -l
# 在redis的解压目录下备份redis.conf
mkdir myredis
cp redis.conf myredis # 拷一个备份，养成良好的习惯，我们就修改这个文件
# 修改配置保证可以后台应用
vim redis.conf
- A. The daemonize daemon thread in the redis.conf configuration file defaults to NO.
- B. daemonize is used to specify whether redis should be started as a daemon thread.
daemonize sets the difference between yes or no
-
daemonize:yes
- Redis uses a single-process multi-thread mode. When the daemonize option in redis.conf is set to yes, it means that the daemon process mode is enabled. In this mode, redis will run in the background and write the process pid number to the file set by the redis.conf option pidfile. At this time, redis will always run unless the process is manually killed.
-
daemonize:no
- When the daemonize option is set to no, the current interface will enter the redis command line interface. Forced exit by exit or closing the connection tool (putty, xshell, etc.) will cause the redis process to exit.
- Start the test!
# 【shell】启动redis服务
[root@192 bin]# cd /usr/local/bin
[root@192 bin]# redis-server /opt/redis-5.0.7/redis.conf
# redis客户端连接===> 观察地址的变化，如果连接ok,是直接连上的，redis默认端口号 6379
[root@192 bin]# redis-cli -p 6379
127.0.0.1:6379> ping
PONG
127.0.0.1:6379> set k1 helloworld
OK
127.0.0.1:6379> get k1
"helloworld"
# 【shell】ps显示系统当前进程信息
[root@192 myredis]# ps -ef|grep redis
root 16005 1 0 04:45 ? 00:00:00 redis-server
127.0.0.1:6379
root 16031 15692 0 04:47 pts/0 00:00:00 redis-cli -p 6379
root 16107 16076 0 04:51 pts/2 00:00:00 grep --color=auto redis
# 【redis】关闭连接
127.0.0.1:6379> shutdown
not connected> exit
# 【shell】ps显示系统当前进程信息
[root@192 myredis]# ps -ef|grep redis
root 16140 16076 0 04:53 pts/2 00:00:00 grep --color=auto redis
4. Basic knowledge description
Preparation: Start the redis service and connect the client
redis stress testing tool-----Redis-benchmark
Redis-benchmark is the official Redis performance testing tool that can effectively test the performance of Redis services.
The optional parameters of the redis performance testing tool are as follows:
| serial number | Options | describe | default value |
|---|---|---|---|
| 1 | -h | Specify server hostname | 127.0.0.1 |
| 2 | -p | Specify server port | 6379 |
| 3 | -s | Specify server socket | |
| 4 | -c | Specify the number of concurrent connections | 50 |
| 5 | -n | Specify the number of requests | 10000 |
| 6 | -d | Specifies the data size of the SET/GET value in bytes | 2 |
| 7 | -k | 1=keep alive 0=reconnect | 1 |
| 8 | -r | SET/GET/INCR uses random keys, SADD uses random values | |
| 9 | -P | Pipe requests | 1 |
| 10 | -q | Force quit redis. Show only query/sec values | |
| 11 | –csv | Output in CSV format | |
| 12 | -l | Generate loops that execute tests forever | |
| 13 | -t | Run only a comma-separated list of test commands. | |
| 14 | -I | Idle mode. Open only N idle connections and wait. |
# 测试一：100个并发连接，100000个请求，检测host为localhost 端口为6379的redis服务器性能
redis-benchmark -h localhost -p 6379 -c 100 -n 100000
# 测试出来的所有命令只举例一个！
====== SET ======
100000 requests completed in 1.88 seconds # 对集合写入测试
100 parallel clients # 每次请求有100个并发客户端
3 bytes payload # 每次写入3个字节的数据，有效载荷
keep alive: 1 # 保持一个连接，一台服务器来处理这些请求
17.05% <= 1 milliseconds
97.35% <= 2 milliseconds
99.97% <= 3 milliseconds
100.00% <= 3 milliseconds # 所有请求在 3 毫秒内完成
53248.14 requests per second # 每秒处理 53248.14 次请求
Basic database knowledge
There are 16 databases by default. Similar to array subscripts starting from zero, the initial default is to use library number zero.
查看 redis.conf ，里面有默认的配置
databases 16
# Set the number of databases. The default database is DB 0, you can select
# a different one on a per-connection basis using SELECT <dbid> where
# dbid is a number between 0 and 'databases'-1
databases 16
Select command switches database
127.0.0.1:6379> select 7
OK
127.0.0.1:6379[7]>
# 不同的库可以存不同的数据
Dbsize checks the number of keys in the current database
127.0.0.1:6379> select 7
OK
127.0.0.1:6379[7]> DBSIZE
(integer) 0
127.0.0.1:6379[7]> select 0
OK
127.0.0.1:6379> DBSIZE
(integer) 5
127.0.0.1:6379> keys * # 查看具体的key
1) "counter:__rand_int__"
2) "mylist"
3) "k1"
4) "myset:__rand_int__"
5) "key:__rand_int__"
Flushdb: Clear the current library
Flushall: Clear all libraries
127.0.0.1:6379> DBSIZE
(integer) 5
127.0.0.1:6379> FLUSHDB
OK
127.0.0.1:6379> DBSIZE
(integer) 0
Why is the default port 6379? Fan effect!
Why redis is single-threaded
We must first understand that Redis is fast! The official stated that because Redis is a memory-based operation, the CPU is not the bottleneck of Redis. The bottleneck of Redis is most likely the size of the machine memory or the network bandwidth. Since single-threading is easy to implement and the CPU will not become a bottleneck, it is logical to adopt the single-threaded solution!
Redis uses a memory-based KV database that uses a single-process single-thread model. It is written in C language. The officially provided data can reach 100,000+ QPS (number of queries per second). This data is no worse than Memcached, the same memory-based KV database that uses single process and multi-threading!
Why is Redis so fast?
1) There has always been a misunderstanding in the past, thinking that high-performance servers must be implemented with multi-threads
The reason is very simple and is caused by misunderstanding 2: multi-threading must be more efficient than single-threading, but it is not!
Before talking about this, I hope everyone has an understanding of the speed of CPU, memory, and hard disk!
2) The core of redis is that if all my data is in memory, my single-threaded operation will be the most efficient. Why? Because the essence of multi-threading is that the CPU simulates multiple threads. This simulated situation is There is a price, which is context switching. For a memory system, it is the most efficient without context switching. Redis uses a single CPU to bind a piece of memory data, and then when reading and writing multiple times to the data in this memory, it is all done on one CPU, so it handles this matter in a single thread. In the case of memory, this solution is the best solution.
Because a CPU context switch takes about 1500ns. Reading 1MB of continuous data from memory takes about 250us. Assume that 1MB of data is read 1000 times by multiple threads, then there are 1000 time context switches, then there are 1500ns * 1000 = 1500us. I It only takes 250us to read 1MB of data in a single thread. It takes 1500us just to switch the time context. I don't include the time it takes you to read a little data each time.
3. Five major data types
Official documentation
Full translation:
Redis is an open source (BSD licensed) in-memory data structure store used as a database, cache, and message broker. It supports data structures such as strings, hashes, lists, sets, sorted sets with range queries, bitmaps, hyperlogs, geospatial indexes with radius queries and streams. Redis has built-in replication, Lua scripts, LRU eviction, transactions and different levels of disk durability, and provides high availability through Redis Sentinel and Redis Cluster automatic partitioning.
String (string type)
String is the most basic type of redis. You can understand it as the same type as Memcached. One key corresponds to one value.
The String type is binary safe, which means that the redis string can contain any data, such as jpg images or serialized objects.
The String type is the most basic data type of redis. The string value in a redis can be up to 512M.
Hash (Hash, similar to Map in Java)
Redis hash is a collection of key-value pairs.
Redis hash is a mapping table of String type fields and values. Hash is particularly suitable for storing objects.
Similar to Map<String,Object> in Java
List
Redis lists are simple lists of strings, sorted in insertion order. You can add an element to the head (left) or tail (right) of the list.
Its bottom layer is actually a linked list!
Set
Redis's Set is an unordered collection of String type, which is implemented through HashTable!
Zset (sorted set: ordered set)
Redis zset, like set, is also a collection of String type elements and does not allow duplicate members.
The difference is that each element is associated with a double type score.
Redis uses scores to sort the members in the set from small to large. The members of zset are unique, but the scores (Score) can be repeated.
1. Redis key (key)
# keys * 查看所有的key
127.0.0.1:6379> keys *
(empty list or set)
127.0.0.1:6379> set name qinjiang
OK
127.0.0.1:6379> keys *
1) "name"
# exists key 的名字，判断某个key是否存在
127.0.0.1:6379> EXISTS name
(integer) 1
127.0.0.1:6379> EXISTS name1
(integer) 0
# move key db ---> 当前库就没有了，被移除了
127.0.0.1:6379> move name 1
(integer) 1
127.0.0.1:6379> keys *
(empty list or set)
# expire key 秒钟：为给定 key 设置生存时间，当 key 过期时(生存时间为 0 )，它会被自动删
除。
# ttl key 查看还有多少秒过期，-1 表示永不过期，-2 表示已过期
127.0.0.1:6379> set name qinjiang
OK
127.0.0.1:6379> EXPIRE name 10
(integer) 1
127.0.0.1:6379> ttl name
(integer) 4
127.0.0.1:6379> ttl name
(integer) 3
127.0.0.1:6379> ttl name
(integer) 2
127.0.0.1:6379> ttl name
(integer) 1
127.0.0.1:6379> ttl name
(integer) -2
127.0.0.1:6379> keys *
(empty list or set)
# type key 查看你的key是什么类型
127.0.0.1:6379> set name qinjiang
OK
127.0.0.1:6379> get name
"qinjiang"
127.0.0.1:6379> type name
string
2. String
Single value Single Value
Common command descriptions:
# ===================================================
# set、get、del、append、strlen
# ===================================================
127.0.0.1:6379> set key1 value1 # 设置值
OK
127.0.0.1:6379> get key1 # 获得key
"value1"
127.0.0.1:6379> del key1 # 删除key
(integer) 1
127.0.0.1:6379> keys * # 查看全部的key
(empty list or set)
127.0.0.1:6379> exists key1 # 确保 key1 不存在
(integer) 0
127.0.0.1:6379> append key1 "hello" # 对不存在的 key 进行 APPEND ，等同于 SET
key1 "hello"
(integer) 5 # 字符长度
127.0.0.1:6379> APPEND key1 "-2333" # 对已存在的字符串进行 APPEND
(integer) 10 # 长度从 5 个字符增加到 10 个字符
127.0.0.1:6379> get key1
"hello-2333"
127.0.0.1:6379> STRLEN key1 # # 获取字符串的长度
(integer) 10
# ===================================================
# incr、decr 一定要是数字才能进行加减，+1 和 -1。
# incrby、decrby 命令将 key 中储存的数字加上指定的增量值。
# ===================================================
127.0.0.1:6379> set views 0 # 设置浏览量为0
OK
127.0.0.1:6379> incr views # 浏览 + 1
(integer) 1
127.0.0.1:6379> incr views # 浏览 + 1
(integer) 2
127.0.0.1:6379> decr views # 浏览 - 1
(integer) 1
127.0.0.1:6379> incrby views 10 # +10
(integer) 11
127.0.0.1:6379> decrby views 10 # -10
(integer) 1
# ===================================================
# range [范围]
# getrange 获取指定区间范围内的值，类似between...and的关系，从零到负一表示全部
# ===================================================
127.0.0.1:6379> set key2 abcd123456 # 设置key2的值
OK
127.0.0.1:6379> getrange key2 0 -1 # 获得全部的值
"abcd123456"
127.0.0.1:6379> getrange key2 0 2 # 截取部分字符串
"abc"
# ===================================================
# setrange 设置指定区间范围内的值，格式是setrange key值 具体值
# ===================================================
127.0.0.1:6379> get key2
"abcd123456"
127.0.0.1:6379> SETRANGE key2 1 xx # 替换值
(integer) 10
127.0.0.1:6379> get key2
"axxd123456"
# ===================================================
# setex（set with expire）键秒值
# setnx（set if not exist）
# ===================================================
127.0.0.1:6379> setex key3 60 expire # 设置过期时间
OK
127.0.0.1:6379> ttl key3 # 查看剩余的时间
(integer) 55
127.0.0.1:6379> setnx mykey "redis" # 如果不存在就设置，成功返回1
(integer) 1
127.0.0.1:6379> setnx mykey "mongodb" # 如果存在就设置，失败返回0
(integer) 0
127.0.0.1:6379> get mykey
"redis"
# ===================================================
# mset Mset 命令用于同时设置一个或多个 key-value 对。
# mget Mget 命令返回所有(一个或多个)给定 key 的值。
# 如果给定的 key 里面，有某个 key 不存在，那么这个 key 返回特殊值 nil 。
# msetnx 当所有 key 都成功设置，返回 1 。
# 如果所有给定 key 都设置失败(至少有一个 key 已经存在)，那么返回 0 。原子操
作
# ===================================================
127.0.0.1:6379> mset k10 v10 k11 v11 k12 v12
OK
127.0.0.1:6379> keys *
1) "k12"
2) "k11"
3) "k10"
127.0.0.1:6379> mget k10 k11 k12 k13
1) "v10"
2) "v11"
3) "v12"
4) (nil)
127.0.0.1:6379> msetnx k10 v10 k15 v15 # 原子性操作！
(integer) 0
127.0.0.1:6379> get key15
(nil)
# 传统对象缓存
set user:1 value(json数据)
# 可以用来缓存对象
mset user:1:name zhangsan user:1:age 2
mget user:1:name user:1:age
# ===================================================
# getset（先get再set）
# ===================================================
127.0.0.1:6379> getset db mongodb # 没有旧值，返回 nil
(nil)
127.0.0.1:6379> get db
"mongodb"
127.0.0.1:6379> getset db redis # 返回旧值 mongodb
"mongodb"
127.0.0.1:6379> get db
"redis"
The String data structure is a simple key-value type. In fact, value can be not only String, but also a number.
General key-value caching application:
Conventional counts: number of Weibo posts, number of fans, etc.
3. List List
Single value multiple Value
# ===================================================
# Lpush：将一个或多个值插入到列表头部。（左）
# rpush：将一个或多个值插入到列表尾部。（右）
# lrange：返回列表中指定区间内的元素，区间以偏移量 START 和 END 指定。
# 其中 0 表示列表的第一个元素， 1 表示列表的第二个元素，以此类推。
# 你也可以使用负数下标，以 -1 表示列表的最后一个元素， -2 表示列表的倒数第二个元素，以此类推。
# ===================================================
127.0.0.1:6379> LPUSH list "one"
(integer) 1
127.0.0.1:6379> LPUSH list "two"
(integer) 2
127.0.0.1:6379> RPUSH list "right"
(integer) 3
127.0.0.1:6379> Lrange list 0 -1
1) "two"
2) "one"
3) "right"
127.0.0.1:6379> Lrange list 0 1
1) "two"
2) "one"
# ===================================================
# lpop 命令用于移除并返回列表的第一个元素。当列表 key 不存在时，返回 nil 。
# rpop 移除列表的最后一个元素，返回值为移除的元素。
# ===================================================
127.0.0.1:6379> Lpop list
"two"
127.0.0.1:6379> Rpop list
"right"
127.0.0.1:6379> Lrange list 0 -1
1) "one"
# ===================================================
# Lindex，按照索引下标获得元素（-1代表最后一个，0代表是第一个）
# ===================================================
127.0.0.1:6379> Lindex list 1
(nil)
127.0.0.1:6379> Lindex list 0
"one"
127.0.0.1:6379> Lindex list -1
"one"
# ===================================================
# llen 用于返回列表的长度。
# ===================================================
127.0.0.1:6379> flushdb
OK
127.0.0.1:6379> Lpush list "one"
(integer) 1
127.0.0.1:6379> Lpush list "two"
(integer) 2
127.0.0.1:6379> Lpush list "three"
(integer) 3
127.0.0.1:6379> Llen list # 返回列表的长度
(integer) 3
# ===================================================
# lrem key 根据参数 COUNT 的值，移除列表中与参数 VALUE 相等的元素。
# ===================================================
127.0.0.1:6379> lrem list 1 "two"
(integer) 1
127.0.0.1:6379> Lrange list 0 -1
1) "three"
2) "one"
# ===================================================
# Ltrim key 对一个列表进行修剪(trim)，就是说，让列表只保留指定区间内的元素，不在指定区间之内的元素都将被删除。
# ===================================================
127.0.0.1:6379> RPUSH mylist "hello"
(integer) 1
127.0.0.1:6379> RPUSH mylist "hello"
(integer) 2
127.0.0.1:6379> RPUSH mylist "hello2"
(integer) 3
127.0.0.1:6379> RPUSH mylist "hello3"
(integer) 4
127.0.0.1:6379> ltrim mylist 1 2
OK
127.0.0.1:6379> lrange mylist 0 -1
1) "hello"
2) "hello2"
# ===================================================
# rpoplpush 移除列表的最后一个元素，并将该元素添加到另一个列表并返回。
# ===================================================
127.0.0.1:6379> rpush mylist "hello"
(integer) 1
127.0.0.1:6379> rpush mylist "foo"
(integer) 2
127.0.0.1:6379> rpush mylist "bar"
(integer) 3
127.0.0.1:6379> rpoplpush mylist myotherlist
"bar"
127.0.0.1:6379> lrange mylist 0 -1
1) "hello"
2) "foo"
127.0.0.1:6379> lrange myotherlist 0 -1
1) "bar"
# ===================================================
# lset key index value 将列表 key 下标为 index 的元素的值设置为 value 。
# ===================================================
127.0.0.1:6379> exists list # 对空列表(key 不存在)进行 LSET
(integer) 0
127.0.0.1:6379> lset list 0 item # 报错
(error) ERR no such key
127.0.0.1:6379> lpush list "value1" # 对非空列表进行 LSET
(integer) 1
127.0.0.1:6379> lrange list 0 0
1) "value1"
127.0.0.1:6379> lset list 0 "new" # 更新值
OK
127.0.0.1:6379> lrange list 0 0
1) "new"
127.0.0.1:6379> lset list 1 "new" # index 超出范围报错
(error) ERR index out of range
# ===================================================
# linsert key before/after pivot value 用于在列表的元素前或者后插入元素。
# 将值 value 插入到列表 key 当中，位于值 pivot 之前或之后。
# ===================================================
redis> RPUSH mylist "Hello"
(integer) 1
redis> RPUSH mylist "World"
(integer) 2
redis> LINSERT mylist BEFORE "World" "There"
(integer) 3
redis> LRANGE mylist 0 -1
1) "Hello"
2) "There"
3) "World"
Performance summary
- It is a string linked list, left and right can be inserted and added
- If the key does not exist, create a new linked list
- If the key already exists, add the content
- If all values are removed, the corresponding keys will disappear.
- The operation of the linked list is extremely efficient on both the head and tail, but if the operation is performed on the middle elements, the efficiency is very dismal.
A list is a linked list, and anyone with some knowledge of data structures should be able to understand its structure. Using the Lists structure, we can easily implement functions such as ranking of the latest news. Another application of List is the message queue. You can use the PUSH operation of List to store tasks in the List, and then the worker thread uses the POP operation to take out the tasks for execution. Redis also provides an API for operating a certain segment in the List. You can directly query and delete elements of a certain segment in the List.
Redis's list is a doubly linked list in which each sub-element is of type String. Elements can be added or deleted from the head or tail of the list through push and pop operations, so that the List can be used as a stack or a queue.
4. Set
single value multiple value
# ===================================================
# sadd 将一个或多个成员元素加入到集合中，不能重复
# smembers 返回集合中的所有的成员。
# sismember 命令判断成员元素是否是集合的成员。
# ===================================================
127.0.0.1:6379> sadd myset "hello"
(integer) 1
127.0.0.1:6379> sadd myset "kuangshen"
(integer) 1
127.0.0.1:6379> sadd myset "kuangshen"
(integer) 0
127.0.0.1:6379> SMEMBERS myset
1) "kuangshen"
2) "hello"
127.0.0.1:6379> SISMEMBER myset "hello"
(integer) 1
127.0.0.1:6379> SISMEMBER myset "world"
(integer) 0
# ===================================================
# scard，获取集合里面的元素个数
# ===================================================
127.0.0.1:6379> scard myset
(integer) 2
# ===================================================
# srem key value 用于移除集合中的一个或多个成员元素
# ===================================================
127.0.0.1:6379> srem myset "kuangshen"
(integer) 1
127.0.0.1:6379> SMEMBERS myset
1) "hello"
# ===================================================
# srandmember key 命令用于返回集合中的一个随机元素。
# ===================================================
127.0.0.1:6379> SMEMBERS myset
1) "kuangshen"
2) "world"
3) "hello"
127.0.0.1:6379> SRANDMEMBER myset
"hello"
127.0.0.1:6379> SRANDMEMBER myset 2
1) "world"
2) "kuangshen"
127.0.0.1:6379> SRANDMEMBER myset 2
1) "kuangshen"
2) "hello"
# ===================================================
# spop key 用于移除集合中的指定 key 的一个或多个随机元素
# ===================================================
127.0.0.1:6379> SMEMBERS myset
1) "kuangshen"
2) "world"
3) "hello"
127.0.0.1:6379> spop myset
"world"
127.0.0.1:6379> spop myset
"kuangshen"
127.0.0.1:6379> spop myset
"hello"
# ===================================================
# smove SOURCE DESTINATION MEMBER
# 将指定成员 member 元素从 source 集合移动到 destination 集合。
# ===================================================
127.0.0.1:6379> sadd myset "hello"
(integer) 1
127.0.0.1:6379> sadd myset "world"
(integer) 1
127.0.0.1:6379> sadd myset "kuangshen"
(integer) 1
127.0.0.1:6379> sadd myset2 "set2"
(integer) 1
127.0.0.1:6379> smove myset myset2 "kuangshen"
(integer) 1
127.0.0.1:6379> SMEMBERS myset
1) "world"
2) "hello"
127.0.0.1:6379> SMEMBERS myset2
1) "kuangshen"
2) "set2"
# ===================================================
- 数字集合类
- 差集： sdiff
- 交集： sinter
- 并集： sunion
# ===================================================
127.0.0.1:6379> sadd key1 "a"
(integer) 1
127.0.0.1:6379> sadd key1 "b"
(integer) 1
127.0.0.1:6379> sadd key1 "c"
(integer) 1
127.0.0.1:6379> sadd key2 "c"
(integer) 1
127.0.0.1:6379> sadd key2 "d"
(integer) 1
127.0.0.1:6379> sadd key2 "e"
(integer) 1
127.0.0.1:6379> SDIFF key1 key2 # 差集
1) "a"
2) "b"
127.0.0.1:6379> SINTER key1 key2 # 交集
1) "c"
127.0.0.1:6379> SUNION key1 key2 # 并集
1) "a"
2) "b"
3) "c"
4) "e"
5) "d"
In the Weibo application, all the followers of a user can be stored in a collection, and all the fans can be stored in a collection. Redis also provides operations such as intersection, union, and difference for collections, which can be very convenient to implement functions such as joint attention, common preferences, and second-degree friends. For all the above collection operations, you can also use different command selections. Return the results to the client or save them in a new collection.
5. Hash
The kv mode remains unchanged, but V is a key-value pair
# ===================================================
# hset、hget 命令用于为哈希表中的字段赋值 。
# hmset、hmget 同时将多个field-value对设置到哈希表中。会覆盖哈希表中已存在的字段。
# hgetall 用于返回哈希表中，所有的字段和值。
# hdel 用于删除哈希表 key 中的一个或多个指定字段
# ===================================================
127.0.0.1:6379> hset myhash field1 "kuangshen"
(integer) 1
127.0.0.1:6379> hget myhash field1
"kuangshen"
127.0.0.1:6379> HMSET myhash field1 "Hello" field2 "World"
OK
127.0.0.1:6379> HGET myhash field1
"Hello"
127.0.0.1:6379> HGET myhash field2
"World"
127.0.0.1:6379> hgetall myhash
1) "field1"
2) "Hello"
3) "field2"
4) "World"
127.0.0.1:6379> HDEL myhash field1
(integer) 1
127.0.0.1:6379> hgetall myhash
1) "field2"
2) "World"
# ===================================================
# hlen 获取哈希表中字段的数量。
# ===================================================
127.0.0.1:6379> hlen myhash
(integer) 1
127.0.0.1:6379> HMSET myhash field1 "Hello" field2 "World"
OK
127.0.0.1:6379> hlen myhash
(integer) 2
# ===================================================
# hexists 查看哈希表的指定字段是否存在。
# ===================================================
127.0.0.1:6379> hexists myhash field1
(integer) 1
127.0.0.1:6379> hexists myhash field3
(integer) 0
# ===================================================
# hkeys 获取哈希表中的所有域（field）。
# hvals 返回哈希表所有域(field)的值。
# ===================================================
127.0.0.1:6379> HKEYS myhash
1) "field2"
2) "field1"
127.0.0.1:6379> HVALS myhash
1) "World"
2) "Hello"
# ===================================================
# hincrby 为哈希表中的字段值加上指定增量值。
# ===================================================
127.0.0.1:6379> hset myhash field 5
(integer) 1
127.0.0.1:6379> HINCRBY myhash field 1
(integer) 6
127.0.0.1:6379> HINCRBY myhash field -1
(integer) 5
127.0.0.1:6379> HINCRBY myhash field -10
(integer) -5
# ===================================================
# hsetnx 为哈希表中不存在的的字段赋值 。
# ===================================================
127.0.0.1:6379> HSETNX myhash field1 "hello"
(integer) 1 # 设置成功，返回 1 。
127.0.0.1:6379> HSETNX myhash field1 "world"
(integer) 0 # 如果给定字段已经存在，返回 0 。
127.0.0.1:6379> HGET myhash field1
"hello"
Redis hash is a mapping table of string type fields and values. Hash is particularly suitable for storing objects.
Store some changed data, such as user information, etc.
6. Ordered set Zset
Based on the set, add a score value. Previously set was k1 v1 v2 v3, now zset is k1 score1 v1 score2 v2
# ===================================================
# zadd 将一个或多个成员元素及其分数值加入到有序集当中。
# zrange 返回有序集中，指定区间内的成员
# ===================================================
127.0.0.1:6379> zadd myset 1 "one"
(integer) 1
127.0.0.1:6379> zadd myset 2 "two" 3 "three"
(integer) 2
127.0.0.1:6379> ZRANGE myset 0 -1
1) "one"
2) "two"
3) "three"
# ===================================================
# zrangebyscore 返回有序集合中指定分数区间的成员列表。有序集成员按分数值递增(从小到大)次序排列。
# ===================================================
127.0.0.1:6379> zadd salary 2500 xiaoming
(integer) 1
127.0.0.1:6379> zadd salary 5000 xiaohong
(integer) 1
127.0.0.1:6379> zadd salary 500 kuangshen
(integer) 1
# Inf无穷大量+∞,同样地,-∞可以表示为-Inf。
127.0.0.1:6379> ZRANGEBYSCORE salary -inf +inf # 显示整个有序集
1) "kuangshen"
2) "xiaoming"
3) "xiaohong"
127.0.0.1:6379> ZRANGEBYSCORE salary -inf +inf withscores # 递增排列
1) "kuangshen"
2) "500"
3) "xiaoming"
4) "2500"
5) "xiaohong"
6) "5000"
127.0.0.1:6379> ZREVRANGE salary 0 -1 WITHSCORES # 递减排列
1) "xiaohong"
2) "5000"
3) "xiaoming"
4) "2500"
5) "kuangshen"
6) "500"
127.0.0.1:6379> ZRANGEBYSCORE salary -inf 2500 WITHSCORES # 显示工资 <=2500
的所有成员
1) "kuangshen"
2) "500"
3) "xiaoming"
4) "2500"
# ===================================================
# zrem 移除有序集中的一个或多个成员
# ===================================================
127.0.0.1:6379> ZRANGE salary 0 -1
1) "kuangshen"
2) "xiaoming"
3) "xiaohong"
127.0.0.1:6379> zrem salary kuangshen
(integer) 1
127.0.0.1:6379> ZRANGE salary 0 -1
1) "xiaoming"
2) "xiaohong"
# ===================================================
# zcard 命令用于计算集合中元素的数量。
# ===================================================
127.0.0.1:6379> zcard salary
(integer) 2
OK
# ===================================================
# zcount 计算有序集合中指定分数区间的成员数量。
# ===================================================
127.0.0.1:6379> zadd myset 1 "hello"
(integer) 1
127.0.0.1:6379> zadd myset 2 "world" 3 "kuangshen"
(integer) 2
127.0.0.1:6379> ZCOUNT myset 1 3
(integer) 3
127.0.0.1:6379> ZCOUNT myset 1 2
(integer) 2
# ===================================================
# zrank 返回有序集中指定成员的排名。其中有序集成员按分数值递增(从小到大)顺序排列。
# ===================================================
127.0.0.1:6379> zadd salary 2500 xiaoming
(integer) 1
127.0.0.1:6379> zadd salary 5000 xiaohong
(integer) 1
127.0.0.1:6379> zadd salary 500 kuangshen
(integer) 1
127.0.0.1:6379> ZRANGE salary 0 -1 WITHSCORES # 显示所有成员及其 score 值
1) "kuangshen"
2) "500"
3) "xiaoming"
4) "2500"
5) "xiaohong"
6) "5000"
127.0.0.1:6379> zrank salary kuangshen # 显示 kuangshen 的薪水排名，最少
(integer) 0
127.0.0.1:6379> zrank salary xiaohong # 显示 xiaohong 的薪水排名，第三
(integer) 2
# ===================================================
# zrevrank 返回有序集中成员的排名。其中有序集成员按分数值递减(从大到小)排序。
# ===================================================
127.0.0.1:6379> ZREVRANK salary kuangshen # 狂神第三
(integer) 2
127.0.0.1:6379> ZREVRANK salary xiaohong # 小红第一
(integer) 0
Compared with set, sorted set adds a weight parameter score, so that the elements in the set can be arranged in order according to score. For example, a sorted set that stores the grades of the whole class, the set value can be the student number of the classmate, and score It can be its test score, so that when the data is inserted into the collection, it will be naturally sorted. You can use sorted set to create a weighted queue. For example, the score of ordinary messages is 1, and the score of important messages is 2. Then the worker thread can choose to obtain the work tasks in the reverse order of the scores. Prioritize important tasks.
Ranking application, take TOP N operations!
4. Three special data types
1. GEO geographical location
Introduction
The GEO feature of Redis was launched in Redis version 3.2. This feature can store the geographical location information given by the user and operate on this information. To implement functions that rely on geographical location information such as nearby locations and shake. The data type of geo is zset.
GEO's data structure has a total of six common commands:
- geoadd
- geopos
- geodist
- georadius
- georadiusbymember
- gethash
Official document: https://www.redis.net.cn/order/3685.html
geoadd
Analysis:
# 语法
geoadd key longitude latitude member ...
# 将给定的空间元素(纬度、经度、名字)添加到指定的键里面。
# 这些数据会以有序集合的形式被储存在键里面，从而使得georadius和georadiusbymember这样的命令可以在之后通过位置查询取得这些元素。
# geoadd命令以标准的x,y格式接受参数,所以用户必须先输入经度,然后再输入纬度。
# geoadd能够记录的坐标是有限的:非常接近两极的区域无法被索引。
# 有效的经度介于-180-180度之间，有效的纬度介于-85.05112878 度至 85.05112878 度之间。，当用户尝试输入一个超出范围的经度或者纬度时,geoadd命令将返回一个错误。
Test: Baidu search longitude and latitude query, simulating real data
127.0.0.1:6379> geoadd china:city 116.23 40.22 北京
(integer) 1
127.0.0.1:6379> geoadd china:city 121.48 31.40 上海 113.88 22.55 深圳 120.21
30.20 杭州
(integer) 3
127.0.0.1:6379> geoadd china:city 106.54 29.40 重庆 108.93 34.23 西安 114.02
30.58 武汉
(integer) 3
geopos
Analysis:
# 语法
geopos key member [member...]
#从key里返回所有给定位置元素的位置（经度和纬度）
test:
127.0.0.1:6379> geopos china:city 北京
1) 1) "116.23000055551528931"
2) "40.2200010338739844"
127.0.0.1:6379> geopos china:city 上海 重庆
1) 1) "121.48000091314315796"
2) "31.40000025319353938"
2) 1) "106.54000014066696167"
2) "29.39999880018641676"
127.0.0.1:6379> geopos china:city 新疆
1) (nil)
geodist
Analysis:
# 语法
geodist key member1 member2 [unit]
# 返回两个给定位置之间的距离，如果两个位置之间的其中一个不存在,那么命令返回空值。
# 指定单位的参数unit必须是以下单位的其中一个：
# m表示单位为米
# km表示单位为千米
# mi表示单位为英里
# ft表示单位为英尺
# 如果用户没有显式地指定单位参数,那么geodist默认使用米作为单位。
#geodist命令在计算距离时会假设地球为完美的球形,在极限情况下,这一假设最大会造成0.5%的误
差。
test:
127.0.0.1:6379> geodist china:city 北京 上海
"1088785.4302"
127.0.0.1:6379> geodist china:city 北京 上海 km
"1088.7854"
127.0.0.1:6379> geodist china:city 重庆 北京 km
"1491.6716"
georadius
Analysis:
# 语法
georadius key longitude latitude radius m|km|ft|mi [withcoord][withdist]
[withhash][asc|desc][count count]
# 以给定的经纬度为中心， 找出某一半径内的元素
Test: Reconnect redis-cli and add the parameter --raw to force Chinese output, otherwise it will be garbled.
[root@kuangshen bin]# redis-cli --raw -p 6379
# 在 china:city 中寻找坐标 100 30 半径为 1000km 的城市
127.0.0.1:6379> georadius china:city 100 30 1000 km
重庆
西安
# withdist 返回位置名称和中心距离
127.0.0.1:6379> georadius china:city 100 30 1000 km withdist
重庆
635.2850
西安
963.3171
# withcoord 返回位置名称和经纬度
127.0.0.1:6379> georadius china:city 100 30 1000 km withcoord
重庆
106.54000014066696167
29.39999880018641676
西安
108.92999857664108276
34.23000121926852302
# withdist withcoord 返回位置名称 距离 和经纬度 count 限定寻找个数
127.0.0.1:6379> georadius china:city 100 30 1000 km withcoord withdist count
1
重庆
635.2850
106.54000014066696167
29.39999880018641676
127.0.0.1:6379> georadius china:city 100 30 1000 km withcoord withdist count
2
重庆
635.2850
106.54000014066696167
29.39999880018641676
西安
963.3171
108.92999857664108276
34.23000121926852302
georadiusbymember
Analysis:
# 语法
georadiusbymember key member radius m|km|ft|mi [withcoord][withdist]
[withhash][asc|desc][count count]
# 找出位于指定范围内的元素，中心点是由给定的位置元素决定
test:
127.0.0.1:6379> GEORADIUSBYMEMBER china:city 北京 1000 km
北京
西安
127.0.0.1:6379> GEORADIUSBYMEMBER china:city 上海 400 km
杭州
上海
geohash
Analysis:
# 语法
geohash key member [member...]
# Redis使用geohash将二维经纬度转换为一维字符串，字符串越长表示位置更精确,两个字符串越相似
表示距离越近。
test:
127.0.0.1:6379> geohash china:city 北京 重庆
wx4sucu47r0
wm5z22h53v0
127.0.0.1:6379> geohash china:city 北京 上海
wx4sucu47r0
wtw6sk5n300
age
GEO does not provide a command to delete members, but because the underlying implementation of GEO is zset, you can use the zrem command to delete geographical location information.
127.0.0.1:6379> geoadd china:city 116.23 40.22 beijin
1
127.0.0.1:6379> zrange china:city 0 -1 # 查看全部的元素
重庆
西安
深圳
武汉
杭州
上海
beijin
北京
127.0.0.1:6379> zrem china:city beijin # 移除元素
1
127.0.0.1:6379> zrem china:city 北京 # 移除元素
1
127.0.0.1:6379> zrange china:city 0 -1
重庆
西安
深圳
武汉
杭州
上海
2、HyperLogLog
Introduction
Redis added the HyperLogLog structure in version 2.8.9.
Redis HyperLogLog is an algorithm used for cardinality statistics. The advantage of HyperLogLog is that when the number or volume of input elements is very large, the space required to calculate the cardinality is always fixed and very small.
In Redis, each HyperLogLog key only requires 12 KB of memory to calculate the cardinality of nearly 2^64 different elements. This is in sharp contrast to a collection that consumes more memory when calculating cardinality. The more elements there are, the more memory is consumed.
HyperLogLog is an algorithm that provides an inexact deduplication counting solution.
For example: If I want to count the UV of a web page (the number of browsing users, multiple visits by the same user in a day can only be counted once), the traditional solution is to use Set to save the user ID, and then count the number of elements in the Set. Get the page UV. However, this solution can only host a small number of users. Once the number of users increases, it will consume a lot of space to store user IDs. My purpose is to count the number of users rather than save them. This is a thankless solution! HyperLogLog using Redis requires up to 12k to count a large number of users. Although it has an error rate of about 0.81%, it is negligible for counting UV data that does not require very accurate data.
What is cardinality?
For example, if the data set is {1, 3, 5, 7, 5, 7, 8}, then the cardinality set of this data set is {1, 3, 5,7, 8}, and the cardinality (non-repeating elements) is 5. Cardinality estimation is to quickly calculate the cardinality within the acceptable error range.
basic commands
| Order | describe |
|---|---|
| [PFADD key element [element …] | Adds the specified element to HyperLogLog. |
| [PFCOUNT key [key …] | Returns the cardinality estimate for the given HyperLogLog. |
| [PFMERGE destkey sourcekey [sourcekey …] | Merge multiple HyperLogLogs into one HyperLogLog and calculate the union |
test
127.0.0.1:6379> PFADD mykey a b c d e f g h i j
1
127.0.0.1:6379> PFCOUNT mykey
10
127.0.0.1:6379> PFADD mykey2 i j z x c v b n m
1
127.0.0.1:6379> PFMERGE mykey3 mykey mykey2
OK
127.0.0.1:6379> PFCOUNT mykey3
15
3、BitMap
Introduction
During development, you may encounter this situation: you need to count some information of the user, such as active or inactive, logged in or not logged in; or you need to record the user's check-in status for a year. If you check in, it is 1, if you have not checked in, it is 1. 0. If ordinary key/value storage is used, 365 records will be recorded. If the number of users is large, the space required will also be large, so Redis provides the Bitmap bitmap data structure. Bitmap operates on binary bits. To record, it is 0 and 1; if you want to record the check-in status for 365 days, the Bitmap representation is roughly as follows: 0101000111000111... What are the benefits of this? Of course, it saves memory. 365 days is equivalent to 365 bits, and 1 byte = 8 bits, so it is equivalent to using 46 bytes.
BitMap uses a bit to represent the value or status corresponding to an element. The key is the corresponding element itself. In fact, the bottom layer is also implemented through the operation of strings. Redis has added several bitmap related commands such as setbit, getbit, bitcount and so on since version 2.2.
setbit set operation
SETBIT key offset value: Set the offset bit of key to value (1 or 0)
# 使用 bitmap 来记录上述事例中一周的打卡记录如下所示：
# 周一：1，周二：0，周三：0，周四：1，周五：1，周六：0，周天：0 （1 为打卡，0 为不打卡）
127.0.0.1:6379> setbit sign 0 1
0
127.0.0.1:6379> setbit sign 1 0
0
127.0.0.1:6379> setbit sign 2 0
0
127.0.0.1:6379> setbit sign 3 1
0
127.0.0.1:6379> setbit sign 4 1
0
127.0.0.1:6379> setbit sign 5 0
0
127.0.0.1:6379> setbit sign 6 0
0
getbit Get operation
GETBIT key offset Gets the value set by offset. If it is not set, it returns 0 by default.
127.0.0.1:6379> getbit sign 3 # 查看周四是否打卡
1
127.0.0.1:6379> getbit sign 6 # 查看周七是否打卡
0
bitcount statistical operation
bitcount key [start, end] counts the number of keys with upper bit 1
# 统计这周打卡的记录，可以看到只有3天是打卡的状态：
127.0.0.1:6379> bitcount sign
3
5. Redis.conf
1. Familiar with basic configuration
Location
The Redis configuration file is located in the Redis installation directory and the file name is redis.conf
config get * # 获取全部的配置
Configuration file address:
Under normal circumstances, we will make a separate copy for operation. to ensure the security of the initial file.
Units Units
1. Configure size units. Some basic measurement units are defined at the beginning. Only bytes are supported, bits are not supported.
2. Insensitive to upper and lower case
INCLUDES contains
Similar to the Spring configuration file, it can be included through includes. redis.conf can be used as the total file and can include other files!
NETWORK network configuration
bind 127.0.0.1 # 绑定的ip
protected-mode yes # 保护模式
port 6379 # 默认端口
GENERAL General
daemonize yes # 默认情况下，Redis不作为守护进程运行。需要开启的话，改为 yes
supervised no # 可通过upstart和systemd管理Redis守护进程
pidfile /var/run/redis_6379.pid # 以后台进程方式运行redis，则需要指定pid 文件
loglevel notice # 日志级别。可选项有：
# debug（记录大量日志信息，适用于开发、测试阶段）；
# verbose（较多日志信息）；
# notice（适量日志信息，使用于生产环境）；
# warning（仅有部分重要、关键信息才会被记录）。
logfile "" # 日志文件的位置，当指定为空字符串时，为标准输出
databases 16 # 设置数据库的数目。默认的数据库是DB 0
always-show-logo yes # 是否总是显示logo
SNAPSHOPTING Snapshot
# 900秒（15分钟）内至少1个key值改变（则进行数据库保存--持久化）
save 900 1
# 300秒（5分钟）内至少10个key值改变（则进行数据库保存--持久化）
save 300 10
# 60秒（1分钟）内至少10000个key值改变（则进行数据库保存--持久化）
save 60 10000
stop-writes-on-bgsave-error yes # 持久化出现错误后，是否依然进行继续进行工作
rdbcompression yes # 使用压缩rdb文件 yes：压缩，但是需要一些cpu的消耗。no：不压缩，需要更多的磁盘空间
rdbchecksum yes # 是否校验rdb文件，更有利于文件的容错性，但是在保存rdb文件的时候，会有大概10%的性能损耗
dbfilename dump.rdb # dbfilenamerdb文件名称
dir ./ # dir 数据目录，数据库的写入会在这个目录。rdb、aof文件也会写在这个目录
REPLICATION We will talk about master-slave replication later and explain it to you! Skip here first!
SECURITYSecurity
View, set and cancel access passwords
# 启动redis
# 连接客户端
# 获得和设置密码
config get requirepass
config set requirepass "123456"
#测试ping，发现需要验证
127.0.0.1:6379> ping
NOAUTH Authentication required.
# 验证
127.0.0.1:6379> auth 123456
OK
127.0.0.1:6379> ping
PONG
limit
maxclients 10000 # 设置能连上redis的最大客户端连接数量
maxmemory <bytes> # redis配置的最大内存容量
maxmemory-policy noeviction # maxmemory-policy 内存达到上限的处理策略
#volatile-lru：利用LRU算法移除设置过过期时间的key。
#volatile-random：随机移除设置过过期时间的key。
#volatile-ttl：移除即将过期的key，根据最近过期时间来删除（辅以TTL）
#allkeys-lru：利用LRU算法移除任何key。
#allkeys-random：随机移除任何key。
#noeviction：不移除任何key，只是返回一个写错误。
append only mode
appendonly no # 是否以append only模式作为持久化方式，默认使用的是rdb方式持久化，这种方式在许多应用中已经足够用了
appendfilename "appendonly.aof" # appendfilename AOF 文件名称
appendfsync everysec # appendfsync aof持久化策略的配置
# no表示不执行fsync，由操作系统保证数据同步到磁盘，速度最快。
# always表示每次写入都执行fsync，以保证数据同步到磁盘。
# everysec表示每秒执行一次fsync，可能会导致丢失这1s数据。
We will explain the details later when we explain the persistence configuration of Redis! First understand, listen to the ear!
2. Introduction to common configurations
1. Redis does not run as a daemon process by default. You can modify this configuration item and use yes to enable the daemon process.
daemonize no
2. When Redis runs in daemon mode, Redis will write the pid to the /var/run/redis.pid file by default, which can be specified through pidfile
pidfile /var/run/redis.pid
3. Specify the Redis listening port. The default port is 6379. The author explained in one of his blog posts why 6379 was chosen as the default port because 6379 is the number corresponding to MERZ on the phone button, and MERZ is taken from the name of the Italian singer Alessia Merz.
port 6379
4. The bound host address
bind 127.0.0.1
5. When the client is idle for a long time, the connection is closed. If it is specified as 0, it means that this function is turned off.
timeout 300
6. Specify the logging level. Redis supports a total of four levels: debug, verbose, notice, and warning. The default is verbose.
loglevel verbose
7. Logging mode, the default is standard output. If Redis is configured to run in daemon mode, and the logging mode is configured as standard output, the log will be sent to /dev/null.
logfile stdout
8. Set the number of databases. The default database is 0. You can use the SELECT command to specify the database id on the connection.
databases 16
9. Specify the period of time and the number of update operations to synchronize the data to the data file. Multiple conditions can be matched
Three conditions are provided in the save Redis default configuration file:
save 900 1
save 300 10
save 60 10000
respectively means 1 change within 900 seconds (15 minutes), 10 changes within 300 seconds (5 minutes) and 60 seconds. There are 10,000 changes.
10. Specify whether to compress the data when storing it in the local database. The default is yes. Redis uses LZF compression. If you want to save CPU time, you can turn off this option, but it will cause the database file to become huge.
rdbcompression yes
11. Specify the local database file name. The default value is dump.rdb.
dbfilename dump.rdb
12. Specify the local database storage directory
dir ./
13. Set when the local machine serves the slav service, set the IP address and port of the master service. When Redis starts, it will automatically synchronize data from the master.
slaveof
14. When the master service is password protected, the password for the slav service to connect to the master
masterauth
15. Set the Redis connection password. If the connection password is configured, the client needs to provide the password through the AUTH command when connecting to Redis. It is closed by default.
requirepass foobared
16. Set the maximum number of client connections at the same time. The default is unlimited. The number of client connections that Redis can open at the same time is the maximum number of file descriptors that the Redis process can open. If maxclients 0 is set, it means there is no limit. When the number of client connections reaches the limit, Redis will close the new connection and return the max number of clients reached error message to the client.
maxclients 128
17. Specify the maximum memory limit of Redis. Redis will load data into the memory when it starts. After reaching the maximum memory, Redis will first try to clear the keys that have expired or are about to expire. After this method is processed, the maximum memory is still reached. settings, writes will no longer be possible, but reads will still be possible. Redis' new vm mechanism will store Key in memory and Value in swap area.
maxmemory
18. Specify whether to perform logging after each update operation. Redis writes data to disk asynchronously by default. If it is not enabled, data may be lost for a period of time during a power outage. Because redis's own synchronized data files are synchronized according to the above save conditions, some data will only exist in memory for a period of time. Default is no
appendonly no
19. Specify the update log file name, the default is appendonly.aof
appendfilename appendonly.aof
20. Specify the update log conditions. There are 3 optional values:
no: means waiting for the operating system to synchronize the data cache to the disk (fast)
always: means manually calling fsync() to write the data to the disk after each update operation (slow, safe)
everysec: means synchronizing once per second (compromise, default value) )
appendfsync everysec
21. Specify whether to enable the virtual memory mechanism. The default value is no. To give a brief introduction, the VM mechanism stores data in pages. Redis swaps the pages with less access, that is, cold data, to the disk. The pages with more access are automatically transferred to the disk. Swap out to memory (I will carefully analyze the VM mechanism of Redis in a later article)
vm-enabled no
22. Virtual memory file path, the default value is /tmp/redis.swap, which cannot be shared by multiple Redis instances.
vm-swap-file /tmp/redis.swap
23. Store all data larger than vm-max-memory in virtual memory. No matter how small the vm-max-memory setting is, all index data is stored in memory (the index data of Redis is keys). In other words, when vm When -max-memory is set to 0, all values actually exist on the disk. The default value is 0
vm-max-memory 0
24. The Redis swap file is divided into many pages. One object can be saved on multiple pages, but one page cannot be shared by multiple objects. The vm-page-size is set according to the size of the stored data. Author It is recommended that if you store many small objects, the page size should be set to 32 or 64 bytes; if you store very large objects, you can use a larger page. If you are not sure, use the default value.
vm-page-size 32
25. Set the number of pages in the swap file. Since the page table (a bitmap indicating that a page is free or used) is placed in memory, every 8 pages on the disk will consume 1 byte of memory.
vm-pages 134217728
26. Set the number of threads to access the swap file. It is best not to exceed the number of cores of the machine. If it is set to 0, then all operations on the swap file will be serial, which may cause a long delay. The default value is 4
vm-max-threads 4
27. Set whether to combine smaller packets into one packet and send it when responding to the client. The default is enabled.
glueoutputbuf yes
28. Specify that a special hash algorithm is used when a certain number is exceeded or the largest element exceeds a certain critical value.
hash-max-zipmap-entries 64
hash-max-zipmap-value 512
29. Specify whether to activate reset hashing. The default is on (details will be introduced later when introducing the Redis hash algorithm)
activerehashing yes
30. By specifying other configuration files, you can use the same configuration file between multiple Redis instances on the same host, and each instance has its own specific configuration file.
include /path/to/local.conf
6. Redis persistence
Redis is an in-memory database. If the database state in memory is not saved to disk, once the server process exits, the database state in the server will also disappear. So Redis provides persistence functionality!
1、RDB（Redis DataBase）
What is RDB
Write a snapshot of the data set in the memory to the disk within a specified time interval, which is also called a Snapshot in the jargon. When it is restored, the snapshot file is read directly into the memory.
Redis will create (fork) a separate sub-process for persistence. It will first write the data to a temporary file. After the persistence process is completed, this temporary file will be used to replace the last persisted file. During the entire process, the main process does not perform any IO operations. This ensures extremely high performance. If large-scale data recovery is required and the integrity of data recovery is not very sensitive, the RDB method is more efficient than the AOF method. The disadvantage of RDB is that data after the last persistence may be lost.
Fork
The function of Fork is to copy a process that is the same as the current process. All the data (variables, environment variables, program counter, etc.) of the new process have the same values as the original process, but it is a completely new process and serves as a child process of the original process.
Rdb saves the dump.rdb file
Configuration location and SNAPSHOTTING analysis
We can modify and test the trigger condition mechanism here:
save 120 10 # 120秒内修改10次则触发RDB
RDB is a compressed Snapshot that integrates memory. The data structure of RDB can configure compound snapshot triggering conditions.
default:
- Changed 10,000 times in 1 minute
- Changed 10 times in 5 minutes
- Changed once in 15 minutes
If you want to disable the RDB persistence strategy, just don't set any save instructions, or pass an empty string parameter to save.
If the modification needs to take effect immediately, you can manually use the save command! Effective immediately!
Parsing the rest of the commands
Stop-writes-on-bgsave-error: If configured to no, it means that you do not care about data inconsistency or have other means to detect and control it. The default is yes.
rbdcompression: For snapshots stored on disk, you can set whether to compress and store them. If so, redis will use the LZF algorithm for compression. If you don't want to consume CPU for compression, you can set this feature to turn off.
rdbchecksum: After storing the snapshot, you can also let redis use the CRC64 algorithm for data verification, but doing so will increase performance consumption by about 10%. If you want to get the maximum performance improvement, you can turn off this function. The default is yes.
How to trigger RDB snapshot
1. The default snapshot configuration in the configuration file. It is recommended to use an extra machine as a backup and copy dump.rdb.
2. Command save or bgsave
- When saving, just save, ignore other things, and block everything.
- bgsave, Redis will perform snapshot operations asynchronously in the background, and the snapshot can also respond to client requests. You can use the lastsave command to obtain the time of the last successful snapshot execution.
3. Executing the flushall command will also generate the dump.rdb file, but it is empty and meaningless!
4. The dump.rdb file will also be generated when exiting!
How to restore
1. Move the backup file (dump.rdb) to the redis installation directory and start the service.
2. CONFIG GET dir to get the directory
127.0.0.1:6379> config get dir
dir
/usr/local/bin
pros and cons
Advantages :
1. Suitable for large-scale data recovery
2. Low requirements for data integrity and consistency
Disadvantages :
1. Make a backup at a certain interval, so if redis accidentally goes down, all modifications after the last snapshot will be lost.
2. When Fork, the data in the memory is cloned, and roughly 2 times the expansion needs to be considered.
summary
2、AOF（Append Only File）
what is
Each write operation is recorded in the form of a log, and all instructions executed by Redis are recorded (read operations are not recorded). Only files can be appended but not rewritten. When redis is started, the file will be read to reconstruct the data. In other words, when redis is restarted, the write instructions will be executed from front to back according to the contents of the log file to complete the data recovery work.
Aof saves the appendonly.aof file
Configuration
appendonly no # 是否以append only模式作为持久化方式，默认使用的是rdb方式持久化，这种方式在许多应用中已经足够用了
appendfilename "appendonly.aof" # appendfilename AOF 文件名称
appendfsync everysec # appendfsync aof持久化策略的配置
# no表示不执行fsync，由操作系统保证数据同步到磁盘，速度最快。
# always表示每次写入都执行fsync，以保证数据同步到磁盘。
# everysec表示每秒执行一次fsync，可能会导致丢失这1s数据。
No-appendfsync-on-rewrite #重写时是否可以运用Appendfsync，用默认no即可，保证数据安全性
Auto-aof-rewrite-min-size # 设置重写的基准值
Auto-aof-rewrite-percentage #设置重写的基准值
AOF startup/repair/recovery
Normal recovery:
- Startup: Set Yes, modify the default appendonly no, and change it to yes
- Copy the aof file with data and save it to the corresponding directory (config get dir)
- Recovery: restart redis and reload
Exception recovery:
- Startup: Set Yes
- Deliberate destruction of the appendonly.aof file!
- Fix: redis-check-aof --fix appendonly.aof to fix
- Recovery: restart redis and reload
Rewrite
What is:
AOF uses file appending, and the file will become larger and larger. To avoid this situation, a new rewriting mechanism is added. When the size of the AOF file exceeds the set threshold, Redis will start the content compression of the AOF file. Only keep the minimum set of instructions that can recover data, you can use the command bgrewriteaof!
Rewriting principle:
When the AOF file continues to grow and becomes too large, a new process will be forked to rewrite the file (the temporary file is written first and then renamed
), and the data in the memory of the new process is traversed. Each record has a Set statement. The operation of rewriting the AOF file does not read the old AOF file. This is somewhat similar to a snapshot!
Trigger mechanism:
Redis will record the AOF size when it was last rewritten. The default configuration is to trigger when the AOF file size is the size after the last rewrite and the file is larger than 64M.
Once an expert makes a move, there will only be no success. Experts will look at the door, while laymen will only look at the fun.
pros and cons
advantage:
1. Synchronize every modification: appendfsync always synchronizes persistence. Every time a data change occurs, it will be immediately recorded to the disk. The performance is poor but the data integrity is better.
2. Synchronization every second: appendfsync everysec asynchronous operation, recorded every second. If the machine goes down within one second, data will be lost.
3. Not synchronized: appendfsync no never synchronizes
shortcoming:
1. For the same data set, aof files are much larger than rdb files, and the recovery speed is slower than rdb.
2. The running efficiency of Aof is slower than that of RDB, the synchronization strategy per second is more efficient, and the out-of-synchronization efficiency is the same as RDB.
Summary
3. Summary
1. RDB persistence method can snapshot and store your data within a specified time interval.
2. AOF persistence mode records each write operation to the server. When the server restarts, these commands will be re-executed to restore the original data. The AOF command uses the Redis protocol to append and save each write operation to the end of the file. Redis can also Background rewriting of AOF files to prevent the size of AOF files from being too large.
3. Only cache. If you only want your data to exist when the server is running, you can also not use any persistence.
4. Enable two persistence methods at the same time
- 在这种情况下，当redis重启的时候会优先载入AOF文件来恢复原始的数据，因为在通常情况下AOF文件保存的数据集要比RDB文件保存的数据集要完整。
- RDB 的数据不实时，同时使用两者时服务器重启也只会找AOF文件，那要不要只使用AOF呢？作者建议不要，因为RDB更适合用于备份数据库（AOF在不断变化不好备份），快速重启，而且不会有AOF可能潜在的Bug，留着作为一个万一的手段。
5、性能建议
- 因为RDB文件只用作后备用途，建议只在Slave上持久化RDB文件，而且只要15分钟备份一次就够了，只保留 save 900 1 这条规则。
- 如果Enable AOF ，好处是在最恶劣情况下也只会丢失不超过两秒数据，启动脚本较简单只load自己的AOF文件就可以了，代价一是带来了持续的IO，二是AOF rewrite 的最后将 rewrite 过程中产生的新数据写到新文件造成的阻塞几乎是不可避免的。只要硬盘许可，应该尽量减少AOF rewrite 的频率，AOF重写的基础大小默认值64M太小了，可以设到5G以上，默认超过原大小100%大小重写可以改到适当的数值。
- 如果不Enable AOF ，仅靠 Master-Slave Repllcation 实现高可用性也可以，能省掉一大笔IO，也减少了rewrite时带来的系统波动。代价是如果Master/Slave 同时倒掉，会丢失十几分钟的数据，启动脚本也要比较两个 Master/Slave 中的 RDB文件，载入较新的那个，微博就是这种架构。
七、Redis事务
1、理论
Redis事务的概念：
Redis 事务的本质是一组命令的集合。事务支持一次执行多个命令，一个事务中所有命令都会被序列化。在事务执行过程，会按照顺序串行化执行队列中的命令，其他客户端提交的命令请求不会插入到事务执行命令序列中。
总结说：redis事务就是一次性、顺序性、排他性的执行一个队列中的一系列命令。
Redis事务没有隔离级别的概念：
批量操作在发送 EXEC 命令前被放入队列缓存，并不会被实际执行！
Redis does not guarantee atomicity:
In Redis, a single command is executed atomically, but transactions are not guaranteed to be atomic and there is no rollback. If any command in the transaction fails to execute, the remaining commands will still be executed.
Three stages of Redis transactions:
- start transaction
- command to queue
- Execute transaction
Redis transaction related commands:
watch key1 key2 ... #监视一或多个key,如果在事务执行之前，被监视的key被其他命令改动，则事务被打断 （ 类似乐观锁 ）
multi # 标记一个事务块的开始（ queued ）
exec # 执行所有事务块的命令 （ 一旦执行exec后，之前加的监控锁都会被取消掉 ）
discard # 取消事务，放弃事务块中的所有命令
unwatch # 取消watch对所有key的监控
2. Practice
Normal execution
abandon transaction
If there is a command error in the transaction queue (similar to a Java compilation error), all commands will not be executed when the EXEC command is executed.
If there is a syntax error in the transaction queue (similar to Java's 1/0 runtime exception), when the EXEC command is executed, other correct commands will be executed, and the wrong command will throw an exception.
Watch monitoring
Pessimistic Lock:
Pessimistic Lock, as the name suggests, is very pessimistic. Every time you go to get the data, you think that others will modify it, so you will lock it every time you get the data, so that if others want to get the data, they will block until it gets the lock. Many such lock mechanisms are used in traditional relational databases, such as row locks, table locks, read locks, write locks, etc., which are all locked before operations.
Optimistic locking:
Optimistic Lock, as the name suggests, is very optimistic. Every time you go to get the data, you think that others will not modify it, so you will not lock it. However, when updating, it will be judged whether others have updated the data during this period. You can use mechanisms such as version numbers. Optimistic locking is suitable for multi-read application types, which can improve throughput. Optimistic locking strategy: The submitted version must be greater than The current version must be recorded in order to perform updates.
test:
1. Initialize the available balance and outstanding balance of the credit card
127.0.0.1:6379> set balance 100
OK
127.0.0.1:6379> set debt 0
OK
2. Use watch to detect balance. The balance data does not change during the transaction and the transaction is executed successfully.
127.0.0.1:6379> watch balance
OK
127.0.0.1:6379> MULTI
OK
127.0.0.1:6379> decrby balance 20
QUEUED
127.0.0.1:6379> incrby debt 20
QUEUED
127.0.0.1:6379> exec
1) (integer) 80
2) (integer) 20
3. Use watch to detect balance. The balance data changes during the transaction and the transaction execution fails!
# 窗口一
127.0.0.1:6379> watch balance
OK
127.0.0.1:6379> MULTI # 执行完毕后，执行窗口二代码测试
OK
127.0.0.1:6379> decrby balance 20
QUEUED
127.0.0.1:6379> incrby debt 20
QUEUED
127.0.0.1:6379> exec # 修改失败！
(nil)
# 窗口二
127.0.0.1:6379> get balance
"80"
127.0.0.1:6379> set balance 200
OK
# 窗口一：出现问题后放弃监视，然后重来！
127.0.0.1:6379> UNWATCH # 放弃监视
OK
127.0.0.1:6379> watch balance
OK
127.0.0.1:6379> MULTI
OK
127.0.0.1:6379> decrby balance 20
QUEUED
127.0.0.1:6379> incrby debt 20
QUEUED
127.0.0.1:6379> exec # 成功！
1) (integer) 180
2) (integer) 40
illustrate:
Once EXEC is executed to start transaction execution, WARCH's monitoring of variables will be canceled regardless of whether the transaction is executed successfully.
Therefore, when the transaction execution fails, you need to re-execute the WATCH command to monitor the variables and start a new transaction for operation.
3. Summary
The watch instruction is similar to optimistic locking. When the transaction is committed, if the value of any KEY among the multiple KEYs monitored by the watch has been changed by other clients, when EXEC is used to execute the transaction, the transaction queue will not be executed and Nullmulti- will be returned. Bulk response to notify the caller that transaction execution failed.
8. Redis publish and subscribe
what is
Redis publish and subscribe (pub/sub) is a message communication model: the sender (pub) sends messages and the subscribers (sub) receive messages.
Redis clients can subscribe to any number of channels.
Subscribe/publish message graph:
The following figure shows the relationship between channel channel1 and the three clients that subscribe to this channel - client2, client5 and client1:
When a new message is sent to channel channel1 through the PUBLISH command, the message will be sent to the three clients that subscribe to it:
Order
These commands are widely used to build instant messaging applications, such as online chat rooms, real-time broadcasts, real-time reminders, etc.
test
The following example demonstrates how publish-subscribe works. In our example we create a subscription channel named redisChat :
redis 127.0.0.1:6379> SUBSCRIBE redisChat
Reading messages... (press Ctrl-C to quit)
1) "subscribe"
2) "redisChat"
3) (integer) 1
Now, we first restart a redis client, and then publish the message twice on the same channel redisChat, so that subscribers can receive the message.
redis 127.0.0.1:6379> PUBLISH redisChat "Hello,Redis"
(integer) 1
redis 127.0.0.1:6379> PUBLISH redisChat "Hello，Kuangshen"
(integer) 1
# 订阅者的客户端会显示如下消息
1) "message"
2) "redisChat"
3) "Hello,Redis"
1) "message"
2) "redisChat"
3) "Hello，Kuangshen"
principle
Redis is implemented in C. By analyzing the pubsub.c file in the Redis source code, we can understand the underlying implementation of the publishing and subscription mechanism, and thereby deepen our understanding of Redis.
Redis implements publishing and subscribing functions through commands such as PUBLISH, SUBSCRIBE and PSUBSCRIBE.
After subscribing to a channel through the SUBSCRIBE command, a dictionary is maintained in redis-server. The key of the dictionary is each channel, and the value of the dictionary is a linked list. The linked list stores all clients that subscribe to this channel. The key to the SUBSCRIBE command is to add the client to the subscription list of a given channel.
To send messages to subscribers through the PUBLISH command, redis-server will use the given channel as a key, search the linked list of all clients that subscribe to this channel in the channel dictionary it maintains, traverse this linked list, and publish the message to all subscriber.
Pub/Sub literally means Publish and Subscribe. In Redis, you can set up message publishing and message subscription for a certain key value. When a message is published on a key value, all Clients that subscribe to it will receive corresponding messages. The most obvious use of this function is as a real-time messaging system, such as ordinary instant chat, group chat and other functions.
scenes to be used
Pub/Sub builds a real-time messaging system
Redis's Pub/Sub system can build a real-time messaging system
, such as many examples of real-time chat systems built with Pub/Sub.
9. Redis master-slave replication
1. Concept
Master-slave replication refers to copying data from one Redis server to other Redis servers. The former is called the master node (master/leader), and the latter is called the slave node (slave/follower); data replication is one-way and can only be from the master node to the slave node. Master is mainly for writing, and Slave is mainly for reading.
By default, each Redis server is a master node; and a master node can have multiple slave nodes (or no slave nodes), but a slave node can only have one master node.
The main functions of master-slave replication include:
1. Data redundancy: Master-slave replication realizes hot backup of data, which is a data redundancy method in addition to persistence.
2. Fault recovery: When a problem occurs on the master node, the slave node can provide services to achieve rapid fault recovery; it is actually a kind of service redundancy.
3. Load balancing: Based on master-slave replication, combined with read-write separation, the master node can provide write services and the slave nodes can provide read services (that is, when writing Redis data, the application connects to the master node, and when reading Redis data, the application connects to the slave node) node) to share the server load; especially in scenarios where there is less writing and more reading, sharing the read load through multiple slave nodes can greatly increase the concurrency of the Redis server.
4. The cornerstone of high availability: In addition to the above functions, master-slave replication is also the basis for the implementation of sentinels and clusters. Therefore, master-slave replication is the basis of Redis high availability.
Generally speaking, to use Redis in engineering projects, it is absolutely impossible to use only one Redis for the following reasons:
1. Structurally, a single Redis server will have a single point of failure, and one server needs to handle all request loads, which puts great pressure;
2. In terms of capacity, the memory capacity of a single Redis server is limited. Even if the memory capacity of a Redis server is 256G, all the memory cannot be used as Redis storage memory. Generally speaking, the maximum memory used by a single Redis server should not exceed 20G.
Products on e-commerce websites are generally uploaded once and viewed countless times. To put it professionally, it means "read more and write less".
For this scenario, we can use the following architecture:
2. Environment configuration
basic configuration
With the slave library but not the master library, the slave library configuration:
slaveof 主库ip 主库端口 # 配置主从
Info replication # 查看信息
Every time you disconnect from the master, you need to reconnect, unless you configure it in the redis.conf file!
Modify the configuration file!
Preparation work: We configure master-slave replication, at least three are required, one master and two slaves! Configure three clients!
1. Copy multiple redis.conf files
2. Specify port 6379, and so on.
3. Turn on daemonize yes
4. The name of the Pid file is pidfile /var/run/redis_6379.pid, and so on.
5. Log file name logfile “6379.log”, and so on.
6. Dump.rdb name dbfilename dump6379.rdb, and so on.
After the above configuration is completed and the three services are started through three different configuration files, our preparation environment is OK!
3. One master and two slaves
One master and two servants
1. Environment initialization
By default, all three are Master nodes.
2. Configure as one Master and two Slaves
3. The values set on the host machine can be obtained on the slave machine! The slave cannot write value!
Test 1: The host hangs up, check the slave information, the host recovers, and check the information again
Test 2: The slave machine hangs up, check the host information, the slave machine recovers, and check the slave machine information
layer-by-layer links
The previous Slave can be the next slave and Master. The Slave can also receive connection and synchronization requests from other slaves. Then the slave serves as the next master in the chain, which can effectively reduce the write pressure of the master!
Test: After setting the value of 6379, both 6380 and 6381 can be obtained! OK!
Conspiracy to usurp the throne
In the case of one master and two slaves, if the master is disconnected, the slave can use the command SLAVEOF NO ONE to change itself to the master! At this time the remaining slaves are linked to this node. Executing the command SLAVEOF NO ONE on a slave server will cause the slave server to turn off the replication function and transition from the slave server back to the master server. The original synchronized data set will not be discarded.
When the master machine comes back, it will be just a polished commander, and the slave machines will run to the new master machine for normal use!
Copy principle
After the Slave starts and successfully connects to the master, it will send a sync command.
After receiving the command, the master starts the background save process and collects all received commands for modifying the data set. After the background process is completed, the master will transfer the entire data file to the slave and complete a complete synchronization.
Full copy: After receiving the database file data, the slave service saves it and loads it into memory.
Incremental replication: Master continues to pass all new collected modification commands to slave in order to complete synchronization.
But as long as the master is reconnected, a full synchronization (full replication) will be automatically performed.
4. Sentry mode
Overview
The method of master-slave switching technology is: when the master server goes down, you need to manually switch a slave server to the master server, which requires manual intervention, is time-consuming and laborious, and will also cause the service to be unavailable for a period of time. This is not a recommended approach, more often we give priority to Sentry mode. Redis has officially provided the Sentinel architecture since 2.8 to solve this problem.
The automatic version that seeks to usurp the throne can monitor whether the host fails in the background. If it fails, it will automatically switch from the slave database to the main database based on the number of votes.
Sentinel mode is a special mode. First, Redis provides sentinel commands. Sentinel is an independent process. As a process, it will run independently. The principle is that the sentinel monitors multiple running Redis instances by sending commands and waiting for the Redis server to respond .
The sentry here has two functions
- By sending commands, the Redis server returns to monitor its running status, including the master server and slave servers.
- 当哨兵监测到master宕机，会自动将slave切换成master，然后通过发布订阅模式通知其他的从服务器，修改配置文件，让它们切换主机。
然而一个哨兵进程对Redis服务器进行监控，可能会出现问题，为此，我们可以使用多个哨兵进行监控。各个哨兵之间还会进行监控，这样就形成了多哨兵模式。
假设主服务器宕机，哨兵1先检测到这个结果，系统并不会马上进行failover过程，仅仅是哨兵1主观的认为主服务器不可用，这个现象成为主观下线。当后面的哨兵也检测到主服务器不可用，并且数量达到一定值时，那么哨兵之间就会进行一次投票，投票的结果由一个哨兵发起，进行failover[故障转移]操作。切换成功后，就会通过发布订阅模式，让各个哨兵把自己监控的从服务器实现切换主机，这个过程称为客观下线。
配置测试
1、调整结构，6379带着80、81
2、自定义的 /myredis 目录下新建 sentinel.conf 文件，名字千万不要错
3、配置哨兵，填写内容
- sentinel monitor 被监控主机名字 127.0.0.1 6379 1
- 上面最后一个数字1，表示主机挂掉后slave投票看让谁接替成为主机，得票数多少后成为主机
4、启动哨兵
- Redis-sentinel /myredis/sentinel.conf
- 上述目录依照各自的实际情况配置，可能目录不同
5、正常主从演示
6、原有的Master 挂了
7、投票新选
8、重新主从继续开工，info replication 查查看
9、问题：如果之前的master 重启回来，会不会双master 冲突？ 之前的回来只能做小弟了
哨兵模式的优缺点
优点
- 哨兵集群模式是基于主从模式的，所有主从的优点，哨兵模式同样具有。
- 主从可以切换，故障可以转移，系统可用性更好。
- 哨兵模式是主从模式的升级，系统更健壮，可用性更高。
缺点
- Redis较难支持在线扩容，在集群容量达到上限时在线扩容会变得很复杂。
- The configuration to implement sentinel mode is not simple, it can even be said to be a bit cumbersome.
Sentinel configuration instructions
# Example sentinel.conf
# 哨兵sentinel实例运行的端口 默认26379
port 26379
# 哨兵sentinel的工作目录
dir /tmp
# 哨兵sentinel监控的redis主节点的 ip port
# master-name 可以自己命名的主节点名字 只能由字母A-z、数字0-9 、这三个字符".-_"组成。
# quorum 配置多少个sentinel哨兵统一认为master主节点失联 那么这时客观上认为主节点失联了
# sentinel monitor <master-name> <ip> <redis-port> <quorum>
sentinel monitor mymaster 127.0.0.1 6379 2
# 当在Redis实例中开启了requirepass foobared 授权密码 这样所有连接Redis实例的客户端都
要提供密码
# 设置哨兵sentinel 连接主从的密码 注意必须为主从设置一样的验证密码
# sentinel auth-pass <master-name> <password>
sentinel auth-pass mymaster MySUPER--secret-0123passw0rd
# 指定多少毫秒之后 主节点没有应答哨兵sentinel 此时 哨兵主观上认为主节点下线 默认30秒
# sentinel down-after-milliseconds <master-name> <milliseconds>
sentinel down-after-milliseconds mymaster 30000
# 这个配置项指定了在发生failover主备切换时最多可以有多少个slave同时对新的master进行 同
步，这个数字越小，完成failover所需的时间就越长，
但是如果这个数字越大，就意味着越 多的slave因为replication而不可用。
可以通过将这个值设为 1 来保证每次只有一个slave 处于不能处理命令请求的状态。
# sentinel parallel-syncs <master-name> <numslaves>
sentinel parallel-syncs mymaster 1
# 故障转移的超时时间 failover-timeout 可以用在以下这些方面：
#1. 同一个sentinel对同一个master两次failover之间的间隔时间。
#2. 当一个slave从一个错误的master那里同步数据开始计算时间。直到slave被纠正为向正确的
master那里同步数据时。
#3.当想要取消一个正在进行的failover所需要的时间。
#4.当进行failover时，配置所有slaves指向新的master所需的最大时间。不过，即使过了这个超
时，slaves依然会被正确配置为指向master，但是就不按parallel-syncs所配置的规则来了
# 默认三分钟
# sentinel failover-timeout <master-name> <milliseconds>
sentinel failover-timeout mymaster 180000
# SCRIPTS EXECUTION
#配置当某一事件发生时所需要执行的脚本，可以通过脚本来通知管理员，例如当系统运行不正常时发邮
件通知相关人员。
#对于脚本的运行结果有以下规则：
#若脚本执行后返回1，那么该脚本稍后将会被再次执行，重复次数目前默认为10
#若脚本执行后返回2，或者比2更高的一个返回值，脚本将不会重复执行。
#如果脚本在执行过程中由于收到系统中断信号被终止了，则同返回值为1时的行为相同。
#一个脚本的最大执行时间为60s，如果超过这个时间，脚本将会被一个SIGKILL信号终止，之后重新执
行。
#通知型脚本:当sentinel有任何警告级别的事件发生时（比如说redis实例的主观失效和客观失效等
等），将会去调用这个脚本，这时这个脚本应该通过邮件，SMS等方式去通知系统管理员关于系统不正常
运行的信息。调用该脚本时，将传给脚本两个参数，一个是事件的类型，一个是事件的描述。如果
sentinel.conf配置文件中配置了这个脚本路径，那么必须保证这个脚本存在于这个路径，并且是可执
行的，否则sentinel无法正常启动成功。
#通知脚本
# sentinel notification-script <master-name> <script-path>
sentinel notification-script mymaster /var/redis/notify.sh
# 客户端重新配置主节点参数脚本
# 当一个master由于failover而发生改变时，这个脚本将会被调用，通知相关的客户端关于master
地址已经发生改变的信息。
# 以下参数将会在调用脚本时传给脚本:
# <master-name> <role> <state> <from-ip> <from-port> <to-ip> <to-port>
# 目前<state>总是“failover”,
# <role>是“leader”或者“observer”中的一个。
# 参数 from-ip, from-port, to-ip, to-port是用来和旧的master和新的master(即旧的
slave)通信的
# 这个脚本应该是通用的，能被多次调用，不是针对性的。
# sentinel client-reconfig-script <master-name> <script-path>
sentinel client-reconfig-script mymaster /var/redis/reconfig.sh
10. Cache penetration and avalanche
The use of Redis cache greatly improves the performance and efficiency of applications, especially in data query. But at the same time, it also brings some problems. Among them, the most critical issue is the consistency of data. Strictly speaking, this problem has no solution. If the consistency requirements of the data are very high, then caching cannot be used.
Other typical problems are cache penetration, cache avalanche and cache breakdown. At present, the industry also has relatively popular solutions.
1. Cache penetration
concept
The concept of cache penetration is very simple. The user wants to query a piece of data and finds that the redis memory database does not have it, that is, the cache does not hit, so it queries the persistence layer database. It was found that there was none, so this query failed. When there are many users, the cache does not hit, so they all request the persistence layer database. This will put a lot of pressure on the persistence layer database, which is equivalent to cache penetration.
solution
bloom filter
Bloom filter is a data structure that stores all possible query parameters in hash form. It is first verified at the control layer and discarded if they do not match, thus avoiding query pressure on the underlying storage system;
Cache empty objects
When the storage layer misses, even the returned empty object will be cached, and an expiration time will be set. Accessing the data later will be obtained from the cache, protecting the back-end data source;
But there are two problems with this approach:
1. If null values can be cached, this means that the cache requires more space to store more keys, because there may be many null value keys;
2. Even if the expiration time is set for the null value, the data in the cache layer and the storage layer will still be inconsistent for a period of time, which will have an impact on businesses that need to maintain consistency.
2. Cache breakdown
Overview
Here you need to pay attention to the difference from cache breakdown. Cache breakdown means that a key is very hot and is constantly carrying large concurrency. Large concurrency focuses on accessing this point. When the key fails at the moment, it continues to Large concurrency breaks through the cache and directly requests the database, which is like cutting a hole in a barrier.
When a key expires, there are a large number of requests for concurrent access. This type of data is generally hot data. Because the cache expires, the database will be accessed at the same time to query the latest data, and the cache will be written back, which will cause excessive pressure on the database in an instant. .
solution
Set hotspot data to never expire
From a cache perspective, there is no expiration time set, so there will be no problems caused by the hotspot key expiration.
Add mutex lock
Distributed lock: Use distributed lock to ensure that only one thread queries the back-end service for each key at the same time. Other threads do not have permission to obtain the distributed lock, so they only need to wait. This method transfers the pressure of high concurrency to distributed locks, so it poses a great challenge to distributed locks.
3. Cache avalanche
concept
Cache avalanche refers to the centralized expiration of caches in a certain period of time.
One of the reasons for the avalanche is that when I write this article, it is almost midnight on Double 12, and there will soon be a wave of panic buying. This wave of products will be put into the cache in a relatively concentrated time. Assume that the cache is for one hour. . Then at one o'clock in the morning, the cache of this batch of products has expired. The access queries for this batch of products all fall on the database, which will generate periodic pressure peaks for the database. As a result, all requests will reach the storage layer, and the number of calls to the storage layer will increase dramatically, causing the storage layer to also hang up.
In fact, centralized expiration is not very fatal. A more fatal cache avalanche is when a node of the cache server goes down or is disconnected from the network. Because of the naturally occurring cache avalanche, caches must be created intensively during a certain period of time. At this time, the database can also withstand the pressure. It's nothing more than periodic pressure on the database. The downtime of the cache service node will put unpredictable pressure on the database server, and it is very likely that the database will be overwhelmed in an instant.
solution
redis high availability
The meaning of this idea is that since redis may fail, then I add a few more redis so that the others can continue to work after one fails. In fact, it is a cluster.
Current limiting downgrade
The idea of this solution is to control the number of threads that read the database and write the cache through locks or queues after the cache expires. For example, only one thread is allowed to query data and write cache for a certain key, while other threads wait.
Data warm-up
The meaning of data heating is that before formal deployment, I first access the possible data in advance, so that some data that may be accessed in large quantities will be loaded into the cache. Manually trigger the loading of different keys in the cache before large concurrent access occurs, and set different expiration times to make the cache invalidation time points as even as possible.
11. Jedis
Jedis is the officially recommended Java connection development tool for Redis. To use Redis middleware well in Java development, you must be familiar with Jedis to write beautiful code.
1. Test China Unicom
1. Create a new ordinary Maven project
2. Import redis dependencies!
<!-- https://mvnrepository.com/artifact/redis.clients/jedis -->
<dependency>
<groupId>redis.clients</groupId>
<artifactId>jedis</artifactId>
<version>3.2.0</version>
</dependency>
<dependency>
<groupId>com.alibaba</groupId>
<artifactId>fastjson</artifactId>
<version>1.2.58</version>
</dependency>
3. Write test code
package com.kuang.ping;
import redis.clients.jedis.Jedis;
public class Ping {
public static void main(String[] args) {
Jedis jedis = new Jedis("127.0.0.1",6379);
System.out.println("连接成功");
//查看服务是否运行
System.out.println("服务正在运行: "+jedis.ping());
}
}
4. Start the redis service
5. Start the test and the results
连接成功
服务正在运行: PONG
2. Commonly used APIs
Basic operations
public class TestPassword {
public static void main(String[] args) {
Jedis jedis = new Jedis("127.0.0.1", 6379);
//验证密码，如果没有设置密码这段代码省略
// jedis.auth("password");
jedis.connect(); //连接
jedis.disconnect(); //断开连接
jedis.flushAll(); //清空所有的key
}
}
Commands for key operations
public class TestKey {
public static void main(String[] args) {
Jedis jedis = new Jedis("127.0.0.1", 6379);
System.out.println("清空数据："+jedis.flushDB());
System.out.println("判断某个键是否存在："+jedis.exists("username"));
System.out.println("新增<'username','kuangshen'>的键值对："+jedis.set("username", "kuangshen"));
System.out.println("新增<'password','password'>的键值对："+jedis.set("password", "password"));
System.out.print("系统中所有的键如下：");
Set<String> keys = jedis.keys("*");
System.out.println(keys);
System.out.println("删除键password:"+jedis.del("password"));
System.out.println("判断键password是否存在："+jedis.exists("password"));
System.out.println("查看键username所存储的值的类型："+jedis.type("username"));
System.out.println("随机返回key空间的一个："+jedis.randomKey());
System.out.println("重命名key："+jedis.rename("username","name"));
System.out.println("取出改后的name："+jedis.get("name"));
System.out.println("按索引查询："+jedis.select(0));
System.out.println("删除当前选择数据库中的所有key："+jedis.flushDB());
System.out.println("返回当前数据库中key的数目："+jedis.dbSize());
System.out.println("删除所有数据库中的所有key："+jedis.flushAll());
}
}
Commands for String operations
public class TestString {
public static void main(String[] args) {
Jedis jedis = new Jedis("127.0.0.1", 6379);
jedis.flushDB();
System.out.println("===========增加数据===========");
System.out.println(jedis.set("key1","value1"));
System.out.println(jedis.set("key2","value2"));
System.out.println(jedis.set("key3", "value3"));
System.out.println("删除键key2:"+jedis.del("key2"));
System.out.println("获取键key2:"+jedis.get("key2"));
System.out.println("修改key1:"+jedis.set("key1", "value1Changed"));
System.out.println("获取key1的值："+jedis.get("key1"));
System.out.println("在key3后面加入值："+jedis.append("key3", "End"));
System.out.println("key3的值："+jedis.get("key3"));
System.out.println("增加多个键值对："+jedis.mset("key01","value01","key02","value02","key03","value03"));
System.out.println("获取多个键值对："+jedis.mget("key01","key02","key03"));
System.out.println("获取多个键值对："+jedis.mget("key01","key02","key03","key04"));
System.out.println("删除多个键值对："+jedis.del("key01","key02"));
System.out.println("获取多个键值对："+jedis.mget("key01","key02","key03"));
jedis.flushDB();
System.out.println("===========新增键值对防止覆盖原先值==============");
System.out.println(jedis.setnx("key1", "value1"));
System.out.println(jedis.setnx("key2", "value2"));
System.out.println(jedis.setnx("key2", "value2-new"));
System.out.println(jedis.get("key1"));
System.out.println(jedis.get("key2"));
System.out.println("===========新增键值对并设置有效时间=============");
System.out.println(jedis.setex("key3", 2, "value3"));
System.out.println(jedis.get("key3"));
try {
TimeUnit.SECONDS.sleep(3);
} catch (InterruptedException e) {
e.printStackTrace();
}
System.out.println(jedis.get("key3"));
System.out.println("===========获取原值，更新为新值==========");
System.out.println(jedis.getSet("key2", "key2GetSet"));
System.out.println(jedis.get("key2"));
System.out.println("获得key2的值的字串："+jedis.getrange("key2", 2,4));
}
}
List operation commands
public class TestList {
public static void main(String[] args) {
Jedis jedis = new Jedis("127.0.0.1", 6379);
jedis.flushDB();
System.out.println("===========添加一个list===========");
jedis.lpush("collections", "ArrayList", "Vector", "Stack","HashMap", "WeakHashMap", "LinkedHashMap");
jedis.lpush("collections", "HashSet");
jedis.lpush("collections", "TreeSet");
jedis.lpush("collections", "TreeMap");
System.out.println("collections的内容："+jedis.lrange("collections",0, -1));//-1代表倒数第一个元素，-2代表倒数第二个元素,end为-1表示查询全部
System.out.println("collections区间0-3的元素："+jedis.lrange("collections",0,3));
System.out.println("===============================");
// 删除列表指定的值 ，第二个参数为删除的个数（有重复时），后add进去的值先被删，类似于出栈
System.out.println("删除指定元素个数："+jedis.lrem("collections", 2,"HashMap"));
System.out.println("collections的内容："+jedis.lrange("collections",0, -1));
System.out.println("删除下表0-3区间之外的元素："+jedis.ltrim("collections", 0, 3));
System.out.println("collections的内容："+jedis.lrange("collections",0, -1));
System.out.println("collections列表出栈（左端）："+jedis.lpop("collections"));
System.out.println("collections的内容："+jedis.lrange("collections",0, -1));
System.out.println("collections添加元素，从列表右端，与lpush相对应："+jedis.rpush("collections", "EnumMap"));
System.out.println("collections的内容："+jedis.lrange("collections",0, -1));
System.out.println("collections列表出栈（右端）："+jedis.rpop("collections"));
System.out.println("collections的内容："+jedis.lrange("collections",0, -1));
System.out.println("修改collections指定下标1的内容："+jedis.lset("collections", 1, "LinkedArrayList"));
System.out.println("collections的内容："+jedis.lrange("collections",0, -1));
System.out.println("===============================");
System.out.println("collections的长度："+jedis.llen("collections"));
System.out.println("获取collections下标为2的元素："+jedis.lindex("collections", 2));
System.out.println("===============================");
jedis.lpush("sortedList", "3","6","2","0","7","4");
System.out.println("sortedList排序前："+jedis.lrange("sortedList", 0,-1));
System.out.println(jedis.sort("sortedList"));
System.out.println("sortedList排序后："+jedis.lrange("sortedList", 0,-1));
}
}
Operation commands for Set
public class TestSet {
public static void main(String[] args) {
Jedis jedis = new Jedis("127.0.0.1", 6379);
jedis.flushDB();
System.out.println("============向集合中添加元素（不重复）============");
System.out.println(jedis.sadd("eleSet","e1","e2","e4","e3","e0","e8","e7","e5"));
System.out.println(jedis.sadd("eleSet", "e6"));
System.out.println(jedis.sadd("eleSet", "e6"));
System.out.println("eleSet的所有元素为："+jedis.smembers("eleSet"));
System.out.println("删除一个元素e0："+jedis.srem("eleSet", "e0"));
System.out.println("eleSet的所有元素为："+jedis.smembers("eleSet"));
System.out.println("删除两个元素e7和e6："+jedis.srem("eleSet","e7","e6"));
System.out.println("eleSet的所有元素为："+jedis.smembers("eleSet"));
System.out.println("随机的移除集合中的一个元素："+jedis.spop("eleSet"));
System.out.println("随机的移除集合中的一个元素："+jedis.spop("eleSet"));
System.out.println("eleSet的所有元素为："+jedis.smembers("eleSet"));
System.out.println("eleSet中包含元素的个数："+jedis.scard("eleSet"));
System.out.println("e3是否在eleSet中："+jedis.sismember("eleSet","e3"));
System.out.println("e1是否在eleSet中："+jedis.sismember("eleSet","e1"));
System.out.println("e1是否在eleSet中："+jedis.sismember("eleSet","e5"));
System.out.println("=================================");
System.out.println(jedis.sadd("eleSet1","e1","e2","e4","e3","e0","e8","e7","e5"));
System.out.println(jedis.sadd("eleSet2","e1","e2","e4","e3","e0","e8"));
System.out.println("将eleSet1中删除e1并存入eleSet3中："+jedis.smove("eleSet1", "eleSet3", "e1"));//移到集合元素
System.out.println("将eleSet1中删除e2并存入eleSet3中："+jedis.smove("eleSet1", "eleSet3", "e2"));
System.out.println("eleSet1中的元素："+jedis.smembers("eleSet1"));
System.out.println("eleSet3中的元素："+jedis.smembers("eleSet3"));
System.out.println("============集合运算=================");
System.out.println("eleSet1中的元素："+jedis.smembers("eleSet1"));
System.out.println("eleSet2中的元素："+jedis.smembers("eleSet2"));
System.out.println("eleSet1和eleSet2的交集:"+jedis.sinter("eleSet1","eleSet2"));
System.out.println("eleSet1和eleSet2的并集:"+jedis.sunion("eleSet1","eleSet2"));
System.out.println("eleSet1和eleSet2的差集:"+jedis.sdiff("eleSet1","eleSet2"));//eleSet1中有，eleSet2中没有
jedis.sinterstore("eleSet4","eleSet1","eleSet2");//求交集并将交集保存到dstkey的集合
System.out.println("eleSet4中的元素："+jedis.smembers("eleSet4"));
}
}
Operation commands for Hash
public class TestHash {
public static void main(String[] args) {
Jedis jedis = new Jedis("127.0.0.1", 6379);
jedis.flushDB();
Map<String,String> map = new HashMap<>();
map.put("key1","value1");
map.put("key2","value2");
map.put("key3","value3");
map.put("key4","value4");
//添加名称为hash（key）的hash元素
jedis.hmset("hash",map);
//向名称为hash的hash中添加key为key5，value为value5元素
jedis.hset("hash", "key5", "value5");
System.out.println("散列hash的所有键值对为："+jedis.hgetAll("hash"));//return Map<String,String>
System.out.println("散列hash的所有键为："+jedis.hkeys("hash"));//returnSet<String>
System.out.println("散列hash的所有值为："+jedis.hvals("hash"));//returnList<String>
System.out.println("将key6保存的值加上一个整数，如果key6不存在则添加key6："+jedis.hincrBy("hash", "key6", 6));
System.out.println("散列hash的所有键值对为："+jedis.hgetAll("hash"));
System.out.println("将key6保存的值加上一个整数，如果key6不存在则添加key6："+jedis.hincrBy("hash", "key6", 3));
System.out.println("散列hash的所有键值对为："+jedis.hgetAll("hash"));
System.out.println("删除一个或者多个键值对："+jedis.hdel("hash","key2"));
System.out.println("散列hash的所有键值对为："+jedis.hgetAll("hash"));
System.out.println("散列hash中键值对的个数："+jedis.hlen("hash"));
System.out.println("判断hash中是否存在key2："+jedis.hexists("hash","key2"));
System.out.println("判断hash中是否存在key3："+jedis.hexists("hash","key3"));
System.out.println("获取hash中的值："+jedis.hmget("hash","key3"));
System.out.println("获取hash中的值："+jedis.hmget("hash","key3","key4"));
}
}
3. Affairs
Basic operations
package com.kuang.multi;
import com.alibaba.fastjson.JSONObject;
import redis.clients.jedis.Jedis;
import redis.clients.jedis.Transaction;
public class TestMulti {
public static void main(String[] args) {
//创建客户端连接服务端，redis服务端需要被开启
Jedis jedis = new Jedis("127.0.0.1", 6379);
jedis.flushDB();
JSONObject jsonObject = new JSONObject();
jsonObject.put("hello", "world");
jsonObject.put("name", "java");
//开启事务
Transaction multi = jedis.multi();
String result = jsonObject.toJSONString();
try{
//向redis存入一条数据
multi.set("json", result);
//再存入一条数据
multi.set("json2", result);
//这里引发了异常，用0作为被除数
int i = 100/0;
//如果没有引发异常，执行进入队列的命令
multi.exec();
}catch(Exception e){
e.printStackTrace();
//如果出现异常，回滚
multi.discard();
}finally{
System.out.println(jedis.get("json"));
System.out.println(jedis.get("json2"));
//最终关闭客户端
jedis.close();
}
}
}
12. SpringBoot integration
1. Basic use
Overview
In SpringBoot, the methods provided by RedisTemplate are generally used to operate Redis. So what are the steps required to integrate Redis using SpringBoot.
1. JedisPoolConfig (this is to configure the connection pool)
2. RedisConnectionFactory This is to configure connection information. RedisConnectionFactory here is an interface. We need to use its implementation class. The following four factory models are provided in the SpringD Data Redis solution:
- JredisConnectionFactory
- JedisConnectionFactory
- LettuceConnectionFactory
- SrpConnectionFactory
3. RedisTemplate basic operations
Import dependencies
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-data-redis</artifactId>
</dependency>
yaml configuration
spring:
redis:
host: 127.0.0.1
port: 6379
password: 123456
jedis:
pool:
max-active: 8
max-wait: -1ms
max-idle: 500
min-idle: 0
lettuce:
shutdown-timeout: 0ms
test
@SpringBootTest
class SpringbootRedisApplicationTests {
@Autowired
private RedisTemplate<String,String> redisTemplate;
@Test
void contextLoads() {
redisTemplate.opsForValue().set("myKey","myValue");
System.out.println(redisTemplate.opsForValue().get("myKey"));
}
}
2. Packaging tools
1. Create a new SpringBoot project
2. Import the redis launcher
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-data-redis</artifactId>
</dependency>
3. Configure redis, you can view the RedisProperties analysis
# Redis服务器地址
spring.redis.host=127.0.0.1
# Redis服务器连接端口
spring.redis.port=6379
4. Analyze the RedisAutoConfiguration automatic configuration class
@Configuration(proxyBeanMethods = false)
@ConditionalOnClass(RedisOperations.class)
@EnableConfigurationProperties(RedisProperties.class)
@Import({
LettuceConnectionConfiguration.class,JedisConnectionConfiguration.class })
public class RedisAutoConfiguration {
@Bean
@ConditionalOnMissingBean(name = "redisTemplate")
public RedisTemplate<Object, Object> redisTemplate(RedisConnectionFactory redisConnectionFactory) throws UnknownHostException {
RedisTemplate<Object, Object> template = new RedisTemplate<>();
template.setConnectionFactory(redisConnectionFactory);
return template;
}
@Bean
@ConditionalOnMissingBean
public StringRedisTemplate stringRedisTemplate(RedisConnectionFactory redisConnectionFactory) throws UnknownHostException {
StringRedisTemplate template = new StringRedisTemplate();
template.setConnectionFactory(redisConnectionFactory);
return template;
}
}
As can be seen from the source code, SpringBoot automatically generates a RedisTemplate and a
StringRedisTemplate for us in the container.
However, the generic type of this RedisTemplate is <Object, Object>, which is inconvenient to write code and requires writing a lot of type conversion code; we need a RedisTemplate whose generic type is in the form of <String, Object>.
Moreover, this RedisTemplate does not set the serialization method of key and value when data is stored in Redis.
After seeing this @ConditionalOnMissingBean annotation, you know that if there is a RedisTemplate object in the Spring container, this automatically configured RedisTemplate will not be instantiated. Therefore, we can directly write a configuration class ourselves to configure RedisTemplate.
5. Since automatic configuration is not easy to use, reconfigure a RedisTemplate.
package com.kuang.config;
import com.fasterxml.jackson.annotation.JsonAutoDetect;
import com.fasterxml.jackson.annotation.PropertyAccessor;
import com.fasterxml.jackson.databind.ObjectMapper;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.data.redis.connection.RedisConnectionFactory;
import org.springframework.data.redis.core.RedisTemplate;
import
org.springframework.data.redis.serializer.Jackson2JsonRedisSerializer;
import org.springframework.data.redis.serializer.StringRedisSerializer;
@Configuration
public class RedisConfig {
@Bean
@SuppressWarnings("all")