From my point of view, this article is a little rough. After all, it doesn't take much time. As an entry point, it may not enter the door?

Redis is a very powerful database. I once read his source code and marveled at the monotony of his bottom layer, but the variability of his top layer. This is in response to the sentence "Tao generates one, life two, two generates three, and three generates all things" (of course, it is not comparable to my favorite operating system, shouting "Linux yyds!").

As the title shows, this blog is mainly about application. Of course, if you want to study the source code, I also have it here.

Let's briefly introduce Redis. As we all know, Redis is a single threaded database written in C language. How does this single thread process requests? Every call from Redis client to server goes through three processes: sending command, executing command and returning result. In the command execution phase, because Redis processes commands in a single thread, all commands arriving at the server will not be executed immediately, and all commands will enter a queue and be executed one by one. And the execution order of commands sent by multiple clients is uncertain. However, it is certain that no two commands will be executed at the same time, and there will be no concurrency problem

Why can Redis be executed quickly?

• Most requests are purely memory operations (very fast)
• Single thread is adopted to avoid unnecessary context switching and competition conditions
• For non blocking io-io multiplexing, Redis uses epoll as the implementation of I/O multiplexing technology. In addition, Redis's own event processing model converts the connection, read-write and shutdown in epoll into time, so as not to waste too much time on I/O

Redis

Pre knowledge

What is Redis?

Redis (remote dictionary server), i.e. remote dictionary service!

It is an open source log and key value database written in ANSl C language, supporting network, memory based and persistent, and provides API s in multiple languages. Free and open source! It is one of the most popular NoSQL technologies at present! Also known as structured database!

redis installation

$ wget https://download.redis.io/releases/redis-6.2.6.tar.gz
$ tar xzf redis-6.2.6.tar.gz
$ cd redis-6.2.6
$ make

Open client

src/redis-cli

Open the server

src/redis-server

Test connection

ping

performance testing

src/redis-benchmark

Basic command

select 3 #Switch the database. The 0 is used by default
dbsize	# View current db size
keys *	# View all key s in the database
flushdb	# clear database 

Why is the redis port 6379?

Phone number corresponding to Italian singer's name (fan effect)

Understand that redis is fast. Redis is based on memory operation. CPU is not the performance bottleneck of redis. The bottleneck of redis is based on the memory and network bandwidth of the machine. Since single thread can be used, single thread can be used.

redis is written in C language. The official data provided is 100000 + QPS, which is no worse than memcache using key value

Basic command

# Generate key value data
set mykey myvalue
# Get value according to key
get mykey
# Does a key exist
exists mykey
# Remove a key from a database
move mykey 1
# Set expiration time
expire mykey second
# View the remaining time of a mykey
ttl mykey
# Determine the data type of the current key
type mykey

data type

string

Implementation: String

# Add a string to a key
append mykey "hello"
# Get the length of the key
strlen mykey
# Self increasing
incr mykey
# Self subtraction
desc mykey
# Increase a certain amount
incrby mykey 10
# Reduce a certain amount
desc mykey 10
# Obtain the interval value, and both sides are closed intervals
getrange mykey startindex endindex
# Intercept the string and reassign it
setrange mykey startindex endindex
# Set the expiration time when defining data
setex mykey 30 "hello"
# Set when the key does not exist (if the key already exists, an error will be reported)
setnx mykey "hello"
# Batch setting value
mset k1 v1 k2 v2
# Batch get value
gset k1 k2
# Set when the batch does not exist (if the key already exists, an error will be reported). The atomic operation either succeeds or fails at the same time
msetnx k1 v1 k2 v2
# get first and then set. If it does not exist, nil is returned
getset db redis

Usage scenario

• timer
• Count multi unit quantity

list

Implementation: bidirectional linked list and jump list

# Insert a value, which is placed on the left by default
lpush mylist one
lpush mylist two
lpush mylist three
# Insert a value and place it on the right
rpush mylist four
# Get range in list
lrange mylist 0 -1
# Remove an element
lpop # Remove the element on the left
rpop # Remove the element on the right
# Get the value by subscript, starting from the left by default
lindex mylist 0
# Returns the length of the list
llen mylist
# Remove n value s
lrem mylist n value
# Intercept some elements and re assign values
ltrim mylist 1 2	# Intercept the first element to the second element
# Remove the last element on the right and insert it to the far left
rpoplpush
# Replacing the fixed subscript value in the list with another value is equivalent to an update operation
lset mylist 1 hello
# Insert value before an element
linsert mylist before "world" "value"
# Insert value after an element
linsert mylist after "world" "value"

set

Implementation: integer set and hash table

# Add element
sadd myset "hello"
# View all values
smembers myset
# Include an element
sismember myset "hello"
# Get the number of elements
scard myset
# Remove a value
srem myset "hello"
# Randomly filter n members. The default is one
srandmember myset n
# Randomly remove elements
spop myset
# Move value to another set
smove myset1 myset2 myvalue
# Difference set
sdiff myset1 myset2
# Find intersection
sinter myset1 myset2
# Union set
sunion myset1 myset2

hashtable

Implementation: compressed list and hash table

# Add element
hset mymap k1 v1
# Get element
hset mymap k1
# Batch add
hset mymap k1 v1 k2 v2
# Batch acquisition
hget mymap k1 k2
# Get all key values
hgetall mymap
# Delete element
hdel mymap k1
# Get the number of elements
hlen mymap
# Judge whether it exists
hexists mymap k1
#Get all key s
hkeys mymap
# Get all value s
hvals mymap
# val = val + n
hincrby mymap k1 n
# val = val - n 
hdescby mymap k1 n
# Add an element. If it is repeated, an error will be reported
hsetnx mymap k1 v1

Applicable to common: it is suitable for information storage with frequent changes of user information

zset

Implementation: compressed list and jump table

# Add an element to which you want to add a numeric value for sorting
zadd myset 1 one
# Gets the value of a range
zrange myset 0 -1
# sort
zrangebyscore myset -inf +inf	# Negative infinity to positive infinity
zrangebyscore myset -inf 2300	# Negative infinity to 2500
# Remove an element
zrem myset k1
# View the number of collections
zcard myset
# Sort from large to small
zrevrange myset 0 -1
# Gets the number of members in the specified interval
zcount myset n m

Redis transaction

Redis transaction essence: a collection of commands! All commands in a transaction will be serialized and executed in sequence during transaction execution! One time, sequential and exclusive! Execute some column commands

• Redis transactions do not have the concept of isolation level
• All commands are not executed directly in a transaction! It will be executed only when the execution command is initiated
• ExecRedis saves atomicity in a single command, but the transaction does not guarantee atomicity

Redis transactions

# Open transaction
multi
# Exit transaction
exec
# Cancel transaction
discard

Compiled exception

There is a problem with the code, and the commands in the transaction will not be executed

Runtime exception IO

If syntax exists in the transaction queue, other commands can be executed normally when the command is executed, and an exception is thrown for the wrong command!

monitor

Pessimistic lock: very pessimistic. I think there will be problems at any time. No matter what I do, I will lock it

Optimistic lock: I am optimistic that there will be no problem at any time. No matter what I do, I will lock it. When updating the data, judge whether anyone has modified the data during this period,

• Get version
• Compare version when updating

# Monitor an element
watch money
# Open transaction
multi
incrby money 20
descby out 20
# Exit transaction
exec
# Cancel monitoring
unwatch

Redis.conf

bind 127.0.0.1 # Available addresses for accessing the database
protected-mode yes # Protection mode
port xxxx# Open port

General configuration

daemonize yes # Daemons are on by default
pidfile # If running in background mode, specify the pid run file
loglevel notice # log information
logfile # File location name of the log
databases # Number of databases
always-show-logo yes # Always show logo

snapshot

# If at least one 1 key has been modified within 900s, we will perform the persistence operation
save 900 1
# If at least 10 key s are modified within 300s, we can perform persistence operation
save 300 10
# If at least 10000 key s are modified within 60s, we will perform persistence operation
save 60 1000

stop-writes-on-bgsave-error yes # If persistence fails, do you want to continue working

rdbcompression yes # Whether to compress rdb files requires some CPU resources

security

# Get redis password
config get requirepass
# Set redis password
config set requirepass "xxx"
# Login with password
auth xx

limit

# Set the maximum number of clients that can connect to redis
maxclients xxx
# Data deletion policy

Persistence

Redis is an in memory database. If the database state in memory is not saved to disk, the database state in the server will disappear once the server process exits. So redis provides persistence

RDB

redis database

Redis will be created separately (fork) a child process will first write data to a temporary file. After the persistence process is completed, the temporary file will be used to replace the last persistent file. During the whole process, the main process will not perform any IO operations. This ensures extremely high performance. If large-scale data recovery is required, and data recovery is very important Integrity is not very sensitive, so RDB is more efficient than AOF. The disadvantage of RDB is that the data may be lost after the last persistence.

save

advantage:

• Suitable for large-scale data recovery
• The integrity of the data is not high

Disadvantages:

• A certain time interval is required for process operation! If redis goes down unexpectedly, there is no need to modify the data for the last time
• The fork process will occupy a certain content space

AOF

Each write operation is recorded in the form of a log. All instructions executed by redis are recorded (read operations are not recorded). Only files can be added, but files cannot be overwritten. Redis will read the file and rebuild the data at the beginning of startup. In other words, if redis restarts, the write instructions will be executed from front to back according to the contents of the Tianzhi file to complete the data recovery

Advantages:

• Each modification is synchronized, and the integrity of the file will be better
• Sync once per second, and you may lose one second of data
• Never synchronized, most efficient

Disadvantages:

• Compared with data files, aof is much larger than rdb, and the repair speed is also slower than rdb!
• aof also runs slower than rdb, so the default configuration of redis is rdb persistence
• By default, aof is the unlimited addition of files, which will become larger and larger

If the aof file is larger than 64m, it is too large! fork a new process to rewrite our file

Redis subscription publication

Redis publish / subscribe (pub/sub) is a message communication mode: the sender (pub) sends messages and the subscriber (sub) receives messages.

Subscription and publication message graph: message sender, message receiver and channel

[the external chain image transfer fails. The source station may have an anti-theft chain mechanism. It is recommended to save the image and upload it directly (img-68hpxtts-1637155483492) (C: \ users \ ASUS \ appdata \ roaming \ typora \ user images \ image-20211117190653564. PNG)]

# Subscribe to one or more channels that match the given pattern.
psubscribe pattern [pattern...]
# View subscription and publishing system status.
pubsub subcommand [argument]
# Sends information to the specified channel.
publish channel message
# Unsubscribe from all channels in a given mode.
punsubscribe [pattern...]
# Subscribe to information about a given channel or channels.
subscribe channel [channel...]
# Unsubscribe from a given channel.
unsubcribe [channel...] 

Usage scenario

• Real time message system
• Real time chat
• subscribe

More complex when using message oriented middleware

Master slave replication

Master-slave replication, read-write separation! 80% of the cases are read operations! Reduce the pressure on the server! Often used in the architecture! One master and two slaves!

Master-slave replication refers to copying data from one Redis server to other Redis servers. The former is called master / leader and the latter is called slave
(slave/follower); data replication is one-way, and can only be from the Master node to the Slave node. The Master mainly writes and the Slave mainly reads.

By default, each Redis server is the master node; and - a master node can have multiple slave nodes (or no slave nodes), but - a slave node can only have one master node.

The main functions of master-slave replication include:

• Data redundancy: master-slave replication realizes the hot backup of data, which is a data redundancy method other than persistence.
• Fault recovery: when the master node has a problem, the slave node can provide services to achieve rapid fault recovery; in fact, it is a kind of service redundancy.
• Load balancing: on the basis of master-slave replication, combined with read-write separation, the master node can provide write services and the slave node can provide read services (that is, the application connects the master node when writing Redis data and the slave node when reading Redis data) , sharing the server load; especially in the scenario of less writing and more reading, sharing the read load through multiple slave nodes can greatly improve the concurrency of Redis server.
• High availability cornerstone: in addition to the above functions, master-slave replication is also the basis for sentinels and clusters to implement. Therefore, master-slave replication is the basis for Redis high availability.

# Print master-slave copy information
info replication

Copy the three configuration files and modify the corresponding information

• port
• pid name
• log file name
• dump.rdb name

By default, each redis is a master. If you want to implement master-slave replication, only the slave is configured

# Transient configuration slave
slaveof host port
# Permanent configuration slave

The master cannot write, and the slave cannot write

principle

• After Slave is successfully started and connected to the master, it will send a sync synchronization command
• After receiving the command, the master starts the background save process and collects all received commands for modifying the dataset. After the background process is executed, the master will transfer the data
• The entire data file is to the slave, and a full synchronization is completed.
• Full copy: after receiving the database file data, the slave service saves it and loads it into memory.
• Incremental replication: the Master continues to transmit all new collected modification commands to the slave in turn to complete the synchronization
• However, as long as the master is reconnected, a full synchronization (full replication) will be performed automatically

If the host is disconnected, we can use SLAVEOF no one to turn ourselves into a host! Other nodes can manually connect to the latest master node (hand)
Move)! If the boss fixes it at this time, reconnect it!

Sentinel mode

See sentinel.conf file for specific configuration

sentinel monitor myredis host port 1 

The following number 1 means that the host hangs up. slave votes to see who takes over as the host. The one with the most votes will become the host!

Start the sentry

redis-sentinel sentinel.conf

advantage:

• Sentinel cluster is based on master-slave replication mode. It has all the advantages of master-slave configuration
• The master-slave can be switched and the fault can be transferred, so the system availability will be better
• Sentinel mode is the upgrade of master-slave mode. It is more robust from manual to automatic!

Disadvantages:

• Redis is not good. Online capacity expansion is very troublesome once the cluster capacity reaches the upper limit!
• The configuration of sentinel mode is actually very troublesome. There are many choices!

Cache penetration and avalanche

The use of Redis cache greatly improves the performance and efficiency of applications, especially in data query. But at the same time, it also brings some - problems. Among them, the most crucial
The problem is the consistency of data. Strictly speaking, this problem has no solution. If data consistency is required, caching cannot be used.

Cache penetration

The concept of cache penetration is very simple. Users want to query one data by one and find that the redis memory database does not hit, that is, the cache does not hit, so they query the persistence layer database. No, so this query failed. When there are many users, the cache misses (second kill!), so they all request the persistence layer database. This will put a lot of pressure on the persistence layer database, which is equivalent to cache penetration.

Solution:

• Bloom filter
• The cache layer is set to null

Bloom filter: bloom filter is a data structure that stores all possible query parameters in the form of hash. It is verified at the control layer first, and discarded if it does not meet the requirements, so as to avoid the query pressure on the underlying storage system;

However, there are two problems with this method:

• If null values can be cached, it means that the cache needs more space to store more keys, because there may be many null keys;
• Even if the expiration time is set for a null value, there will still be inconsistency between the data of the cache layer and the storage layer for a period of time, which will have an impact on the business that needs to maintain consistency.

Buffer breakdown

Here, we need to pay attention to the difference between cache breakdown and cache breakdown. Cache breakdown means that a key is very hot. It is constantly carrying large concurrency. Large concurrency focuses on accessing these points. When the key fails, the continuous large concurrency will break through the cache and directly request the database, just like cutting a hole in a barrier.

When a key expires, a large number of requests are accessed concurrently. This kind of data is generally hot data. Because the cache expires, the database will be accessed at the same time to query the latest data
According to, and write back to the cache will lead to excessive pressure on the database.

Solution:

• Set hotspot data never to expire
• Add mutex

The difference between penetration and breakdown: penetration cannot be found, the amount of breakdown is too large, and the cache expires

Cache avalanche

Cache avalanche means that the cache set expires in a certain period of time. Redis is down!

One of the reasons for the avalanche. For example, when writing this article, it is about to arrive at double twelve o'clock, and there will be a wave of rush buying soon. This wave of goods will be put into the cache for a concentrated time. Suppose the cache is one hour. Then, at o'clock in the morning, the cache of these goods will expire. The access and query of these commodities fall on the database, which will produce periodic pressure peaks. Therefore, all requests will reach the storage layer, and the call volume of the storage layer will increase sharply, resulting in the storage layer hanging up.

In fact, centralized expiration is not very fatal. The more fatal cache avalanche is the downtime or disconnection of a node of the cache server. Because of the naturally formed cache avalanche, - the cache must be created centrally in a certain period of time. At this time, the database can withstand the pressure. It is nothing more than periodic pressure on the database. The downtime of the cache service node will cause unpredictable pressure on the database server, which is likely to crush the database in an instant.

Solution:

• Redis high availability: this idea means that since redis may hang up, I will add several more redis. After one is hung up, others can continue to work. In fact, it is a cluster.
• Current limiting degradation:
• The idea of this solution is to control the number of threads reading and writing to the database cache by locking or queuing after the cache expires. For example, for a key, only one thread is allowed to query data and write cache, while other threads wait.
• Data preheating: Data heating means that before formal deployment, I first access the possible data in advance, so that some data that may be accessed in large quantities will be loaded into the cache. Before a large concurrent access is about to occur, manually trigger the loading of different cache key s and set different expiration times to make the time point of cache invalidation as uniform as possible.