From my point of view, this article is a little rough. After all, it doesn't take much time. As an entry point, it may not enter the door?
Redis is a very powerful database. I once read his source code and marveled at the monotony of his bottom layer, but the variability of his top layer. This is in response to the sentence "Tao generates one, life two, two generates three, and three generates all things" (of course, it is not comparable to my favorite operating system, shouting "Linux yyds!").
As the title shows, this blog is mainly about application. Of course, if you want to study the source code, I also have it here.
Let's briefly introduce Redis. As we all know, Redis is a single threaded database written in C language. How does this single thread process requests? Every call from Redis client to server goes through three processes: sending command, executing command and returning result. In the command execution phase, because Redis processes commands in a single thread, all commands arriving at the server will not be executed immediately, and all commands will enter a queue and be executed one by one. And the execution order of commands sent by multiple clients is uncertain. However, it is certain that no two commands will be executed at the same time, and there will be no concurrency problem
Why can Redis be executed quickly?
- Most requests are purely memory operations (very fast)
- Single thread is adopted to avoid unnecessary context switching and competition conditions
- For non blocking io-io multiplexing, Redis uses epoll as the implementation of I/O multiplexing technology. In addition, Redis's own event processing model converts the connection, read-write and shutdown in epoll into time, so as not to waste too much time on I/O
What is Redis?
Redis (remote dictionary server), i.e. remote dictionary service!
It is an open source log and key value database written in ANSl C language, supporting network, memory based and persistent, and provides API s in multiple languages. Free and open source! It is one of the most popular NoSQL technologies at present! Also known as structured database!
$ wget https://download.redis.io/releases/redis-6.2.6.tar.gz $ tar xzf redis-6.2.6.tar.gz $ cd redis-6.2.6 $ make
Open the server
select 3 #Switch the database. The 0 is used by default dbsize # View current db size keys * # View all key s in the database flushdb # clear database
Why is the redis port 6379?
Phone number corresponding to Italian singer's name (fan effect)
Understand that redis is fast. Redis is based on memory operation. CPU is not the performance bottleneck of redis. The bottleneck of redis is based on the memory and network bandwidth of the machine. Since single thread can be used, single thread can be used.
redis is written in C language. The official data provided is 100000 + QPS, which is no worse than memcache using key value
# Generate key value data set mykey myvalue # Get value according to key get mykey # Does a key exist exists mykey # Remove a key from a database move mykey 1 # Set expiration time expire mykey second # View the remaining time of a mykey ttl mykey # Determine the data type of the current key type mykey
# Add a string to a key append mykey "hello" # Get the length of the key strlen mykey # Self increasing incr mykey # Self subtraction desc mykey # Increase a certain amount incrby mykey 10 # Reduce a certain amount desc mykey 10 # Obtain the interval value, and both sides are closed intervals getrange mykey startindex endindex # Intercept the string and reassign it setrange mykey startindex endindex # Set the expiration time when defining data setex mykey 30 "hello" # Set when the key does not exist (if the key already exists, an error will be reported) setnx mykey "hello" # Batch setting value mset k1 v1 k2 v2 # Batch get value gset k1 k2 # Set when the batch does not exist (if the key already exists, an error will be reported). The atomic operation either succeeds or fails at the same time msetnx k1 v1 k2 v2 # get first and then set. If it does not exist, nil is returned getset db redis
- Count multi unit quantity
Implementation: bidirectional linked list and jump list
# Insert a value, which is placed on the left by default lpush mylist one lpush mylist two lpush mylist three # Insert a value and place it on the right rpush mylist four # Get range in list lrange mylist 0 -1 # Remove an element lpop # Remove the element on the left rpop # Remove the element on the right # Get the value by subscript, starting from the left by default lindex mylist 0 # Returns the length of the list llen mylist # Remove n value s lrem mylist n value # Intercept some elements and re assign values ltrim mylist 1 2 # Intercept the first element to the second element # Remove the last element on the right and insert it to the far left rpoplpush # Replacing the fixed subscript value in the list with another value is equivalent to an update operation lset mylist 1 hello # Insert value before an element linsert mylist before "world" "value" # Insert value after an element linsert mylist after "world" "value"
Implementation: integer set and hash table
# Add element sadd myset "hello" # View all values smembers myset # Include an element sismember myset "hello" # Get the number of elements scard myset # Remove a value srem myset "hello" # Randomly filter n members. The default is one srandmember myset n # Randomly remove elements spop myset # Move value to another set smove myset1 myset2 myvalue # Difference set sdiff myset1 myset2 # Find intersection sinter myset1 myset2 # Union set sunion myset1 myset2
Implementation: compressed list and hash table
# Add element hset mymap k1 v1 # Get element hset mymap k1 # Batch add hset mymap k1 v1 k2 v2 # Batch acquisition hget mymap k1 k2 # Get all key values hgetall mymap # Delete element hdel mymap k1 # Get the number of elements hlen mymap # Judge whether it exists hexists mymap k1 #Get all key s hkeys mymap # Get all value s hvals mymap # val = val + n hincrby mymap k1 n # val = val - n hdescby mymap k1 n # Add an element. If it is repeated, an error will be reported hsetnx mymap k1 v1
Applicable to common: it is suitable for information storage with frequent changes of user information
Implementation: compressed list and jump table
# Add an element to which you want to add a numeric value for sorting zadd myset 1 one # Gets the value of a range zrange myset 0 -1 # sort zrangebyscore myset -inf +inf # Negative infinity to positive infinity zrangebyscore myset -inf 2300 # Negative infinity to 2500 # Remove an element zrem myset k1 # View the number of collections zcard myset # Sort from large to small zrevrange myset 0 -1 # Gets the number of members in the specified interval zcount myset n m
Redis transaction essence: a collection of commands! All commands in a transaction will be serialized and executed in sequence during transaction execution! One time, sequential and exclusive! Execute some column commands
- Redis transactions do not have the concept of isolation level
- All commands are not executed directly in a transaction! It will be executed only when the execution command is initiated
- ExecRedis saves atomicity in a single command, but the transaction does not guarantee atomicity
# Open transaction multi # Exit transaction exec # Cancel transaction discard
There is a problem with the code, and the commands in the transaction will not be executed
Runtime exception IO
If syntax exists in the transaction queue, other commands can be executed normally when the command is executed, and an exception is thrown for the wrong command!
Pessimistic lock: very pessimistic. I think there will be problems at any time. No matter what I do, I will lock it
Optimistic lock: I am optimistic that there will be no problem at any time. No matter what I do, I will lock it. When updating the data, judge whether anyone has modified the data during this period,
- Get version
- Compare version when updating
# Monitor an element watch money # Open transaction multi incrby money 20 descby out 20 # Exit transaction exec # Cancel monitoring unwatch
bind 127.0.0.1 # Available addresses for accessing the database protected-mode yes # Protection mode port xxxx# Open port
daemonize yes # Daemons are on by default pidfile # If running in background mode, specify the pid run file loglevel notice # log information logfile # File location name of the log databases # Number of databases always-show-logo yes # Always show logo
# If at least one 1 key has been modified within 900s, we will perform the persistence operation save 900 1 # If at least 10 key s are modified within 300s, we can perform persistence operation save 300 10 # If at least 10000 key s are modified within 60s, we will perform persistence operation save 60 1000 stop-writes-on-bgsave-error yes # If persistence fails, do you want to continue working rdbcompression yes # Whether to compress rdb files requires some CPU resources
# Get redis password config get requirepass # Set redis password config set requirepass "xxx" # Login with password auth xx
# Set the maximum number of clients that can connect to redis maxclients xxx # Data deletion policy
Redis is an in memory database. If the database state in memory is not saved to disk, the database state in the server will disappear once the server process exits. So redis provides persistence
Redis will be created separately (fork) a child process will first write data to a temporary file. After the persistence process is completed, the temporary file will be used to replace the last persistent file. During the whole process, the main process will not perform any IO operations. This ensures extremely high performance. If large-scale data recovery is required, and data recovery is very important Integrity is not very sensitive, so RDB is more efficient than AOF. The disadvantage of RDB is that the data may be lost after the last persistence.
- Suitable for large-scale data recovery
- The integrity of the data is not high
- A certain time interval is required for process operation! If redis goes down unexpectedly, there is no need to modify the data for the last time
- The fork process will occupy a certain content space
Each write operation is recorded in the form of a log. All instructions executed by redis are recorded (read operations are not recorded). Only files can be added, but files cannot be overwritten. Redis will read the file and rebuild the data at the beginning of startup. In other words, if redis restarts, the write instructions will be executed from front to back according to the contents of the Tianzhi file to complete the data recovery
- Each modification is synchronized, and the integrity of the file will be better
- Sync once per second, and you may lose one second of data
- Never synchronized, most efficient
- Compared with data files, aof is much larger than rdb, and the repair speed is also slower than rdb!
- aof also runs slower than rdb, so the default configuration of redis is rdb persistence
- By default, aof is the unlimited addition of files, which will become larger and larger
If the aof file is larger than 64m, it is too large! fork a new process to rewrite our file
Redis subscription publication
Redis publish / subscribe (pub/sub) is a message communication mode: the sender (pub) sends messages and the subscriber (sub) receives messages.
Subscription and publication message graph: message sender, message receiver and channel
[the external chain image transfer fails. The source station may have an anti-theft chain mechanism. It is recommended to save the image and upload it directly (img-68hpxtts-1637155483492) (C: \ users \ ASUS \ appdata \ roaming \ typora \ user images \ image-20211117190653564. PNG)]
# Subscribe to one or more channels that match the given pattern. psubscribe pattern [pattern...] # View subscription and publishing system status. pubsub subcommand [argument] # Sends information to the specified channel. publish channel message # Unsubscribe from all channels in a given mode. punsubscribe [pattern...] # Subscribe to information about a given channel or channels. subscribe channel [channel...] # Unsubscribe from a given channel. unsubcribe [channel...]
- Real time message system
- Real time chat
More complex when using message oriented middleware
Master slave replication
Master-slave replication, read-write separation! 80% of the cases are read operations! Reduce the pressure on the server! Often used in the architecture! One master and two slaves!
Master-slave replication refers to copying data from one Redis server to other Redis servers. The former is called master / leader and the latter is called slave
(slave/follower); data replication is one-way, and can only be from the Master node to the Slave node. The Master mainly writes and the Slave mainly reads.
By default, each Redis server is the master node; and - a master node can have multiple slave nodes (or no slave nodes), but - a slave node can only have one master node.
The main functions of master-slave replication include:
- Data redundancy: master-slave replication realizes the hot backup of data, which is a data redundancy method other than persistence.
- Fault recovery: when the master node has a problem, the slave node can provide services to achieve rapid fault recovery; in fact, it is a kind of service redundancy.
- Load balancing: on the basis of master-slave replication, combined with read-write separation, the master node can provide write services and the slave node can provide read services (that is, the application connects the master node when writing Redis data and the slave node when reading Redis data) , sharing the server load; especially in the scenario of less writing and more reading, sharing the read load through multiple slave nodes can greatly improve the concurrency of Redis server.
- High availability cornerstone: in addition to the above functions, master-slave replication is also the basis for sentinels and clusters to implement. Therefore, master-slave replication is the basis for Redis high availability.
# Print master-slave copy information info replication
Copy the three configuration files and modify the corresponding information
- pid name
- log file name
- dump.rdb name
By default, each redis is a master. If you want to implement master-slave replication, only the slave is configured
# Transient configuration slave slaveof host port # Permanent configuration slave
The master cannot write, and the slave cannot write
- After Slave is successfully started and connected to the master, it will send a sync synchronization command
- After receiving the command, the master starts the background save process and collects all received commands for modifying the dataset. After the background process is executed, the master will transfer the data
- The entire data file is to the slave, and a full synchronization is completed.
- Full copy: after receiving the database file data, the slave service saves it and loads it into memory.
- Incremental replication: the Master continues to transmit all new collected modification commands to the slave in turn to complete the synchronization
- However, as long as the master is reconnected, a full synchronization (full replication) will be performed automatically
If the host is disconnected, we can use SLAVEOF no one to turn ourselves into a host! Other nodes can manually connect to the latest master node (hand)
Move)! If the boss fixes it at this time, reconnect it!
See sentinel.conf file for specific configuration
sentinel monitor myredis host port 1
The following number 1 means that the host hangs up. slave votes to see who takes over as the host. The one with the most votes will become the host!
Start the sentry
- Sentinel cluster is based on master-slave replication mode. It has all the advantages of master-slave configuration
- The master-slave can be switched and the fault can be transferred, so the system availability will be better
- Sentinel mode is the upgrade of master-slave mode. It is more robust from manual to automatic!
- Redis is not good. Online capacity expansion is very troublesome once the cluster capacity reaches the upper limit!
- The configuration of sentinel mode is actually very troublesome. There are many choices!
Cache penetration and avalanche
The use of Redis cache greatly improves the performance and efficiency of applications, especially in data query. But at the same time, it also brings some - problems. Among them, the most crucial
The problem is the consistency of data. Strictly speaking, this problem has no solution. If data consistency is required, caching cannot be used.
The concept of cache penetration is very simple. Users want to query one data by one and find that the redis memory database does not hit, that is, the cache does not hit, so they query the persistence layer database. No, so this query failed. When there are many users, the cache misses (second kill!), so they all request the persistence layer database. This will put a lot of pressure on the persistence layer database, which is equivalent to cache penetration.
- Bloom filter
- The cache layer is set to null
Bloom filter: bloom filter is a data structure that stores all possible query parameters in the form of hash. It is verified at the control layer first, and discarded if it does not meet the requirements, so as to avoid the query pressure on the underlying storage system;
However, there are two problems with this method:
- If null values can be cached, it means that the cache needs more space to store more keys, because there may be many null keys;
- Even if the expiration time is set for a null value, there will still be inconsistency between the data of the cache layer and the storage layer for a period of time, which will have an impact on the business that needs to maintain consistency.
Here, we need to pay attention to the difference between cache breakdown and cache breakdown. Cache breakdown means that a key is very hot. It is constantly carrying large concurrency. Large concurrency focuses on accessing these points. When the key fails, the continuous large concurrency will break through the cache and directly request the database, just like cutting a hole in a barrier.
When a key expires, a large number of requests are accessed concurrently. This kind of data is generally hot data. Because the cache expires, the database will be accessed at the same time to query the latest data
According to, and write back to the cache will lead to excessive pressure on the database.
- Set hotspot data never to expire
- Add mutex
The difference between penetration and breakdown: penetration cannot be found, the amount of breakdown is too large, and the cache expires
Cache avalanche means that the cache set expires in a certain period of time. Redis is down!
One of the reasons for the avalanche. For example, when writing this article, it is about to arrive at double twelve o'clock, and there will be a wave of rush buying soon. This wave of goods will be put into the cache for a concentrated time. Suppose the cache is one hour. Then, at o'clock in the morning, the cache of these goods will expire. The access and query of these commodities fall on the database, which will produce periodic pressure peaks. Therefore, all requests will reach the storage layer, and the call volume of the storage layer will increase sharply, resulting in the storage layer hanging up.
In fact, centralized expiration is not very fatal. The more fatal cache avalanche is the downtime or disconnection of a node of the cache server. Because of the naturally formed cache avalanche, - the cache must be created centrally in a certain period of time. At this time, the database can withstand the pressure. It is nothing more than periodic pressure on the database. The downtime of the cache service node will cause unpredictable pressure on the database server, which is likely to crush the database in an instant.
- Redis high availability: this idea means that since redis may hang up, I will add several more redis. After one is hung up, others can continue to work. In fact, it is a cluster.
- Current limiting degradation:
- The idea of this solution is to control the number of threads reading and writing to the database cache by locking or queuing after the cache expires. For example, for a key, only one thread is allowed to query data and write cache, while other threads wait.
- Data preheating: Data heating means that before formal deployment, I first access the possible data in advance, so that some data that may be accessed in large quantities will be loaded into the cache. Before a large concurrent access is about to occur, manually trigger the loading of different cache key s and set different expiration times to make the time point of cache invalidation as uniform as possible.