Complete and detailed Redis learning notes

Keywords: Redis Back-end Middleware

Learning reference Videos: [crazy God says Java] Redis

Redis

1, Overview of redis

redis is a non relational database.

1. Introduction to non relational database

A non relational database, that is, NoSQL (Not Only SQL).

Relational database: column + row. The data structure under the same table is the same.

Non relational database: data storage has no fixed format and can be expanded horizontally.

NoSQL generally refers to non relational databases. With the birth of web2.0 Internet, traditional relational databases are difficult to deal with the web2.0 era! In particular, large-scale and highly concurrent communities have exposed many insurmountable problems. NoSQL is developing very rapidly in today's big data environment, and Redis is the fastest growing community.

Features of NoSQL

  1. Easy to expand (there is no relationship between data, it is easy to expand!)
  2. Large amount of data and high performance (Redis can write 80000 times and read 110000 times a second. NoSQL's cache record level is a fine-grained cache with high performance!)
  3. Data types are diverse! (there is no need to design the database in advance, and it can be used at any time)

Classification of NoSQL

typePartial representativecharacteristic
Column storageHbase
Cassandra
HypertableHypertable
As the name suggests, data is stored by column.
The biggest feature is to facilitate the storage of structured and semi-structured data, facilitate data compression, and have great IO advantages for queries against a column or several columns.
Document storageMongoDB
CouchDB
Document storage is generally stored in a format similar to json, and the stored content is document type.
In this way, there is an opportunity to index some fields and realize some functions of relational database.
Key value storageTokyo Cabinet/Tyrant
Berkeley DB
MemcacheDB
Redis
You can quickly query its value through key.
Generally speaking, regardless of the format in which the value is stored, all orders are received. (Redis includes other functions)
Graph storageNeo4J
FlockDB
Optimal storage of graphical relationships.
Using traditional relational database to solve the problem has low performance and inconvenient design and use.
Object storagedb4o
Versant
Operate the database through syntax similar to object-oriented language and access data through object.
xml DatabaseBerkeley DB XML
BaseX
Efficiently store XML data and support the internal query syntax of XML,

2. What is redis

Redis is an in memory cache database. The full name of redis is Remote Dictionary Server (remote data service), which is written in C language. Redis is a key value storage system (key value storage system), which supports rich data types, such as String, list, set, zset and hash.

3.redis application scenario

Many languages support Redis. Because Redis exchanges data quickly, it is often used to store some data that needs to be accessed frequently in the server, which saves memory overhead and greatly improves the speed.

Some hot data is stored in Redis. When it needs to be used, it is directly fetched from memory, which greatly improves the speed and saves the server overhead.

1. Session cache (most commonly used)

2. Message queue (payment)

3. Activity ranking or counting

4. Publish and subscribe to messages (message notification)

5. Product list and comment list

2, Five data types

Before introducing the five data types, let's first understand the basic commands for key operation

commandexplain
keys *View all key s in the current database
existsJudge whether the specified key exists
delDelete the specified key
expireSet the expiration time of the key
typeView the type of key
127.0.0.1:6379> keys *
1) "name"
127.0.0.1:6379> exists name
(integer) 1
127.0.0.1:6379> type name
string
127.0.0.1:6379> expire name 10 #Set 10s expiration time
(integer) 1
127.0.0.1:6379> exists name
(integer) 1
127.0.0.1:6379> exists name
(integer) 1
127.0.0.1:6379> exists name #Expired after 10s
(integer) 0
127.0.0.1:6379> keys *   #When viewing all the keys in the current database, it is found that there are no keys
(empty list or set)

String (string)

Since redis will be prompted after entering the command, I will write his keyword here

commanddescribe
APPENDAppends a string to the value of the specified key
DECR/INCRThe value value of the specified key is + 1 / - 1 (for numbers only)
INCRBY/DECRBYAdds and subtracts values in specified steps
INCRBYFLOATAdds a floating-point value to the value
STRLENGets the string length of the key save value
GETRANGEGet the string according to the start and end position (closed interval, start and end positions)
SETRANGEReplace the value from offset in the key with the specified value
GETSETSet the value of the given key to value and return the old value of the key.
SETNXset only when the key does not exist
127.0.0.1:6379> keys *
1) "name"
2) "age"
127.0.0.1:6379> get name
"thenie"
127.0.0.1:6379> get age
"18"
127.0.0.1:6379> append name Shuai  #Adds Shuai to the specified string
(integer) 11
127.0.0.1:6379> get name
"thenieShuai"
127.0.0.1:6379> DECR age #Self subtraction
(integer) 17
127.0.0.1:6379> INCR age #Self increasing
(integer) 18
127.0.0.1:6379> DECRBY age 5 #Specify reduction
(integer) 13
127.0.0.1:6379> INCRBY age 5
(integer) 18
127.0.0.1:6379> STRLEN age #Return string length
(integer) 2
127.0.0.1:6379> getrange name 0 -1 #Returns the full string
"thenieShuai"
127.0.0.1:6379> getrange name 0 6 #Returns a string with subscripts 0 to 6
"thenieS"
127.0.0.1:6379> setrange name 6 Haokan #Cover with Haokan from position 6
(integer) 12
127.0.0.1:6379> get name
"thenieHaokan"
127.0.0.1:6379> GETSET name kuaijieshule #Returns the value of name and makes the specified modification
"thenieHaokan"
127.0.0.1:6379> GET name
"kuaijieshule"
127.0.0.1:6379> setnx name ,  #Failed to create because the key is name
(integer) 0
127.0.0.1:6379> get name
"kuaijieshule"

String similar usage scenario: value can be either a string or a number. For example:

  • Counter
  • Count the quantity of multiple units: uid:123666: follow 0
  • Number of fans
  • Object storage cache

Hash (hash)

Redis hash is a mapping table of field and value of string type. Hash is especially suitable for storing objects.

commanddescribe
HSETSet the value of the field field in the hash table key to value. Setting the same field repeatedly will overwrite and return 0
HMSETSet multiple field value pairs to the hash table key at the same time.
HSETNXSet the value of the hash table field only if the field field does not exist.
HEXISTSCheck whether the specified field exists in the hash table key.
HGETGets the value of the specified field stored in the hash table
HMGETGets the value of all the given fields
HGETALLGet all fields and values in the hash table key
HKEYSGet all fields in the hash table key
HLENGets the number of fields in the hash table
HVALSGets all values in the hash table
HDELDelete one or more field fields in the hash table key
HINCRBYAdd the increment n to the integer value of the specified field in the hash table key and return the increment. The result is the same. It is only applicable to integer fields
HINCRBYFLOATAdds the increment n to the floating-point value of the specified field in the hash table key.
127.0.0.1:6379> HSET mymap k1 v1 #Initialize a collection whose key is mymap
(integer) 1
127.0.0.1:6379> HSET mymap k2 v2
(integer) 1
127.0.0.1:6379> HMSET mymap k3 v3 k4 v4
OK
127.0.0.1:6379> HEXISTS mymap k1 #Determine whether k1 exists in mymap
(integer) 1
127.0.0.1:6379> HEXISTS mymap k5
(integer) 0
127.0.0.1:6379> HGETALL mymap  #Return all key s and value s in mymap
1) "k1"
2) "v1"
3) "k2"
4) "v2"
5) "k3"
6) "v3"
7) "k4"
8) "v4"
127.0.0.1:6379> HGET mymap k1
"v1"
127.0.0.1:6379> HKEYS mymap #Return all key s in mymap
1) "k1"
2) "k2"
3) "k3"
4) "k4"
127.0.0.1:6379> HVALS mymap
1) "v1"
2) "v2"
3) "v3"
4) "v4"
127.0.0.1:6379> HDEL mymap k2 #Delete k2 in mymap
(integer) 1
127.0.0.1:6379> HKEYS mymap
1) "k1"
2) "k3"
3) "k4"
127.0.0.1:6379> HSET mymap number 10
(integer) 1
127.0.0.1:6379> hget mymap number
"10"
127.0.0.1:6379> HINCRBY mymap number 10
(integer) 20

The data changed by hash is user name and age, especially user information, which changes frequently! Hash is more suitable for object storage, and Sring is more suitable for string storage!

List

Redis list is a simple string list, sorted by insertion order. You can add an element to the head (left) or tail (right) of the list

A list can contain up to 232 - 1 elements (4294967295, more than 4 billion elements per list).

First of all, we can change the list into queue, stack, double ended queue, etc. through rule definition

As shown in the figure, List in Redis can be operated at both ends, so the commands are divided into LXXX and RLLL. Sometimes L also represents List, such as LLEN

commanddescribe
LPUSH/RPUSHPUSH values (one or more) from the left / right to the list.
LRANGEGet the start and end elements of the list = = (the index is incremented from left to right)==
LPUSHX/RPUSHXpush values (one or more) into existing column names
LINSERT key BEFORE|AFTER pivot valueInserts value before / after the specified list element
LLENView list length
LINDEXGet list elements by index
LSETSet values for elements by index
LPOP/RPOPRemove value from leftmost / rightmost and return
RPOPLPUSH source destinationPop up the last value at the end (right) of the list, return it, and then add it to the head of another list
LTRIMIntercept the list within the specified range by subscript
LREM key count valueIn the List, duplicate values are allowed. Count > 0: start the search from the head and delete the specified values. At most count is deleted. Count < 0: start the search from the tail... count = 0: delete all the specified values in the List.
BLPOP/BRPOPMove out and get the first / last element of the list. If there are no elements in the list, the list will be blocked until the waiting timeout or pop-up elements are found.
127.0.0.1:6379> rpush list a b c d e f  #Initialize the list whose key is list
(integer) 6
127.0.0.1:6379> LRANGE list 0 -1   #Return to list from left to right
1) "a"
2) "b"
3) "c"
4) "d"
5) "e"
6) "f"
127.0.0.1:6379> LPUSH list 1  #Insert 1 to the left
(integer) 7
127.0.0.1:6379> LRANGE list 0 -1
1) "1"
2) "a"
3) "b"
4) "c"
5) "d"
6) "e"
7) "f"
127.0.0.1:6379> RPUSH list 2  #Insert 2 to the right
(integer) 8
127.0.0.1:6379> LRANGE list 0 -1  #Return to list from left to right
1) "1"
2) "a"
3) "b"
4) "c"
5) "d"
6) "e"
7) "f"
8) "2"
127.0.0.1:6379> llen list
(integer) 8
127.0.0.1:6379> LINDEX list 2 
"b"
127.0.0.1:6379> LPUSH list1 1  #Initialize a list1 list
(integer) 1
127.0.0.1:6379> RPOPLPUSH list list1  #Pop up a value from the right side of the list and insert it to the left side of the list
"2"
127.0.0.1:6379> LRANGE list1 0 -1
1) "2"
2) "1"
127.0.0.1:6379> LPOP list1
"2"
127.0.0.1:6379> LPOP list1
"1"
127.0.0.1:6379> LTRIM list 1 8
OK
127.0.0.1:6379> LRANGE list 0 -1
1) "a"
2) "b"
3) "c"
4) "d"
5) "e"
6) "f"

Summary

  • List is actually a linked list. Before node, after, left and right can insert values
  • If the key does not exist, create a new linked list
  • If the key exists, add content
  • If all values are removed, the empty linked list also means that it does not exist
  • Insert or change values on both sides for the highest efficiency! Modifying intermediate elements is relatively inefficient

Application:

Message queuing! Message queue (Lpush Rpop), stack (Lpush Rpop)

Set

Redis Set is an unordered Set of string type. Collection members are unique, which means that duplicate data cannot appear in the collection.

Collections in Redis are implemented through hash tables, so the complexity of adding, deleting and searching is O(1).

The largest number of members in the collection is 232 - 1 (4294967295, each collection can store more than 4 billion members).

commanddescribe
SADDAdd one or more members to the collection unordered
SCARDGets the number of members of the collection
SMEMBERSReturns all members of the collection
SISMEMBERQuery whether the member element is a member of the collection, and the result is unordered
SRANDMEMBERRandomly return count members in the collection. The default value of count is 1
SPOPRandomly remove and return count members in the collection. The default value of count is 1
SMOVE source destination memberMove the member of the source collection to the destination collection
SREMRemove one or more members from the collection
SDIFFReturns the difference set of all sets key1- key2 -
SDIFFSTORE destination key1[key2..]Based on SDIFF, save the results to the set = = (overwrite) = =. Cannot save to other types of key s!
SINTER key1 [key2..]Returns the intersection of all sets
SINTERSTORE destination key1[key2..]On the basis of SINTER, the results are stored in the collection. cover
SUNION key1 [key2..]Returns the union of all sets
SUNIONSTORE destination key1 [key2..]On the basis of SUNION, store the results to and sheets. cover
SSCAN KEY [MATCH pattern] [COUNT count]In a large amount of data environment, use this command to traverse the elements in the collection, traversing parts each time
127.0.0.1:6379> sadd set a b c  #Initialize the set with key set
(integer) 3
127.0.0.1:6379> sadd set1 d e f #Initialize the set whose key is set1
(integer) 3
127.0.0.1:6379> SMEMBERS set  #View all members of the specified collection
1) "b"
2) "c"
3) "a"
127.0.0.1:6379> SISMEMBER set b #Determine whether b exists in the set
(integer) 1
127.0.0.1:6379> SISMEMBER set c
(integer) 1
127.0.0.1:6379> SISMEMBER set e
(integer) 0
127.0.0.1:6379> SRANDMEMBER set 2 #Random return of two members
1) "b"
2) "c"
127.0.0.1:6379> SRANDMEMBER set 2
1) "a"
2) "c"
127.0.0.1:6379> SADD set d
(integer) 1
127.0.0.1:6379> SMOVE set set1 a #Move a in set to set1
(integer) 1
127.0.0.1:6379> SMEMBERS set
1) "b"
2) "c"
3) "d"
127.0.0.1:6379> SMEMBERS set1
1) "a"
2) "f"
3) "e"
4) "d"
127.0.0.1:6379> SREM set1 a  #Delete the a element in set1
(integer) 1
127.0.0.1:6379> SMEMBERS set1
1) "f"
2) "e"
3) "d"

ZSet (sorted set)

The difference is that each element is associated with a score of type double. redis sorts the members of the collection from small to large through scores.

Same score: sort in dictionary order

Members of an ordered set are unique, but scores can be repeated.

commanddescribe
ZADD key score member1 [score2 member2]Add one or more members to an ordered collection, or update the scores of existing members
ZCARD keyGets the number of members of an ordered collection
ZCOUNT key min maxCalculates the number of members of the specified interval score in the ordered set
ZINCRBY key n memberAdds the increment n to the score of the specified member in the ordered set
ZSCORE key memberReturns the score value of a member in an ordered set
ZRANK key memberReturns the index of the specified member in an ordered collection
ZRANGE key start endReturns an ordered set through an index interval to synthesize members in a specified interval
ZRANGEBYLEX key min maxReturns the members of an ordered set through a dictionary interval
ZRANGEBYSCORE key min maxReturn the members in the specified interval of the ordered set through scores = = - inf and + inf represent the minimum and maximum values respectively, and only open intervals () are supported==
ZLEXCOUNT key min maxCalculates the number of members in the specified dictionary interval in an ordered set
ZREM key member1 [member2..]Remove one or more members from an ordered collection
ZREMRANGEBYLEX key min maxRemoves all members of a given dictionary interval from an ordered set
ZREMRANGEBYRANK key start stopRemoves all members of a given ranking interval from an ordered set
ZREMRANGEBYSCORE key min maxRemoves all members of a given score interval from an ordered set
ZREVRANGE key start endReturns the members in the specified interval in the ordered set. Through the index, the score is from high to low
ZREVRANGEBYSCORRE key max minReturns the members within the specified score range in the ordered set, and the scores are sorted from high to low
ZREVRANGEBYLEX key max minReturns the members in the specified dictionary interval in the ordered set, in reverse dictionary order
ZREVRANK key memberReturns the ranking of the specified members in the ordered set, and the members of the ordered set are sorted by decreasing points (from large to small)
ZINTERSTORE destination numkeys key1 [key2 ..]Calculate the intersection of one or more given ordered sets and store the result set in the new ordered set key. numkeys: represents the number of sets involved in the operation, and add the score as the score of the result
ZUNIONSTORE destination numkeys key1 [key2..]Calculates the intersection of a given one or more ordered sets and stores the result set in a new ordered set key
ZSCAN key cursor [MATCH pattern\] [COUNT count]Iterate the elements in the ordered set (including element members and element scores)
-------------------ZADD--ZCARD--ZCOUNT--------------
127.0.0.1:6379> ZADD myzset 1 m1 2 m2 3 m3 # Add the member m1 score=1 and the member m2 score=2 to the ordered set myzset
(integer) 2
127.0.0.1:6379> ZCARD myzset # Gets the number of members of an ordered collection
(integer) 2
127.0.0.1:6379> ZCOUNT myzset 0 1 # Get the number of members whose score is in the [0,1] interval
(integer) 1
127.0.0.1:6379> ZCOUNT myzset 0 2
(integer) 2

----------------ZINCRBY--ZSCORE--------------------------
127.0.0.1:6379> ZINCRBY myzset 5 m2 # Add the score of member m2 + 5
"7"
127.0.0.1:6379> ZSCORE myzset m1 # Get the score of member m1
"1"
127.0.0.1:6379> ZSCORE myzset m2
"7"

--------------ZRANK--ZRANGE-----------------------------------
127.0.0.1:6379> ZRANK myzset m1 # Get the index of member m1. The index is sorted by score, and the index values with the same score are increased in dictionary order
(integer) 0
127.0.0.1:6379> ZRANK myzset m2
(integer) 2
127.0.0.1:6379> ZRANGE myzset 0 1 # Get members with index 0 ~ 1
1) "m1"
2) "m3"
127.0.0.1:6379> ZRANGE myzset 0 -1 # Get all members
1) "m1"
2) "m3"
3) "m2"

#Testset = > {ABC, add, amaze, apple, back, Java, redis} scores are all 0
------------------ZRANGEBYLEX---------------------------------
127.0.0.1:6379> ZRANGEBYLEX testset - + # Return all members
1) "abc"
2) "add"
3) "amaze"
4) "apple"
5) "back"
6) "java"
7) "redis"
127.0.0.1:6379> ZRANGEBYLEX testset - + LIMIT 0 3 # Display 0,1,2 records of query results by index in pages
1) "abc"
2) "add"
3) "amaze"
127.0.0.1:6379> ZRANGEBYLEX testset - + LIMIT 3 3 # Display 3, 4 and 5 records
1) "apple"
2) "back"
3) "java"
127.0.0.1:6379> ZRANGEBYLEX testset (- [apple # Show members in (-, apple] interval
1) "abc"
2) "add"
3) "amaze"
4) "apple"
127.0.0.1:6379> ZRANGEBYLEX testset [apple [java # Displays the members of the [apple,java] dictionary section
1) "apple"
2) "back"
3) "java"

-----------------------ZRANGEBYSCORE---------------------
127.0.0.1:6379> ZRANGEBYSCORE myzset 1 10 # Returns members whose score is between [1,10]
1) "m1"
2) "m3"
3) "m2"
127.0.0.1:6379> ZRANGEBYSCORE myzset 1 5
1) "m1"
2) "m3"

--------------------ZLEXCOUNT-----------------------------
127.0.0.1:6379> ZLEXCOUNT testset - +
(integer) 7
127.0.0.1:6379> ZLEXCOUNT testset [apple [java
(integer) 3

------------------ZREM--ZREMRANGEBYLEX--ZREMRANGBYRANK--ZREMRANGEBYSCORE--------------------------------
127.0.0.1:6379> ZREM testset abc # Remove member abc
(integer) 1
127.0.0.1:6379> ZREMRANGEBYLEX testset [apple [java # Remove all members in the dictionary interval [apple,java]
(integer) 3
127.0.0.1:6379> ZREMRANGEBYRANK testset 0 1 # Remove all members ranking 0 ~ 1
(integer) 2
127.0.0.1:6379> ZREMRANGEBYSCORE myzset 0 3 # Remove member with score at [0,3]
(integer) 2


# Testset = > {ABC, add, apple, amaze, back, Java, redis} scores are all 0
# myzset=> {(m1,1),(m2,2),(m3,3),(m4,4),(m7,7),(m9,9)}
----------------ZREVRANGE--ZREVRANGEBYSCORE--ZREVRANGEBYLEX-----------
127.0.0.1:6379> ZREVRANGE myzset 0 3 # Sort by score, and then return 0 ~ 3 of the results by index
1) "m9"
2) "m7"
3) "m4"
4) "m3"
127.0.0.1:6379> ZREVRANGE myzset 2 4 # Returns 2 ~ 4 of the index of the sorting result
1) "m4"
2) "m3"
3) "m2"
127.0.0.1:6379> ZREVRANGEBYSCORE myzset 6 2 # Returns the members in the collection whose scores are between [2,6] in descending order of score
1) "m4"
2) "m3"
3) "m2"
127.0.0.1:6379> ZREVRANGEBYLEX testset [java (add # Returns the members of the (add,java] dictionary interval in the collection in reverse dictionary order
1) "java"
2) "back"
3) "apple"
4) "amaze"

-------------------------ZREVRANK------------------------------
127.0.0.1:6379> ZREVRANK myzset m7 # Returns the member m7 index in descending order of score
(integer) 1
127.0.0.1:6379> ZREVRANK myzset m2
(integer) 4


# Mathscore = > {(XM, 90), (XH, 95), (XG, 87)} math scores of Xiao Ming, Xiao Hong and Xiao Gang
# Enscore = > {(XM, 70), (XH, 93), (XG, 90)} English scores of Xiao Ming, Xiao Hong and Xiao Gang
-------------------ZINTERSTORE--ZUNIONSTORE-----------------------------------
127.0.0.1:6379> ZINTERSTORE sumscore 2 mathscore enscore # Merge mathcore and enscore, and store the results in sumcore
(integer) 3
127.0.0.1:6379> ZRANGE sumscore 0 -1 withscores # The merged score is the sum of all scores in the previous set
1) "xm"
2) "160"
3) "xg"
4) "177"
5) "xh"
6) "188"

127.0.0.1:6379> ZUNIONSTORE lowestscore 2 mathscore enscore AGGREGATE MIN # Take the minimum member score of two sets as the result
(integer) 3
127.0.0.1:6379> ZRANGE lowestscore 0 -1 withscores
1) "xm"
2) "70"
3) "xg"
4) "87"
5) "xh"
6) "93"

Application case:

  • set sort to store the sorting of class grade table and salary table!
  • Ordinary message, 1. Important message, 2. Judgment with weight
  • Leaderboard application implementation, taking the Top N test

3, Business

A single Redis command guarantees atomicity, while Redis transactions do not guarantee atomicity.

1. How to use transactions

  • Open transaction (multi)

  • Order to join the team

  • Execute transaction (exec)

    The commands after the transaction is started will not be executed immediately, but will be executed together after exec input

    127.0.0.1:6379> keys *
    (empty list or set)
    127.0.0.1:6379> MULTI  #open
    OK
    127.0.0.1:6379> set name thenie    #Order to join the team
    QUEUED
    127.0.0.1:6379> set age 18
    QUEUED
    127.0.0.1:6379> get name
    QUEUED
    127.0.0.1:6379> get name
    QUEUED
    127.0.0.1:6379> get age
    QUEUED
    127.0.0.1:6379> EXEC  #End -- execute
    1) OK
    2) OK
    3) "thenie"
    4) "thenie"
    5) "18"
    

    In redis, batch processing is more appropriate than transaction.

    If you don't want to execute the transaction in the middle of starting the transaction, enter DISCARD to give up.

2. What happens to command errors in transactions

Code syntax error (compile time exception) all commands are not executed

127.0.0.1:6379> KEYS *
(empty list or set)
127.0.0.1:6379> MULTI  #open
OK
127.0.0.1:6379> set name thenie
QUEUED
127.0.0.1:6379> set age 18
QUEUED
127.0.0.1:6379> setset name er   #Wrong command
(error) ERR unknown command `setset`, with args beginning with: `name`, `er`, 
127.0.0.1:6379> get name
QUEUED
127.0.0.1:6379> EXEC  #No execution after completion
(error) EXECABORT Transaction discarded because of previous errors.

Code logic error (runtime exception) * * other commands can be executed normally * * > >, so transaction atomicity is not guaranteed

127.0.0.1:6379> KEYS *
(empty list or set)
127.0.0.1:6379> MULTI
OK
127.0.0.1:6379> set name thenie
QUEUED
127.0.0.1:6379> set age 18
QUEUED
127.0.0.1:6379> INCR name  #Logical error, the latest string is self incremented
QUEUED
127.0.0.1:6379> INCR age
QUEUED
127.0.0.1:6379> EXEC
1) OK
2) OK
3) (error) ERR value is not an integer or out of range  #The third command was wrong, but all other commands were executed
4) (integer) 19

4, Spring boot integrates reids

You can read my blog Spring boot integrates reids

5, Persistence - RDB

The data in redis is cached in memory, and the data in memory is lost immediately after power failure, which requires us to write the data in memory to the hard disk after a specified time interval

This process is called persistence, and RDB (Redis Databases) is one of them.

What is RDB

After the specified time interval, write the dataset snapshot in memory to the database; During recovery, directly read the snapshot file for data recovery;

By default, Redis saves the database snapshot in a binary file named dump.rdb. The file name can be customized in the configuration file.

How to trigger

  • Trigger persistence rule

    The persistence rules are configured in a configuration file named redis.conf:

  • Flush command

    Persistence is also triggered by default when this command is executed.

    After restarting the redis database, restore the data.

  • save

    Using the save command will immediately persist the data in the current memory, but it will block, that is, it will not accept other operations;

    Since the save command is a synchronization command, it will occupy the main process of Redis. If Redis data is very large, the execution speed of the save command will be very slow, blocking the requests of all clients.

  • bgsave

    bgsave is performed asynchronously. redis can also continue to respond to client requests during persistence;

    That is, fork() a child process to persist, while the main process continues to work.

working principle

During RDB, the main thread of redis will not perform io operations, and the main thread will fork a sub thread to complete the operation;

  1. Redis calls forks. Have both parent and child processes.
  2. The child process writes the data set to a temporary RDB file.
  3. When the child process finishes writing the new RDB file, Redis replaces the original RDB file with the new RDB file and deletes the old RDB file.

This way of working enables Redis to benefit from the copy on write mechanism (because the child process is used for write operations, and the parent process can still receive requests from the client.)

Advantages and disadvantages

advantage:

  1. Suitable for large-scale data recovery
  2. The requirements for data integrity are not high

Disadvantages:

  1. Operations need to be performed at a certain interval. If redis goes down unexpectedly, the last modified data will not be available.
  2. The fork process will occupy a certain content space.

6, Persistence - AOF

What is AOF

Record all our commands, history, and execute all the files again when recovering

Each write operation is recorded in the form of a log. All instructions executed by redis are recorded (read operations are not recorded). Only files are allowed to be added, but files cannot be overwritten. At the beginning of redis startup, the file will be read to rebuild the data. In other words, if redis restarts, the write instructions will be executed from front to back according to the contents of the log file to complete the data recovery.

How to open

It is not enabled by default. We need to configure it manually and restart redis to take effect!

If you want to use AOF, you need to modify the configuration file redis.conf:

appendonly no yes indicates that AOF is enabled

There is a problem with the transfer file

If the aof file is misplaced, redis cannot be started at this time. I need to modify the aof file

Redis provides us with a tool redis check AOF -- fix, which will eliminate the faulty commands in this file

Advantages and disadvantages

advantage

  1. Each modification will be synchronized, and the integrity of the file will be better
  2. If you do not synchronize every second, you may lose data for one second
  3. Never synchronized, most efficient

shortcoming

  1. Compared with data files, aof is much larger than rdb, and the repair speed is slower than rdb!
  2. Aof also runs slower than rdb, so the default configuration of redis is rdb persistence

7, redis master-slave replication

concept

In the redis Master-Slave architecture, the Master node is responsible for processing write requests, while the Slave node only processes read requests. For scenarios where there are few write requests and many read requests, such as the e-commerce details page, the concurrency can be greatly improved through this read-write separation operation. By increasing the number of redis Slave nodes, the QPS of redis can reach 10W +.

When the Master node writes data, it will be synchronized to the Slave node. The simple version architecture diagram is as follows:

Simulate one master and two slaves (redis) on a server

  1. Set and configure three redis service processes

    There are three profiles:

    #Modify the configuration files of the three redis respectively
    --------------Master( redis.conf)-------------
    #Port number
    port 6379
    #pid file name
    pidfile /var/run/redis_6379.pid
    #Log file name
    logfile "6379.log"
    #rdb file name
    dbfilename dump.rdb
    --------------From( slave1.conf)-------------
    port 6380
    pidfile /var/run/redis_6380.pid
    logfile "6380.log"
    dbfilename dump6380.rdb
    --------------From( slave2.conf)-------------
    port 6381
    pidfile /var/run/redis_6381.pid
    logfile "6381.log"
    dbfilename dump6381.rdb
    

    Service started successfully:

  2. Establish contact

    By default, each redis is a master node, so we only need to configure the slave

    Use SLAVEOF host port to configure the host for the slave.

    Configure successfully and use info replication to view master-slave status information:

    Use rules

    • The slave can only read, but the host can read, but it is mostly used for writing

    • When the host is powered off and down, the role of the slave will not change by default. The cluster only loses the write operation. After the host is restored, the slave will be connected and restored to its original state.

    • After the slave is powered off and down, if the slave is not configured using the configuration file, the data of the previous host cannot be obtained as the host after restart. If the slave is reconfigured at this time, all the data of the host can be obtained. Here is a synchronization principle.

    • As mentioned in Article 2, by default, new hosts will not appear after host failure. There are two ways to generate new hosts:

      • The slave manually executes the command slave of no one. After execution, the slave becomes a host independently
      • Use sentinel mode (automatic election)

8, Sentinel mode

When the primary server is down, it is obviously unrealistic to manually switch the primary server. Sentinel mode is more often used.

summary

Sentinel mode is a special mode. Firstly, Redis provides sentinel commands. Sentinel is an independent process. As a process, it will run independently. The principle is that the sentinel sends a command and waits for the response of the Redis server, so as to monitor multiple running Redis instances.

The role of the sentry here:

  • Send commands to monitor redis and return it to running status (including master server and slave server)
  • When the sentinel detects that the master is down, it will automatically switch one slave to the master, and then notify other slave servers through publish subscribe mode to modify the configuration file and let them switch hosts.

However, there may be problems when a sentinel process monitors the Redis server. Therefore, we can use multiple sentinels for monitoring. Each sentinel will also be monitored, which forms a multi sentinel mode.

failover process

Assuming that the main server goes down, sentry 1 detects this result first, and the system will not fail immediately. Sentry 1 subjectively thinks that the main server is unavailable, which becomes a subjective offline phenomenon. When the following sentinels also detect that the primary server is unavailable and the number reaches a certain value, a vote will be held between sentinels. The voting result is initiated by a sentinel for failover. After the switch is successful, each sentinel will switch its monitored slave server to the host through the publish and subscribe mode. This process is called objective offline. In this way, everything is transparent to the client.

9, Cache penetration and avalanche

Cache penetration

concept

By default, when a user requests data, it will first find it in the cache, and then find it in the database if it misses. However, if the access volume is too large and the cache misses, it will be transferred to the database, resulting in too much pressure, which may lead to database crash. In network security, some people maliciously use this means to attack, which is called flood attack.

Solution

Bloom filter

As the name suggests, bron raises a shield in front.

Hash all possible query parameters to quickly judge whether the data is available; Intercept verification is performed at the control layer. If there is no, it will be returned directly.

Cache empty objects

If a request is not found in the cache and database, an empty object is placed in the cache for subsequent processing.

There is a drawback to this: storing empty objects also requires space. A large number of empty objects will consume a certain space, and the storage efficiency is not high. The way to solve this problem is to set a shorter expiration time;

Even if the expiration time is set for a null value, there will still be inconsistency between the data of the cache layer and the storage layer for a period of time, which will have an impact on the business that needs to maintain consistency.

Buffer breakdown

concept

Compared with cache penetration, cache breakdown is more purposeful. When a key expires in the cache and multiple requests access the key at the same time, these requests will breakdown into the DB, resulting in a large number of transient DB requests and a sudden increase in pressure. This means that the cache is broken down. It is only because the cache of one key is unavailable, but other keys can still use the cache response.

For example, on the hot search ranking, a hot news is accessed in large numbers at the same time, which may lead to cache breakdown.

Solution

  1. Set hotspot data never to expire

    In this way, the hot data will not expire, but it will occupy more space. Once there are too many hot data, it will occupy more space. When the redis memory is full, it will also clean up some data.

  2. Add mutex lock (distributed lock)

    Before accessing the key, SETNX (set if not exists) is used to set another short-term key to lock the access of the current key. After the access is completed, the short-term key is deleted. Ensure that only one thread can access at the same time. In this way, the requirements for locks are very high.

Cache avalanche

concept

A large number of key s set the same expiration time, resulting in all caches failing at the same time, resulting in large instantaneous DB requests, sudden pressure increase and avalanche.

Solution

  • redis high availability

    The meaning of this idea is that since redis may hang up, I will add several more redis. After one is hung up, others can continue to work. In fact, it is a cluster

  • Current limiting degradation

    The idea of this solution is to control the number of threads reading and writing to the database cache by locking or queuing after the cache expires. For example, for a key, only one thread is allowed to query data and write cache, while other threads wait.

  • Data preheating

    The meaning of data heating is that before the formal deployment, I first access the possible data in advance, so that some data that may be accessed in large quantities will be loaded into the cache. Before a large concurrent access is about to occur, manually trigger the loading of different cache key s and set different expiration times to make the time point of cache invalidation as uniform as possible.
    If the expiration time is set for a null value, there will still be inconsistency between the data of the cache layer and the storage layer for a period of time, which will have an impact on the business that needs to maintain consistency.

If the article is helpful to you, please praise it and add a close~~

Posted by tofi84 on Sat, 20 Nov 2021 18:29:15 -0800