This article is included in the column
❤️<Redis Factory Essential Skills Pack❤️
Thousands of people praise the collection, the complete set of Redis learning materials, the essential skills of the factory!
Catalog
3.2 pipeline code implementation
1. Demand
Restrict a user's behavior to N occurrences within a specified time T.Assume T is 1 second and N is 1000 times.
2. Common design errors
The programmer designed a flow restriction scheme that allows only 1000 accesses per minute, as shown in Figure 01:00s-02:00s below. The biggest problem with this design is that requests may be requested 1000 times between 01:59s-02:00s and 1000 times between 02:00s-02:01s. In this case, requests may be made 2000 times between 01:59s-02:01s.Obviously this design is wrong.
3. Sliding window algorithm
3.1 Solution
Only N occurrences are allowed within the specified time T.We can think of this specified time T as a sliding time window (fixed width).We circle this sliding time window using Redis's zset base data type score.In the actual zset operation, we only need to keep the data within this sliding time window, other data can not be processed.
- Each user's behavior is stored in a zset store, score is a millisecond timestamp, and value uses a millisecond timestamp (which saves more memory than UUID)
- Keep only a record of the behavior during the sliding window. If the zset is empty, remove the zset and no longer consume memory (save memory)
3.2 pipeline code implementation
The logic behind the implementation of the code is to count the number of behaviors in the zset within the sliding window and to directly compare with the threshold maxCount to determine if the current behavior is allowed.There are multiple redis operations involved, so using pipeline can greatly increase efficiency
package com.lizba.redis.limit;
import redis.clients.jedis.Jedis;
import redis.clients.jedis.Pipeline;
import redis.clients.jedis.Response;
/**
* <p>
* Limiting current by sliding window algorithm through zset
* </p>
*
* @Author: Liziba
* @Date: 2021/9/6 18:11
*/
public class SimpleSlidingWindowByZSet {
private Jedis jedis;
public SimpleSlidingWindowByZSet(Jedis jedis) {
this.jedis = jedis;
}
/**
* Judging whether an action is allowed
*
* @param userId User id
* @param actionKey Behavior key
* @param period Current Limiting Cycle
* @param maxCount Maximum number of requests (sliding window size)
* @return
*/
public boolean isActionAllowed(String userId, String actionKey, int period, int maxCount) {
String key = this.key(userId, actionKey);
long ts = System.currentTimeMillis();
Pipeline pipe = jedis.pipelined();
pipe.multi();
pipe.zadd(key, ts, String.valueOf(ts));
// Remove data other than sliding windows
pipe.zremrangeByScore(key, 0, ts - (period * 1000));
Response<Long> count = pipe.zcard(key);
// Set the expiration time of the behavior, and if the data is cold, zset will be deleted to save memory space
pipe.expire(key, period);
pipe.exec();
pipe.close();
return count.get() <= maxCount;
}
/**
* Current limiting key
*
* @param userId
* @param actionKey
* @return
*/
public String key(String userId, String actionKey) {
return String.format("limit:%s:%s", userId, actionKey);
}
}
Test code:
package com.lizba.redis.limit;
import redis.clients.jedis.Jedis;
/**
*
* @Author: Liziba
* @Date: 2021/9/6 20:10
*/
public class TestSimpleSlidingWindowByZSet {
public static void main(String[] args) {
Jedis jedis = new Jedis("192.168.211.108", 6379);
SimpleSlidingWindowByZSet slidingWindow = new SimpleSlidingWindowByZSet(jedis);
for (int i = 1; i <= 15; i++) {
boolean actionAllowed = slidingWindow.isActionAllowed("liziba", "view", 60, 5);
System.out.println("No." + i +"Secondary operation" + (actionAllowed ? "Success" : "fail"));
}
jedis.close();
}
}
Test results:
From the test output data, it can be seen that the current limiting effect has been achieved. Request operations have failed since the 11th time, but this and our allowed five errors are still large.The reason for this problem is that the milliseconds we test System.currentTimeMillis() may be the same, and at this point the value is the same as System.currentTimeMillis(), which causes element overrides in zset!
Modify Code Testing:
Sleep for one millisecond in a cycle, and the test results are as expected!
TimeUnit.MILLISECONDS.sleep(1);
3.3 lua code implementation
We will use more atomic Lua steps in our projects to achieve flow restriction, so a version of lua based on zset operation is also provided here
package com.lizba.redis.limit;
import com.google.common.collect.ImmutableList;
import redis.clients.jedis.Jedis;
import redis.clients.jedis.Pipeline;
import redis.clients.jedis.Response;
/**
* <p>
* Limiting current by sliding window algorithm through zset
* </p>
*
* @Author: Liziba
* @Date: 2021/9/6 18:11
*/
public class SimpleSlidingWindowByZSet {
private Jedis jedis;
public SimpleSlidingWindowByZSet(Jedis jedis) {
this.jedis = jedis;
}
/**
* lua Script Current Limiting
*
* @param userId
* @param actionKey
* @param period
* @param maxCount
* @return
*/
public boolean isActionAllowedByLua(String userId, String actionKey, int period, int maxCount) {
String luaScript = this.buildLuaScript();
String key = key(userId, actionKey);
long ts = System.currentTimeMillis();
System.out.println(ts);
ImmutableList<String> keys = ImmutableList.of(key);
ImmutableList<String> args = ImmutableList.of(String.valueOf(ts),String.valueOf((ts - period * 1000)), String.valueOf(period));
Number count = (Number) jedis.eval(luaScript, keys, args);
return count != null && count.intValue() <= maxCount;
}
/**
* Current limiting key
*
* @param userId
* @param actionKey
* @return
*/
private String key(String userId, String actionKey) {
return String.format("limit:%s:%s", userId, actionKey);
}
/**
* Limit flow using lua scripts for a key
*
* @return
*/
private String buildLuaScript() {
return "redis.call('ZADD', KEYS[1], tonumber(ARGV[1]), ARGV[1])" +
"\nlocal c" +
"\nc = redis.call('ZCARD', KEYS[1])" +
"\nredis.call('ZREMRANGEBYSCORE', KEYS[1], 0, tonumber(ARGV[2]))" +
"\nredis.call('EXPIRE', KEYS[1], tonumber(ARGV[3]))" +
"\nreturn c;";
}
}
The test code is unchanged, so you can test it yourself. Remember to consider the equality of System.currentTimeMillis() when we test, if you don't trust you to output System.currentTimeMillis()!Think more, technology is in your heart!