Java from 0 to architect Directory: [Java from 0 to architect] learning record
Some concepts: PV (Page View) visits, UV (Unique Visitor) independent visitors
Basic use of Jedis
Reference article: Redis notes Java operations redis (Jedis)
<dependencies>
<dependency>
<groupId>redis.clients</groupId>
<artifactId>jedis</artifactId>
<version>3.1.0</version>
</dependency>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>4.13</version>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<configuration>
<source>1.8</source>
<target>1.8</target>
</configuration>
</plugin>
</plugins>
</build>
Redis data elimination strategy
- lft: eliminated according to the number of times the key is used
- lru: obsolete according to the latest usage time of the key
In Redis, users are allowed to set the maximum memory size (configured in redis.conf)
maxmemory 1G maxmemory-policy noeviction
maxmemory-policy:
- Volatile LRU deletes the least commonly used data from the data for which the timeout is set
- Allkeys LRU queries the least frequently used data of all keys to delete
- Volatile random is randomly deleted from the data for which the timeout has been set
- Allkeys random queries all keys and deletes them randomly
- Volatile TTL queries all the data with set timeout, sorts and deletes the data about to expire
- noeviction is the default policy. If this property is set, the deletion operation will not be performed. If the memory overflows, an error will be reported and returned
- Volatile LFU removes the least frequently used key from all keys configured with expiration time
- Allkeys LFU removes the least frequently used key from all keys
Advanced development of Redis
Command execution process:
pipline - command batch processing to reduce the network overhead of a large number of commands and improve operation performance
pipline application:
-
Multiple commands can be executed in parallel without correlation
pipline does not guarantee atomicity, but sends a bunch of commands to execute; mset this command is atomic
-
For operations that need to be performed in batch processing, multiple different data need to be put into memory at the same time
-
The time consumption is more at the network level. The time required to execute commands is us (microsecond)
Code implementation:
public class JedisDemo {
private Jedis jedis = null;
@Before
public void init() throws Exception {
jedis = new Jedis("192.168.48.101", 6379);
}
@Test
public void testJedis() throws Exception {
jedis.flushAll();
Long start = System.currentTimeMillis(); // Get start time
for (int i = 0; i < 100000; i++) {
jedis.set("name" + i, i + "");
}
Long end = System.currentTimeMillis(); // Get end time
System.out.println(end - start);
jedis.close();
}
@Test
public void testPipeline() throws Exception {
jedis.flushAll();
Long start = System.currentTimeMillis(); // Get start time
Pipeline pipelined = jedis.pipelined();
for (int i = 0; i < 100000; i++) {
pipelined.set("name" + i, i + "");
}
pipelined.sync();
Long end = System.currentTimeMillis(); // Get end time
System.out.println(end - start);
jedis.close();
}
}
Publish subscribe - Subscribe
Understand that there is a special message middleware for publishing and subscribing
Roles in publications and Subscriptions:
- Publisher publisher
- Subscriber subscriber
- channel
# Subscribe to one or more channels subscribe < channel > subscribe sohu:tv # Publish < channel > < message > publish sohu:tv "hello world" # Unsubscribe unsubscribe
Test case:
- producer:
public class PublisherDemo {
private Jedis jedis = null;
@Before
public void init() throws Exception {
jedis = new Jedis("192.168.48.101", 6379);
}
@After
public void destroy() throws Exception {
jedis.close();
}
@Test
public void publish() throws Exception {
for (int i = 0; i < 100; i++) {
jedis.publish("channel02", "value" + i);
TimeUnit.MICROSECONDS.sleep(10);
}
}
}
- consumer:
public class Subscriber {
private Jedis jedis = null;
@Before
public void init() throws Exception {
jedis=new Jedis("192.168.48.101", 6379);
}
@After
public void destroy() throws Exception {
jedis.close();
}
@Test
public void subscribe() throws Exception {
JedisPubSub jedisPubSub = new JedisPubSub(){
@Override
public void onMessage(String channel, String message) {
System.out.println("channel = " + channel);
System.out.println("message = " + message);
}
};
jedis.subscribe(jedisPubSub, "channel01", "channel02");
}
}
Bitmap - a continuous string of binary digits (string), each bit at an offset
Bitmap is a series of consecutive binary digits (0 or 1), and the position of each bit is offset
setbit,getbit,bitcount:
# Set the specified bit 127.0.0.1:6379> setbit qq:uv 1002 1 127.0.0.1:6379> setbit qq:uv 1003 1 # Gets the value of the specified location and returns 1 or 0 127.0.0.1:6379> getbit qq:uv 1003 (integer) 1 # bitcount: number of with statistical value of 1 127.0.0.1:6379> BITCOUNT qq:uv (integer) 1
bitop:
# Find a logical sum (default and 1) for one or more key s, and save the results to destkey BITOP AND destkey key [key ...] 1# Find a logical or (default and 0) for one or more key s and save the results to destkey BITOP OR destkey key [key ...] # Find logical XOR for one or more key s and save the results to destkey BITOP XOR destkey key [key ...] # Find the logical negation of the given key and save the result to destkey BITOP NOT destkey key
bitpos:
# bitpos: used to return the index position of the operation # 5: Means to start searching from the 5th byte, and 8: means to start searching from the 8th byte 127.0.0.1:6379> BITPOS qq:uv 1 5 8
Application: count the number of logged in users to record the day when users log in. Only record whether they log in every day. The repeated login status is considered logged in. There is no need to record the user's operation behavior, last login time and ip address
For example, there are now the following users: id name 1 Zhang San 3 Li Si 8 Wang Wu # bitset key offset [0,1] # When Li Si logs in, it is recorded that Li Si logs in today bitset login_20191206 3 1 # At this time, the following information is stored in the bitmap: 00000000, 00001000 # Wang Wu logged in. Wang Wu logged in today bitset login_20191206 8 1 # At this time, the following information is stored in the bitmap: 00000001 00001000
Code implementation:
- Initialization data:
@Test
public void initData() throws Exception {
// Add logged in users
jedis.setbit("user:login:20190919", 1, "1"); // The user with id 1 has logged in
jedis.setbit("user:login:20190919", 8, "1"); // The user with id 8 has logged in
jedis.setbit("user:login:20190919", 12, "1"); // The user with id 12 has logged in
jedis.setbit("user:login:20190920", 8, "1"); // The user with id 8 has logged in
jedis.setbit("user:login:20190920", 22, "1"); // The user with id 22 has logged in
jedis.setbit("user:login:20190921", 8, "1"); // The user with id 8 has logged in
jedis.setbit("user:login:20190921", 24, "1"); // The user with id 22 has logged in
}
- Statistics of login number:
@Test
public void useData() throws Exception {
// 1. Count the number of login users in 20190919
System.out.println(jedis.bitcount("user:login:20190919"));
// 2 count the total number of login users in the last three days
jedis.bitop(BitOP.OR, "user:login:last3_1",
"user:login:20190919",
"user:login:20190920",
"user:login:20190921");
System.out.println(jedis.bitcount("user:login:last3_1"));
// 3 count the number of users who have logged in for three consecutive days
jedis.bitop(BitOP.AND, "user:login:last3_2",
"user:login:20190919",
"user:login:20190920",
"user:login:20190921");
System.out.println(jedis.bitcount("user:login:last3_2"));
}
Hyperlog - a cardinality statistical algorithm
Hyperlog is an algorithm used for cardinality Statistics (inaccurate, fuzzy value)
- When the number or volume of input elements is very large, the space required to calculate the cardinality is always fixed and small
In Redis, each hyperlog key only needs 12KB of memory to calculate the cardinality of nearly 2 ^ 64 (about 4.2 billion) different elements
This is in sharp contrast to collections where the more elements consume more memory when calculating cardinality
Considerations for using hyperlog:
- Can errors be tolerated (0.81%)
- Need a single piece of data
- Do you need to use small memory
# pfadd: add any number of elements to the specified hyperlog pfadd qq:tv:uv 003 004 005 # pfcount: used to count the number of elements 127.0.0.1:6379> pfcount qq:tv:uv # pfmerge: merge multiple keys into one key to automatically filter duplicate elements pfmerge tv:uv qq:tv:uv aiqiyi:tv:uv
Code implementation:
// Add logged in users add 10000 users
Pipeline pipelined = jedis.pipelined();
for (int i = 0; i < 5000000; i++) {
pipelined.pfadd("pf:user:login:20190919","user"+i);
}
pipelined.sync();
System.out.println(jedis.pfcount("pf:user:login:20190919"));
GEO - geographic information positioning, storage of longitude and latitude, calculation of distance between two places, calculation range
geoadd Add a geographic location geopos Get geographic location information geodist Get two geographic locations georadius Gets the data within the specified range
Longitude and latitude query: https://lbs.amap.com/tools/picker
Code implementation:
- Package latitude and longitude information:
@Data
@NoArgsConstructor
@AllArgsConstructor
public class CityPosition {
// longitude
private double lng;
// latitude
private double lat;
// Merchant id
private String key;
}
- Initialize the corresponding city data:
@Test
public void initData() throws Exception {
CityPosition s1 = new CityPosition (113.264385, 23.129112, "Guangzhou");
CityPosition s2 = new CityPosition (121.473658, 31.230378, "Shanghai");
CityPosition s3 = new CityPosition (114.085947, 22.547, "Shenzhen");
CityPosition s4 = new CityPosition (113.746262, 23.046237, "Dongguan");
jedis.geoadd("city", s1.getLng(), s1.getLat(), s1.getKey());
jedis.geoadd("city", s2.getLng(), s2.getLat(), s2.getKey());
jedis.geoadd("city", s3.getLng(), s3.getLat(), s3.getKey());
jedis.geoadd("city", s4.getLng(), s4.getLat(), s4.getKey());
}
- Get the corresponding city information:
@Test
public void getInfo() throws Exception {
System.out.println(jedis.geodist("city", "Guangzhou", "Dongguan", GeoUnit.KM));
List<GeoRadiusResponse> datas = jedis.georadius("city", 113.264385,23.129112, 100, GeoUnit.KM);
for (GeoRadiusResponse data : datas) {
System.out.println(data);
System.out.println("city: = " + new String(data.getMember()));
}
}