Redis Source Learning Brief (7) object Principle and Personal Understanding

Object is an encapsulation system in redis.It encapsulates string, list, set, zset and hash as a unified object named robj.This data structure stores data of type, encoding, number of references, data and LRU replacement algorithm.Specifically, first look at the definition of this data structure, which is defined in server.h.

1.  //redis belongs to the key-value database  
2.  //nosql database, this mapping relationship is maintained using dict  
3.  //The dict entity data (dictentry) has a field void *value  
4. typedef struct redisObject {  
5.     unsigned type:4;  
6.     /* 
7.         string 0 
8.         list 1 
9.         set 2 
10.         zset 3 
11.         hash 4 
12.         4 Bits 
13.  
14.     */  
15.   
16.     unsigned encoding:4;  
17.     //Encoding 4 bit s  
18.     unsigned lru:LRU_BITS; /* LRU time (relative to global lru_clock) or 
19.                             * LFU data (least significant 8 bits frequency 
20.                             * and most significant 16 bits access time). */  
21.     //24bit, LRU Replacement Algorithm  
22.     int refcount;//Reference Count 32bit 4 bytes  
23.     void *ptr;//32-bit system 4 bytes 64-bit system 8 bytes  
24.     //Point to real data  
25. } robj;  

Let's talk more about this.First, type, defined as follows, is actually four bit s used to store 0-4 data.

1. /* The actual Redis Object */  
2. #define OBJ_STRING 0    /* String object. */  
3. #define OBJ_LIST 1      /* List object. */  
4. #define OBJ_SET 2       /* Set object. */  
5. #define OBJ_ZSET 3      /* Sorted set object. */  
6. #define OBJ_HASH 4      /* Hash object. */  

Then encoding.In general, there are 11 encoding modes, which are also stored in 4 bits, with 4 bits being the maximum of 16 encoding modes.

1. #define OBJ_ENCODING_RAW 0     /* Raw representation */  
2. #define OBJ_ENCODING_INT 1     /* Encoded as integer */  
3. #define OBJ_ENCODING_HT 2      /* Encoded as hash table */  
4. #define OBJ_ENCODING_ZIPMAP 3  /* Encoded as zipmap */  
5. #define OBJ_ENCODING_LINKEDLIST 4 /* No longer used: old list encoding. */  
6. #define OBJ_ENCODING_ZIPLIST 5 /* Encoded as ziplist */  
7. #define OBJ_ENCODING_INTSET 6  /* Encoded as intset */  
8. #define OBJ_ENCODING_SKIPLIST 7  /* Encoded as skiplist */  
9. #define OBJ_ENCODING_EMBSTR 8  /* Embedded sds string encoding */  
10. #define OBJ_ENCODING_QUICKLIST 9 /* Encoded as linked list of ziplists */  
11. #define OBJ_ENCODING_STREAM 10 /* Encoded as a radix tree of listpacks */  

Next comes lru, which stores the LRU replacement algorithm. For now, let's dig into it and use three bytes of LRU_BITS (24bits) storage.

refcount is a reference pointer, which acts like a smart pointer, calculating the number of references to an object and releasing its usage space when the number of references to an object is 0.

The last void *ptr is the final pointer to the data.These are basically the types of data that were previously analyzed.

The data structure is hard to understand, and then look at its basic implementation function.They are basically implemented in object.c.

First, it is created, which returns a robj pointer based on the type and the data ptr points to.

1. robj *createObject(int type, void *ptr) {  
2.   
3.     robj *o = zmalloc(sizeof(*o));//Allocate space  
4.     o->type = type;//Setting Type  
5.     o->encoding = OBJ_ENCODING_RAW;//Native encoding mode  
6.     //There are many different types here, and we'll discuss them in more detail later.  
7.     o->ptr = ptr;//Execute real data  
8.     o->refcount = 1;//Reference Count  
9.   
10.     /* Set the LRU to the current lruclock (minutes resolution), or 
11.      * alternatively the LFU counter. */  
12.     //Page replacement algorithms are not considered first, but are studied later.  
13.     if (server.maxmemory_policy & MAXMEMORY_FLAG_LFU) {  
14.         o->lru = (LFUGetTimeInMinutes()<<8) | LFU_INIT_VAL;  
15.     } else {  
16.         o->lru = LRU_CLOCK();  
17.     }  
18.     return o;  
19. }  

Set a special refcount.Make the object shred.Increasing references and decreasing references check this particular reference count.Here's how you can make objects shared by setting the refcout value to int_max.

1. robj *makeObjectShared(robj *o) {  
2.     serverAssert(o->refcount == 1);  
3.     o->refcount = OBJ_SHARED_REFCOUNT;  
4.     //Make the object shared and set obj_shared_refcount to int_max  
5.     return o;  
6. }  

Next are two ways to create string objects.

Let's start by introducing how to create native strings.Simply call sdsnewlen's method, then get a string pointer and create it by calling create.

1. /* Create a string object with encoding OBJ_ENCODING_RAW, that is a plain 
2.  * string object where o->ptr points to a proper sds string. */  
3. robj *createRawStringObject(const char *ptr, size_t len) {  
4.     //Create RawStringObject Type  
5.     //Is actually an sds type, and by default encoding is OBJ_ENCODING_RAW  
6.     /* 
7.         #define OBJ_STRING 0     
8.         #define OBJ_LIST 1      
9.         #define OBJ_SET 2        
10.         #define OBJ_ZSET 3       
11.         #define OBJ_HASH 4       
12.     */  
13.     return createObject(OBJ_STRING, sdsnewlen(ptr,len));  
14. }  

Another way to create it is to add a string directly behind the object, which is stored directly behind the object.Specifies that the entire object is no more than 64 bytes.

1. /* Create a string object with encoding OBJ_ENCODING_EMBSTR, that is 
2.  * an object where the sds string is actually an unmodifiable string 
3.  * allocated in the same chunk as the object itself. */  
4. robj *createEmbeddedStringObject(const char *ptr, size_t len) {  
5.     //Step by step through the code  
6.     robj *o = zmalloc(sizeof(robj)+sizeof(struct sdshdr8)+len+1);  
7.     //sds headspace allocated robj data and sdshdr8 plus len+1 length, the extra 1 length stores null  
8.     struct sdshdr8 *sh = (void*)(o+1);  
9.     //o+1 grows based on the bytes of robj's structure  
10.     //robj is 16 bytes in total, so plus 16 is just the head of sdshdr8  
11.     o->type = OBJ_STRING;  
12.     o->encoding = OBJ_ENCODING_EMBSTR;  
13.     o->ptr = sh+1;  
14.     //sh+1 is the number of bytes plus sdshdr8, whereas sdshdr8 is 3 bytes len+alloc+flags  
15.     //Then after adding 1, ptr points to sh->buf  
16.     /* 
17.     An interesting phenomenon was found in studying the bytes occupied by characters. 
18.         struct __attribute__ ((__packed__)) sdshdr8 { 
19.                 uint8_t len;  
20.                 uint8_t alloc;  
21.                 unsigned char flags;  
22.                 char buf[]; 
23.             }; 
24.     Because char buf[]] causes the calculation of the size of sdshdr8 to be less than a buf[] odd 
25.     With a pointer, 8 bytes are added for storage 
26.     The length of [] will also be increased if it has a value and will not be calculated if it is not written.The buf's address is the last flags. 
27.  
28.     */  
29.     o->refcount = 1;  
30.     //Initialize Reference Count  
31.     if (server.maxmemory_policy & MAXMEMORY_FLAG_LFU) {  
32.         o->lru = (LFUGetTimeInMinutes()<<8) | LFU_INIT_VAL;  
33.     } else {  
34.         o->lru = LRU_CLOCK();  
35.     }  
36.     //Page algorithms are not studied first.  
37.     //Initialization of sds  
38.     sh->len = len;  
39.     sh->alloc = len;  
40.     sh->flags = SDS_TYPE_8;  
41.     if (ptr == SDS_NOINIT)  
42.         //sds_noint is a static const char *  
43.         //Determine whether to initialize  
44.   
45.         sh->buf[len] = '\0';  
46.     else if (ptr) {  
47.         //Initialize to see if the ptr value is NULL  
48.         memcpy(sh->buf,ptr,len);  
49.         sh->buf[len] = '\0';  
50.     } else {  
51.         memset(sh->buf,0,len+1);  
52.     }  
53.     return o;  
54. }  

The encoding of these two strings is invoked through a total string interface.As mentioned above, EmbeddedString consists of the following sections

robj 16-byte sdshdr8 header 3 bytes plus a'0'byte, last stored 44 bytes

3+16+1+44=64.It happens that 64 sections have all the data loaded.

1. /* Create a string object with EMBSTR encoding if it is smaller than 
2.  * OBJ_ENCODING_EMBSTR_SIZE_LIMIT, otherwise the RAW encoding is 
3.  * used. 
4.  * 
5.  * The current limit of 44 is chosen so that the biggest string object 
6.  * we allocate as EMBSTR will still fit into the 64 byte arena of jemalloc. */  
7. #define OBJ_ENCODING_EMBSTR_SIZE_LIMIT 44  
8. robj *createStringObject(const char *ptr, size_t len) {  
9.     //Ensure that the size of the embedded dedstringbox is 64 or call creteRawStringObject  
10.     //Set the maximum length to 44+16+3+1=64  
11.     //In fact, sdshdr8 can store 256 characters  
12.     if (len <= OBJ_ENCODING_EMBSTR_SIZE_LIMIT)  
13.         return createEmbeddedStringObject(ptr,len);  
14.     else  
15.         return createRawStringObject(ptr,len);  
16. }  

Finished encapsulating strings.Take a look at the packaging of integers.Overall, when 8 bytes of 32-bit storage can be used, the ptr in robj is the integer directly.Equivalent to the number of instants in the assembly.Beyond that range, the data is stored as a string.The ptr is then used as the pointer.

1. robj *createStringObjectFromLongLong(long long value) {  
2.     robj *o;  
3.     if (value >= 0 && value < OBJ_SHARED_INTEGERS) {  
4.         incrRefCount(shared.integers[value]);  
5.         //You probably know that shared is an institution that contains a lot of shared data.  
6.         //struct sharedObjectsStruct  
7.         o = shared.integers[value];  
8.     } else {  
9.         if (value >= LONG_MIN && value <= LONG_MAX) {  
10.             o = createObject(OBJ_STRING, NULL);  
11.             o->encoding = OBJ_ENCODING_INT;  
12.             o->ptr = (void*)((long)value);  
13.             //If you can use 8 bytes of storage, storing it directly in ptr is equivalent to the number of assemblies immediately.  
14.         } else {  
15.             //Use sds's schema storage if you can't store it  
16.             o = createObject(OBJ_STRING,sdsfromlonglong(value));  
17.         }  
18.     }  
19.     return o;  
20. }  

Floating-point numbers are stored in the form of strings.

1. /* Create a string object from a long double. If humanfriendly is non-zero 
2.  * it does not use exponential format and trims trailing zeroes at the end, 
3.  * however this results in loss of precision. Otherwise exp format is used 
4.  * and the output of snprintf() is not modified. 
5.  * 
6.  * The 'humanfriendly' option is used for INCRBYFLOAT and HINCRBYFLOAT. */  
7. robj *createStringObjectFromLongDouble(long double value, int humanfriendly) {  
8.     char buf[MAX_LONG_DOUBLE_CHARS];  
9.     //Human friendly is used to handle floating-point numbers, which you can ignore first  
10.     int len = ld2string(buf,sizeof(buf),value,humanfriendly);  
11.     //Convert a long double to sds and store it  
12.     return createStringObject(buf,len);  
13. }  

Replication of strings between objects is limited to data stored in strings (including integers stored using strings), or integers stored using immediate numbers.

1. /* Duplicate a string object, with the guarantee that the returned object 
2.  * has the same encoding as the original one. 
3.  * 
4.  * This function also guarantees that duplicating a small integere object 
5.  * (or a string object that contains a representation of a small integer) 
6.  * will always result in a fresh object that is unshared (refcount == 1). 
7.  * 
8.  * The resulting object always has refcount set to 1. */  
9. robj *dupStringObject(const robj *o) {  
10.     robj *d;  
11.   
12.     serverAssert(o->type == OBJ_STRING);  
13.   
14.     switch(o->encoding) {  
15.     case OBJ_ENCODING_RAW:  
16.         return createRawStringObject(o->ptr,sdslen(o->ptr));  
17.     case OBJ_ENCODING_EMBSTR:  
18.         return createEmbeddedStringObject(o->ptr,sdslen(o->ptr));  
19.     case OBJ_ENCODING_INT:  
20.         d = createObject(OBJ_STRING, NULL);  
21.         d->encoding = OBJ_ENCODING_INT;  
22.         d->ptr = o->ptr;  
23.         return d;  
24.     default:  
25.         serverPanic("Wrong encoding.");  
26.         break;  
27.     }  
28. }  

Following are some other types of creation, most of which are created using the create function of that type and then releasing space, which are relatively simple.There are also functions that reduce the reference count, which are straightforward and simple.Just look straight at it.

1. robj *createQuicklistObject(void) {  
2.     quicklist *l = quicklistCreate();  
3.     robj *o = createObject(OBJ_LIST,l);  
4.     o->encoding = OBJ_ENCODING_QUICKLIST;  
5.     return o;  
6. }  
7.   
8. robj *createZiplistObject(void) {  
9.     unsigned char *zl = ziplistNew();  
10.     robj *o = createObject(OBJ_LIST,zl);  
11.     o->encoding = OBJ_ENCODING_ZIPLIST;  
12.     return o;  
13. }  
14.   
15. robj *createSetObject(void) {  
16.     dict *d = dictCreate(&setDictType,NULL);  
17.     robj *o = createObject(OBJ_SET,d);  
18.     o->encoding = OBJ_ENCODING_HT;  
19.     return o;  
20. }  
21.   
22. robj *createIntsetObject(void) {  
23.     intset *is = intsetNew();  
24.     robj *o = createObject(OBJ_SET,is);  
25.     o->encoding = OBJ_ENCODING_INTSET;  
26.     return o;  
27. }  
28.   
29. robj *createHashObject(void) {  
30.     unsigned char *zl = ziplistNew();  
31.     robj *o = createObject(OBJ_HASH, zl);  
32.     o->encoding = OBJ_ENCODING_ZIPLIST;  
33.     return o;  
34. }  
35.   
36. robj *createZsetObject(void) {  
37.     zset *zs = zmalloc(sizeof(*zs));  
38.     robj *o;  
39.   
40.     zs->dict = dictCreate(&zsetDictType,NULL);  
41.     zs->zsl = zslCreate();  
42.     o = createObject(OBJ_ZSET,zs);  
43.     o->encoding = OBJ_ENCODING_SKIPLIST;  
44.     return o;  
45. }  
46.   
47. robj *createZsetZiplistObject(void) {  
48.     unsigned char *zl = ziplistNew();  
49.     robj *o = createObject(OBJ_ZSET,zl);  
50.     o->encoding = OBJ_ENCODING_ZIPLIST;  
51.     return o;  
52. }  
53.   
54. robj *createStreamObject(void) {  
55.     stream *s = streamNew();  
56.     robj *o = createObject(OBJ_STREAM,s);  
57.     o->encoding = OBJ_ENCODING_STREAM;  
58.     return o;  
59. }  
60.   
61. robj *createModuleObject(moduleType *mt, void *value) {  
62.     moduleValue *mv = zmalloc(sizeof(*mv));  
63.     mv->type = mt;  
64.     mv->value = value;  
65.     return createObject(OBJ_MODULE,mv);  
66. }  
67.   
68. void freeStringObject(robj *o) {  
69.     if (o->encoding == OBJ_ENCODING_RAW) {  
70.         sdsfree(o->ptr);  
71.     }  
72. }  
73.   
74. void freeListObject(robj *o) {  
75.     if (o->encoding == OBJ_ENCODING_QUICKLIST) {  
76.         quicklistRelease(o->ptr);  
77.     } else {  
78.         serverPanic("Unknown list encoding type");  
79.     }  
80. }  
81.   
82. void freeSetObject(robj *o) {  
83.     switch (o->encoding) {  
84.     case OBJ_ENCODING_HT:  
85.         dictRelease((dict*) o->ptr);  
86.         break;  
87.     case OBJ_ENCODING_INTSET:  
88.         zfree(o->ptr);  
89.         break;  
90.     default:  
91.         serverPanic("Unknown set encoding type");  
92.     }  
93. }  
94.   
95. void freeZsetObject(robj *o) {  
96.     zset *zs;  
97.     switch (o->encoding) {  
98.     case OBJ_ENCODING_SKIPLIST:  
99.         zs = o->ptr;  
100.         dictRelease(zs->dict);  
101.         zslFree(zs->zsl);  
102.         zfree(zs);  
103.         break;  
104.     case OBJ_ENCODING_ZIPLIST:  
105.         zfree(o->ptr);  
106.         break;  
107.     default:  
108.         serverPanic("Unknown sorted set encoding");  
109.     }  
110. }  
111.   
112. void freeHashObject(robj *o) {  
113.     switch (o->encoding) {  
114.     case OBJ_ENCODING_HT:  
115.         dictRelease((dict*) o->ptr);  
116.         break;  
117.     case OBJ_ENCODING_ZIPLIST:  
118.         zfree(o->ptr);  
119.         break;  
120.     default:  
121.         serverPanic("Unknown hash encoding type");  
122.         break;  
123.     }  
124. }  
125.   
126. void freeModuleObject(robj *o) {  
127.     moduleValue *mv = o->ptr;  
128.     mv->type->free(mv->value);  
129.     zfree(mv);  
130. }  
131.   
132. void freeStreamObject(robj *o) {  
133.     freeStream(o->ptr);  
134. }  
135.   
136. void incrRefCount(robj *o) {  
137.     if (o->refcount != OBJ_SHARED_REFCOUNT) o->refcount++;  
138. }  
139.   
140. void decrRefCount(robj *o) {  
141.     if (o->refcount == 1) {  
142.         switch(o->type) {  
143.         case OBJ_STRING: freeStringObject(o); break;  
144.         case OBJ_LIST: freeListObject(o); break;  
145.         case OBJ_SET: freeSetObject(o); break;  
146.         case OBJ_ZSET: freeZsetObject(o); break;  
147.         case OBJ_HASH: freeHashObject(o); break;  
148.         case OBJ_MODULE: freeModuleObject(o); break;  
149.         case OBJ_STREAM: freeStreamObject(o); break;  
150.         default: serverPanic("Unknown object type"); break;  
151.         }  
152.         zfree(o);  
153.     } else {  
154.         if (o->refcount <= 0) serverPanic("decrRefCount against refcount <= 0");  
155.         if (o->refcount != OBJ_SHARED_REFCOUNT) o->refcount--;  
156.     }  
157. }  
158.   
159. /* This variant of decrRefCount() gets its argument as void, and is useful 
160.  * as free method in data structures that expect a 'void free_object(void*)' 
161.  * prototype for the free method. */  
162. void decrRefCountVoid(void *o) {  
163.     decrRefCount(o);  
164. }