Grape
All videos: https://segmentfault.com/a/11...

intset is a kind of data structure in Redis. It has the status of ziplist and dict.

Definition of intset?

Intset is one of the underlying implementations of Redis collections. When all the data added are integers, intset is used; otherwise dict is used. In particular, Redis converts the data structure to dict when it encounters adding data as strings, i.e. not integers, that is, it moves all the data in the intset to dict.

What is the meaning of intset?

Intset stores integer elements in an array in order, and reduces the time complexity of finding elements by dichotomy. When the amount of data is large, commands relying on "find" such as SISMEMBER will encounter certain bottlenecks due to the time complexity of O(logn), so dict will replace Intset when the amount of data is large. But the advantage of intset is that it saves more memory than dict, and when the amount of data is small, O(logn) may not be slower than O(1) hash function. This is why intset exists.

Why does intset save more memory?

First, let's look at the structure of Intset:

typedef struct intset {
uint32_t encoding;   //Type encoding of intset
uint32_t length;      //Number of member elements
int8_t contents[];    //Flexible arrays for storing members
} intset;
  

Then the structure of dict is compared:

typedef struct dict {
dictEntry **table;
dictType *type;
unsigned long size;
unsigned long sizemask;
unsigned long used;
void *privdata;
} dict;

//It is important to note that the declaration of content group members as int8_t does not mean that there are members of int8_t type in content. This type declaration can be used for content.    
//It is considered meaningless because the type of intset member depends entirely on the value of the encoding variable. Enoding provides the following three values:
/* Note that these encodings are ordered, so:
 * INTSET_ENC_INT16 < INTSET_ENC_INT32 < INTSET_ENC_INT64. */
#define INTSET_ENC_INT16 (sizeof(int16_t))
#define INTSET_ENC_INT32 (sizeof(int32_t))
#define INTSET_ENC_INT64 (sizeof(int64_t))

//If the encoding of the intset is INTSET_ENC_INT16, the "logical type" of each member of the contents is int16_t.

The spatial size of Intset and Dict structures was observed, and the results were obvious. When the amount of data is small, O(logn) may not be slower than O(1) hash, so the existence of intset becomes necessary.
In addition, when we look at the structure of Intset and look at the source code, we know that there are only three types of encoding. It seems a bit wasteful to use unit32 to store. So can we save some space when we build the structure? The author simply throws a brick to construct the structure as follows:

        typedef struct intset {
        uint32_t length;
        uint8_t encoding;
        int8_t contents[];
        } intset;

Do you think so?

In addition, here we should pay attention to the problem that Intset stores Intset member variables in memory in the same byte order (small end) on any machine. Why?
If we honestly assign values by contents[x], we don't need to consider the byte order problem, but intset specifies the address offset of elements based on the encoding value and violently operates on memory. If the data is truncated, large-end machines and small-end machines will show inconsistency. To avoid this, intset member variables are stored in memory in the same byte order (small end) on any machine.

So under what circumstances will the address offset of the element occur? Don't worry, as we will see in Intset's operation below, watch carefully.

inset's sauce operation?

First of all, we can see a series of operations of Intset from the source code:

    intset *intsetNew(void);
    intset *intsetAdd(intset *is, int64_t value, uint8_t *success);
    intset *intsetRemove(intset *is, int64_t value, int *success);
    uint8_t intsetFind(intset *is, int64_t value);
    int64_t intsetRandom(intset *is);
    uint8_t intsetGet(intset *is, uint32_t pos, int64_t *value);
    uint32_t intsetLen(const intset *is);
    size_t intsetBlobLen(intset *is);

So limited to the length of the article, the author will take the insertion to specific analysis, if other interested, you can check the source code on your own.
For Intset insertion, there are two cases, respectively:

1. The encoding of the inserted value is greater than the encoding of the intset to be inserted
2. The encoding of the inserted value is smaller than the encoding of the intset to be inserted.

   In the first case, if the encoding of value is greater than the encoding of intset to be inserted, the intsetUpgrade AndAdd is called to upgrade the encoding of intset directly and insert it into the head or tail. If the encoding of value is less than the encoding of the intset to be inserted, the encoding of intset need not be upgraded. Inset Search is called to find the appropriate insertion location, and then the data from that location to the end of contents are all moved to the right one grid, and finally the value is inserted into pos.
   Yes, it's very simple. Compare the encoding of the insertion value with the existing encoding value when inserting the element. If it's less than, query the insertion location yourself, or upgrade the intset to insert the head and tail. For this query, the bottom is a binary search. Interested readers can see why it is inserted at the beginning and the end, because the value that may be inserted after extended coding is negative.

Inserted source code:

 /* Insert an integer in the intset *///Success passing null in indicates that the outer caller does not need to know whether the insertion is successful (value already exists), otherwise success is used for this purpose.
    intset *intsetAdd(intset *is, int64_t value, uint8_t *success) {
        uint8_t valenc = _intsetValueEncoding(value);//Calculate the encoding of value according to the size of value
        uint32_t pos;
        if (success) *success = 1;
    
        /* Upgrade encoding if necessary. If we need to upgrade, we know that
         * this value should be either appended (if > 0) or prepended (if < 0),
         * because it lies outside the range of existing values. */
        if (valenc > intrev32ifbe(is->encoding)) {
            //This insertion requires a change in encoding (no search is required, because the encoding change means that value must be inserted at the beginning or end of the content)
            /* This always succeeds, so we don't need to curry *success. */
            return intsetUpgradeAndAdd(is,value);
        } else {
            /* Abort if the value is already present in the set.
             * This call will populate "pos" with the right position to insert
             * the value when it cannot be found. */
            if (intsetSearch(is,value,&pos)) {
                if (success) *success = 0;//The value already exists in the intset and the return fails
                return is;
            }
    
            is = intsetResize(is,intrev32ifbe(is->length)+1);
            if (pos < intrev32ifbe(is->length)) intsetMoveTail(is,pos,pos+1);//Move one grid to the right
        }
    
        _intsetSet(is,pos,value);//insert values
        is->length = intrev32ifbe(intrev32ifbe(is->length)+1);
        return is;
    }
    
    /* Return the required encoding for the provided value. *///The required encoding type static uint8_t_intsetValueEncoding (int64_t v) is determined by the size of the V value.{
        if (v < INT32_MIN || v > INT32_MAX)
            return INTSET_ENC_INT64;
        else if (v < INT16_MIN || v > INT16_MAX)
            return INTSET_ENC_INT32;
        else
            return INTSET_ENC_INT16;
    }
    
    /* Upgrades the intset to a larger encoding and inserts the given integer. *///This function is executed on the premise that the value parameter exceeds the current encoding // reallocates memory for is - > content and modifies the encoding to add value to the intsetstatic intset *intsetUpgradeAndAdd(intset *is, int64_t value){
        uint8_t curenc = intrev32ifbe(is->encoding);//Current encoding type
        uint8_t newenc = _intsetValueEncoding(value);//New coding types
        int length = intrev32ifbe(is->length);
        int prepend = value < 0 ? 1 : 0;//Because value must exceed the coding limit, it depends on whether value is greater than or less than 0 to determine whether value is placed in content[0] or content[length]
    
        /* First set new encoding and resize */
        is->encoding = intrev32ifbe(newenc);
        is = intsetResize(is,intrev32ifbe(is->length)+1);
    
        /* Upgrade back-to-front so we don't overwrite values.
         * Note that the "prepend" variable is used to make sure we have an empty
         * space at either the beginning or the end of the intset. */
        while(length--)
            //Take curenc as the encoding reverse order to extract all the values and assign them to the new location
            _intsetSet(is,length+prepend,_intsetGetEncoded(is,length,curenc));
    
        /* Set the value at the beginning or the end. */
        if (prepend)
            _intsetSet(is,0,value);
        else
            _intsetSet(is,intrev32ifbe(is->length),value);
        is->length = intrev32ifbe(intrev32ifbe(is->length)+1);
        return is;
    }
    
    /* Resize the intset *///Remove the memory allocation of is and reallocate the memory static intset *intsetResize(intset *is, uint32_t len) of intset length len{
        uint32_t size = len*intrev32ifbe(is->encoding);
        is = zrealloc(is,sizeof(intset)+size);
        return is;
    }
    
    //Replicate the whole block of data indexed from the end of intset to index (the value of from remains unchanged after replication, but can be overwritten) static void intsetMoveTail (intset*is, uint32_t from, uint32_t to){
        void *src, *dst;
        uint32_t bytes = intrev32ifbe(is->length)-from;
        uint32_t encoding = intrev32ifbe(is->encoding);
    
        if (encoding == INTSET_ENC_INT64) {
            src = (int64_t*)is->contents+from;
            dst = (int64_t*)is->contents+to;
            bytes *= sizeof(int64_t);
        } else if (encoding == INTSET_ENC_INT32) {
            src = (int32_t*)is->contents+from;
            dst = (int32_t*)is->contents+to;
            bytes *= sizeof(int32_t);
        } else {
            src = (int16_t*)is->contents+from;
            dst = (int16_t*)is->contents+to;
            bytes *= sizeof(int16_t);
        }
        memmove(dst,src,bytes);
    }

summary

Through the implementation of intset, we can find that the time complexity of executing some commands that need to be queried based on sequential storage of integer set is not O(1) in the document, and the average time complexity of query is O(logn) in the operation of a member insertion. So when the amount of integer set data increases, redis uses dict as the underlying implementation of the set, reducing the time complexity of commands such as SADD, SREM, SISMEMBER to O(1), which, of course, consumes more memory than intset. So Redis only uses intset when the data volume is small.