Computer memory
External memory characteristics
Advantages: permanent storage capacity, portability
Disadvantages: long access time
Principle: minimize the number of accesses to external memory
External memory data access mode
It is divided into two stages: location and access
External memory is divided into fixed length storage space
The data access of external memory is carried out in blocks, so as to reduce the positioning times of external memory and the time consumption of external memory reading and writing
Document organization and management
A file is a data structure stored in external memory and a collection of a large number of records with the same nature. Records are data blocks with independent logical meaning and basic data units

By record type

Operating system files
A continuous sequence of characters with no obvious structure

Database file
A structured set of records. Each record consists of one or more data items. Each data item is a basic data unit that cannot be subdivided


According to the length of recorded information

Fixed length file

Variable length file

Document organization
The operating system organizes data in the form of files and completes the mapping from the logical structure of files to the physical structure of external memory

File organization logic
Fixed length record, variable length record and key access record

file physical structure
Sequential file, hash file, index file, inverted file
C + + file stream
File stream is a data stream that takes external storage files as input and output objects
Including istream,ostream,iostream and fstream, ifstream and OFSTREAM
External sorting
The data file in external memory is divided into several segments. One segment is read into memory each time and sorted by internal sorting
These sorted segments or sub files are called sequential or merged segments
Write the sequence back to external memory to make room for memory, and then process other unordered segments
Merge the sequence after processing
 Permutation selection sorting: initialize the external storage file into an ordered string set as long as possible
 Merge and sort: merge and sort the sequential string set one by one to form a globally ordered external memory file
Time composition
 The time required to generate the inner sort of the initial sequence
 Read and write time required to initialize the sequence and merge process
 Time required for internal consolidation
Reducing the number of reads and writes of external memory information is the key to improve the efficiency of external sorting
Permutation selection sort
objective
Generate several initial sequence strings from the file (the longer the sequence string, the better, and the fewer the sequence strings, the better)
realization
This is done with the help of the heap in RAM
 Initialize minimum heap: improve sorting efficiency in RAM
 Read M records from the buffer and put them into the array RAM
 Set tail flag: LAST=M1
 Create a minimum heap
 Repeat the following steps until the heap is empty (end condition) (i.e. last < 0)
 Send the record (root node) with the minimum key value to the output buffer
 Let R be the next record in the input buffer
 If the key of R is not less than the key value just output, put r in the root node
 Otherwise, use the record of the last position in the array to replace the root node, then put R in the last position (wait for the next sequential processing), and set LAST=LAST1
 Rearrange the heap and screen out the root node
The heap size is M, and the minimum length of the sequence is M
 At least those records in the original heap will become part of the sequence
 At best, it is possible to generate an entire file into a sequence at one time
 The average length is 2M
algorithm
//A is the array stored after reading n elements from external memory template <class Elem> void ReplacementSelection(Elem * A, int n, const char * in, const char * out) { Elem mval;//Minimum value to store the minimum heap Elem r;//Stores the elements read from the input buffer FILE * iptF;//Input file handle FILE * optF;//Output file handle Buffer<Elem> input;//Input buffer Buffer<Elem> output;// Output buffer initFiles(inputFile, outputFile, in, out);//Initialize I / O file initMinHeapArry(inputFile, n, A);//Initialize the data of the heap and read in n data MinHeap<Elem> H(A, n, n);//Establish minimum heap initInputBuffer(input, inputFile);//Initialize inputbuffer and read in some data for(int last =n1; last >= 0;) { mval = H.heapArray[0];//Minimum value of heap sendToOutputBuffer(input,output,iptF,optF, mval); input.read(r);//Reads a record from the input buffer if(!less(r, mval)) H.heapArray[0] = r; else//Otherwise, replace the root node with the last position record and put r in the last { H.heapArray[0] = H.heapArray[last]; H.heapArray[last] = r; H.setSize(last); last; } if (last!=0) H.SiftDown(0);//Heap adjustment } endUp(output,inputFile,outputFile);//Process output buffer }
Merge sort
Merging property
Two way merging
Merge tree height ⌈ log 2 m ⌉ + 1 \lceil\log_{2}m\rceil+1 ⌈ log2 ⌉ m ⌉ + 1, proceed ⌈ log 2 m ⌉ \lceil\log_{2}m\rceil ⌈ log2 ⌉ m ⌉ scanning
Two input buffers and one output buffer are required
kway merging
kway merging, scanning k strings each time, the number of merging times is [ log k m ] [\log_{k}m] [logkm]
k input buffers and one output buffer are required
Best merge tree
The arrangement of merging order affects the number of reads and writes. Taking the length of the initial sequence as the weight is essentially the problem of Huffman tree optimization
process
Take the number of blocks of all initial sequences as the leaf nodes of the tree. If it is kway merging, a kfork Huffman tree is established. Such a Huffman tree is the best merging tree
Multiway merging tree
When doing kway merging, you need to compare K1 times each time to find the required records, which is expensive. You want to improve the efficiency of finding the minimum value among the current values of K merging strings
Winner tree
Using complete binary tree as storage structure
Leaf nodes are represented by arrays L[1... n], and internal nodes are represented by arrays B[1... n1]
What is stored in array B is actually the index of array L
The internal node records the winner
Relationship of nodes
nway merging, the winner tree has 2n1 nodes

The number of external nodes is n, the number of internal nodes is n1, and the depth of the competition tree is s = ⌈ log 2 n ⌉ − 1 s=\lceil\log_{2}n\rceil1 s=⌈log2n⌉−1

The number of the lowest and leftmost internal nodes is 2s

The number of internal nodes in the lowest layer is n2s

The number of external nodes at the bottom layer is twice the number of internal nodes at the bottom layer, i.e L o w E x t = 2 ( n − 2 s ) LowExt=2(n2^s) LowExt=2(n−2s)

The number (offset) of all nodes above the lowest external node is o f f s e t = 2 s + 1 − 1 offset=2^{s+1}1 offset=2s+1−1

The relationship between the external node L[i] and the internal parent node B[p] is shown as follows
{ ( i + o f f s e t ) / 2 i ≤ L o w E x t ( i − L o w E x t + n − 1 ) / 2 i > L o w E x t \begin{cases}(i+offset)/2 &i\le LowExt\\(iLowExt+n1)/2 &i>LowExt \end{cases} {(i+offset)/2(i−LowExt+n−1)/2i≤LowExti>LowExt
Characteristics of winner tree

Determine the winner of a game by comparing the scores of two players
From the bottom of the tree, there is a competition between every two leaves. The losers are eliminated, and the winners continue to compete upward. The tree root records the winners of the whole competition

If the score of player L[i] changes, you can modify the winner tree
Along the path from L[i] to the root node, compare with the values of brother nodes, and modify the values of binary tree nodes according to the competition structure without modifying the competition results of other parts
Loser tree
Node records losers. Add a root node B[0] on the root node to record the last winner
When reconstructing the loser tree, it only needs to be compared with the nodes on the path, not with the sibling nodes, which simplifies the reconstruction process
Competition process

Match the new node entering the tree with its parent node
Store the loser in the parent node, and then compete the winner with the parent node of the upper level

The game continues until node B[1]
Put the loser's index on node B[1] and the winner's index on node B[0]
algorithm
template<class T> class LoserTree { private: int MaxSize;// Maximum number of players int n;// Current players int LowExt;// Number of external nodes at the bottom layer int offset;// Total number of nodes above the lowest external node int *B;// The loser tree array actually stores subscripts T *L;// Element array void Play(int p, int lc, int rc, int(*winner)(T A[], int b, int c), int(*loser)(T A[], int b, int c));// In the internal node, branch up from the right public: LoserTree(int Treesize = MAX); ~LoserTree(){delete [] B;} void Initialize(T A[], int size,int (*winner)(T A[], int b, int c), int(*loser)(T A[], int b, int c));// Initialize loser tree int Winner();// Returns the winner index void RePlay(int i, int(*winner)(T A[], int b, int c), int (*loser)(T A[], int b, int c));// Reconstruct the loser tree };
Initialize loser tree
template<class T> void LoserTree<T>::Initialize(T A[], int size, int(*winner)(T A[], int b, int c), int(*loser)(T A[], int b, int c)) { int i,s; n = size;// Initializing member variables L = A; for (s = 1; 2*s <= n1; s += s);//Calculate the number of nodes in the penultimate layer LowExt = 2*(ns); offset = 2*s1; for (i = 2; i <= LowExt; i += 2)// Bottom external node competition Play((offset+i)/2, i1, i, winner, loser); // Process the remaining external nodes if (n%2)//n is an odd number, and the internal node is compared with the external node once { // The ratio of the left winner in the temporary parent node to the external right child node Play(n/2, B[(n1)], LowExt+1, winner, loser); i = LowExt+3; } else i = LowExt+2; for (; i <= n; i += 2)// Competition of remaining external nodes Play((iLowExt+n1)/2, i1, i, winner, loser); }
Generate loser tree
template<class T> void LoserTree<T>::PLAY(int p, int lc, int rc, int(*winner)(T A[], int b, int c), int(*loser)(T A[], int b, int c)) { B[p] = loser(L, lc, rc);//The loser index is placed in B[p] int temp1, temp2; temp1 = winner(L, lc, rc); while (p>1 && p%2) { //p is an odd upward game temp2 = winner(L, temp1, B[p/2]);//Winner vs. father B[p/2] = loser(L, temp1, B[p/2]);//Losers stay temp1 = temp2;//Put the winner in temp1 p/=2;//p points up to the parent node } B[p/2] = temp1;//B[p] is the left child or p=1 }
Reconstruct the loser tree
void LoserTree<T>::RePlay(int i, int (*winner)(T A[], int b, int c), int (*loser)(T A[], int b, int c)) { int p;//Temporary variable used to calculate the index of the parent node if (i <= LowExt)//Determine the location of the parent node p = (i+offset)/2; else p = (iLowExt+n1)/2; B[0] = winner(L, i, B[p]);// Save the winner's index in B[0] B[p] = loser(L, i, B[p]);// Save the loser's index in B[p] for (; (p/2) >= 1; p/=2)// Race up the path { int temp;// Index to temporarily store winners temp = winner(L,B[p/2], B[0]); B[p/2] = loser(L,B[p/2], B[0]); B[0] = temp; } }
Efficiency of multiway merging
Merge k sequential strings
 Original method: the time to find each minimum value is O(k), and the total time to generate a sequence of size n is O(kn)
 Loser tree method: O(k) is required to initialize the loser tree containing K players, O(logk) is required to read in a new value and reconstruct the loser tree, and the total time to generate a sequence of size n is O(k+nlogk)