1, Decision tree
 Decision Tree (Decision Tree) is a decision analysis method that calculates the probability that the expected value of net present value is greater than or equal to zero by forming a Decision Tree on the basis of knowing the occurrence probability of various situations, evaluates the project risk and judges its feasibility. It is a graphical method of intuitively using probability analysis. Because this decision branch is drawn in a graph, it is very similar to the branch of a tree, so it is called Decision Tree In learning, the Decision Tree is a prediction model, which represents a mapping relationship between object attributes and object values. Entropy = the disorder degree of the system. The spanning tree algorithm uses entropy using algorithms ID3, C4.5 and C5.0. This measurement is based on the concept of entropy in informatics theory.
 Decision tree is a tree structure, in which each internal node represents a test on an attribute, each branch represents a test output, and each leaf node represents a category.
 Classification tree (decision tree) It is a very common classification method. It is a kind of supervised learning. The socalled supervised learning is that given a pile of samples, each sample has a set of attributes and a category. These categories are determined in advance, then a classifier can be obtained through learning, which can give correct classification to new objects. Such machine learning is called supervised learning .
advantage:
 The decision tree is easy to understand and realize. People do not need users to know a lot of background knowledge in the learning process. At the same time, it can directly reflect the characteristics of data. As long as it is explained, it has the ability to understand the meaning expressed by the decision tree.
 For the decision tree, the preparation of data is often simple or unnecessary, and it can deal with data and conventional attributes at the same time, and can make feasible and effective results for large data sources in a relatively short time.
 It is easy to evaluate the model through static test and measure the reliability of the model; if an observed model is given, it is easy to deduce the corresponding logical expression according to the generated decision tree.
Disadvantages:
 Fields of continuity are difficult to predict.
 For data with time sequence, a lot of preprocessing work is required.
 When there are too many categories, errors may increase faster.
 When the general algorithm classifies, it only classifies according to one field.
2, Related concepts
2.1 Gini index
 Gini index (Gini impure) represents the probability that a randomly selected sample in the sample set is misdivided.
 Note: the smaller the Gini index, the smaller the probability that the selected samples in the set are mixed up, that is, the higher the purity of the set, on the contrary, the more impure the set. When all samples in the set are one class, the Gini index is 0
 The Gini index is calculated as follows:
2.2 information entropy
 The socalled information entropy is a quite abstract concept in mathematics. Here, it is advisable to understand information entropy as the occurrence probability of a specific information. Information entropy and thermodynamic entropy are closely related. According to Charles H. Bennett's reinterpretation of Maxwell's Demon, the destruction of information is an irreversible process, so the destruction of information conforms to the second law of thermodynamics Generating information is the process of introducing negative (thermodynamic) entropy into the system, so the symbol of information entropy should be opposite to thermodynamic entropy.
 Generally speaking, when a kind of information has a higher probability of occurrence, it indicates that it is more widely spread, or more cited. We can think that from the perspective of information dissemination, information entropy can represent the value of information. In this way, we have a standard to measure the value of information and can make more inferences about the problem of knowledge circulation.
 Calculation formula:
 Where, X represents random variables, and the corresponding set is the set of all possible outputs, which is defined as a symbolic set. The output of random variables is represented by X. P(x) represents the output probability function. The greater the uncertainty of variables, the greater the entropy, and the greater the amount of information needed to make it clear
3, ID3 algorithm
3.1 concept
 ID3 algorithm was first proposed by J. Ross Quinlan at the University of Sydney in 1975. The core of the algorithm is "information entropy" By calculating the information gain of each attribute, ID3 algorithm considers that the attribute with high information gain is a good attribute. Each division selects the attribute with the highest information gain as the division standard, and repeats this process until a decision tree that can perfectly classify training samples is generated.
3.2 steps
 step1: calculate the initial information entropy and use the above formula to calculate the information entropy
 Step 2: calculate the information entropy of each feature. The calculation method is to first calculate the information entropy of each feature value according to the above formula, multiply it by the proportion of the feature value, add it is the entropy of the feature, and then subtract the information entropy calculated from the win, which is the information gain. The following figure takes color as an example
 step3: sort by information gain
 step4: select the feature of maximum information gain and take this feature as the dividing node
 step5: delete the feature from the feature list, return to step 4 and continue to divide until there is no feature
3.3 code implementation
 Import package
import numpy as np import pandas as pd import math import collections
 Import data
def import_data(): data = pd.read_csv('..\\source\\watermalon.txt') data.head(10) data=np.array(data).tolist() # Characteristic value list labels = ['color and lustre', 'Root', 'Knock', 'texture', 'Umbilicus', 'Tactile sensation'] # All possible cases corresponding to the feature labels_full = {} for i in range(len(labels)): labelList = [example[i] for example in data] uniqueLabel = set(labelList) labels_full[labels[i]] = uniqueLabel return data,labels,labels_full
 Call function to get data
data,labels,labels_full=import_data()
 Calculate the initial information entropy, that is, the information entropy before classification
def calcShannonEnt(dataSet): """ Calculate the information entropy of a given data set(Shannon entropy) :param dataSet: :return: """ # Calculate the total number of data sets numEntries = len(dataSet) # Used to count tags labelCounts = collections.defaultdict(int) # Loop the whole data set to get the classification label of the data for featVec in dataSet: # Get the current label currentLabel = featVec[1] # # If the current tag is no longer in the tag set, add it (in the book) # if currentLabel not in labelCounts.keys(): # labelCounts[currentLabel] = 0 # # # The number of corresponding tags in the tag set plus one # labelCounts[currentLabel] += 1 # It can also be written as follows labelCounts[currentLabel] += 1 # Default information entropy shannonEnt = 0.0 for key in labelCounts: # Calculate the proportion of the current category label in the total label prob = float(labelCounts[key]) / numEntries # Find logarithm based on 2 shannonEnt = prob * math.log2(prob) return shannonEnt
 View initial information entropy
print(calcShannonEnt(data)) # Output: 0.99750254631653
 Obtain the number of each eigenvalue, which is to prepare for the later calculation of information gain
def splitDataSet(dataSet, axis, value): """ Divide the data set according to the given eigenvalue :param dataSet: data set :param axis: Coordinates of a given eigenvalue :param value: If a given eigenvalue satisfies the condition, only the given eigenvalue is equal to this value Will return when :return: """ # Create a new list to prevent modifications to the original list retDataSet = [] # Traverse the entire dataset for featVec in dataSet: # If the given eigenvalue is equal to the desired eigenvalue if featVec[axis] == value: # Save the contents in front of the characteristic value reducedFeatVec = featVec[:axis] # Save the content after the eigenvalue, so the given eigenvalue is removed reducedFeatVec.extend(featVec[axis + 1:]) # Add to return list retDataSet.append(reducedFeatVec) return retDataSet
 Calculate the information gain to determine the best data set partition
def chooseBestFeatureToSplit(dataSet, labels): """ The best data set partition feature is selected and calculated according to the information gain value :param dataSet: :return: """ # Get the total number of eigenvalues of the data numFeatures = len(dataSet[0])  1 # Calculate the basic information entropy baseEntropy = calcShannonEnt(dataSet) # The gain of basic information is 0.0 bestInfoGain = 0.0 # Best eigenvalue bestFeature = 1 # Calculate the information entropy of each eigenvalue for i in range(numFeatures): # Get a list of all current eigenvalues in the dataset featList = [example[i] for example in dataSet] # Make the current feature unique, that is, how many kinds of current feature values are there uniqueVals = set(featList) # The new entropy represents the entropy of the current eigenvalue newEntropy = 0.0 # Possibility of traversing existing features for value in uniqueVals: # At the current feature position of all data sets, find the set whose feature value is equal to the current value subDataSet = splitDataSet(dataSet=dataSet, axis=i, value=value) # Calculate the weight prob = len(subDataSet) / float(len(dataSet)) # Calculate the entropy of the current eigenvalue newEntropy += prob * calcShannonEnt(subDataSet) # Calculate the "information gain" infoGain = baseEntropy  newEntropy #print('current characteristic value: '+ labels[i] +', corresponding information gain value: '+ str(infoGain)+"i equals" + str(i)) #If the current information gain is greater than the original if infoGain > bestInfoGain: # Best information gain bestInfoGain = infoGain # The new best eigenvalue is used to divide bestFeature = i #print('features with maximum information gain: '+ labels[bestFeature]) return bestFeature
 Judge whether the attributes of each sample set are consistent
def judgeEqualLabels(dataSet): """ Judge whether each attribute set of the data set is completely consistent :param dataSet: :return: """ # Calculate the total number of attributes in the sample set, and the last one is category feature_leng = len(dataSet[0])  1 # Calculate the total number of data data_leng = len(dataSet) # Tag what is the first attribute value in each attribute first_feature = '' # Are all attribute sets completely consistent is_equal = True # Traverse all attributes for i in range(feature_leng): # Get the ith attribute of the first sample first_feature = dataSet[0][i] # Compare with all the data in the sample set to see if the attribute is consistent for _ in range(1, data_leng): # If inequality is found, False is returned directly if first_feature != dataSet[_][i]: return False return is_equal
 Draw decision tree (Dictionary)
def createTree(dataSet, labels): """ Create decision tree :param dataSet: data set :param labels: Feature label :return: """ # Get the classification labels of all data sets classList = [example[1] for example in dataSet] # Count the number of occurrences of the first label and compare it with the total number of labels. If it is equal, it means that all labels in the current list are one kind of labels. At this time, the division is stopped if classList.count(classList[0]) == len(classList): return classList[0] # Calculate the number of data in the first row. If there is only one, it means that all feature attributes have been traversed. The remaining one is the category label, or all samples are consistent in all attributes if len(dataSet[0]) == 1 or judgeEqualLabels(dataSet): # Returns the one that appears more frequently in the remaining tags return majorityCnt(classList) # Select the best partition feature and get the subscript of the feature bestFeat = chooseBestFeatureToSplit(dataSet=dataSet, labels=labels) print(bestFeat) # Get the name of the best feature bestFeatLabel = labels[bestFeat] print(bestFeatLabel) # A dictionary is used to store the tree structure, and the bifurcation is the divided feature name myTree = {bestFeatLabel: {}} # Delete the characteristic value of this division from the list del(labels[bestFeat]) # Get all possible values of the current feature label featValues = [example[bestFeat] for example in dataSet] # Uniqueness, removing duplicate eigenvalues uniqueVals = set(featValues) # Traverse all eigenvalues for value in uniqueVals: # Get the remaining feature tags subLabels = labels[:] subTree = createTree(splitDataSet(dataSet=dataSet, axis=bestFeat, value=value), subLabels) # Recursive call divides all data in the data set whose feature is equal to the current feature value into the current node. During recursive call, the current feature needs to be removed first myTree[bestFeatLabel][value] = subTree return myTree
 Call the function and print, and you can see a dictionary type tree
mytree=createTree(data,labels) print(mytree)
 Draw visual tree
import matplotlib.pylab as plt import matplotlib # Can display Chinese matplotlib.rcParams['font.sansserif'] = ['SimHei'] matplotlib.rcParams['font.serif'] = ['SimHei'] # Bifurcation node, that is, decision node decisionNode = dict(boxstyle="sawtooth", fc="0.8") # Leaf node leafNode = dict(boxstyle="round4", fc="0.8") # Arrow style arrow_args = dict(arrowstyle="<") def plotNode(nodeTxt, centerPt, parentPt, nodeType): """ Draw a node :param nodeTxt: Text information describing the node :param centerPt: Coordinates of text :param parentPt: The coordinates of the point, which also refers to the coordinates of the parent node :param nodeType: Node type,It is divided into leaf node and decision node :return: """ createPlot.ax1.annotate(nodeTxt, xy=parentPt, xycoords='axes fraction', xytext=centerPt, textcoords='axes fraction', va="center", ha="center", bbox=nodeType, arrowprops=arrow_args) def getNumLeafs(myTree): """ Gets the number of leaf nodes :param myTree: :return: """ # Count the total number of leaf nodes numLeafs = 0 # Get the current first key, that is, the root node firstStr = list(myTree.keys())[0] # Get the content corresponding to the first key secondDict = myTree[firstStr] # Recursively traversing leaf nodes for key in secondDict.keys(): # If the key corresponds to a dictionary, it is called recursively if type(secondDict[key]).__name__ == 'dict': numLeafs += getNumLeafs(secondDict[key]) # If not, it means that it is a leaf node at this time else: numLeafs += 1 return numLeafs def getTreeDepth(myTree): """ Number of depth layers obtained :param myTree: :return: """ # Used to save the maximum number of layers maxDepth = 0 # Get root node firstStr = list(myTree.keys())[0] # Get the content corresponding to the key secondDic = myTree[firstStr] # Traverse all child nodes for key in secondDic.keys(): # If the node is a dictionary, it is called recursively if type(secondDic[key]).__name__ == 'dict': # Depth of child node plus 1 thisDepth = 1 + getTreeDepth(secondDic[key]) # This indicates that this is a leaf node else: thisDepth = 1 # Replace maximum layers if thisDepth > maxDepth: maxDepth = thisDepth return maxDepth def plotMidText(cntrPt, parentPt, txtString): """ Calculate the middle position between the parent node and the child node, and fill in the information :param cntrPt: Child node coordinates :param parentPt: Parent node coordinates :param txtString: Filled text information :return: """ # Calculate the middle position of the xaxis xMid = (parentPt[0]cntrPt[0])/2.0 + cntrPt[0] # Calculate the middle position of the yaxis yMid = (parentPt[1]cntrPt[1])/2.0 + cntrPt[1] # Draw createPlot.ax1.text(xMid, yMid, txtString) def plotTree(myTree, parentPt, nodeTxt): """ Draw all nodes of the tree and draw recursively :param myTree: tree :param parentPt: Coordinates of the parent node :param nodeTxt: Text information of the node :return: """ # Calculate the number of leaf nodes numLeafs = getNumLeafs(myTree=myTree) # Calculate the depth of the tree depth = getTreeDepth(myTree=myTree) # Get the information content of the root node firstStr = list(myTree.keys())[0] # Calculate the middle coordinates of the current root node in all child nodes, that is, the offset of the current Xaxis plus the calculated center position of the root node as the xaxis (for example, the first time: the initial xoffset is:  1/2W, the calculated center position of the root node is: (1+W)/2W, add to get: 1 / 2), and the current Yaxis offset is the yaxis cntrPt = (plotTree.xOff + (1.0 + float(numLeafs))/2.0/plotTree.totalW, plotTree.yOff) # Draw the connection between the node and the parent node plotMidText(cntrPt, parentPt, nodeTxt) # Draw the node plotNode(firstStr, cntrPt, parentPt, decisionNode) # Get the subtree corresponding to the current root node secondDict = myTree[firstStr] # Calculate the new yaxis offset and move down 1/D, that is, the drawing yaxis of the next layer plotTree.yOff = plotTree.yOff  1.0/plotTree.totalD # Loop through all key s for key in secondDict.keys(): # If the current key is a dictionary and there are subtrees, it will be traversed recursively if isinstance(secondDict[key], dict): plotTree(secondDict[key], cntrPt, str(key)) else: # Calculate the new Xaxis offset, that is, the xaxis coordinate drawn by the next leaf moves 1/W to the right plotTree.xOff = plotTree.xOff + 1.0/plotTree.totalW # Open the annotation to observe the coordinate changes of leaf nodes # print((plotTree.xOff, plotTree.yOff), secondDict[key]) # Draw leaf node plotNode(secondDict[key], (plotTree.xOff, plotTree.yOff), cntrPt, leafNode) # Draw the content of the middle line between the leaf node and the parent node plotMidText((plotTree.xOff, plotTree.yOff), cntrPt, str(key)) # Before returning to recursion, you need to increase the offset of the yaxis and move it up by 1/D, that is, return to draw the yaxis of the previous layer plotTree.yOff = plotTree.yOff + 1.0/plotTree.totalD def createPlot(inTree): """ Decision tree to be drawn :param inTree: Decision tree dictionary :return: """ # Create an image fig = plt.figure(1, facecolor='white') fig.clf() axprops = dict(xticks=[], yticks=[]) createPlot.ax1 = plt.subplot(111, frameon=False, **axprops) # Calculate the total width of the decision tree plotTree.totalW = float(getNumLeafs(inTree)) # Calculate the total depth of the decision tree plotTree.totalD = float(getTreeDepth(inTree)) # The initial xaxis offset, that is  1/2W, moves 1/W to the right each time, that is, the x coordinates drawn by the first leaf node are: 1/2W, the second: 3/2W, the third: 5/2W, and the last: (W1)/2W plotTree.xOff = 0.5/plotTree.totalW # The initial yaxis offset, moving down or up 1/D each time plotTree.yOff = 1.0 # Call the function to draw the node image plotTree(inTree, (0.5, 1.0), '') # draw plt.show() if __name__ == '__main__': createPlot(mytree)
 give the result as follows
 There are some labels missing from the tree. You need to complete the tree and output it again
def makeTreeFull(myTree, labels_full, default): """ Complete the nonexistent feature labels in the tree to the category with the most occurrences in the parent node :param myTree: Generated tree :param labels_full: All labels for the feature :param parentClass: Maximum categories in parent node :param default: If the parent node in the missing label cannot judge the category, this value is used :return: """ # The parent node mentioned here is the current root node, and the feature label that does not exist under the current root node is taken as the child node # Get the current root node root_key = list(myTree.keys())[0] # Get all the classifications under the root node, which may be child nodes (good melon or bad melon) or not child nodes (attribute values divided again) sub_tree = myTree[root_key] # If it is a leaf node, it ends if isinstance(sub_tree, str): return # Find the category that uses the most under the current node classification, and use the classification result as the classification of the new feature label. For example, if there is no light white under the color, use the cyan classification in the color as the light white classification root_class = [] # Record the classified results for sub_key in sub_tree.keys(): if isinstance(sub_tree[sub_key], str): root_class.append(sub_tree[sub_key]) # Find the category that appears most in this layer, and the same situation may occur. Take one if len(root_class): most_class = collections.Counter(root_class).most_common(1)[0][0] else: most_class = None# There are no classified attributes under the current node # print(most_class) # Loop through all feature labels and add nonexistent labels for label in labels_full[root_key]: if label not in sub_tree.keys(): if most_class is not None: sub_tree[label] = most_class else: sub_tree[label] = default # Recursive processing for sub_key in sub_tree.keys(): if isinstance(sub_tree[sub_key], dict): makeTreeFull(myTree=sub_tree[sub_key], labels_full=labels_full, default=default)
 Call and draw the tree again
makeTreeFull(mytree,labels_full,default='unknown') createPlot(mytree)
 Implementing ID3 using sklearn
# Import package import pandas as pd from sklearn import tree import graphviz
 Read the file and you can see the read data
df = pd.read_csv('..\\source\\watermalon.txt') df.head(10)
 Convert all eigenvalues to numbers
df['color and lustre']=df['color and lustre'].map({'plain ':1,'dark green':2,'Black':3}) df['Root']=df['Root'].map({'Slightly curled':1,'Curl up':2,'Stiff':3}) df['stroke ']=df['stroke '].map({'Crisp':1,'Turbid sound':2,'Dull':3}) df['texture']=df['texture'].map({'clear':1,'Slightly paste':2,'vague':3}) df['Umbilicus']=df['Umbilicus'].map({'flat':1,'Slightly concave':2,'sunken':3}) df['Tactile sensation'] = np.where(df['Tactile sensation']=="Hard slip",1,2) df['Good melon'] = np.where(df['Good melon']=="yes",1,0) x_train=df[['color and lustre','Root','stroke ','texture','Umbilicus','Tactile sensation']] y_train=df['Good melon'] print(df) id3=tree.DecisionTreeClassifier(criterion='entropy') id3=id3.fit(x_train,y_train) print(id3)
 Get the following pictures
 Training and visualization. The parameter of DecisionTreeClassifier is entropy id3 algorithm. The default is CART algorithm, and there is no C4.5 algorithm
id3=tree.DecisionTreeClassifier(criterion='entropy') id3=id3.fit(x_train,y_train) labels = ['color and lustre', 'Root', 'Knock', 'texture', 'Umbilicus', 'Tactile sensation'] dot_data = tree.export_graphviz(id3 ,feature_names=labels ,class_names=["Good melon","Bad melon"] ,filled=True ,rounded=True ) graph = graphviz.Source(dot_data) graph
4, C4.5 algorithm
4.1 introduction
 C4.5 is a decision tree algorithm, which is an improved algorithm of the core algorithm ID3 of the decision tree (the decision tree is the organization of decisionmaking nodes like a tree, which is actually an inverted tree)
4.2 steps
 As mentioned above, the core of C4.5 algorithm is still ID3, so the steps are the same, except that the information gain is changed into the information gain rate, that is, the information gain is calculated and then divided by the information entropy of the feature
4.3 change code
 Because it becomes less, we just add another step to get the information gain to get the information gain rate
## Implement C4.5 algorithm def chooseBestFeatureToSplit_4(dataSet, labels): """ The best data set partition feature is selected and calculated according to the information gain value :param dataSet: :return: """ # Get the total number of eigenvalues of the data numFeatures = len(dataSet[0])  1 # Calculate the basic information entropy baseEntropy = calcShannonEnt(dataSet) # The gain of basic information is 0.0 bestInfoGain = 0.0 # Best eigenvalue bestFeature = 1 # Calculate the information entropy of each eigenvalue for i in range(numFeatures): # Get a list of all current eigenvalues in the dataset featList = [example[i] for example in dataSet] # Make the current feature unique, that is, how many kinds of current feature values are there uniqueVals = set(featList) # The new entropy represents the entropy of the current eigenvalue newEntropy = 0.0 # Possibility of traversing existing features for value in uniqueVals: # At the current feature position of all data sets, find the set whose feature value is equal to the current value subDataSet = splitDataSet(dataSet=dataSet, axis=i, value=value) # Calculate the weight prob = len(subDataSet) / float(len(dataSet)) # Calculate the entropy of the current eigenvalue newEntropy += prob * calcShannonEnt(subDataSet) # Calculate the "information gain" infoGain = baseEntropy  newEntropy infoGain = infoGain/newEntropy #print('current characteristic value: '+ labels[i] +', corresponding information gain value: '+ str(infoGain)+"i equals" + str(i)) #If the current information gain is greater than the original if infoGain > bestInfoGain: # Best information gain bestInfoGain = infoGain # The new best eigenvalue is used to divide bestFeature = i #print('features with maximum information gain: '+ labels[bestFeature]) return bestFeature
 I still changed the function of creating the tree to avoid confusion
def createTree_4(dataSet, labels): """ Create decision tree :param dataSet: data set :param labels: Feature label :return: """ # Get the classification labels of all data sets classList = [example[1] for example in dataSet] # Count the number of occurrences of the first label and compare it with the total number of labels. If it is equal, it means that all labels in the current list are one kind of labels. At this time, the division is stopped if classList.count(classList[0]) == len(classList): return classList[0] # Calculate the number of data in the first row. If there is only one, it means that all feature attributes have been traversed. The remaining one is the category label, or all samples are consistent in all attributes if len(dataSet[0]) == 1 or judgeEqualLabels(dataSet): # Returns the one that appears more frequently in the remaining tags return majorityCnt(classList) # Select the best partition feature and get the subscript of the feature bestFeat = chooseBestFeatureToSplit_4(dataSet=dataSet, labels=labels) print(bestFeat) # Get the name of the best feature bestFeatLabel = labels[bestFeat] print(bestFeatLabel) # A dictionary is used to store the tree structure, and the bifurcation is the divided feature name myTree = {bestFeatLabel: {}} # Delete the characteristic value of this division from the list del(labels[bestFeat]) # Get all possible values of the current feature label featValues = [example[bestFeat] for example in dataSet] # Uniqueness, removing duplicate eigenvalues uniqueVals = set(featValues) # Traverse all eigenvalues for value in uniqueVals: # Get the remaining feature tags subLabels = labels[:] subTree = createTree(splitDataSet(dataSet=dataSet, axis=bestFeat, value=value), subLabels) # Recursive call divides all data in the data set whose feature is equal to the current feature value into the current node. During recursive call, the current feature needs to be removed first myTree[bestFeatLabel][value] = subTree return myTree
 Call the function and look at the dictionary tree
mytree_4=createTree_4(data,labels) print(mytree_4)
 Then complete it and then visualize it
makeTreeFull(mytree_4,labels_full,default='unknown') createPlot(mytree_4)
5, CART algorithm
5.1 introduction
CART algorithm is a binary recursive segmentation technology, which divides the current sample into two sub samples, so that each non leaf node has two branches. Therefore, the decision tree generated by CART algorithm is a binary tree with simple structure. Because CART algorithm is a binary tree, it can only be "yes" or "no" in each step of decisionmaking. Even if a feature has multiple values, it also divides the data into two parts. The CART algorithm is mainly divided into two steps
(1) Recursive partition of samples for tree building process
(2) Pruning with validation data
5.2 steps

step1: select an independent variable, and then select a value to divide the dimensional space into two parts. All points of one part are satisfied, and all points of the other part are satisfied. For discontinuous variables, there are only two values of attribute value, that is, equal to or not equal to the value.

Step 2: recursive processing. Select a new attribute from the above two parts according to step 1 and continue to divide until the whole dimensional space is divided. There is a problem in the division. What criteria is it divided according to? For a variable attribute, its partition point is the midpoint of a pair of continuous variable attribute values. Assuming that an attribute of a set of samples has a continuous value, there will be a split point, and each split point is the mean of two adjacent continuous values. The division of each attribute is sorted according to the amount of impurities that can be reduced, and the reduction of impurities is defined as the sum of the impurities before division minus the proportion of impurity mass division of each node after division. The Gini index is commonly used in impurity measurement methods. Assuming that a sample has a common class, the Gini impurity of a node can be defined as

Where Pi represents the probability of belonging to class i. When Gini(A)=0, all samples belong to the same class. When all classes appear in the node with equal probability, Gini(A) is maximized. At this time, Pi has the above theoretical basis. The actual recursive partition process is as follows: if all samples of the current node do not belong to the same class or only one sample is left, then this node is a non leaf node, Therefore, each attribute of the sample and the corresponding splitting point of each attribute will be tried to find the partition with the largest impurity variable, and the subtree divided by this attribute is the optimal branch.
5.3 implementation using sklearn Library
 Import package
import pandas as pd from sklearn import tree import graphviz
 Import data and read
df = pd.read_csv('..\\source\\watermalon.txt') df.head(10)
 Convert eigenvalues to numbers
df['color and lustre']=df['color and lustre'].map({'plain ':1,'dark green':2,'Black':3}) df['Root']=df['Root'].map({'Slightly curled':1,'Curl up':2,'Stiff':3}) df['stroke ']=df['stroke '].map({'Crisp':1,'Turbid sound':2,'Dull':3}) df['texture']=df['texture'].map({'clear':1,'Slightly paste':2,'vague':3}) df['Umbilicus']=df['Umbilicus'].map({'flat':1,'Slightly concave':2,'sunken':3}) df['Tactile sensation'] = np.where(df['Tactile sensation']=="Hard slip",1,2) df['Good melon'] = np.where(df['Good melon']=="yes",1,0) x_train=df[['color and lustre','Root','stroke ','texture','Umbilicus','Tactile sensation']] y_train=df['Good melon']
 Train the tree and visualize it
# Build models and train gini=tree.DecisionTreeClassifier() gini=gini.fit(x_train,y_train) #Visualization of decision tree gini_data = tree.export_graphviz(gini ,feature_names=labels ,class_names=["Good melon","Bad melon"] ,filled=True ,rounded=True ) gini_graph = graphviz.Source(gini_data) gini_graph
6, Summary
It is not difficult to understand the decision tree, that is, it is necessary to know the division basis of each algorithm. The steps are roughly to calculate the evaluation criteria first, then sort according to the evaluation results, and finally recursively build the tree according to the order.
7, Reference
Decision tree
Information entropy
CART algorithm
Gini Gini index
Implementation of ID3 decision tree in watermelon book