Chapter IV decision tree
1 basic process
2 division selection
With the continuous division process, we hope that the samples contained in the branch nodes of the decision tree belong to the same category as much as possible, that is, the "purity" of the nodes is getting higher and higher.
2.1 information gain
2.1.1 what is information entropy
What is entropy: the uncertainty of a thing.
Information: something that eliminates uncertainty.
Function of information: adjust probability; Eliminate interference; Determine the situation (for example, the melon seller said that it is cooked and sweet).
Noise: something that cannot eliminate someone's uncertainty about something.
Data = information + noise
2.1.2 how to quantify entropy - equal probability
Refer to an uncertain event as a unit. For example, a coin toss is recorded as 1bit.
For example, eight uncertain cases with equal probability are equivalent to tossing coins three times, that is, 2 ^ 3 possible cases, and the entropy is 3bit.
For example, 10 uncertain cases with equal probability are equivalent to tossing log10 coins, that is, 2^log10 possible cases. The entropy is log10bit, and the logarithm is based on 2.
2.1.3 how to quantify entropy - unequal probability
2.1.4 how to quantify information
The difference in entropy before and after knowing the information.
Pre information entropy: log4 = 2
Post information entropy: 31/6log6 + 1/2*log2
#Calculate the information gain of color attribute import numpy as np #The root node information entropy is calculated according to the proportion of good and bad melons Good = 8 Bad = 9 Total = 17 EntD = - 8/17 * np.log2(8/17) - 9/17 * np.log2(9/17) #Calculate color information #Calculate the number of good and bad melons in all colors Green_good = 3 Green_bad = 3 Black_good = 4 Black_bad = 2 White_good = 1 White_bad = 4 #Calculate the occurrence probability of good and bad melons P_Green_good = 3/6 P_Green_bad = 3/6 P_Green = 6/17 P_Black_good = 4/6 P_Black_bad = 2/6 P_Black = 6/17 P_White_good = 1 /5 P_White_bad = 4/5 P_White = 5/17 #Calculate the information entropy of each branch node EntD_Green = -P_Green_good*np.log2(P_Green_good) -P_Green_bad*np.log2(P_Green_bad) EntD_White = -P_White_good*np.log2(P_White_good) -P_White_bad*np.log2(P_White_bad) EntD_Black = -P_Black_good*np.log2(P_Black_good) -P_Black_bad*np.log2(P_Black_bad) #The information entropy of color attribute is obtained by weighting EntD_color = P_Green*EntD_Green + P_White*EntD_White + P_Black*EntD_Black #The information gain of color attribute is obtained by subtraction Gain = EntD - EntD_color Gain
According to the above similar method, the information gain of different attributes is calculated.
The root node is divided according to the attribute with the largest information gain - texture
Based on the known texture category, the information gain of each attribute is calculated to obtain the decision tree.
2.2 gain rate
The information gain criterion has a preference for attributes with a large number of values.
To solve this problem, Quinlan(1993) used the gain ratio to select the optimal partition attribute, which is defined as follows:
Firstly, the attributes with higher information gain than the average level are found from the candidate partition attributes, and then the number of attribute partition decisions with the highest gain rate is selected.
2.3 Gini index
2.3.1 classification tree: Gini index minimum criterion.
Data set Gini index:
Attribute Gini index:
#Gini index of data set: two bifurcation as an example def gini_index_single(a,b): single_gini = 1 - ((a/(a+b))**2) - ((b/(a+b))**2) return single_gini print(gini_index_single(105,39),gini_index_single(130,14)) #It can be seen that the higher the purity, the smaller the Gini index
#Attribute Gini index: weighting of data sets def gini_index(a, b, c, d): zuo = gini_index_single(a,b) you = gini_index_single(c,d) gini_index = zuo*((a+b)/(a+b+c+d)) + you*((c+d)/(a+b+c+d)) return gini_index gini_index(105, 39, 34, 125)
gini_index(37, 127, 100, 33)
gini_index(92, 31, 45, 129)
Select the attribute with the smallest Gini index for the first bifurcation, that is, good blood circulation
Continue with the second bifurcation
gini_index(13, 98, 24, 29)
gini_index(24, 25, 13, 102)
According to the above figure, Gini impurity of Blocked arteries is 0.29, which is better than chestpain. (the smaller the Gini index, the better)
And the Gini index after bifurcation is lower than 0.34, so the second bifurcation should be carried out
Therefore, take Blocked arteries as the second bifurcation attribute. Get:
Since 0.32 < 0.49, the third bifurcation should be carried out.
The final bifurcation results are as follows: