1, Construction of decision tree
decision tree is a kind of common machine learning algorithm. It makes decisions based on tree structure. Step by step from the root node to the leaf node (decision). All data will eventually fall to leaf nodes, which can be classified or regressed.
Decision tree
Advantages: the calculation complexity is not high, the output result is easy to understand, is not sensitive to the loss of intermediate value, and can process irrelevant feature data.
Disadvantages: over matching may occur.
Applicable data types: numerical type and nominal type.
When constructing a decision tree, the first problem we need to solve is which feature on the current data set plays a decisive role in dividing data classification. In order to find the decisive features and divide the best results, we must evaluate each feature. After the test, the original data is divided into several data subsets. These data subsets are distributed on all branches of the first decision point. If the data under a branch belongs to the same type, the data set will be further divided here and correctly classified. If the data in the data subset does not belong to the same type, you need to repeat the process of dividing the data subset. The algorithm of how to divide the data subset is the same as that of the original data set, until all data with the same type are in one data subset.
General process of decision tree
(1) Collect data: any method can be used.
(2) Prepare data: the tree construction algorithm is only applicable to nominal data, so numerical data must be discretized.
(3) Analyze data: any method can be used. After constructing the tree, we should check whether the graph meets the expectations.
(4) Training algorithm: construct the data structure of the tree.
(5) Test algorithm: use the experience tree to calculate the error rate.
(6) Using algorithm: this step can be applied to any supervised learning algorithm, and using decision tree can better understand the internal meaning of data.
1. Information gain
The general principle of dividing data sets is to make disordered data more orderly
The change of information before and after dividing the data set is called information gain. Knowing how to calculate the information gain, we can calculate the information gain obtained by dividing the data set by each eigenvalue. The feature with the highest information gain is the best choice.
Before we can evaluate which data division method is the best data division, we must learn how to calculate the information gain. The measurement of set information is called Shannon entropy or entropy for short.
Entropy is defined as the expected value of information. Before clarifying this concept, we must know the definition of information. If the thing to be classified may be divided into multiple classifications, the symbol
x
i
x_i
xi's information is defined as
l
(
x
i
)
=
−
log
2
p
(
x
i
)
l(x_i) = -\log_2p(x_i)
l(xi)=−log2p(xi)
among
p
(
x
i
)
p(x_i)
p(xi) is the probability of selecting reclassification
In order to calculate entropy, we need to calculate the expected value of information contained in all possible values of all categories, which is obtained by the following formula:
H
=
−
∑
i
=
1
n
p
(
x
i
)
log
2
p
(
x
i
)
H = - \sum_{i = 1}^n p(x_i)\log_2p(x_i)
H=−i=1∑np(xi)log2p(xi)
Where n is the number of classifications.
When
p
(
x
i
)
=
0
p(x_i) = 0
p(xi) = 0 or
p
(
x
i
)
=
1
p(x_i) = 1
When p(xi) = 1,
H
=
0
H = 0
H=0, random variables have no uncertainty at all.
When
p
(
x
i
)
=
0.5
p(x_i) = 0.5
When p(xi) = 0.5,
H
=
1
H = 1
H=1, the uncertainty of random variables is the largest
Calculate Shannon entropy for a given dataset:
'''
Parameters:
dataSet - data set
Returns:
shannonEnt - Returns the Shannon entropy calculated for a given dataset
'''
def calcShannonEnt(dataSet):
# Returns the number of rows in the dataset
numEntries = len(dataSet)
# Save a dictionary of the number of occurrences of each label
labelCounts = {}
# Each group of eigenvectors is counted
for featVec in dataSet:
# Extract label information
currentLabel = featVec[-1]
# If the label is not put into the dictionary of statistical times, add it
if currentLabel not in labelCounts.keys():
labelCounts[currentLabel] = 0
# Label count
labelCounts[currentLabel] += 1
# Shannon entropy
shannonEnt = 0.0
# Calculate Shannon entropy
for key in labelCounts:
# The probability of selecting the label
prob = float(labelCounts[key])/numEntries
# Calculation of Shannon entropy by formula
shannonEnt -= prob * log(prob, 2)
# Return Shannon entropy
return shannonEnt
First, calculate the total number of instances in the dataset. Then, create a data dictionary whose key value is the value of the last column. If the current key value does not exist, expand the dictionary and add the current key value to the dictionary. Each key value records the number of occurrences of the current category. Finally, the occurrence frequency of all class labels is used to calculate the probability of class occurrence. We will use this probability to calculate Shannon entropy and count the times of all class labels.
Marine biological data:
| Can we survive without coming to the surface | Are there fins | Belonging to fish | |
|---|---|---|---|
| 1 | yes | yes | yes |
| 2 | yes | yes | yes |
| 3 | yes | no | no |
| 4 | no | yes | no |
| 5 | no | yes | no |
Create a simple fish identification dataset from the table above
"""
Parameters:
nothing
Returns:
dataSet - data set
labels - Classification properties
"""
def createDataSet():
# data set
dataSet = [[1, 1, 'yes'],
[1, 1, 'yes'],
[1, 0, 'no'],
[0, 1, 'no'],
[0, 1, 'no']]
# Classification properties
labels = ['no surfacing', 'flippers']
# Return dataset and classification properties
return dataSet, labels
if __name__ == '__main__':
dataSet, labels = createDataSet()
print(dataSet)
print(calcShannonEnt(dataSet))
>>> [[1, 1, 'yes'], [1, 1, 'yes'], [1, 0, 'no'], [0, 1, 'no'], [0, 1, 'no']] 0.9709505944546686
The higher the entropy, the more mixed data. We can add more classifications to the data set and observe how the entropy changes. Add a third category named may to test the change of entropy:
if __name__ == '__main__':
dataSet, labels = createDataSet()
dataSet[0][-1] = 'maybe'
print(dataSet)
print(calcShannonEnt(dataSet))
>>> [[1, 1, 'maybe'], [1, 1, 'yes'], [1, 0, 'no'], [0, 1, 'no'], [0, 1, 'no']] 1.3709505944546687
The calculated entropy increases with the increase of classification. After obtaining the entropy, we can divide the data set according to the method of obtaining the maximum information gain.
2. Divide data sets
In addition to measuring the information entropy, the classification algorithm also needs to divide the data set and measure the entropy of the divided data set, so as to judge whether the data set is divided correctly. The information entropy is calculated once for the result of dividing the data set according to each feature, and then it is judged which feature is the best way to divide the data set.
Divide the data set according to the given characteristics:
"""
Parameters:
dataSet - Data set to be divided
axis - Characteristics of partitioned data sets
value - The value of the feature to be returned
Returns:
retDataSet - Partitioned dataset
"""
def splitDataSet(dataSet, axis, value):
# Create a list of returned datasets
retDataSet = []
# Traversal dataset
for featVec in dataSet:
if featVec[axis] == value:
# Remove axis feature
reducedFeatVec = featVec[:axis]
# Add eligible to the returned dataset
reducedFeatVec.extend(featVec[axis + 1:])
retDataSet.append(reducedFeatVec)
# Returns the partitioned dataset
return retDataSet
Test the function splitDataSet() on the previous simple sample data:
if __name__ == '__main__':
dataSet, labels = createDataSet()
print(dataSet)
print(splitDataSet(dataSet, 0 ,1))
print(splitDataSet(dataSet, 0 ,0))
>>> [[1, 1, 'yes'], [1, 1, 'yes'], [1, 0, 'no'], [0, 1, 'no'], [0, 1, 'no']] [[1, 'yes'], [1, 'yes'], [0, 'no']] [[1, 'no'], [1, 'no']]
Next, traverse the entire data set, and cycle through the entropy and splitDataSet() function to find the best feature division method. Entropy calculation will tell us how to divide data sets is the best way to organize data.
Select the best data set division method:
"""
Parameters:
dataSet - data set
Returns:
bestFeature - Maximum information gain(optimal)Index value of the feature
"""
def chooseBestFeatureToSplit(dataSet):
# Number of features
numFeatures = len(dataSet[0]) - 1
# Calculate Shannon entropy of data set
baseEntropy = calcShannonEnt(dataSet)
# information gain
bestInfoGain = 0.0
# Index value of optimal feature
bestFeature = -1
# Traverse all features
for i in range(numFeatures):
#Get the i th all features of dataSet
featList = [example[i] for example in dataSet]
# Create set set {}, elements cannot be repeated
uniqueVals = set(featList)
# Empirical conditional entropy
newEntropy = 0.0
# Calculate information gain
for value in uniqueVals:
# The subset of the subDataSet after partition
subDataSet = splitDataSet(dataSet, i, value)
# Calculate the probability of subsets
prob = len(subDataSet) / float(len(dataSet))
# The empirical conditional entropy is calculated according to the formula
newEntropy += prob * calcShannonEnt(subDataSet)
# information gain
infoGain = baseEntropy - newEntropy
# Print information gain for each feature
print("The first%d The gain of each feature is%.3f" % (i, infoGain))
# Calculate information gain
if (infoGain > bestInfoGain):
# Update the information gain to find the maximum information gain
bestInfoGain = infoGain
# Record the index value of the feature with the largest information gain
bestFeature = i
# Returns the index value of the feature with the largest information gain
return bestFeature
if __name__ == '__main__':
dataSet, labels = createDataSet()
print(dataSet)
print(chooseBestFeatureToSplit(dataSet))
>>> [[1, 1, 'yes'], [1, 1, 'yes'], [1, 0, 'no'], [0, 1, 'no'], [0, 1, 'no']] The gain of the 0th feature is 0.420 The gain of the first feature is 0.171 0
The best index is 0, that is, the first feature is the best feature used to divide the dataset.
3. Recursive construction of decision tree
Working principle: get the original data set, and then divide the data set based on the best attribute value. Since there may be more than two eigenvalues, there may be data set division greater than two branches. After the first partition, the data will be passed down to the next node of the tree branch. On this node, we can partition the data again. Therefore, we can use the principle of recursion to deal with data sets.
The condition for the end of recursion: the program traverses all the attributes that divide the data set, or all instances under each branch have the same classification. If all instances have the same classification, a leaf node or termination block is obtained. Any data that reaches the leaf node must belong to the classification of the leaf node. If the dataset has processed all attributes, but the class label is still not unique, we need to decide how to define the leaf node. In this case, we usually use the majority voting method to determine the classification of the leaf node.
Majority vote:
"""
Parameters:
classList - Class label list
Returns:
sortedClassCount[0][0] - Elements that appear most here(Class label)
"""
def majorityCnt(classList):
classCount = {}
# Count the number of occurrences of each element in the classList
for vote in classList:
if vote not in classCount.keys():classCount[vote] = 0
classCount[vote] += 1
# Sort by dictionary values in descending order
sortedClassCount = sorted(classCount.items(), key = operator.itemgetter(1), reverse = True)
# Returns the most frequent element in the classList
return sortedClassCount[0][0]
To create a decision tree:
"""
Parameters:
dataSet - Training data set
labels - Classification attribute label
Returns:
myTree - Decision tree
"""
def createTree(dataSet, labels):
# Take the classification label
classList = [example[-1] for example in dataSet]
# If all class labels are identical, the class label is returned directly
if classList.count(classList[0]) == len(classList):
return classList[0]
# When all features are traversed, the class label with the most occurrences is returned
if len(dataSet[0]) == 1:
return majorityCnt(classList)
# Select the optimal feature
bestFeat = chooseBestFeatureToSplit(dataSet)
# Optimal feature label
bestFeatLabel = labels[bestFeat]
# Label spanning tree based on optimal features
myTree = {bestFeatLabel:{}}
# Delete used feature labels
del(labels[bestFeat])
# The attribute values of all optimal features in the training set are obtained
featValues = [example[bestFeat] for example in dataSet]
# Remove duplicate attribute values
uniqueVals = set(featValues)
# Traverse the features and create a decision tree.
for value in uniqueVals:
# The class label is copied and stored in the new list variable subLabels
subLabels = labels[:]
myTree[bestFeatLabel][value] = createTree(splitDataSet(dataSet, bestFeat, value), subLabels)
return myTree
Simple data set before testing
if __name__ == '__main__':
dataSet, labels = createDataSet()
print(dataSet)
myTree = createTree(dataSet, labels)
print(myTree)
>>>
[[1, 1, 'yes'], [1, 1, 'yes'], [1, 0, 'no'], [0, 1, 'no'], [0, 1, 'no']]
{'no surfacing': {0: 'no', 1: {'flippers': {0: 'no', 1: 'yes'}}}}
The variable myTree contains many nested dictionaries representing tree structure information. Starting from the left, the first keyword no surfacing is the feature name of the first partitioned dataset, and the value of this keyword is also another data dictionary. The second keyword is the data set divided by the no surfacing feature. The values of these keywords are the child nodes of the no surfacing node. These values may be class labels (such as' flippers') or another data dictionary. If the value is a class label, the child node is a leaf node; If the value is another data dictionary, the child node is a judgment node. This format structure repeats continuously to form the whole tree. This tree contains three leaf nodes and two judgment nodes.
2, Drawing a tree in Python using the Matplotlib annotation
1. Matplotlib
Draw tree nodes with text annotations:
# Define text box and arrow formats
decisionNode = dict(boxstyle="sawtooth", fc="0.8")
leafNode = dict(boxstyle="round4", fc="0.8")
arrow_args = dict(arrowstyle="<-")
"""
Parameters:
nodeTxt - Node name
centerPt - Text position
parentPt - Arrow position of dimension
nodeType - Node format
Returns:
nothing
"""
def plotNode(nodeTxt, centerPt, parentPt, nodeType):
createPlot.ax1.annotate(nodeTxt, xy=parentPt, xycoords='axes fraction',
xytext=centerPt, textcoords='axes fraction',
va="center", ha="center", bbox=nodeType, arrowprops=arrow_args)
def createPlot():
fig = plt.figure(1, facecolor='white')
fig.clf()
createPlot.ax1 = plt.subplot(111, frameon=False)
plotNode('a decision node', (0.5, 0.1), (0.1, 0.5), decisionNode)
plotNode('a leaf node', (0.8, 0.1), (0.3, 0.8), leafNode)
plt.show()
if __name__ == '__main__':
createPlot()
Test effect:
2. Construct annotation tree
Although there are x and Y coordinates, how to place all tree nodes is a problem. You must know how many leaf nodes there are so that you can correctly determine the length of the x-axis; You also need to know how many layers the tree has so that you can correctly determine the height of the y-axis.
Get the number of leaf nodes and tree layers:
"""
Parameters:
myTree - Decision tree
Returns:
numLeafs - Number of leaf nodes of decision tree
"""
def getNumLeafs(myTree):
# Initialize leaf
numLeafs = 0
#myTree.keys() in Python 3 returns dict_keys, not a list,
#Therefore, you cannot use the method of myTree.keys()[0] to obtain node properties. You can use list(myTree.keys())[0]
firstStr = next(iter(myTree))
# Get the next set of dictionaries
secondDict = myTree[firstStr]
for key in secondDict.keys():
# Test whether the node is a dictionary. If it is not a dictionary, it means that the node is a leaf node
if type(secondDict[key]).__name__=='dict':
numLeafs += getNumLeafs(secondDict[key])
else: numLeafs +=1
return numLeafs
"""
Parameters:
myTree - Decision tree
Returns:
maxDepth - Layers of decision tree
"""
def getTreeDepth(myTree):
# Initialize decision tree depth
maxDepth = 0
#myTree.keys() in Python 3 returns dict_keys, not a list,
#Therefore, you cannot use the method of myTree.keys()[0] to obtain node properties. You can use list(myTree.keys())[0]
firstStr = next(iter(myTree))
# Get next dictionary
secondDict = myTree[firstStr]
for key in secondDict.keys():
#Test whether the node is a dictionary. If it is not a dictionary, it means that the node is a leaf node
if type(secondDict[key]).__name__=='dict':
thisDepth = 1 + getTreeDepth(secondDict[key])
else: thisDepth = 1
# Update layers
if thisDepth > maxDepth: maxDepth = thisDepth
return maxDepth
The function retrieveTree outputs the pre stored tree information, avoiding the trouble of creating a tree from the data every time you test the code.
def retrieveTree(i):
"""
Predefined tree structure for testing
"""
listOfTrees = [{'no surfacing': {0: 'no', 1: {'flippers': {0: 'no', 1: 'yes'}}}},
{'no surfacing': {0: 'no', 1: {'flippers': {0: {'head': {0: 'no', 1: 'yes'}}, 1: 'no'}}}}
]
return listOfTrees[i]
if __name__ == '__main__':
print(retrieveTree(1))
myTree = retrieveTree(0)
print(getNumLeafs(myTree))
print(getTreeDepth(myTree))
>>>
{'no surfacing': {0: 'no', 1: {'flippers': {0: {'head': {0: 'no', 1: 'yes'}}, 1: 'no'}}}}
3
2
Both the getNumLeafs() function and the gettreedepts() function returned the correct results. Next, draw a complete tree.
Dimension directed edge attribute values:
"""
Parameters:
cntrPt,parentPt - Used to calculate dimension locations
txtString - Marked content
Returns:
nothing
"""
def plotMidText(cntrPt, parentPt, txtString):
xMid = (parentPt[0]-cntrPt[0])/2.0 + cntrPt[0]#Calculate dimension location
yMid = (parentPt[1]-cntrPt[1])/2.0 + cntrPt[1]
createPlot.ax1.text(xMid, yMid, txtString, va="center", ha="center", rotation=30)
Draw decision tree:
"""
Parameters:
myTree - Decision tree(Dictionaries)
parentPt - Marked content
nodeTxt - Node name
Returns:
nothing
"""
def plotTree(myTree, parentPt, nodeTxt):
# Set node format
decisionNode = dict(boxstyle="sawtooth", fc="0.8")
# Format leaf nodes
leafNode = dict(boxstyle="round4", fc="0.8")
# Get the number of decision leaf nodes, which determines the width of the tree
numLeafs = getNumLeafs(myTree)
# Get the number of decision tree layers
depth = getTreeDepth(myTree)
# Next dictionary
firstStr = next(iter(myTree))
# Center position
cntrPt = (plotTree.xOff + (1.0 + float(numLeafs))/2.0/plotTree.totalW, plotTree.yOff)
# Dimension directed edge attribute values
plotMidText(cntrPt, parentPt, nodeTxt)
# Draw node
plotNode(firstStr, cntrPt, parentPt, decisionNode)
# The next dictionary is to continue drawing child nodes
secondDict = myTree[firstStr]
# y offset
plotTree.yOff = plotTree.yOff - 1.0/plotTree.totalD
for key in secondDict.keys():
# Test whether the node is a dictionary. If it is not a dictionary, it means that the node is a leaf node, not a leaf node. Call recursively to continue drawing
if type(secondDict[key]).__name__=='dict':
plotTree(secondDict[key],cntrPt,str(key))
# If it is a leaf node, draw the leaf node and mark the directed edge attribute value
else:
plotTree.xOff = plotTree.xOff + 1.0/plotTree.totalW
plotNode(secondDict[key], (plotTree.xOff, plotTree.yOff), cntrPt, leafNode)
plotMidText((plotTree.xOff, plotTree.yOff), cntrPt, str(key))
plotTree.yOff = plotTree.yOff + 1.0/plotTree.totalD
To create a paint panel:
"""
Parameters:
inTree - Decision tree(Dictionaries)
Returns:
nothing
"""
def createPlot(inTree):
# Create fig
fig = plt.figure(1, facecolor='white')
# Empty fig
fig.clf()
axprops = dict(xticks=[], yticks=[])
# Remove the x and y axes
createPlot.ax1 = plt.subplot(111, frameon=False, **axprops)
# Get the number of decision nodes
plotTree.totalW = float(getNumLeafs(inTree))
# Get the number of decision tree layers
plotTree.totalD = float(getTreeDepth(inTree))
# x offset
plotTree.xOff = -0.5/plotTree.totalW; plotTree.yOff = 1.0;
# Draw decision tree
plotTree(inTree, (0.5,1.0), '')
# Display drawing results
plt.show()
if __name__ == '__main__':
myTree = retrieveTree(0)
createPlot(myTree)
3, Test and store classifiers
1. Test algorithm: use decision tree to perform classification
After constructing the decision tree based on the training data, we can use it to classify the actual data. When performing data classification, the decision tree and the label vector used to construct the tree are needed. Then, the program compares the test data with the values on the decision tree, and recursively executes the process until it enters the leaf node; Finally, the test data is defined as the type of leaf node.
Use the classification function of the decision tree:
"""
Parameters:
inputTree - Generated decision tree
featLabels - Store the selected optimal feature label
testVec - Test data list, the order corresponds to the optimal feature label
Returns:
classLabel - Classification results
"""
def classify(inputTree, featLabels, testVec):
# Get decision tree node
firstStr = next(iter(inputTree))
# Next dictionary
secondDict = inputTree[firstStr]
featIndex = featLabels.index(firstStr)
for key in secondDict.keys():
if testVec[featIndex] == key:
if type(secondDict[key]).__name__ == 'dict':
classLabel = classify(secondDict[key], featLabels, testVec)
else: classLabel = secondDict[key]
return classLabel
if __name__ == '__main__':
dataSet, labels = createDataSet()
print(labels)
myTree = retrieveTree(0)
print(myTree)
print(classify(myTree, labels, [1, 0]))
print(classify(myTree, labels, [1, 1]))
>>>
['no surfacing', 'flippers']
{'no surfacing': {0: 'no', 1: {'flippers': {0: 'no', 1: 'yes'}}}}
no
yes
2. Use algorithm: storage of decision tree
Constructing a decision tree is a time-consuming task. It takes several seconds to process a small data set, such as the previous sample data. If the data set is large, it will consume a lot of computing time. However, using the created decision tree to solve the classification problem can be completed quickly. Therefore, in order to save computing time, it is best to call the constructed decision tree every time the classification is executed.
Storage decision tree:
"""
Parameters:
inputTree - Generated decision tree
filename - Storage file name of the decision tree
Returns:
nothing
"""
def storeTree(inputTree, filename):
with open(filename, 'wb') as fw:
pickle.dump(inputTree, fw)
Read decision tree:
"""
Parameters:
filename - Storage file name of the decision tree
Returns:
pickle.load(fr) - Decision tree dictionary
"""
def grabTree(filename):
fr = open(filename, 'rb')
return pickle.load(fr)
if __name__ == '__main__':
myTree = retrieveTree(0)
storeTree(myTree, 'classifierStorage.txt')
grabTree('classifierStorage.txt')
Through the above code, we can store the classifier on the hard disk instead of learning it again every time we classify the data, which is also one of the advantages of the decision tree.
4, Using decision tree to predict vehicle condition
The condition of the vehicle is divided into four categories:
- unacc (Unacceptable condition is very poor)
- acc (Acceptable condition is normal)
- good (Good)
- vgood (Very good)
Judge the condition of the vehicle through the following classification attributes:
- buying (purchase price: vhigh, high, med, low)
- maint (maintenance price: vhigh, high, med, low)
- Doors (several doors: 2, 3, 4, 5more)
- persons (capacity: 2, 4, more)
- lug_boot (storage space: small, med, big)
- safety: low, Med, high
Data sample:
Test code:
if __name__ == '__main__':
fr = open('carTrain.txt')
cars = [inst.strip().split('\t') for inst in fr.readlines()]
carLabels = ['buying', 'maint', 'doors', 'persons', 'lub_boot', 'safety']
carsTree = createTree(cars, carLabels)
print(carsTree)
createPlot(carsTree)
Test results:
>>>
{'safety': {'low': 'unacc', 'med': {'persons': {'2': 'unacc', '4': {'buying': {'low': {'maint': {'low': 'acc', 'med': 'good', 'high': 'acc', 'vhigh': 'acc'}}, 'med': {'maint': {'low': 'acc', 'med': 'acc', 'high': 'unacc', 'vhigh': 'acc'}}, 'high': {'maint': {'low': 'unacc', 'med': 'acc', 'high': 'unacc', 'vhigh': 'unacc'}}, 'vhigh': {'maint': {'low': 'unacc', 'med': 'acc', 'high': 'unacc', 'vhigh': 'unacc'}}}}, 'more': {'buying': {'low': {'maint': {'med': {'doors': {'2': 'good', '3': 'good', '4': 'acc'}}, 'vhigh': {'doors': {'2': 'acc', '3': 'acc', '4': 'unacc'}}}}, 'med': {'doors': {'2': 'acc', '3': 'acc', '4': {'maint': {'med': 'acc', 'vhigh': 'unacc'}}}}, 'high': {'maint': {'med': {'doors': {'2': 'acc', '3': 'acc', '4': 'unacc'}}, 'vhigh': 'unacc'}}, 'vhigh': {'maint': {'med': {'doors': {'2': 'acc', '3': 'acc', '4': 'unacc'}}, 'vhigh': 'unacc'}}}}}}, 'high': {'persons': {'4': {'buying': {'low': {'maint': {'med': {'doors': {'2': 'vgood', '3': 'good', '4': 'good'}}, 'vhigh': 'acc'}}, 'med': {'doors': {'2': {'maint': {'med': 'vgood', 'vhigh': 'acc'}}, '3': 'acc', '4': 'acc'}}, 'high': {'maint': {'med': 'acc', 'vhigh': 'unacc'}}, 'vhigh': {'maint': {'med': 'acc', 'vhigh': 'unacc'}}}}, '2': 'unacc', 'more': {'buying': {'low': {'doors': {'4': 'vgood', '3': 'vgood', '5more': {'maint': {'low': 'good', 'high': 'acc'}}}}, 'med': {'maint': {'low': {'doors': {'4': 'vgood', '3': 'vgood', '5more': 'good'}}, 'high': 'acc'}}, 'high': 'acc', 'vhigh': {'maint': {'low': 'acc', 'high': 'unacc'}}}}}}}}
Along different branches of the decision tree, the condition of the vehicle can be judged by some attributes of the vehicle itself.
The decision tree matches the experimental data very well, but there may be too many matching options. This problem is called over matching. In order to reduce the over matching problem, we can cut the decision tree and remove some unnecessary leaf nodes. If a leaf node can only add a little information, it can be deleted and incorporated into other leaf nodes. ID3 algorithm cannot directly process numerical data. Although we can convert numerical data into nominal data by quantitative method, if there are too many feature divisions, ID3 algorithm will still face other problems.
5, Improved algorithm
1. C4.5 algorithm
C4.5 algorithm is a classical algorithm used to generate decision tree. It is an extension and optimization of ID3 algorithm. C4.5 algorithm improves ID3 algorithm. The main improvements are:
- Using information gain rate to select partition features overcomes the deficiency of using information gain, but the information gain rate has a preference for attributes with a small number of values;
- It can handle discrete and continuous attribute types, that is, discrete continuous attributes;
- Capable of processing training data with missing attribute values;
- Pruning during tree construction;
Information gain rate
The information gain criterion has a preference for attributes with a large number of values. In order to reduce the possible adverse effects of this preference, C4.5 algorithm uses the information gain rate to select the optimal partition attributes. Gain rate formula:
G
a
i
n
_
r
a
t
i
o
(
D
,
a
)
=
G
a
i
n
(
D
,
a
)
I
V
(
a
)
Gain\_ratio(D,a) = \frac{ Gain(D,a)}{IV(a)} \quad
Gain_ratio(D,a)=IV(a)Gain(D,a)
among
I
V
(
a
)
=
−
∑
v
=
1
V
∣
D
v
∣
∣
D
∣
log
2
∣
D
v
∣
∣
D
∣
IV(a) = -\sum_{v=1}^V \frac{|D^v|}{|D|}\log_2 \frac{|D^v|}{|D|} \quad \quad
IV(a)=−v=1∑V∣D∣∣Dv∣log2∣D∣∣Dv∣
It is called the "intrinsic value" of attribute A. The more possible values of attribute a (i.e. the greater V), the
I
V
(
a
)
IV(a)
The larger the value of IV(a) is usually
The information gain rate criterion has a preference for attributes with a small number of values. Therefore, C4.5 algorithm does not directly select the candidate partition attribute with the largest information gain rate, but first find the attribute with higher information gain than the average level from the candidate partition attributes, and then select the attribute with the highest information gain rate.
Processing of continuous features
When the attribute type is discrete, there is no need to discretize the data;
When the attribute type is continuous, the data needs to be discretized. The specific ideas are as follows:
Specific ideas:
The continuous characteristic a of M samples has m values, arranged from small to large
a
1
,
a
2
,
...
,
a
n
a_1,a_2,{\ldots} ,a_n
a1, a2,..., an, take the average of two adjacent sample values as the division point, there are m-1 in total, and the ith division point Ti is expressed as:
T
i
=
(
a
i
+
a
i
+
1
)
/
2
T_i = (a_i + a_{i+1})/2
Ti=(ai+ai+1)/2 .
The information gain rate when these m-1 points are used as binary segmentation points is calculated respectively. The point with the largest information gain rate is selected as the best segmentation point of the continuous feature. For example, the point with the largest information gain rate is
a
t
a_t
at, then less than
a
t
a_t
The value of at is category 1, greater than
a
t
a_t
The value of at is class 2, so the discretization of continuous features is achieved.
2. CART
CART algorithm constructs a decision tree for classification: the size of Gini index is used to measure the advantages and disadvantages of each partition point.
The definition of Gini index is: in the classification problem, assuming that D has k classes, the probability that the sample point belongs to class k is
p
k
p_k
pk, then the probability
The Gini value of the distribution is defined as:
G
i
n
i
(
D
)
=
∑
k
=
1
K
p
k
(
1
−
p
k
)
=
1
−
∑
k
=
1
K
p
k
2
Gini(D) = \sum_{k=1}^K p_k(1 - p_k) = 1 - \sum_{k=1}^K p_k^2
Gini(D)=k=1∑Kpk(1−pk)=1−k=1∑Kpk2
G
i
n
i
(
D
)
Gini(D)
The smaller Gini(D), the higher the purity of data set D;
Given dataset D, the Gini index of attribute a is defined as:
G
i
n
i
i
n
d
e
x
(
D
,
a
)
=
∑
v
=
1
V
∣
D
v
∣
∣
D
∣
G
i
n
i
(
D
v
)
Gini_index(D,a) = \sum_{v=1}^V \frac{|D^v|}{|D|}Gini(D^v) \quad
Giniindex(D,a)=v=1∑V∣D∣∣Dv∣Gini(Dv)
In the candidate attribute set A, select the attribute that minimizes the Gini index after partition as the optimal partition attribute.