ID3 decision tree algorithm
background knowledge
ID3 algorithm was first proposed by J. Ross Quinlan at the University of Sydney in 1975. The core of the algorithm is "information entropy". By calculating the information gain of each attribute, ID3 algorithm considers that the attribute with high information gain is a good attribute. Each division selects the attribute with the highest information gain as the division standard, and repeats this process until a decision tree that can perfectly classify training samples is generated.
Decision tree is used to classify data to achieve the purpose of prediction. The decision tree method first forms a decision tree according to the training set data. If the tree can not give correct classification to all objects, select some exceptions to the training set data and repeat the process until a correct decision set is formed. Decision tree represents the tree structure of decision set.
The decision tree consists of decision nodes, branches and leaves. The top node in the decision tree is the root node, and each branch is a new decision node or a leaf of the tree. Each decision node represents a problem or decision, which usually corresponds to the attributes of the object to be classified. Each leaf node represents a possible classification result. In the process of traversing from top to bottom along the decision tree, a test will be encountered at each node. Different test outputs of problems on each node will lead to different branches, and finally reach a leaf node. This process is the process of classification using the decision tree and using several variables to judge the category.
data description
The sample data used has certain requirements. ID3 is:
- Description - attributes - attributes with the same value must describe each example and have a fixed number of values.
- Predefined class instance properties must have been defined, that is, they are not learning ID3.
- Discrete classes - classes must be sharp and sharp. The decomposition of continuous classes into fuzzy categories (e.g. metals are "hard, difficult, flexible, gentle and soft") is unreliable.
- Enough examples - because inductive generalizations are used (i.e. undetectable), enough test cases must be selected to distinguish effective patterns and eliminate the influence of special coincidence factors.
ID3 determines which attributes are the best. A statistical feature, called information gain, uses entropy to get a given attribute to measure the training examples. The information with the highest information gain (information is the most beneficial classification) is selected. In order to clarify the gain, we first borrow a definition from information theory, called entropy. Each attribute has an entropy.
summary
ID3 decision tree is a very important structure used to deal with classification problems. It looks like a nested N-level IF... ELSE structure, but its judgment standard is no longer a relational expression, but the information gain of the corresponding module. It selects a branch from the root node through the size of the information gain, just like entering the statement of an IF structure, through the attribute value Different values enter the statement of the new IF structure until it reaches the leaf node and finds the "classification" label to which it belongs.
Its flowchart is a multi fork tree that cannot guarantee balance. Each parent node is a judgment module. Through judgment, the current vector will enter one of its child nodes. This child node is the judgment module or termination module (leaf node). If and only if this vector reaches the leaf node, it will find its "classification" label.
ID3 decision tree can form a permanent "tree" through a fixed training set. This course tree can be saved and applied to different test sets. The only requirement is that the test set and training set need to be structurally equivalent. This training process is the process of creating rules according to the training set, which is also the process of machine learning.
realization
data conversion
Watermelon dataset.txt:
Turquoise curling, turbid sound, clear depression, hard and smooth
Dark curled, dull, clear, sunken, hard and slippery
Black curl, turbid sound, clear depression, hard and smooth
Turquoise curled, dull, clear, sunken, hard and smooth
Light white curled, turbid sound, clear depression, hard and smooth
Turquoise slightly curled, turbid, clear, slightly concave, soft and sticky
Dark, slightly curled, turbid, noisy, slightly pasted, slightly concave, soft and sticky
Dark, slightly curled, turbid, clear, slightly concave, hard and smooth
Dark, slightly curled, dull, slightly pasted, slightly concave, hard and smooth
Green, hard, crisp, clear, flat, soft and sticky
Light white hard crisp fuzzy flat hard smooth no
Light white curled turbid sound fuzzy flat soft sticky no
Turquoise slightly curled, turbid, noisy, slightly pasted, concave, hard and slippery or not
Light white, slightly curled, dull, slightly pasted, sunken, hard and slippery or not
Dark, slightly curled, turbid, clear, slightly concave, soft and sticky
Light white curled turbid sound fuzzy flat hard smooth no
Green curled up, dull, slightly pasted, slightly concave, hard and slippery
To convert a txt document to csv: txt to csv
Import package
import numpy as np
import pandas as pd
import sklearn.tree as st
import math
import matplotlib
import os
import matplotlib.pyplot as plt
data = pd.read_csv('F:/Watermelon dataset.csv',header=None)
data
entropy
def calcEntropy(dataSet):
mD = len(dataSet)
dataLabelList = [x[-1] for x in dataSet]
dataLabelSet = set(dataLabelList)
ent = 0
for label in dataLabelSet:
mDv = dataLabelList.count(label)
prop = float(mDv) / mD
ent = ent - prop * np.math.log(prop, 2)
return ent
Split dataset
# Index - the index of the feature to split
# Feature - the feature to split
# Return value - a set in which the feature of the index in the dataSet is feature and the index column is removed
def splitDataSet(dataSet, index, feature):
splitedDataSet = []
mD = len(dataSet)
for data in dataSet:
if(data[index] == feature):
sliceTmp = data[:index]
sliceTmp.extend(data[index + 1:])
splitedDataSet.append(sliceTmp)
return splitedDataSet
Select the optimal feature
# Return value - the subscript of the best feature
def chooseBestFeature(dataSet):
entD = calcEntropy(dataSet)
mD = len(dataSet)
featureNumber = len(dataSet[0]) - 1
maxGain = -100
maxIndex = -1
for i in range(featureNumber):
entDCopy = entD
featureI = [x[i] for x in dataSet]
featureSet = set(featureI)
for feature in featureSet:
splitedDataSet = splitDataSet(dataSet, i, feature) # Split dataset
mDv = len(splitedDataSet)
entDCopy = entDCopy - float(mDv) / mD * calcEntropy(splitedDataSet)
if(maxIndex == -1):
maxGain = entDCopy
maxIndex = i
elif(maxGain < entDCopy):
maxGain = entDCopy
maxIndex = i
return maxIndex
Find most as labels
# Return value - label
def mainLabel(labelList):
labelRec = labelList[0]
maxLabelCount = -1
labelSet = set(labelList)
for label in labelSet:
if(labelList.count(label) > maxLabelCount):
maxLabelCount = labelList.count(label)
labelRec = label
return labelRec
Spanning tree
def createFullDecisionTree(dataSet, featureNames, featureNamesSet, labelListParent):
labelList = [x[-1] for x in dataSet]
if(len(dataSet) == 0):
return mainLabel(labelListParent)
elif(len(dataSet[0]) == 1): #There are no separable properties
return mainLabel(labelList) #Select the most label as the label of the dataset
elif(labelList.count(labelList[0]) == len(labelList)): # All belong to the same Label
return labelList[0]
bestFeatureIndex = chooseBestFeature(dataSet)
bestFeatureName = featureNames.pop(bestFeatureIndex)
myTree = {bestFeatureName: {}}
featureList = featureNamesSet.pop(bestFeatureIndex)
featureSet = set(featureList)
for feature in featureSet:
featureNamesNext = featureNames[:]
featureNamesSetNext = featureNamesSet[:][:]
splitedDataSet = splitDataSet(dataSet, bestFeatureIndex, feature)
myTree[bestFeatureName][feature] = createFullDecisionTree(splitedDataSet, featureNamesNext, featureNamesSetNext, labelList)
return myTree
initialization
# Return value
# dataSet dataset
# featureNames label
# featureNamesSet column label
def readWatermelonDataSet():
dataSet = data.values.tolist()
featureNames =['color and lustre', 'Root', 'Knock', 'texture', 'Umbilicus', 'Tactile sensation']
#Get featureNamesSet
featureNamesSet = []
for i in range(len(dataSet[0]) - 1):
col = [x[i] for x in dataSet]
colSet = set(col)
featureNamesSet.append(list(colSet))
return dataSet, featureNames, featureNamesSet
Drawing
# Can display Chinese
matplotlib.rcParams['font.sans-serif'] = ['SimHei']
matplotlib.rcParams['font.serif'] = ['SimHei']
# Bifurcation node, that is, decision node
decisionNode = dict(boxstyle="sawtooth", fc="0.8")
# Leaf node
leafNode = dict(boxstyle="round4", fc="0.8")
# Arrow style
arrow_args = dict(arrowstyle="<-")
def plotNode(nodeTxt, centerPt, parentPt, nodeType):
"""
Draw a node
:param nodeTxt: Text information describing the node
:param centerPt: Coordinates of text
:param parentPt: The coordinates of the point, which also refers to the coordinates of the parent node
:param nodeType: Node type,It is divided into leaf node and decision node
:return:
"""
createPlot.ax1.annotate(nodeTxt, xy=parentPt, xycoords='axes fraction',
xytext=centerPt, textcoords='axes fraction',
va="center", ha="center", bbox=nodeType, arrowprops=arrow_args)
def getNumLeafs(myTree):
"""
Gets the number of leaf nodes
:param myTree:
:return:
"""
# Count the total number of leaf nodes
numLeafs = 0
# Get the current first key, that is, the root node
firstStr = list(myTree.keys())[0]
# Get the content corresponding to the first key
secondDict = myTree[firstStr]
# Recursively traversing leaf nodes
for key in secondDict.keys():
# If the key corresponds to a dictionary, it is called recursively
if type(secondDict[key]).__name__ == 'dict':
numLeafs += getNumLeafs(secondDict[key])
# If not, it means that it is a leaf node at this time
else:
numLeafs += 1
return numLeafs
def getTreeDepth(myTree):
"""
Number of depth layers obtained
:param myTree:
:return:
"""
# Used to save the maximum number of layers
maxDepth = 0
# Get root node
firstStr = list(myTree.keys())[0]
# Get the content corresponding to the key
secondDic = myTree[firstStr]
# Traverse all child nodes
for key in secondDic.keys():
# If the node is a dictionary, it is called recursively
if type(secondDic[key]).__name__ == 'dict':
# Depth of child node plus 1
thisDepth = 1 + getTreeDepth(secondDic[key])
# This indicates that this is a leaf node
else:
thisDepth = 1
# Replace maximum layers
if thisDepth > maxDepth:
maxDepth = thisDepth
return maxDepth
def plotMidText(cntrPt, parentPt, txtString):
"""
Calculate the middle position between the parent node and the child node, and fill in the information
:param cntrPt: Child node coordinates
:param parentPt: Parent node coordinates
:param txtString: Filled text information
:return:
"""
# Calculate the middle position of the x-axis
xMid = (parentPt[0]-cntrPt[0])/2.0 + cntrPt[0]
# Calculate the middle position of the y-axis
yMid = (parentPt[1]-cntrPt[1])/2.0 + cntrPt[1]
# Draw
createPlot.ax1.text(xMid, yMid, txtString)
def plotTree(myTree, parentPt, nodeTxt):
"""
Draw all nodes of the tree and draw recursively
:param myTree: tree
:param parentPt: Coordinates of the parent node
:param nodeTxt: Text information of the node
:return:
"""
# Calculate the number of leaf nodes
numLeafs = getNumLeafs(myTree=myTree)
# Calculate the depth of the tree
depth = getTreeDepth(myTree=myTree)
# Get the information content of the root node
firstStr = list(myTree.keys())[0]
# Calculate the middle coordinates of the current root node in all child nodes, that is, the offset of the current X-axis plus the calculated center position of the root node as the x-axis (for example, the first time: the initial x-offset is: - 1/2W, the calculated center position of the root node is: (1+W)/2W, add to get: 1 / 2), and the current Y-axis offset is the y-axis
cntrPt = (plotTree.xOff + (1.0 + float(numLeafs))/2.0/plotTree.totalW, plotTree.yOff)
# Draw the connection between the node and the parent node
plotMidText(cntrPt, parentPt, nodeTxt)
# Draw the node
plotNode(firstStr, cntrPt, parentPt, decisionNode)
# Get the subtree corresponding to the current root node
secondDict = myTree[firstStr]
# Calculate the new y-axis offset and move down 1/D, that is, the drawing y-axis of the next layer
plotTree.yOff = plotTree.yOff - 1.0/plotTree.totalD
# Loop through all key s
for key in secondDict.keys():
# If the current key is a dictionary and there are subtrees, it will be traversed recursively
if isinstance(secondDict[key], dict):
plotTree(secondDict[key], cntrPt, str(key))
else:
# Calculate the new X-axis offset, that is, the x-axis coordinate drawn by the next leaf moves 1/W to the right
plotTree.xOff = plotTree.xOff + 1.0/plotTree.totalW
# Open the annotation to observe the coordinate changes of leaf nodes
# print((plotTree.xOff, plotTree.yOff), secondDict[key])
# Draw leaf node
plotNode(secondDict[key], (plotTree.xOff, plotTree.yOff), cntrPt, leafNode)
# Draw the content of the middle line between the leaf node and the parent node
plotMidText((plotTree.xOff, plotTree.yOff), cntrPt, str(key))
# Before returning to recursion, you need to increase the offset of the y-axis and move it up by 1/D, that is, return to draw the y-axis of the previous layer
plotTree.yOff = plotTree.yOff + 1.0/plotTree.totalD
def createPlot(inTree):
"""
Decision tree to be drawn
:param inTree: Decision tree dictionary
:return:
"""
# Create an image
fig = plt.figure(1, facecolor='white')
fig.clf()
axprops = dict(xticks=[], yticks=[])
createPlot.ax1 = plt.subplot(111, frameon=False, **axprops)
# Calculate the total width of the decision tree
plotTree.totalW = float(getNumLeafs(inTree))
# Calculate the total depth of the decision tree
plotTree.totalD = float(getTreeDepth(inTree))
# The initial x-axis offset, that is - 1/2W, moves 1/W to the right each time, that is, the x coordinates drawn by the first leaf node are: 1/2W, the second: 3/2W, the third: 5/2W, and the last: (W-1)/2W
plotTree.xOff = -0.5/plotTree.totalW
# The initial y-axis offset, moving down or up 1/D each time
plotTree.yOff = 1.0
# Call the function to draw the node image
plotTree(inTree, (0.5, 1.0), '')
# draw
plt.show()
result
dataSet, featureNames, featureNamesSet=readWatermelonDataSet() testTree= createFullDecisionTree(dataSet, featureNames, featureNamesSet,featureNames) createPlot(testTree)
