## 1, Introduction to CART decision tree algorithm

CART (Classification And Regression Trees) algorithm is a tree construction algorithm, which can be used for both classification tasks and regression. Compared with ID3 and C4.5, which can only be used for discrete data and classification tasks, CART algorithm has a much wider application. It can be used for both discrete data and continuous data, and both classification and regression tasks can be processed.

This paper only discusses the construction of basic CART classification decision tree, not regression tree and pruning.

First of all, we should clarify the following points:

1. CART algorithm is a common method of binary classification. The decision tree generated by CART algorithm is a binary tree, while the decision tree generated by ID3 and C4.5 algorithm is a multi tree. From the perspective of operation efficiency, the binary tree model will be more efficient than the multi tree.

2. CART algorithm selects the optimal feature through Gini index.

## 2, Gini coefficient

Gini coefficient represents the impurity of the model. The smaller the Gini coefficient, the lower the impurity. Note that this is just opposite to the definition of information gain ratio in C4.5.

In the classification problem, assuming that there are k classes and the probability that the sample points belong to class k is pk, the Gini coefficient of the probability distribution is defined as:

If CART is used for class II classification problems (not only for class II Classification), the Gini coefficient of probability distribution can be simplified to

Assuming that feature A is used to divide data set D into two parts D1 and D2, the Gini coefficient of the data set divided according to feature A is:

## 3, CART decision tree generation algorithm

Input: training data set D, conditions for stopping calculation

Output: CART decision tree

According to the training data set, starting from the root node, recursively perform the following operations on each node to build a binary decision tree:

(1) Calculate the Gini index of the existing features to the data set, as shown above;

(2) Select the feature corresponding to the minimum value of Gini index as the optimal feature, and the corresponding tangent point as the optimal tangent point (if there are multiple features or tangent points corresponding to the minimum value, just take one);

(3) According to the optimal features and optimal cut points, two sub nodes are generated from the current node, and the data in the training data set are allocated to the two sub nodes according to the features and attributes;

(4) Recursively call (1) (2) (3) to two child nodes until the stop condition is met.

(5) Generate CART tree.

Conditions for stopping the algorithm: the number of samples in the node is less than the predetermined threshold, or the Gini index of the sample set is less than the predetermined threshold (samples basically belong to the same class, if they all belong to the same class, it is 0), or the feature set is empty.

Note: the optimal tangent point is a necessary condition to divide the current sample into two classes (because we want to construct a binary tree). For the discrete case, the optimal tangent point is a value of the current optimal feature; For the continuous case, the optimal tangent point can be a specific value. In the specific application, we need to traverse all the possible optimal tangent points to find the optimal tangent point we need.

## 4, Python implementation of CART algorithm

from math import log # Construct dataset def create_dataset(): dataset = [['youth', 'no', 'no', 'just so-so', 'no'], ['youth', 'no', 'no', 'good', 'no'], ['youth', 'yes', 'no', 'good', 'yes'], ['youth', 'yes', 'yes', 'just so-so', 'yes'], ['youth', 'no', 'no', 'just so-so', 'no'], ['midlife', 'no', 'no', 'just so-so', 'no'], ['midlife', 'no', 'no', 'good', 'no'], ['midlife', 'yes', 'yes', 'good', 'yes'], ['midlife', 'no', 'yes', 'great', 'yes'], ['midlife', 'no', 'yes', 'great', 'yes'], ['geriatric', 'no', 'yes', 'great', 'yes'], ['geriatric', 'no', 'yes', 'good', 'yes'], ['geriatric', 'yes', 'no', 'good', 'yes'], ['geriatric', 'yes', 'no', 'great', 'yes'], ['geriatric', 'no', 'no', 'just so-so', 'no']] features = ['age', 'work', 'house', 'credit'] return dataset, features # Calculates the Gini coefficient of the current set def calcGini(dataset): # Find the total number of samples num_of_examples = len(dataset) labelCnt = {} # Traverse the entire sample set for example in dataset: # The tag value of the current sample is the last element of the list currentLabel = example[-1] # Count how many times each label appears if currentLabel not in labelCnt.keys(): labelCnt[currentLabel] = 0 labelCnt[currentLabel] += 1 # After getting the number of samples of each label in the current set, calculate their p value for key in labelCnt: labelCnt[key] /= num_of_examples labelCnt[key] = labelCnt[key] * labelCnt[key] # Calculate Gini coefficient Gini = 1 - sum(labelCnt.values()) return Gini # Extract subset # Function: first find all samples with axis tag value = value from the dataSet # Then delete the axis tag value of these samples, and then extract them all into a new sample set def create_sub_dataset(dataset, index, value): sub_dataset = [] for example in dataset: current_list = [] if example[index] == value: current_list = example[:index] current_list.extend(example[index + 1 :]) sub_dataset.append(current_list) return sub_dataset # Divide the current sample set into a part with value of feature i and a part with value other than value (two points) def split_dataset(dataset, index, value): sub_dataset1 = [] sub_dataset2 = [] for example in dataset: current_list = [] if example[index] == value: current_list = example[:index] current_list.extend(example[index + 1 :]) sub_dataset1.append(current_list) else: current_list = example[:index] current_list.extend(example[index + 1 :]) sub_dataset2.append(current_list) return sub_dataset1, sub_dataset2 def choose_best_feature(dataset): # Total number of features numFeatures = len(dataset[0]) - 1 # When there is only one feature if numFeatures == 1: return 0 # Initialize optimal Gini coefficient bestGini = 1 # Initialize optimal features index_of_best_feature = -1 # Traverse all features to find the optimal feature and the optimal tangent point under the feature for i in range(numFeatures): # De duplication, each attribute value is unique uniqueVals = set(example[i] for example in dataset) # Each value in the Gini dictionary represents the Gini coefficient after dividing the current set with the key corresponding to the value as the tangent point Gini = {} # For each value of the current feature for value in uniqueVals: # First find the two subsets divided by the value sub_dataset1, sub_dataset2 = split_dataset(dataset,i,value) # Find the proportion coefficient prob1 and prob2 of the two subsets in the original set prob1 = len(sub_dataset1) / float(len(dataset)) prob2 = len(sub_dataset2) / float(len(dataset)) # Calculate Gini coefficients for subset 1 Gini_of_sub_dataset1 = calcGini(sub_dataset1) # Calculate Gini coefficients for subset 2 Gini_of_sub_dataset2 = calcGini(sub_dataset2) # Calculate the final Gini coefficient divided by the current optimal tangent point Gini[value] = prob1 * Gini_of_sub_dataset1 + prob2 * Gini_of_sub_dataset2 # Update the optimal features and optimal cut points if Gini[value] < bestGini: bestGini = Gini[value] index_of_best_feature = i best_split_point = value return index_of_best_feature, best_split_point # Returns the value of the tag with the largest number of samples ('yes' or 'no') def find_label(classList): # Initialize the dictionary for counting the times of each label # The key is each label, and the corresponding value is the number of times the label appears labelCnt = {} for key in classList: if key not in labelCnt.keys(): labelCnt[key] = 0 labelCnt[key] += 1 # Sort classCount values in descending order # For example: sorted_labelCnt = {'yes': 9, 'no': 6} sorted_labelCnt = sorted(labelCnt.items(), key = lambda a:a[1], reverse = True) # There is a problem with the following way of writing # sortedClassCount = sorted(labelCnt.iteritems(), key=operator.itemgetter(1), reverse=True) # Sorted_ The first value of the first element in labelcnt is the desired value return sorted_labelCnt[0][0] def create_decision_tree(dataset, features): # Find the labels of all samples in the training set # For the initial dataset, its label_list = ['no', 'no', 'yes', 'yes', 'no', 'no', 'no', 'yes', 'yes', 'yes', 'yes', 'yes', 'yes', 'yes', 'no'] label_list = [example[-1] for example in dataset] # First write two cases where recursion ends: # If the labels of all samples in the current set are equal (that is, the samples have been classified as "pure") # The label value is directly returned as a leaf node if label_list.count(label_list[0]) == len(label_list): return label_list[0] # If all features of the training set have been used, there are no available features, but the sample has not been classified as "pure" # The label with the most samples is returned as the result if len(dataset[0]) == 1: return find_label(label_list) # The following is the formal establishment process # Select the subscript and the best segmentation point of the best feature for branching index_of_best_feature, best_split_point = choose_best_feature(dataset) # Get the best feature best_feature = features[index_of_best_feature] # Initialize decision tree decision_tree = {best_feature: {}} # Delete the current best feature after it has been used del(features[index_of_best_feature]) # Sub feature = current feature (because the used feature has been deleted just now) sub_labels = features[:] # Recursive call to create_decision_tree to generate new nodes # Generate a binary set divided by the optimal tangent point sub_dataset1, sub_dataset2 = split_dataset(dataset,index_of_best_feature,best_split_point) # Construct left subtree decision_tree[best_feature][best_split_point] = create_decision_tree(sub_dataset1, sub_labels) # Construct right subtree decision_tree[best_feature]['others'] = create_decision_tree(sub_dataset2, sub_labels) return decision_tree # Use the decision tree trained above to classify the new samples def classify(decision_tree, features, test_example): # The attribute represented by the root node first_feature = list(decision_tree.keys())[0] # second_dict is the value of the first classification attribute (also a dictionary) second_dict = decision_tree[first_feature] # The position of the attribute represented by the tree root in the attribute tag, that is, the first attribute index_of_first_feature = features.index(first_feature) # For second_ Every key in Dict for key in second_dict.keys(): # key not equal to 'others' if key != 'others': if test_example[index_of_first_feature] == key: # If current second_ The value of the key of dict is a dictionary if type(second_dict[key]).__name__ == 'dict': # Recursive query is required classLabel = classify(second_dict[key], features, test_example) # If current second_ The value of the key of dict is a separate value else: # Is the tag value to find classLabel = second_dict[key] # If the value of the test sample in the current feature is not equal to key, it means that its value in the current feature belongs to 'others' else: # If second_ If the value of dict ['others'] is a string, it is output directly if isinstance(second_dict['others'],str): classLabel = second_dict['others'] # If second_ If the value of dict ['others'] is a dictionary, the query is recursive else: classLabel = classify(second_dict['others'], features, test_example) return classLabel if __name__ == '__main__': dataset, features = create_dataset() decision_tree = create_decision_tree(dataset, features) # Print the generated decision tree print(decision_tree) # Classify the new samples features = ['age', 'work', 'house', 'credit'] test_example = ['midlife', 'yes', 'no', 'great'] print(classify(decision_tree, features, test_example))

If it is a binary classification problem, then the functions calcGini and choose_ best_ Features can be simplified as follows:

# Calculate the probability p that the sample belongs to the first class def calcProbabilityEnt(dataset): numEntries = len(dataset) count = 0 label = dataset[0][len(dataset[0]) - 1] for example in dataset: if example[-1] == label: count += 1 probabilityEnt = float(count) / numEntries return probabilityEnt def choose_best_feature(dataset): # Total number of features numFeatures = len(dataset[0]) - 1 # When there is only one feature if numFeatures == 1: return 0 # Initialize optimal Gini coefficient bestGini = 1 # Initialize optimal features index_of_best_feature = -1 for i in range(numFeatures): # De duplication, each attribute value is unique uniqueVals = set(example[i] for example in dataset) # Gini coefficient that defines the value of the characteristic Gini = {} for value in uniqueVals: sub_dataset1, sub_dataset2 = split_dataset(dataset,i,value) prob1 = len(sub_dataset1) / float(len(dataset)) prob2 = len(sub_dataset2) / float(len(dataset)) probabilityEnt1 = calcProbabilityEnt(sub_dataset1) probabilityEnt2 = calcProbabilityEnt(sub_dataset2) Gini[value] = prob1 * 2 * probabilityEnt1 * (1 - probabilityEnt1) + prob2 * 2 * probabilityEnt2 * (1 - probabilityEnt2) if Gini[value] < bestGini: bestGini = Gini[value] index_of_best_feature = i best_split_point = value return index_of_best_feature, best_split_point