Handwriting recognition based on KNN
Task introduction
 This example uses sklearn to train a Knearest neighbor (KNN) classifier to recognize handwritten digits in the data set DBRHD.
 The recognition effect of KNN is compared with that of multilayer perceptron.
Input of KNN

Each picture of DBRHD dataset is a 32 * 32 text matrix composed of 0 or 1;

The input of KNN is a 1024 dimensional vector expanded by the picture matrix.
Handwriting recognition based on KNN
Experimental steps:

Step 1: create the project and lead the sklearn package

Step 2: load training data

Step 3: build KNN classifier

Step 4: test set evaluation
Specific steps
Step 1: create the project and import the sklearn package
(1) Create the sklearnKNN.py file
(2) Import sklearn related packages in the sklearknn.py file
Step 2: load training data
(1) In the sklearnKNN.py file, define the img2vector function to expand the loaded 32 * 32 picture matrix into a column of vectors.
(2) Define the function readDataSet to load training data in the sklearnKNN.py file.
(3) in the sklearnKNN.py file, the read DataSet and img2vector functions are called to load the data, and the trained pictures are stored in train_. In the dataset, the corresponding tag has a train_hwLabels.
Step 3: build KNN classifier
In the sklearnKNN.py file, build KNN classifier: set the search algorithm and the number of neighbor points (k).
 KNN is a lazy learning method. There is no learning process. It only finds the nearest neighbor point during prediction. The input of data set is the process of constructing KNN classifier.
 When building KNN, we also called the fit() function.
Step 4: test set evaluation
(1) Load test set
(2) The constructed KNN classifier is used to predict the test set, and the prediction error rate is calculated
Specific code
import numpy as np # Import numpy Toolkit from os import listdir # Use the listdir module to access local files from sklearn import neighbors def img2vector(fileName): retMat = np.zeros([1024], int) # Define the returned matrix with a size of 1 * 1024 fr = open(fileName) # Open a digital file with a size of 32 * 32 lines = fr.readlines() # Read all lines of the file for i in range(32): # Traverse all lines of the file for j in range(32): # And store the 01 number in retMat retMat[i * 32 + j] = lines[i][j] return retMat def readDataSet(path): fileList = listdir(path) # Get all files in the folder numFiles = len(fileList) # Count the number of files that need to be read dataSet = np.zeros([numFiles, 1024], int) # Used to store all digital files hwLabels = np.zeros([numFiles]) # Used to store the corresponding label (different from neural network) for i in range(numFiles): # Traverse all files filePath = fileList[i] # Get file name / path digit = int(filePath.split('_')[0]) # Get label by file name hwLabels[i] = digit # Store numbers directly, not one hot vectors dataSet[i] = img2vector(path + '/' + filePath) # Read file contents return dataSet, hwLabels # read dataSet train_dataSet, train_hwLabels = readDataSet('digits/trainingDigits') knn = neighbors.KNeighborsClassifier(algorithm='kd_tree', n_neighbors=3) knn.fit(train_dataSet, train_hwLabels) # read testing dataSet dataSet, hwLabels = readDataSet('digits/testDigits') res = knn.predict(dataSet) # Predict the test set error_num = np.sum(res != hwLabels) # Count the number of classification errors num = len(dataSet) # Number of test sets print("Total num:", num, " Wrong num:", \ error_num, " TrueRate:", 1(error_num / float(num)))
Experimental effect
Influence analysis of neighbor number k: set KNN classifiers with K as 1, 3, 5 and 7, and compare their experimental results.
KNN classifier with K set to 1:
Total num: 946 Wrong num: 13 TrueRate: 0.9862579281183932
KNN classifier with K set to 3:
Total num: 946 Wrong num: 12 TrueRate: 0.9873150105708245
KNN classifier with K set to 5:
Total num: 946 Wrong num: 19 TrueRate: 0.9799154334038055
KNN classifier with K set to 7:
Total num: 946 Wrong num: 22 TrueRate: 0.9767441860465116
Conclusion:
When K=3, the accuracy rate is the highest. When k > 3, the accuracy rate begins to decline. This is because when the sample is a sparse data set (there are only 946 samples in this example), the kth neighbor point may be far away from the test point, so it cast an error vote, which affects the final prediction result.
Comparative experiment
KNN classifier vs. MLP multilayer perceptron:
We take the MLP classifier with the highest accuracy (H) and the worst accuracy (L) in the comparison experiments on the number of neurons in different hidden layers, the maximum number of iterations and the learning rate in the previous section. The parameters of each MLP are set as follows:
MLP code  Number of hidden layer neurons  Maximum number of iterations  optimization method  Initial learning rate / learning rate 

MLPYH  200  2000  adam  0.0001 
MLPYL  50  2000  adam  0.0001 
MLPDH  100  2000  adam  0.0001 
MLPDL  100  500  adam  0.0001 
MLPXH  100  2000  sgd  0.1 
MLPXL  100  2000  sgd  0.0001 
Compare the best KNN classifier (K=3) and the worst KNN classifier (K=7) with each MLP classifier as follows:
(for MLP data, we need to look at the previous experiment, mainly real data)
classifier  Number of neurons in MLP hidden layer (MLPY)  MLP iterations (MLPD)  MLP learning rate (MLPX)  Number of KNN neighbors  

best  Error quantity  37  33  33  12 
best  Correct rate  0.9608  0.9651  0.9651  0.9873 
worst  Error quantity  43  54  242  22 
worst  Correct rate  0.9545  0.9429  0.7441  0.9767 
Conclusion:
 The accuracy of KNN is much higher than that of MLP classifier, which is because MLP is easy to over fit on small data sets.
 MLP is sensitive to parameter adjustment. If the parameter setting is unreasonable, it is easy to get poor classification effect. Therefore, parameter setting is very important for MLP.
Last thought
The principle of this experiment, KNN, K nearest neighbor algorithm, can be reviewed in the 12 basic classification models.
This time, the program needs to import the data set digits.rar and put it in the file directory.
The code is not very difficult, and it is marked. Different from the original video, the final output accuracy is changed, and the output result is slightly different from that given in the video.
For experimental comparison, different K values are set as 1, 3, 5 and 7. What needs to be changed is n in kneigborsclassifier()_ The value of neighbors.
In addition, because it needs to be compared with MLP multilayer perceptron, I made a table and went to the last experiment to find the data.
The final conclusion is that the accuracy of KNN is much higher than that of MLP classifier.
It's so cold. I want to eat hot pot and small cake.