Machine learning_ 1:K-nearest neighbor algorithm

Experimental background

This experiment is based on the classical k-nearest neighbor algorithm of machine learning. I will first introduce the principle of k-nearest neighbor algorithm and basic classification experiments, and then introduce how to use k-nearest neighbor algorithm for handwriting recognition.

1.k-nearest neighbor algorithm

1.1 algorithm principle

What is k-nearest neighbor algorithm is to find k nearest points according to a given point to be judged. Among the k points, which label appears the most times, the point to be judged belongs to that label. As shown in the figure, the value of K here is 5, and among the five points, there are four w1, so this judgment point belongs to w1, which is the k-nearest neighbor algorithm.
Pseudo code:
Perform the following operations for each point in the test dataset in turn:
(1) Calculate the distance between the point in the training data set and the current point;
(2) Sort by increasing distance from small to large;
(3) Select the k points with the minimum distance from the current point;
(4) Determine the occurrence frequency of the category where the k points are located (i.e. count the number of each category in the k points)
(5) The category with the highest frequency among the k points is returned as the prediction classification of the current point

1.2 algorithm analysis

The following code is k-The principle code of nearest neighbor algorithm, I will try to analyze most of the code, so that everyone reading this article can understand it

#Scientific computing package numpy
from numpy import *            
#Operator module 
import operator                 
def createDataSet():
    group=array([[1.0,1.1],[1.0,1.0],[0,0],[0,0.1]])
    labels=['A','A','B','B']
    return group,labels

#Four parameters: input vector inx (input test coordinate point), training sample set dataSet (the array group above), label vector labels (the labels above), and k is the number of nearest neighbors (k in kNN algorithm)
def classify0(inx,dataSet,labels,k):
    #Read the length of the first dimension of the matrix
    dataSetSize=dataSet.shape[0]
    #diffMat stores the difference between the input vector inx and the sample set dataSet calculated through tile
    diffMat=tile(inx,(dataSetSize,1))-dataSet
    #Square each number in the diffMat array
    sqDiffMat=diffMat**2
    #Sum these numbers
    sqDistances=sqDiffMat.sum(axis=1)
    #Root the summation result
    distances=sqDistances**0.5
    #Sort data from small to large
    sortedDistIndicies=distances.argsort()
    #Build dictionary
    classCount={}
    #Cycle k times
    for i in range(k):
        #Match labels to sort order
        voteIlabel=labels[sortedDistIndicies[i]]
        #Statistics label frequency
        classCount[voteIlabel]=classCount.get(voteIlabel,0)+1
    #Sort again and sort in descending order according to the order of the second element
    sortedClassCount=sorted(classCount.iteritems(),key=operator.itemgetter(1),reverse=True)
    #Returns the most frequent tag
    return sortedClassCount[0][0]

The distance calculation formula adopts the European distance formula, which is as follows:

It can be simply understood as the square sum of the difference of the corresponding coordinates and then the root sign

1.3 algorithm experiment

As shown in the figure, click the path above, enter cmd to open the command prompt, and enter it in sequence
python
import KNN (the file you named to save the k-nearest neighbor algorithm)
group,labels=KNN.createDataSet() (create group and labels)
KNN.classify0([0,0],group,labels,3)
Errors may appear here

The reason is that the new version of python no longer supports iteritems. Just change it to items
The test result is shown in the figure, and the result is B (if it is not B, please pay attention to whether the code is different)

2. Handwriting recognition based on k-nearest neighbor algorithm

2.1 code analysis

#Convert the 32 * 32 image into a 1 * 1024 vector
def img2vector(filename):
    #initialization
    returnVect=zeros((1,1024))
    #Open file
    fr=open(filename)
    #Cyclic assignment
    for i in range(32):
        lineStr=fr.readline()
        for j in range(32):
            returnVect[0,32*i+j]=int(lineStr[j])
    #Return array
    return returnVect

#Handwriting recognition test set code
def handwritingClassTest():
    #Create an empty label array
    hwLabels=[]
    #Get directory content
    trainingFileList=listdir('trainingDigits')
    #Get the number of training sets
    m=len(trainingFileList)
    #initialization
    trainingMat=zeros((m,1024))
    #Cycle m times
    for i in range(m):
        #Get file name
        fileNameStr=trainingFileList[i]
        #Remove file suffix
        fileStr=fileNameStr.split('.')[0]
        #The file naming format is k_x. Like 9_ 45 (indicating that the picture is the number 9, ranking 45th in this kind of image), and obtain the category
        classNumStr=int(fileStr.split('_')[0])
        #Add category to label array
        hwLabels.append(classNumStr)
        trainingMat[i,:]=img2vector('trainingDigits/%s'%fileNameStr)
    #Import test set
    testFileList=listdir('testDigits')
    #Error statistics
    errorCount=0.0
    #Number of tests
    mTest=len(testFileList)
    #Cycle training
    for i in range(mTest):
        #Get file name
        fileNameStr=testFileList[i]
        #Remove file suffix
        fileStr=fileNameStr.split('.')[0]
        #Get test category
        classNumStr=int(fileStr.split('_')[0])
        #Convert to 1 * 1024 vector
        vectorUnderTest=img2vector('testDigits/%s'%fileNameStr)
        #Judge with classify0
        classifierResult=classify0(vectorUnderTest,trainingMat,hwLabels,3)
        #Display judgment results and real results
        print ("the classifier came back with:%d,the real answer is :%d"%(classifierResult,classNumStr))
        #Judge whether it is consistent, and the number of inconsistent errors + 1
        if(classifierResult!=classNumStr):errorCount+=1.0
    #Number of output errors
    print("\nthe total number of errors is:%d"%errorCount)
    #Output error rate
    print("\nthe total error rate is :%f"%(errorCount/float(mTest)))

2.2 code implementation

As in the previous experiment, enter cmd in the code directory to open the command prompt
Enter in sequence:
python
import KNN
KNN.handwritingClassTest()

As shown in the figure, the test results generally fluctuate around 1%.

3. Experimental summary

This experiment can clearly see the advantages and disadvantages of k-nearest neighbor algorithm
advantage:
1. Simple and effective
2. Easy to understand
Disadvantages:
1. All data sets must be saved. If the training data set is large, it will consume a lot of storage space
2. Compared with other algorithms, it is very time-consuming to calculate the distance of each data
3. The basic structure information of any data cannot be given, so it is impossible to know what characteristics the average instance sample and typical sample have
Through the advantages and disadvantages, we can find that k-nearest neighbor algorithm is obviously a basic algorithm suitable for beginners' machine learning. It may be further optimized, but its design idea determines its upper limit, so it is difficult to make a huge breakthrough.