# Machine learning -- logistic regression (multivariate classification) programming training

reference material:

1.Mr. Huang haiguang: Wu Enda machine learning notes github

This paper is the first part of the third programming training in Wu Enda's machine learning course.

Programming tasks:

Recognize handwritten digits (0 to 9) using logical regression

Dataset: GitHub warehouse from Mr. Huang haiguang, Specific links

This dataset is a. mat file containing two variables X , y X,y X,y

variable X X X ￠ sample pictures containing 5000 handwritten digits, each sample picture is 20 in size × 20 pixels, so each sample is represented as 400 dimensions, where the value represents the gray value of each pixel lattice.

variable y y y is the label corresponding to 5000 samples, such as 1-500 samples y = 10 y = 10 y=10, indicating that the 500 samples are corresponding to class 0, 501-1000 samples y = 1 y = 1 y=1, indicating that the 500 samples correspond to class 1. If this goes on, y y The value of y ranges from 1-10, corresponding to numbers 1-9 and 0 respectively

## 1. Multivariate classification

adopt Machine learning -- logistic regression (classification) programming training We are already familiar with the binary classification problem.

The so-called multivariate classification can be understood as: the reclassification of secondary classification.

For example, we want to classify dataset a into four categories: A, B, C and d. First, we can divide a into a and B, then B into B and C, and finally C into C and d.

This multiple classification problem is based on two classification, which can be called one vs others classification.

Multivariate classification problem is actually realized by using multiple binary classifiers, which is essentially the same as logistic regression binary classification problem.

## 2. Label Vectorization

As shown in the figure above, the label data we obtained from the original dataset is a (5000,1) column vector.

Similarly, we refer to the second classification problem. There are only two kinds of labels for binary classification problems: 0 and 1.

When a is divided into a and B, we can convert the label corresponding to a to 1 and the label corresponding to other data to 0, and successfully separate a. A new label vector is obtained y 1 y_1 y1, containing only 0 and 1.

Similarly, we can use the same method to separate b, c and d. There should be three new label vectors y 2 , y 3 , y 4 y_2,y_3,y_4 y2,y3,y4.

This process is called the label vectorization process.

## 3. Multivariate classification process

- Read data
- Get eigenvalue X X X
- Get tag value y y y
- Tag value Vectorization y [ 0 ] , y [ 1 ] , y [ 2 ] , y [ 3 ] . . . y[0],y[1],y[2],y[3]... y[0],y[1],y[2],y[3]...
- Feature scaling (if necessary)
- Building functional relationships between features and labels (hypothetical functions)
- Constructing regularization cost function
- Find the partial derivative of the regularization cost function
- The minimum regularization cost function is used to obtain the weight at this time
- The functional relationship between features and labels is obtained according to the weight

The process of multivariate classification is only one more label value vectorization compared with the second classification, and the rest are exactly the same as the second classification. It should be noted that here y [ 0 ] y[0] y[0] is equivalent to a two classifier, which corresponds to a group θ [ 0 ] θ[0] θ [0], and y [ 1 ] y[1] y[1] is equivalent to another two classifier, which corresponds to a group θ [ 1 ] θ[1] θ [1] In this way, each label vector is equivalent to a two classifier, corresponding to a set of weight coefficients.

## 4. Difference between (n,) (n, 1)

One problem needs to be clarified before code implementation. This will help us understand the dimension correspondence related to calculation in the code.

numpy In, about(n,) (n,1)Differences between

We can observe the following code:

import numpy as np list_1 = [[1, 2, 3]] a = np.array(list_1) print(a) # [[1 2 3]] print(a.shape) # (1, 3) b = np.arange(1, 4, 1).reshape(3, 1) print(b) # [[1] # [2] # [3]] print(b.shape) #(3, 1) print(a @ b) # [[14]] # The calculation here is the calculation corresponding to the (1,3) (3,1) dimension, which is in line with our matrix calculation rules

import numpy as np list_1 = [[1, 2, 3]] a = np.array(list_1).reshape(3) print(a) # [1 2 3] print(a.shape) # (3,) b = np.arange(1, 7, 1).reshape(3, 2) print(b) # [[1] # [2] # [3]] print(b.shape) # (3, 1) print(a @ b) # [14] # We found that this code gives a result similar to the previous code. This is the calculation corresponding to the (3), (3,1) dimension. If it is based on the matrix calculation rules, can we understand it as follows: # (3) row vector equivalent to (1,3)?

import numpy as np list_1 = [[1, 2, 3]] a = np.array(list_1).reshape(3) print(a) # [1 2 3] print(a.shape) # (3,) b = np.arange(1, 4, 1).reshape(1,3) print(b) # [[1 2 3]] print(b.shape) # (1, 3) print(b @ a) # [14] # In this code, we get the same result as in the second code. This is the calculation corresponding to the dimension (1,3) (3). If it is based on the matrix calculation rules, can we have the following understanding: # (3) column vector equivalent to (3,1)? # When we try to calculate (3), (1,3), we find that there is a syntax error

Through the above code, we get the following conclusions:

**(n,) when performing matrix operation, it can select whether it is expressed as column vector or row vector according to the situation. In this case, on the premise of following the matrix calculation rules, the calculation result must be a one-dimensional array, otherwise an error will be reported** Therefore, transforming (n,1) into (n,) can ignore the dimension correspondence to a certain extent.

## 5. Code implementation

### 5.1 graphical data

variable X X X ￠ sample pictures containing 5000 handwritten digits, each sample picture is 20 in size × 20 pixels, so each sample is represented as 400 dimensions, where the value represents the gray value of each pixel lattice.

variable y y y is the label corresponding to 5000 samples

In order to better understand the true meaning of the data, the data can be graphical through the following code:

import numpy as np import scipy.io as sio import matplotlib.pyplot as plt import matplotlib def load_data(path): # Load data data = sio.loadmat(path) # Read. mat data through path y = data.get('y') # Get tag y(5000,1) y = y.reshape(y.shape[0]) # For the convenience of calculation, y is converted into a one-dimensional array of (5000,) X = data.get('X') # (5000,400) return X, y def plot_an_image(image): # mapping fig, ax = plt.subplots(figsize=(1, 1)) # Define an image of size (1,1) ax.matshow(image.reshape((20, 20)), cmap=matplotlib.cm.binary) # A function that draws a matrix or array into an image plt.xticks(np.array([])) # The x-axis is not displayed plt.yticks(np.array([])) # The y-axis is not displayed X, raw_y = load_data('ex3data1.mat') # Read feature X pick_one = np.random.randint(0, 5000) # Select a sample at random print(pick_one) plot_an_image(X[pick_one, :]) # Drawing according to the characteristics of selected samples, that is, drawing according to the gray value of pixel lattice plt.show() print('this should be {}'.format(raw_y[pick_one]))

Operation results:

4004 # The 4004th sample this should be 8

### 5.2 complete code

Because the process of multivariate classification is almost the same as that of binary classification, the code can be used simply by modifying the code of binary classification. The same code part as binary classification will not be commented in detail. For details, see Machine learning -- regularized logistic regression (classification) programming training Code comments in.

import numpy as np import scipy.io as sio import scipy.optimize as opt from sklearn.metrics import classification_report # This package is an evaluation report def load_data(path): # Load data data = sio.loadmat(path) # Read. mat data through path y = data.get('y') # Get tag y(5000,1) y = y.reshape(y.shape[0]) # For the convenience of calculation, y is converted into a one-dimensional array of (5000,) raw_X = data.get('X') # (5000,400) X = np.insert(raw_X, 0, values=np.ones(raw_X.shape[0]), axis=1) # The first column (all 1) is inserted, which is x_0 = 1 return X, y def label_vectorization(raw_y): # Label Vectorization y_matrix = [] # Define an empty list to store vectorized data for k in range(1, 11): # Label: 1-10, corresponding to 1-9 and 0, and the label is vectorized into 10 vectors y_matrix.append((raw_y == k).astype(int)) y_matrix = [y_matrix[-1]] + y_matrix[:-1] # Since the last vector corresponds to class 0, Class 0 is mentioned to the first, that is, 0-9 y = np.array(y_matrix) # Convert the list data type to the ndarray data type return y def sigmoid(z): # Define sigmoid function return 1 / (1 + np.exp(-z)) def cost(theta, X, y): # To define the cost function, we need to pay attention to the dimensions of X, theta and y, which do not affect the calculation return np.mean(-y * np.log(sigmoid(X @ theta)) - (1 - y) * np.log(1 - sigmoid(X @ theta))) # It is worth mentioning that in numpy, +, -, *, / are only the operations of corresponding elements and do not follow the matrix calculation rules # Matrix calculation in numpy requires the use of @ or. dot() functions # Similarly, ndarray type data can be converted into a matrix through np.matrix() and calculated with * def regularized_cost(theta, X, y, l=1): # The regularization cost function l is the regularization coefficient theta_j1_to_n = theta[1:] # from θ_ 1 to θ_ n. Usually, we are wrong θ_ 0 for regularization regularized_term = (l / (2 * len(X))) * np.power(theta_j1_to_n, 2).sum() # The regular term of the cost function is a numerical value return cost(theta, X, y) + regularized_term def gradient(theta, X, y): # Partial derivative return (1 / len(X)) * X.T @ (sigmoid(X @ theta) - y) def regularized_gradient(theta, X, y, l=1): # The regularized partial derivative l is the regularization coefficient theta_j1_to_n = theta[1:] # # from θ_ 1 to θ_ n regularized_theta = (l / len(X)) * theta_j1_to_n # The regular term of partial derivative of cost function is an array # Because there is no regularization θ_ 0, so splice an array 0 to ensure the array dimension regularized_term = np.concatenate([np.array([0]), regularized_theta]) return gradient(theta, X, y) + regularized_term def logistic_regression(X, y, l=1): theta = np.zeros(X.shape[1]) # definition θ To be the same as the feature data dimension # Using the optimization function in the optimizer, the TNC "truncated Newton method" is adopted res = opt.minimize(fun=regularized_cost, x0=theta, args=(X, y, l), method='TNC', jac=regularized_gradient) final_theta = res.x # Get the final θ value return final_theta def predict(x, theta): # Prediction function prob = sigmoid(x @ theta) return (prob >= 0.5).astype(int) # Get feature X, label y X, raw_y = load_data('ex3data1.mat') y = label_vectorization(raw_y) # Label Vectorization # logistic regression k_theta = np.array([logistic_regression(X, y[k]) for k in range(10)]) # y[k].shape = (5000,) can be regarded as row vector and column vector # k_theta.shape = (10,401) # Prediction and verification prob_matrix = sigmoid(X @ k_theta.T) # h_ θ (x) , because it is composed of multiple two classifiers, it should be a matrix # (5000401) [(10401). T] = = = > (5000,10), which means 5000 samples, and each sample belongs to the possibility of 0-9. The value with the greatest possibility is the predicted value. y_pred = np.argmax(prob_matrix, axis=1) # Returns the index of the maximum value along the axis. axis=1 represents the row. The index is 0-9, which better corresponds to the classification y_ture = raw_y.copy() # Tags 1-9 and 10 not vectorized y_ture[y_ture == 10] = 0 # Replace 10 in the label with 0 print(classification_report(y_ture, y_pred)) # assessment report

## 6. Operation results

precision recall f1-score support 0 0.97 0.99 0.98 500 1 0.95 0.99 0.97 500 2 0.95 0.92 0.93 500 3 0.95 0.91 0.93 500 4 0.95 0.95 0.95 500 5 0.92 0.92 0.92 500 6 0.97 0.98 0.97 500 7 0.95 0.95 0.95 500 8 0.93 0.92 0.92 500 9 0.92 0.92 0.92 500 accuracy 0.94 5000 macro avg 0.94 0.94 0.94 5000 weighted avg 0.94 0.94 0.94 5000

The recognition accuracy of 0 and 6 is very high, reaching 98% and 97%, while the recognition accuracy of 5 and 9 is low, only 92%. The overall recognition rate reached 94%.