preface

Linear regression (Linear Rrgresson) is one of the simplest machine learning algorithms. It is the first algorithm for most new machine learning, which is equivalent to the "Hello World" in machine learning. In order to let everyone better learn linear regression, the author will use numpy (handwritten) batch gradient descent method to realize the code of linear regression.

1, What is linear regression?

As for linear regression, there are many relevant materials on the Internet, such as Wikipedia, Baidu Encyclopedia, the "linear regression" item on the Encyclopedia of MBA think tank, Wu Enda's machine learning lectiure2~4 and other articles written by CSDN blogger. I believe that most people write better than the author, so the author will not repeat the principle part.

2, References

Lecture2~Lecture4 in Wu Enda's Machine Learning|Coursera courseware

3, Implementation steps

The code is as follows

1. Import and storage

First, the numpy library is introduced

import numpy as np

2. Build LR (linear regression) class and set super parameters

Here, due to the problem of calculation accuracy, the optimizer needs a lot of steps or cycles to iterate to complete 0 error, so the threshold is set. That is, if the error is less than the threshold, the iteration will be stopped (previous article) [ random thoughts ] prediction of time series by adaptive filtering method (mentioned)

class LR:
    def __init__(self, variable, label, alpha=0.0001, max_epoch=100):
        self.variable = np.array(variable, dtype=float)  # Characteristics
        self.label = np.array(label)  # Tag value

        self.num_sample = self.variable.shape[0]  # Number of samples
        self.num_variable = self.variable.shape[1]  # Feature number, which is used to determine the number of parameters (assumed function weight)

        self.alpha = alpha  # Learning rate
        self.threshold = 0.0001  # Mean square error change threshold

        self.theta = np.random.normal(size=(1, self.num_variable)).T.flatten()  # Initial weight

        self.max_epoch = max_epoch  # Maximum iterated algebra

3. Construct hypothesis function

Suppose the function expression is as follows: h θ = ∑ i = 0 n θ i x i h_\theta=\sum^{n}_{i=0}\theta_{i}x_{i} hθ​=i=0∑n​θi​xi​
In order to facilitate computer operation and express clearly and concisely, we will write it in the form of vector operation, that is h θ = θ T x \boldsymbol h_\theta = \boldsymbol \theta^T \boldsymbol x hθ​=θTx

 def Hypothesis_func(self, X, Theta):  # Hypothetical function

        return np.dot(X, Theta)

4. Loss function

The loss function is expressed as follows: J = 1 2 m ∑ i = 0 m ( h θ ( x i ) − y i ) 2 J = \frac{1}{2m}\sum^m_{i=0}(h_\theta(\boldsymbol x^i)-y^i)^2 J=2m1​i=0∑m​(hθ​(xi)−yi)2
among m m m is the number of samples

    def loss_func(self, Prediction, label):  # Loss function
        '''
        print('Loss function')
        print('Prediction.shape')
        print(Prediction.shape)
        print('label.shape')
        print(label.shape)
        print('Loss function')
        print('')
        '''
        return (1 / 2 * self.num_variable) * (np.sum((Prediction.flatten() - label)) ** 2)

5. Gradient

For each parameter in the loss function, i.e θ i \theta_i θ i) find the partial derivative and construct it as a vector, that is, the gradient

    def delta_loss_func(self, Prediction, label):  # Loss function derivative
        Prediction = Prediction.flatten()
        h = (Prediction - label).T
        return (1 / self.num_sample) * np.dot((Prediction - label).T, self.variable)

6. Batch gradient descent method

In the previous loss function, the number of loss function samples m m m take the number of all samples

    def BGD(self):
        epoch = 0  # Current steps
        del_mse = self.delta_loss_func(Prediction=self.Hypothesis_func(self.variable, self.theta),
                                       label=self.label)


        while del_mse.all() > self.threshold and (epoch < self.max_epoch):
           
            self.theta = self.theta - self.alpha * self.delta_loss_func(
                Prediction=self.Hypothesis_func(self.variable, self.theta), label=self.label).T
            epoch = epoch + 1
            # print('self.theta')
            # print(self.theta)
            mse = self.loss_func(self.Hypothesis_func(self.variable, self.theta), label=self.label)
            print('The first{}Second iteration,The mean square error is{:.7f}'.format(epoch, mse))
        return self.theta

7. Verification

7-1. Building data sets

We will randomly generate 100 samples and build labels y = 3 x 0 + 1 x 1 + 2 x 2 + 6 x 3 + 5 x 4 y=3x_0+1x_1+2x_2+6x_3+5x_4 y=3x0​+1x1​+2x2​+6x3​+5x4​

if __name__ == "__main__":
    '''Data set construction'''
    data_x = np.random.normal(size=(100, 5))
    weight_0 = np.array([3, 1, 2, 6, 5])

    data_y = np.dot(data_x, weight_0)

    train_x = data_x[0:80]
    train_y = data_y[0:80]

    test_x = data_x[81:100]
    test_y = data_y[81:100]

7-2. Prediction of data sets

'''Forecast data'''
    LR = LR(train_x, train_y, alpha=0.1, max_epoch=1000)
    theta = LR.BGD()
    print(theta)
    y_predict = np.dot(test_x, theta)
    MSE_test = LR.loss_func(Prediction=y_predict, label=test_y)
    print('The error on the test set is: ', MSE_test)

summary

Through this article, I believe you have learned how to use numpy library to build a linear regression model.

Thank you for reading my article. Finally, I wish you all a smooth study and a happy life in the future.