# 1. Linear relationship model

A function that predicts through a linear combination of attributes:

w is the weight, b is the offset term, which can be understood as:

# 2. Linear regression (iteration, self-learning)

Definition: linear regression is a regression analysis modeled between one or more independent variables and dependent variables. It can be a linear combination between one or more independent variables (a kind of linear regression)

Univariate linear regression: only one variable (only one eigenvalue) is involved

Multiple linear regression: involving two or more variables (multiple eigenvalues)

Formula:

Where W and X are matrices:

# 3. Matrix operation (linear regression operation)

# 4. Loss function strategy (error size)

● Is the true value of the ith training sample

● It is the combined prediction function of the eigenvalues of the ith training sample

Definition of total loss (least square method):

The purpose is to reduce this loss

How to find w in the model to minimize the loss? (the purpose is to find the W value corresponding to the minimum loss)

(optimization method) solution 1: normal equation of least square method:

Solution:

X is the eigenvalue matrix and y is the target value matrix

Disadvantages: when the features are too complex, the solution speed is too slow

(optimization method) solution 2: gradient descent of least square method (important)

Take W0 and W1 in a single variable as an example:

α The learning rate needs to be specified manually,Indicates the direction

Understanding: follow the descending direction of this function to find the lowest point of the valley, and then update the W value

Usage: facing the task of huge training data scale

# 5.sklearn linear regression normal equation, gradient descent API

● sklearn.linear_model.LinearRegression -- normal equation

● ordinary least squares linear regression

● coef_: regression coefficient

● sklean.linear_ Model.sgdrepression -- gradient descent

● minimize the linear model by using SGD

● coef_: regression coefficient

# 6. Linear regression example - Boston house price forecast

House price characteristics:

Boston house price data set analysis process:

1. Boston area house price data acquisition

2. Boston area house price data segmentation

3. Standardized processing of training and test data

4. The simplest linear regression model linear regression and gradient descent estimation SGDRegression are used to predict house prices

demo:

from sklearn.datasets import load_boston from sklearn.linear_model import LinearRegression, SGDRegressor from sklearn.model_selection import train_test_split from sklearn.preprocessing import StandardScaler from sklearn.metrics import mean_squared_error def mylinear(): """ Linear regression prediction of house price :return: None """ # get data lb = load_boston() # The data is divided into training set and test set x_train, x_test, y_train, y_test = train_test_split(lb.data, lb.target, test_size=0.25) # print(y_train,y_test) # Carry out standardization processing, and standardize the eigenvalue and target values respectively # Eigenvalue standardization std_x = StandardScaler() x_train = std_x.fit_transform(x_train) x_test = std_x.transform(x_test) # Target value standardization, y_ train,y_ The test format is one-dimensional, and the standard scaler requires two-dimensional, so reshape the shape with reshape std_y = StandardScaler() y_train = std_y.fit_transform(y_train.reshape(-1,1)) y_test = std_y.transform(y_test.reshape(-1,1)) #Algorithm model #Solution of normal equation and prediction results lr = LinearRegression() lr.fit(x_train,y_train) print(lr.coef_) #Forecast test set house price y_lr_predict = std_y.inverse_transform(lr.predict(x_test)) print("The predicted price of each house in the normal equation test set:",y_lr_predict) print("Mean square error of normal equation:",mean_squared_error(std_y.inverse_transform(y_test),y_lr_predict)) #gradient descent sgd = SGDRegressor() sgd.fit(x_train, y_train) print(sgd.coef_) # Forecast test set house price y_sgd_predict = std_y.inverse_transform(sgd.predict(x_test)) print("Forecast price of each house in the test set:", y_sgd_predict) print("Mean square error of gradient descent:", mean_squared_error(std_y.inverse_transform(y_test), y_sgd_predict)) return None if __name__ == "__main__": mylinear()

Operation results:

Regression performance evaluation:

(mean square error (MSE) evaluation mechanism:

● mean_squared_error(y_true,y_pred)

● mean square error regression loss

● y_true: true value

● y_pred: predicted value

● return: floating point number result

Note: the real value and the predicted value are the values before standardization

# 7. Over fitting and under fitting

Over fitting: a hypothesis can get better fitting than other hypotheses on the training data, but it can not fit the data well on the data set outside the training data. At this time, it is considered that this hypothesis has over fitting phenomenon. (the model is too complex)

Under fitting: a hypothesis can not get better fitting on the training data, but it can not fit the data well on the data set other than the training data. At this time, it is considered that this hypothesis has the phenomenon of under fitting. (the model is too simple)

Causes and solutions of under fitting:

● cause: too few characteristics of data are learned

● Solution: increase the number of features of the data

Over fitting causes and Solutions

● Reason: there are too many original features and some noisy features. The model is too complex because the model tries to take into account all test data points

● Solution: select features and eliminate features with high relevance (it is difficult to do); Cross validation (make all data trained); Regularization

L2 regularization:

Function: it can make each element of W very small and close to 0

Advantages: the smaller the parameter, the simpler the model, and the simpler the model, the less likely it is to produce over fitting.

# 8. Ridge regression - linear regression with L2 regularization

● sklean.linear_model.Ridge(alpha=1.0)

● linear least squares with I2 regularization

● alpha: regularization strength λ 0~1 1~10

● coef_: regression coefficient

Note: it will not be equal to 0, but will be infinitely close to 0

demo:

from sklearn.datasets import load_boston from sklearn.linear_model import LinearRegression, SGDRegressor,Ridge from sklearn.model_selection import train_test_split from sklearn.preprocessing import StandardScaler from sklearn.metrics import mean_squared_error def ridge(): """ Linear regression prediction of house price :return: None """ # get data lb = load_boston() # The data is divided into training set and test set x_train, x_test, y_train, y_test = train_test_split(lb.data, lb.target, test_size=0.25) # print(y_train,y_test) # Carry out standardization processing, and standardize the eigenvalue and target values respectively # Eigenvalue standardization std_x = StandardScaler() x_train = std_x.fit_transform(x_train) x_test = std_x.transform(x_test) # Target value standardization, y_ train,y_ The test format is one-dimensional, and the standard scaler requires two-dimensional, so reshape the shape with reshape std_y = StandardScaler() y_train = std_y.fit_transform(y_train.reshape(-1,1)) y_test = std_y.transform(y_test.reshape(-1,1)) #Algorithm model #Ridge regression rd = Ridge(alpha=1.0) rd.fit(x_train, y_train) print(rd.coef_) # Forecast test set house price y_rd_predict = std_y.inverse_transform(rd.predict(x_test)) print("Forecast price of each house in the test set:", y_rd_predict) print("Mean square error of ridge regression:", mean_squared_error(std_y.inverse_transform(y_test), y_rd_predict)) return None if __name__ == "__main__": ridge()

Operation results:

Comparison between linear regression and ridge regression:

● ridge regression: the regression coefficient obtained by regression is more practical and reliable. In addition, it can make the fluctuation range of estimation parameters smaller and more stable. It has great practical value in the research of too many morbid data.

# 9. Classification algorithm - logistic regression (Solving binary classification problem)

Logistic regression formula:

Activation function Sigmoid: regression to classification

# 10. Logistic regression loss function and optimization

It is the same as the principle of linear regression, but because it is a classification problem, the loss function is different and can only be solved by gradient descent

Log likelihood loss function:

Complete loss function:

When y = 1 When, the classification result is 1 (positive): (horizontal axis classification result, vertical axis loss function)

When y = 0 When, the classification result is 0 (negative): (horizontal axis classification result, vertical axis loss function)

The log likelihood loss may fall into the local optimal solution (the mean square error has only one value and there is no local optimal problem):

# 11. Logistic regression API

● sklearn.lnear_model.LogisticRegression(penal='l2',C=1.0)

● Logistic regression classifier

● coef_: regression coefficient

# 12. Logistic regression cases

Data address: https://archive.ics.uci.edu/ml/machine-learning-databases/

demo:

#Model API from sklearn.linear_model import LogisticRegression #Recall API from sklearn.metrics import classification_report #Split dataset API from sklearn.model_selection import train_test_split #Standardized API from sklearn.preprocessing import StandardScaler import pandas as pd import numpy as np def logistic(): """ Logistic regression classification for tumor prediction :return: None """ # Construct column label name column = ['Sample code number', 'Clump Thickness', 'Uniformity of Cell Size', 'Uniformity of Cell Shape', 'Marginal Adhesion', 'Single Epithelial Cell Size', 'Bare Nuclei', 'Bland Chromatin', 'Normal Nucleoli', 'Mitoses', 'Class'] # Read data data = pd.read_csv("https://archive.ics.uci.edu/ml/machine-learning-databases/" "breast-cancer-wisconsin/breast-cancer-wisconsin.data", names=column) # print(data) #Handling of missing values data = data.replace(to_replace="?",value=np.nan) #Delete missing values data = data.dropna() #Dataset segmentation x_train, x_test, y_train, y_test = train_test_split(data[column[1:10]], data[column[10]], test_size=0.25) #Standardized treatment std = StandardScaler() x_train = std.fit_transform(x_train) x_test = std.transform(x_test) #Logistic regression prediction lg = LogisticRegression(C=1.0) lg.fit(x_train,y_train) print(lg.coef_) print("Prediction accuracy:",lg.score(x_test,y_test)) #recall y_predict = lg.predict(x_test) print("Recall rate:",classification_report(y_test,y_predict,labels=[2,4],target_names=["Benign","malignant"])) return None if __name__ == "__main__": logistic()

Operation results:

Summary (II Classification questions):

Application: advertising click through rate prediction, whether it is sick, financial fraud, whether it is a false account

Advantages: it is suitable for scenes where a classification probability needs to be obtained. It is simple and fast

Disadvantages: it is difficult to deal with multi classification problems

# 13. Generation model and discrimination model

The criterion of the two is whether there is a priori probability

# 14. Unsupervised learning K-means clustering algorithm

Algorithm flow:

Steps:

1. K points in the feature space are randomly set as the initial clustering center

2. For each other point, calculate the distance to K centers. For unknown points, select the nearest cluster center as the marker category

3. Then, after facing the marked cluster center, recalculate the new center point (average value) of each cluster

4. If the calculated new center point is the same as the original center point, it shall be terminated, otherwise the second step of the process shall be repeated

K-meansAPI:

● sklean.cluster.KMeans(n_clusters=8,init='k-means++')

● k-means clustering

● n_cluster: number of cluster centers started

● init: initialization method, default to 'k-means + +'

● labels_: The default tag type can be compared with the real value (not the value)

Case demo:

# coding:utf-8 import pandas as pd from sklearn.decomposition import PCA from sklearn.cluster import KMeans import matplotlib.pyplot as plt from sklearn.metrics import silhouette_score # Read data from four tables prior = pd.read_csv('./data/order_products__prior.csv') products = pd.read_csv('./data/products.csv') orders = pd.read_csv('./data/orders.csv') aisles = pd.read_csv('./data/aisles.csv') # Merge four tables into one table (user - item category) _mg = pd.merge(prior, products, on=['product_id', 'product_id']) _mg = pd.merge(_mg, orders, on=['order_id', 'order_id']) mt = pd.merge(_mg, aisles, on=['aisle_id', 'aisle_id']) print(mt.head(10)) # Crosstab (special grouping tool) cross = pd.crosstab(mt['user_id'], mt['aisle']) # Principal component analysis pca = PCA(n_components=0.9) data = pca.fit_transform(cross) print(data.shape) # Reduce the number of samples x = data[:500] # Suppose that users are divided into four categories km = KMeans(n_clusters=4) km.fit(x) predict = km.predict(x) print(predict) # Displays the results of the clustering plt.figure(figsize=(10, 10)) # Create a list of four colors colored = ['orange', 'green', 'blue', 'purple'] color = [colored[i] for i in predict] plt.scatter(x[:, 1], x[:, 20], color=color) plt.xlabel("1") plt.ylabel("20") plt.show() print(silhouette_score(x, predict))

Output results:

order_id product_id ... days_since_prior_order aisle 0 2 33120 ... 8.0 eggs 1 26 33120 ... 7.0 eggs 2 120 33120 ... 10.0 eggs 3 327 33120 ... 8.0 eggs 4 390 33120 ... 9.0 eggs 5 537 33120 ... 3.0 eggs 6 582 33120 ... 10.0 eggs 7 608 33120 ... 12.0 eggs 8 623 33120 ... 3.0 eggs 9 689 33120 ... 3.0 eggs [10 rows x 14 columns] (206209, 27) [1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 3 1 1 1 1 1 1 1 1 1 0 1 1 1 1 1 1 1 1 1 1 1 1 0 1 1 1 0 1 1 1 1 1 1 1 1 0 1 1 1 1 1 1 1 0 1 1 1 0 1 1 1 1 1 1 1 1 1 1 0 1 1 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 3 1 0 1 1 1 0 1 1 1 1 1 3 0 1 1 1 1 1 1 1 0 1 1 1 1 1 1 1 1 1 1 1 0 1 1 1 1 1 1 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 1 1 1 1 0 1 0 1 1 1 1 1 2 1 1 1 0 1 1 1 1 1 1 1 1 3 1 1 1 0 1 1 1 1 0 0 0 1 0 1 1 1 1 1 1 0 1 1 1 1 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 3 1 1 1 1 1 1 1 1 1 1 1 1 1 0 1 3 0 1 1 1 1 1 1 1 2 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 1 1 1 2 1 1 1 1 1 1 1 0 1 3 1 1 1 3 1 1 1 1 1 1 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 2 1 1 1 1 1 1 1 0 1 1 1 1 1 1 1 1 0 1 1 1 1 1 1 1 0 1 1 1 1 0 1 1 1 1 1 3 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 1 1 1 1 1 1 1 1 1 1 1 2 1 1 1 0 1 1 1 0 1 0 1 1 1 1 1 1 1 3 1 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 1 1 1 1 0 0 1 1 1 1 1 1 1 0 3 1 1 1 1]

Cluster scatter diagram:

# 15.K-means clustering effect evaluation criteria

K-means--sklearn clustering effect evaluation API:

● sklearn.metrics.silhouette_score(X,labels)

● calculate the average contour coefficient of all samples

● X: characteristic value

● labels: target value (predicted value) marked by clustering

K-means summary:

Feature analysis: iterative algorithm is adopted, which is intuitive, easy to understand and very practical.

Disadvantages: it is easy to converge to the local optimal solution (multiple clustering)

Note: clustering is usually done before classification