# House price forecast

Original question: https://www.kaggle.com/c/house-prices-advanced-regression-techniques/data?select=train.csv

reference resources: https://blog.csdn.net/qq_29153321/article/details/103967670

The house price prediction experiment analyzes 1460 house price data and the attributes of 80 houses, establishes a regression model, and then predicts the prices of 1459 new house data.

## Data preprocessing

### Import data

import pandas as pd train_path = 'house-prices-advanced-regression-techniques/train.csv' train = pd.read_csv(train_path) test_path = 'house-prices-advanced-regression-techniques/test.csv' test = pd.read_csv(test_path) print(train.head()) print(train.info())

Out:

### Weed out useless columns

It is easy to get from the characteristics and purpose: 'Id' belongs to the sequence and does not help predict house prices. Delete it

train.drop('Id', axis = 1, inplace = True) test.drop('Id', axis = 1, inplace = True) print(train.head())

Out:

(pandas.DataFrame).drop()

Parameter Description:

First parameter: Specifies the index to be deleted

Axis: processed by row (axis = 0) or column (axis = 1). The default value is 0

inplace: whether to directly operate the original data. The default value is False

### Detect outliers

According to the problem prediction analysis, the aboveground living area will be a major factor affecting the house price, so the visual display will check whether there are abnormal values

import matplotlib.pyplot as plt train.plot(kind = 'scatter', x = 'GrLivArea', y = 'SalePrice', alpha = 0.2, figsize = (10, 8)) plt.ylabel('SalePrice', fontsize = 13) plt.xlabel('GrLivArea', fontsize = 13) plt.grid() plt.show()

Out:

It can be seen that there are two outliers in the lower right corner (large area but low price). Delete the outliers directly

train = train.drop(train[(train['GrLivArea']>4000) & (train['SalePrice']<300000)].index)

This statement is divided into several layers:

train[(train ['GrLivArea '] > 4000) & (train ['SalePrice'] < 300000)]: data division, filter out qualified data, format: df [(condition)]

. index: returns the index value

. drop(): deletes the specified data according to the index

### Modified label distribution

For linear regression models, it is an important step to check whether the label distribution conforms to the normal distribution and correct it

Because when the variable follows the normal distribution, the probability of predicting the variable and finding it within a certain range will become very simple

import seaborn as sns sns.distplot(train['SalePrice']) plt.ylabel('Frequency') plt.title('SalePrice distribution') plt.show()

Out:

It can be seen that the price distribution is inclined to the left. If you want to correct it, you need to expand the low price area and compress the high price area. log(1+x) is used to convert and correct the SalePrice. Of course, in the subsequent application of the model, you should pay attention to the label transformation

import numpy as np train['SalePrice'] = np.log1p(train['SalePrice']) sns.distplot(train['SalePrice']) plt.ylabel('Frequency') plt.title('SalePrice distribution') plt.show()

Out:

### Merge datasets

The training set and test set are combined for data preprocessing without repeated work

len_train = len(train) len_test = len(test) y_train = train['SalePrice'] all_data = pd.concat((train, test), axis = 0).reset_index(drop = True) all_data.drop(['SalePrice'], axis = 1, inplace = True)

df.concat((A, B), axis = 0).reset_index(drop = True) means to merge up and down matrices A and B (axis = 0), and delete the original row indexes of the two matrices

### Missing value processing

First, make statistics on which fields have missing data and what are the missing rates

def get_missing_data_summary(my_dataframe): all_data_na = (my_dataframe.isnull().sum() / len(my_dataframe)) * 100 all_data_na = all_data_na.drop(all_data_na[all_data_na == 0].index).sort_values(ascending=False) missing_data = pd.DataFrame({'Missing Ratio (%)' :all_data_na}) return missing_data missing_data = pd.DataFrame({'Missing Ratio (%)' :all_data_na}) print(missing_data)

Out:

It can be seen that there are many features with missing data, which need to be processed according to the specific business significance of the features

Class I: the missing value also belongs to one of the characteristic values (for example, PoolQC is the information related to the swimming pool, and the missing value indicates that there is no swimming pool)

C1 = ['PoolQC', 'MiscFeature', 'Alley', 'Fence', 'FireplaceQu', 'GarageType', 'GarageFinish', 'GarageQual', 'GarageCond','BsmtQual', 'BsmtCond', 'BsmtExposure', 'BsmtFinType1', 'BsmtFinType2', 'MasVnrType'] for col in C1: all_data[col].fillna('None', inplace = True)

Type II: numerical type characteristics, which should be assigned as 0 according to the analysis (for example, if there is no garage, the corresponding relevant information, such as garage area, can be directly assigned as 0)

C2 = ['GarageYrBlt', 'GarageArea', 'GarageCars', 'BsmtFinSF1', 'BsmtFinSF2', 'BsmtUnfSF', 'TotalBsmtSF', 'BsmtFullBath','BsmtHalfBath', 'MasVnrArea'] for col in C2: all_data[col].fillna(0, inplace = True)

The third category: most of the deficiencies that do not belong to the first two categories are illogical, and their modes are used to fill them

C3 = ['MSZoning', "Functional", 'Electrical', 'KitchenQual', 'Exterior1st', 'Exterior2nd', 'SaleType', 'MSSubClass'] for col in C3: all_data[col].fillna(all_data[col].mode()[0], inplace = True)

Category 4: a special feature: 'Utilities'. All non missing values are the same value, which is meaningless. They are simply deleted

all_data.drop('Utilities', axis = 1, inplace = True)

Category 5: a special feature: 'LotFrontage', which is "linear Street feet connected to the property", while another feature, 'Neighborhood', indicates "the geographical area where the house is located". Obviously, the distance between the house and the property in the same area is almost the same, so the missing value can be replaced by the median of the 'LotFrontage' value in the same area

all_data["LotFrontage"] = all_data.groupby("Neighborhood")["LotFrontage"].transform(lambda x: x.fillna(x.median()))

df.groupby('A ') [' B '). transform(lambda x: c): divide the data into several categories according to the value of column A (several values in column A are divided into several categories), and then replace the value of B in each category with c

This statement means: classify the samples according to the feature "Neighborhood", name the "LotFrontage" value of each class as X, and replace the x value of this class with the median of X of this class

Review the characteristics of missing data again

missing_data = pd.DataFrame({'Missing Ratio (%)' :all_data_na}) print(missing_data)

Out:

### Handling special attributes

all_data['MSSubClass'] = all_data['MSSubClass'].apply(str)

Although MSSubClass is a numeric type, it actually uses numeric values to distinguish house types. The values themselves are not comparable in size, so they are converted to character types for subsequent encoding processing

### Combination feature

Generate a new field: total house area

According to business knowledge, this feature may be useful for house price prediction

all_data['TotalSF'] = all_data['TotalBsmtSF'] + all_data['1stFlrSF'] + all_data['2ndFlrSF']

### Processing category data as dummy variables

all_data = pd.get_dummies(all_data, drop_first = True) print(all_data.info())

Out:

The characteristics of non numerical classes in the matrix are treated as dummy variables and inserted back into the original matrix, and the matrix will be expanded accordingly

### Split dataset

X_train = all_data[:len_train].values X_test = all_data[len_train:].values

Re split the data set into training set and test set

### Feature scaling

from sklearn.preprocessing import StandardScaler sc_X = StandardScaler() X_train = sc_X.fit_transform(X_train) X_test = sc_X.transform(X_test) y_train = np.array(y_train).reshape(-1, 1)

Feature scaling is often used as the last step of data preprocessing. It is suggested that feature scaling is needed before building a model

## Select model

### Establish regression model

#Lasso from sklearn.linear_model import Lasso lasso = Lasso(alpha = 0.0005, random_state = 1) #Elastic Net Regression from sklearn.linear_model import ElasticNet enet = ElasticNet(alpha = 0.0005, l1_ratio = 0.9, random_state = 3) #Kernel Ridge Regression from sklearn.kernel_ridge import KernelRidge krr = KernelRidge(alpha = 0.6, kernel = 'polynomial', degree = 2, coef0 = 2.5) #Gradient Boosting Regression from sklearn.ensemble import GradientBoostingRegressor gboost = GradientBoostingRegressor(n_estimators = 3000, learning_rate = 0.05, max_depth = 4, max_features = 'sqrt', min_samples_leaf = 15, min_samples_split = 10, loss = 'huber', random_state = 5) #random forest from sklearn.ensemble import RandomForestRegressor rf = RandomForestRegressor(n_estimators = 500, random_state = 0) #XGBoost import xgboost as xgb xgb = xgb.XGBRegressor(colsample_bytree = 0.4, gamma = 0.05, learning_rate = 0.05, max_depth = 3, min_child_weight = 2, n_estimators = 2200, reg_alpha = 0.5, reg_lambda = 0.9, subsample = 0.5,silent = 1, random_state = 7, nthread = -1) #LightGBM import lightgbm as lgb lgb = lgb.LGBMRegressor(objective = 'regression', num_leaves = 5, learning_rate = 0.05, n_estimators = 720, max_bin = 55, bagging_fraction = 0.8, bagging_freq = 5, feature_fraction = 0.25, feature_fraction_seed = 9, bagging_seed = 9, min_child_samples = 6, min_child_weight = 11)

### Model performance evaluation and selection

k-fold cross validation is generally used in the industry to evaluate the model performance. Here, k=10. There are two evaluation indicators, one is r^2 (r squared), which is used to judge the model performance; the other is mse (mean squared error), which is used to observe the variance. In order to speed up the execution speed, n_jobs = 2 means using two CPUs (two core CPUs) verbose = 1 indicates that information is continuously printed out during execution to help judge how much is executed

#K-fold from sklearn.model_selection import cross_val_score def evaluation_model(model): rmse = np.sqrt(-cross_val_score(estimator = model, X = X_train, y = y_train, scoring = 'neg_mean_squared_error', cv = 10, n_jobs = 2, verbose = 1)) r2_score = cross_val_score(estimator = model, X = X_train, y = y_train, scoring = 'r2', cv = 10, n_jobs = 2, verbose = 1) return (r2_score, rmse)

#running lasso_r2, lasso_rmse = evaluation_model(lasso) enet_r2, enet_rmse = evaluation_model(enet) krr_r2, krr_rmse = evaluation_model(krr) gboost_r2, gboost_rmse = evaluation_model(gboost) rf_r2, rf_rmse = evaluation_model(rf) xgb_r2, xgb_rmse = evaluation_model(xgb) lgb_r2, lgb_rmse = evaluation_model(lgb)

Out:

#print result function def print_result(r2_score, rmse, model_name): print('%s evaluation:r2 = %.4f(std = %.4f), rmse = %.4f(std = %.4f)' %(model_name, r2_score.mean(), r2_score.std(), rmse.mean(), rmse.std())) return 0

#print result print_result(lasso_r2, lasso_rmse, 'Lasso') print_result(enet_r2, enet_rmse, 'Elastic Net') print_result(krr_r2, krr_rmse, 'Kernel Redge') print_result(gboost_r2, gboost_rmse, 'Gradient Boost') print_result(rf_r2, rf_rmse, 'Random Forest') print_result(xgb_r2, xgb_rmse, 'XG Boost') print_result(lgb_r2, lgb_rmse, 'Lightgbm')

Out:

We found that Linghtgbm has the best performance (R2 > 0.9, and mse and std are small), so we chose Lightgbm model for further optimization

### Model performance optimization

Debug the super parameters of the model and optimize the data of the project to obtain better regression data.

LightGBM parameter adjustment method:

- Select a higher learning rate to accelerate the convergence speed and improve the parameter adjustment efficiency
- Adjusting the basic parameters of decision tree
- Regularization parameter tuning
- Reduce learning rate and improve accuracy

#### Get the number of iterations under the current learning rate

Using the cv function of LightGBM, the initial values of model parameters are set simply according to the problem

LightGBM parameters: https://lightgbm.readthedocs.io/en/latest/pythonapi/lightgbm.LGBMRegressor.html#lightgbm.LGBMRegressor

How to use lightgbm.cv function: https://blog.csdn.net/weixin_44414593/article/details/108727307

import lightgbm as lgb params = { 'boosting_type':'gbdt', 'objective':'regression', 'n_jobs':-1 'learning_rate':0.1, 'num_leaves':15, 'max_depth':5, 'subsample':0.8, 'colsample_bytree':0.8 } data_train = lgb.Dataset(X_train, y_train, silent = True) cv_results = lgb.cv(params, data_train, nfold = 5, metrics = 'rmse', num_boost_round = 1000, early_stopping_rounds = 50, verbose_eval = 50, stratified = False, seed = 0) print('best n_estimators:', len(cv_results['rmse-mean'])) print('best cv score:', cv_results['rmse-mean'][-1])

Out:

It can be seen that when the learning rate is 0.1, the optimal number of iterations is 194, so we next use learning_rate = 0.1, num_boost_round = 194

#### Basic commissioning parameters

Use GridSearchCV() in sklearn to perform grid search, coarse adjustment first and then fine adjustment

GridSearchCV() reference: https://blog.csdn.net/weixin_41988628/article/details/83098130?ops_request_misc=%257B%2522request%255Fid%2522%253A%2522163444549716780269819093%2522%252C%2522scm%2522%253A%252220140713.130102334...%2522%257D&request_id=163444549716780269819093&biz_id=0&utm_medium=distribute.pc_search_result.none-task-blog-2alltop_positive~default-1-83098130.first_rank_v2_pc_rank_v29&utm_term=GridSearchCV&spm=1018.2226.3001.4187

Simply set the initial value first

from sklearn.model_selection import GridSearchCV model_lgb = lgb.LGBMRegressor(object = 'regression', num_leaves = 15, learning_rate = 0.1, n_estimators = 194, max_depth = 5, metric = 'rmse', bagging_fraction = 0.8, feature_fraction = 0.8)

Important parameters to improve accuracy

max_depth: the depth of the tree. If the depth is too large, it may be too fitted

Num_leaves: the complexity of the tree. It is required that num_leaves < 2 ^ (max_depth), otherwise it will be over fitted

params_test1 = {'max_depth':range(3, 10, 2), 'num_leaves':range(5, 125, 20)} gsearch1 = GridSearchCV(estimator = model_lgb, param_grid = params_test1, scoring = 'neg_mean_squared_error', cv = 5, verbose = 1, n_jobs = -1) gsearch1.fit(X_train, y_train) gsearch1.best_params_, gsearch1.best_score_

Out:

Here, the lower the mean square error (rmse) used in the metric strategy, the better. sklearn provides neg_mean_squared_error and returns the rmse of metric. Similarly, the closer it is to 0, the better

The range is reduced several times according to the results, and finally the best super parameters are obtained:

params_test1 = {'max_depth':range(2, 5, 1), 'num_leaves':range(5, 10, 1)} gsearch1 = GridSearchCV(estimator = model_lgb, param_grid = params_test1, scoring = 'neg_mean_squared_error', cv = 5, verbose = 1, n_jobs = -1) gsearch1.fit(X_train, y_train) gsearch1.best_params_, gsearch1.best_score_

Out:

Reduce the important parameters of over fitting

min_child_samples: also known as min_data_in_leaf, the minimum number of data on a leaf. It can be used to process over fitting

min_child_weight: also known as min_sum_hessian_in_leaf, the minimum Hessian sum on a leaf can be used to deal with over fitting

params_test2 = {'min_child_samples':range(0, 40, 10), 'min_child_weight':range(0, 20, 2)} model_lgb = lgb.LGBMRegressor(objective = 'regression', num_leaves = 6, learning_rate = 0.1, n_estimators = 194, max_depth = 3, metric = 'rmse', bagging_fraction = 0.8, feature_fraction = 0.8) gsearch2 = GridSearchCV(estimator = model_lgb, param_grid = params_test2, scoring = 'neg_mean_squared_error', cv = 5, verbose = 1, n_jobs = -1) gsearch2.fit(X_train, y_train) gsearch2.best_params_, gsearch2.best_score_

Out:

Further narrow the scope:

params_test2 = {'min_child_samples':range(0, 10, 1), 'min_child_weight':range(5, 10, 1)} model_lgb = lgb.LGBMRegressor(objective = 'regression', num_leaves = 6, learning_rate = 0.1, n_estimators = 194, max_depth = 3, metric = 'rmse', bagging_fraction = 0.8, feature_fraction = 0.8) gsearch2 = GridSearchCV(estimator = model_lgb, param_grid = params_test2, scoring = 'neg_mean_squared_error', cv = 5, verbose = 1, n_jobs = -1) gsearch2.fit(X_train, y_train) gsearch2.best_params_, gsearch2.best_score_

Out:

Reduce the other two parameters of over fitting

feature_fraction: also known as colsample_bytree, it is used for sub sampling of features to prevent over fitting and improve training speed

bagging_fraction: also known as subsample, it is the sample sampling proportion of the training instance, which can make bagging run faster and reduce fitting.

bagging_freq: also called subsample_freq, 0 by default, indicates the frequency of bagging. 0 means that bagging is not used, and K means that bagging is performed every k rounds of iteration.

params_test3 = { 'feature_fraction':[0.1, 0.3, 0.5, 0.7, 0.9, 1], 'bagging_fraction':[0.1, 0.3, 0.5, 0.7, 0.9, 1] 'bagging_freq':range(0, 10) } model_lgb = lgb.LGBMRegressor(objective = 'regression', num_leaves = 6, learning_rate = 0.1, n_estimators = 194, max_depth = 3, metric = 'rmse', bagging_fraction = 0.8, feature_fraction = 0.8, min_child_samples = 6, min_child_weight = 7) gsearch3 = GridSearchCV(estimator = model_lgb, param_grid = params_test3, scoring = 'neg_mean_squared_error', cv = 5, verbose = 1, n_jobs = -1) gsearch3.fit(X_train, y_train) gsearch3.best_params_, gsearch3.best_score_

Out:

params_test3 = { 'feature_fraction':[0.25, 0.3, 0.35, 0.4, 0.45], 'bagging_fraction':[0.86, 0.88, 0.9, 0.92, 0.94]， 'bagging_freq':[1, 2, 3, 4] } model_lgb = lgb.LGBMRegressor(objective = 'regression', num_leaves = 6, learning_rate = 0.1, n_estimators = 194, max_depth = 3, metric = 'rmse', bagging_fraction = 0.8, feature_fraction = 0.8, min_child_samples = 6, min_child_weight = 7) gsearch3 = GridSearchCV(estimator = model_lgb, param_grid = params_test3, scoring = 'neg_mean_squared_error', cv = 5, verbose = 1, n_jobs = -1) gsearch3.fit(X_train, y_train) gsearch3.best_params_, gsearch3.best_score_

Out:

Regularization parameters

lambda_l1: L1 regularization coefficient

lambda_l2: L2 regularization coefficient

params_test4 = { 'lambda_l1': [0, 0.001, 0.01, 0.03, 0.08, 0.3, 0.5], 'lambda_l2': [0, 0.001, 0.01, 0.03, 0.08, 0.3, 0.5] } model_lgb = lgb.LGBMRegressor(objective = 'regression', num_leaves = 6, learning_rate = 0.1, n_estimators = 194, max_depth = 3, metric = 'rmse', bagging_fraction = 0.01, feature_fraction = 0.41, min_child_samples = 6, min_child_weight = 7) gsearch4 = GridSearchCV(estimator = model_lgb, param_grid = params_test4, scoring = 'neg_mean_squared_error', cv = 5, verbose = 1, n_jobs = -1) gsearch4.fit(X_train, y_train) gsearch4.best_params_, gsearch4.best_score_

Out:

#### Reduce learning rate

Higher learning rate was used before because it can make convergence faster, but accuracy is bound to be sacrificed. A lower learning rate and more decision trees are used here n_estimators to train data, trying to get a better model

params = { 'boosting_type':'gbdt', 'objective':'regression', 'n_jobs':-1, 'learning_rate':0.005, 'num_leaves':6, 'max_depth':3, 'bagging_fraction':0.92, 'bagging_freq':2, 'colsample_bytree':0.4, 'min_child_samples':6, 'min_child_weight':7, 'lambda_l1':0, 'lambda_l2':0.01 } data_train = lgb.Dataset(X_train, y_train, silent = True) cv_results = lgb.cv(params, data_train, nfold = 5, metrics = 'rmse', num_boost_round = 10000, early_stopping_rounds = 50, verbose_eval = 100, stratified = False, seed = 0, show_stdv = True) print('best n_estimators:', len(cv_results['rmse-mean'])) print('best cv score:', cv_results['rmse-mean'][-1])

Out:

The optimal number of iterations is 3970

## Establish the best model for prediction

According to the optimization, we get the best model

import lightgbm as lgb regressor = lgb.LGBMRegressor( boosting_type = 'gbdt', objective = 'regression', n_jobs = -1, n_estimators = 3970, learning_rate = 0.005, num_leaves = 6, max_depth = 3, bagging_fraction = 0.92, bagging_freq = 2, colsample_bytree = 0.4, min_child_samples = 6, min_child_weight = 7, lambda_l2 = 0.01 )

Training model

regressor.fit(X_train, y_train) y_pred = regressor.predict(X_test) y_pred = np.expm1(y_pred)#The label distribution is corrected during data preprocessing, which needs to be corrected test_set = pd.DataFrame(np.arange(0, X_test.shape[0]).reshape(-1, 1)) test_set['SalePrice'] = y_pred test_set.to_csv('test_set.csv')

effect

The effect can only be said reluctantly...

This article is my study notes, focusing on learning the whole regression project process. Leaders are welcome to put forward suggestions in the comment area.