Machine learning training camp -- come and explore happiness

Keywords: Python Machine Learning Data Analysis

preface

This learning note is the learning content of Alibaba cloud Tianchi Longzhu plan machine learning training camp. The learning links are:
https://tianchi.aliyun.com/specials/promotion/aicampml

Competition link: Come and explore happiness together!

Friendly reminder: I just started learning machine learning. Some places may not be mature. I refer to some forum notes. I can write it all by the selfless sharing of all leaders. I hope it will help you!

1, Game problem understanding

1.1 experimental environment

In the experimental environment of Alibaba cloud Tianchi, first import the dataset into DSW.

1.2 background introduction

The data of the competition is the data in the survey results of the official 2015 China Comprehensive Social Survey (CGSS) questionnaire (resident questionnaire), which contains 139 dimensions, including individual variables (gender, age, region, occupation, health, marriage and political outlook, etc.) and family variables (parents, spouses, children, family capital, etc.), social attitude (fairness, credit, public service), etc.

The competition requires the use of the given 139 dimensional characteristics and more than 8000 groups of data to predict personal well-being (the predicted values are 1, 2, 3, 4 and 5, of which 1 represents the lowest well-being and 5 represents the highest well-being).

1.3 data information

happiness_index.xlsx: 142 rows and 5 columns. The specific fields of this competition question are introduced, including the questionnaire question corresponding to each variable and the meaning of variable value. In fact, it is the 139 dimensional feature that can be used.

happiness_submit.csv: 2969 lines, 2 columns, and the platform submits the result sample file.
happiness_survey_cgss2015.pdf: 2015 annual questionnaire of China Comprehensive Social Survey (CGSS) (resident questionnaire)
happiness_test_abbr.csv: 2969 rows, 41 columns, reduced version data.
happiness_test_complete.csv: 2969 rows, 139 columns, full version data.
happiness_train_abbr.csv: 8001 rows, 42 columns, reduced version data.
happiness_train_complete.csv: 8001 rows, 140 columns, full version data.

Here, I use the full version of the data.

1.4 evaluation index

The final evaluation index is the mean square error MSE,

The smaller the submitted mean square error, the better the representative result.

2, Exploratory data analysis (EDA) & Feature Engineering

2.1 why exploratory data analysis

Understanding data
– data type size (what configuration is needed, how expensive is the competition)
– whether the data is clean (obviously wrong data, such as height 5m...)
– what type of tag is it and whether format conversion is required?.. (DataFrame.info())
Prepare for data modeling
– whether the construction of offline verification set may cross? (observe the data distribution)
– are there any strange phenomena? Prepare for Feature Engineering: for example, periodic change of time series

2.2 what data should exploratory data analysis look at

1. Data set size and field type: how big the data is and what type each field is
2. Missing value: whether the missing value is serious and whether the missing value has special meaning
3. Redundancy between features: for example, if the height is expressed in cm and m, there is redundancy
4. Whether there is time information: potential crossing problems
5. Label distribution: whether the category distribution is unbalanced, etc
6. Distribution of training set and test set: there are many fields in the test set, but there are no features in the training set
7. Univariate / multivariate distribution: be familiar with the distribution of features and the relationship with labels

2.3 data preprocessing

Import package

import os
import time 
import pandas as pd
import numpy as np
import seaborn as sns
from sklearn.linear_model import LogisticRegression
from sklearn.svm import SVC, LinearSVC
from sklearn.ensemble import RandomForestClassifier
from sklearn.neighbors import KNeighborsClassifier
from sklearn.naive_bayes import GaussianNB
from sklearn.linear_model import Perceptron
from sklearn.linear_model import SGDClassifier
from sklearn.tree import DecisionTreeClassifier
from sklearn import metrics
from datetime import datetime
import matplotlib.pyplot as plt
from sklearn.metrics import roc_auc_score, roc_curve, mean_squared_error,mean_absolute_error, f1_score
import lightgbm as lgb
import xgboost as xgb
from sklearn.ensemble import RandomForestRegressor as rfr
from sklearn.ensemble import ExtraTreesRegressor as etr
from sklearn.linear_model import BayesianRidge as br
from sklearn.ensemble import GradientBoostingRegressor as gbr
from sklearn.linear_model import Ridge
from sklearn.linear_model import Lasso
from sklearn.linear_model import LinearRegression as lr
from sklearn.linear_model import ElasticNet as en
from sklearn.kernel_ridge import KernelRidge as kr
from sklearn.model_selection import  KFold, StratifiedKFold,GroupKFold, RepeatedKFold
from sklearn.model_selection import train_test_split
from sklearn.model_selection import GridSearchCV
from sklearn import preprocessing
import logging
import warnings

warnings.filterwarnings('ignore') #Eliminate warning

#Import data
train = pd.read_csv("happiness_train_complete.csv", parse_dates=['survey_time'],encoding='latin-1') 
test = pd.read_csv("happiness_test_complete.csv", parse_dates=['survey_time'],encoding='latin-1') #latin-1 is downward compatible with ASCII

#Observation data size
train.shape

test.shape

#Simple view data
train.head()

#Check whether the data is missing
train.info(verbose=True,null_counts=True)

output:

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 8000 entries, 0 to 7999
Data columns (total 140 columns):
 #   Column                Non-Null Count  Dtype         
---  ------                --------------  -----         
 0   id                    8000 non-null   int64         
 1   happiness             8000 non-null   int64         
 2   survey_type           8000 non-null   int64         
 3   province              8000 non-null   int64         
 4   city                  8000 non-null   int64         
 5   county                8000 non-null   int64         
 6   survey_time           8000 non-null   datetime64[ns]
 7   gender                8000 non-null   int64         
 8   birth                 8000 non-null   int64         
 9   nationality           8000 non-null   int64         
 10  religion              8000 non-null   int64         
 11  religion_freq         8000 non-null   int64         
 12  edu                   8000 non-null   int64         
 13  edu_other             3 non-null      object        
 14  edu_status            6880 non-null   float64       
 15  edu_yr                6028 non-null   float64       
 16  income                8000 non-null   int64         
 17  political             8000 non-null   int64         
 18  join_party            824 non-null    float64       
 19  floor_area            8000 non-null   float64       
 20  property_0            8000 non-null   int64         
 21  property_1            8000 non-null   int64         
 22  property_2            8000 non-null   int64         
 23  property_3            8000 non-null   int64         
 24  property_4            8000 non-null   int64         
 25  property_5            8000 non-null   int64         
 26  property_6            8000 non-null   int64         
 27  property_7            8000 non-null   int64         
 28  property_8            8000 non-null   int64         
 29  property_other        66 non-null     object        
 30  height_cm             8000 non-null   int64         
 31  weight_jin            8000 non-null   int64         
 32  health                8000 non-null   int64         
 33  health_problem        8000 non-null   int64         
 34  depression            8000 non-null   int64         
 35  hukou                 8000 non-null   int64         
 36  hukou_loc             7996 non-null   float64       
 37  media_1               8000 non-null   int64         
 38  media_2               8000 non-null   int64         
 39  media_3               8000 non-null   int64         
 40  media_4               8000 non-null   int64         
 41  media_5               8000 non-null   int64         
 42  media_6               8000 non-null   int64         
 43  leisure_1             8000 non-null   int64         
 44  leisure_2             8000 non-null   int64         
 45  leisure_3             8000 non-null   int64         
 46  leisure_4             8000 non-null   int64         
 47  leisure_5             8000 non-null   int64         
 48  leisure_6             8000 non-null   int64         
 49  leisure_7             8000 non-null   int64         
 50  leisure_8             8000 non-null   int64         
 51  leisure_9             8000 non-null   int64         
 52  leisure_10            8000 non-null   int64         
 53  leisure_11            8000 non-null   int64         
 54  leisure_12            8000 non-null   int64         
 55  socialize             8000 non-null   int64         
 56  relax                 8000 non-null   int64         
 57  learn                 8000 non-null   int64         
 58  social_neighbor       7204 non-null   float64       
 59  social_friend         7204 non-null   float64       
 60  socia_outing          8000 non-null   int64         
 61  equity                8000 non-null   int64         
 62  class                 8000 non-null   int64         
 63  class_10_before       8000 non-null   int64         
 64  class_10_after        8000 non-null   int64         
 65  class_14              8000 non-null   int64         
 66  work_exper            8000 non-null   int64         
 67  work_status           2951 non-null   float64       
 68  work_yr               2951 non-null   float64       
 69  work_type             2951 non-null   float64       
 70  work_manage           2951 non-null   float64       
 71  insur_1               8000 non-null   int64         
 72  insur_2               8000 non-null   int64         
 73  insur_3               8000 non-null   int64         
 74  insur_4               8000 non-null   int64         
 75  family_income         7999 non-null   float64       
 76  family_m              8000 non-null   int64         
 77  family_status         8000 non-null   int64         
 78  house                 8000 non-null   int64         
 79  car                   8000 non-null   int64         
 80  invest_0              8000 non-null   int64         
 81  invest_1              8000 non-null   int64         
 82  invest_2              8000 non-null   int64         
 83  invest_3              8000 non-null   int64         
 84  invest_4              8000 non-null   int64         
 85  invest_5              8000 non-null   int64         
 86  invest_6              8000 non-null   int64         
 87  invest_7              8000 non-null   int64         
 88  invest_8              8000 non-null   int64         
 89  invest_other          29 non-null     object        
 90  son                   8000 non-null   int64         
 91  daughter              8000 non-null   int64         
 92  minor_child           6934 non-null   float64       
 93  marital               8000 non-null   int64         
 94  marital_1st           7172 non-null   float64       
 95  s_birth               6282 non-null   float64       
 96  marital_now           6230 non-null   float64       
 97  s_edu                 6282 non-null   float64       
 98  s_political           6282 non-null   float64       
 99  s_hukou               6282 non-null   float64       
 100 s_income              6282 non-null   float64       
 101 s_work_exper          6282 non-null   float64       
 102 s_work_status         2565 non-null   float64       
 103 s_work_type           2565 non-null   float64       
 104 f_birth               8000 non-null   int64         
 105 f_edu                 8000 non-null   int64         
 106 f_political           8000 non-null   int64         
 107 f_work_14             8000 non-null   int64         
 108 m_birth               8000 non-null   int64         
 109 m_edu                 8000 non-null   int64         
 110 m_political           8000 non-null   int64         
 111 m_work_14             8000 non-null   int64         
 112 status_peer           8000 non-null   int64         
 113 status_3_before       8000 non-null   int64         
 114 view                  8000 non-null   int64         
 115 inc_ability           8000 non-null   int64         
 116 inc_exp               8000 non-null   float64       
 117 trust_1               8000 non-null   int64         
 118 trust_2               8000 non-null   int64         
 119 trust_3               8000 non-null   int64         
 120 trust_4               8000 non-null   int64         
 121 trust_5               8000 non-null   int64         
 122 trust_6               8000 non-null   int64         
 123 trust_7               8000 non-null   int64         
 124 trust_8               8000 non-null   int64         
 125 trust_9               8000 non-null   int64         
 126 trust_10              8000 non-null   int64         
 127 trust_11              8000 non-null   int64         
 128 trust_12              8000 non-null   int64         
 129 trust_13              8000 non-null   int64         
 130 neighbor_familiarity  8000 non-null   int64         
 131 public_service_1      8000 non-null   int64         
 132 public_service_2      8000 non-null   int64         
 133 public_service_3      8000 non-null   int64         
 134 public_service_4      8000 non-null   int64         
 135 public_service_5      8000 non-null   float64       
 136 public_service_6      8000 non-null   int64         
 137 public_service_7      8000 non-null   int64         
 138 public_service_8      8000 non-null   int64         
 139 public_service_9      8000 non-null   int64         
dtypes: datetime64[ns](1), float64(25), int64(111), object(3)
memory usage: 8.5+ MB

#View label distribution
y_train_=train["happiness"]
y_train_.value_counts()

#Replace - 8 with 3
y_train_=y_train_.map(lambda x:3 if x==-8 else x)

#Review label distribution
y_train_.value_counts()

#Let the label start with 0
y_train_=y_train_.map(lambda x:x-1)
#train and test are linked together
data = pd.concat([train,test],axis=0,ignore_index=True)
#Total data size
data.shape

#Processing time characteristics
data['survey_time'] = pd.to_datetime(data['survey_time'],format='%Y-%m-%d %H:%M:%S')
data["weekday"]=data["survey_time"].dt.weekday
data["year"]=data["survey_time"].dt.year
data["quarter"]=data["survey_time"].dt.quarter
data["hour"]=data["survey_time"].dt.hour
data["month"]=data["survey_time"].dt.month

#Divide the time of the day
def hour_cut(x):
    if 0<=x<6:
        return 0
    elif  6<=x<8:
        return 1
    elif  8<=x<12:
        return 2
    elif  12<=x<14:
        return 3
    elif  14<=x<18:
        return 4
    elif  18<=x<21:
        return 5
    elif  21<=x<24:
        return 6

    
data["hour_cut"]=data["hour"].map(hour_cut)

#Age at the time of questionnaire
data["survey_age"]=data["year"]-data["birth"]

#Let the label start with 0
data["happiness"]=data["happiness"].map(lambda x:x-1)

#Remove the three missing values
data=data.drop(["edu_other"], axis=1)
data=data.drop(["happiness"], axis=1)
data=data.drop(["survey_time"], axis=1)

#Whether to join the party
data["join_party"]=data["join_party"].map(lambda x:0 if pd.isnull(x)  else 1)

#Date of birth
def birth_split(x):
    if 1920<=x<=1930:
        return 0
    elif  1930<x<=1940:
        return 1
    elif  1940<x<=1950:
        return 2
    elif  1950<x<=1960:
        return 3
    elif  1960<x<=1970:
        return 4
    elif  1970<x<=1980:
        return 5
    elif  1980<x<=1990:
        return 6
    elif  1990<x<=2000:
        return 7
    
data["birth_s"]=data["birth"].map(birth_split)

#Revenue grouping
def income_cut(x):
    if x<0:
        return 0
    elif  0<=x<1200:
        return 1
    elif  1200<x<=10000:
        return 2
    elif  10000<x<24000:
        return 3
    elif  24000<x<40000:
        return 4
    elif  40000<=x:
        return 5
 

    
data["income_cut"]=data["income"].map(income_cut)

#Fill data
data["edu_status"]=data["edu_status"].fillna(5)
data["edu_yr"]=data["edu_yr"].fillna(-2)
data["property_other"]=data["property_other"].map(lambda x:0 if pd.isnull(x)  else 1)
data["hukou_loc"]=data["hukou_loc"].fillna(1)
data["social_neighbor"]=data["social_neighbor"].fillna(8)
data["social_friend"]=data["social_friend"].fillna(8)
data["work_status"]=data["work_status"].fillna(0)
data["work_yr"]=data["work_yr"].fillna(0)
data["work_type"]=data["work_type"].fillna(0)
data["work_manage"]=data["work_manage"].fillna(0)
data["family_income"]=data["family_income"].fillna(-2)
data["invest_other"]=data["invest_other"].map(lambda x:0 if pd.isnull(x)  else 1)

#Fill data
data["minor_child"]=data["minor_child"].fillna(0)
data["marital_1st"]=data["marital_1st"].fillna(0)
data["s_birth"]=data["s_birth"].fillna(0)
data["marital_now"]=data["marital_now"].fillna(0)
data["s_edu"]=data["s_edu"].fillna(0)
data["s_political"]=data["s_political"].fillna(0)
data["s_hukou"]=data["s_hukou"].fillna(0)
data["s_income"]=data["s_income"].fillna(0)
data["s_work_exper"]=data["s_work_exper"].fillna(0)
data["s_work_status"]=data["s_work_status"].fillna(0)
data["s_work_type"]=data["s_work_type"].fillna(0)

data=data.drop(["id"], axis=1)

X_train_ = data[:train.shape[0]]
X_test_  = data[train.shape[0]:]

X_train_.shape

X_test_.shape

target_column = 'happiness'
feature_columns=list(X_test_.columns) 
feature_columns