### Introduction

Deep learning is one of the most interesting and promising areas of artificial intelligence (AI) and machine learning currently. With great advances in technology and algorithms in recent years, deep learning has opened the door to a new era of AI applications.

In many of these applications, deep learning algorithms performed equal to human experts and sometimes surpassed them.

Python has become the go-to language for Machine Learning and many of the most popular and powerful deep learning libraries and frameworks like TensorFlow, Keras, and PyTorch are built in Python.

In this series, we'll be using Keras to perform Exploratory Data Analysis (EDA), Data Preprocessing and finally, build a Deep Learning Model and evaluate it.

### Data Preprocessing

In the preprocessing stage, we'll prepare the data to be fed to the Keras model. The first step is clearing the dataset of null values. Then, we'll use one-hot encoding to convert categorical variables to numerical variables. Neural Nets work with numerical data, not categorical.

We'll also split the data into a training and testing set. Finally, we'll scale the data/standardize it so that it ranges from -1 to 1. This standardization helps both train the model better and allows it to converge easier.

#### Dealing with Missing Values

Let's find out the number and percentage of missing values in each variable in the dataset:

```
missing_values = pd.DataFrame({
'Column': df.columns.values,
'# of missing values': df.isna().sum().values,
'% of missing values': 100 * df.isna().sum().values / len(df),
})
missing_values = missing_values[missing_values['# of missing values'] > 0]
print(missing_values.sort_values(by='# of missing values',
ascending=False
).reset_index(drop=True))
```

This code will produce the following table which shows us variables that contain missing values and how many missing values they contain:

Column | # of missing values | % of missing values | |

0 | Pool QC | 2917 | 99.5563 |

1 | Misc Feature | 2824 | 96.3823 |

2 | Alley | 2732 | 93.2423 |

3 | Fence | 2358 | 80.4778 |

4 | Fireplace Qu | 1422 | 48.5324 |

5 | Lot Frontage | 490 | 16.7235 |

6 | Garage Cond | 159 | 5.42662 |

7 | Garage Qual | 159 | 5.42662 |

8 | Garage Finish | 159 | 5.42662 |

9 | Garage Yr Blt | 159 | 5.42662 |

10 | Garage Type | 157 | 5.35836 |

11 | Bsmt Exposure | 83 | 2.83276 |

12 | BsmtFin Type 2 | 81 | 2.76451 |

13 | BsmtFin Type 1 | 80 | 2.73038 |

14 | Bsmt Qual | 80 | 2.73038 |

15 | Bsmt Cond | 80 | 2.73038 |

16 | Mas Vnr Area | 23 | 0.784983 |

17 | Mas Vnr Type | 23 | 0.784983 |

18 | Bsmt Half Bath | 2 | 0.0682594 |

19 | Bsmt Full Bath | 2 | 0.0682594 |

20 | Total Bsmt SF | 1 | 0.0341297 |

Since `Pool QC`

, `Misc Feature`

, `Alley`

, `Fence`

, and `Fireplace Qu`

variables contain a high percentage of missing values as shown in the table, we will simply remove them as they probably won't affect the results much at all:

```
df.drop(['Pool QC', 'Misc Feature', 'Alley', 'Fence', 'Fireplace Qu'],
axis=1, inplace=True)
```

For other variables that contain missing values, we will replace these missing values depending on the data type of the variable: whether it is *numerical* or *categorical*.

If it is numerical, we will replace missing values with the variable mean. If it is categorical, we will replace the missing values with the variable mode. This removes the false bias that can be created with missing values in a neutral way.

To know which variables are numerical and which are categorical, we will print out 5 unique items for each of the variables that contain missing values using this code:

```
cols_with_missing_values = df.columns[df.isna().sum() > 0]
for col in cols_with_missing_values:
print(col)
print(df[col].unique()[:5])
print('*'*30)
```

And we get the following results:

```
Lot Frontage
[141. 80. 81. 93. 74.]
******************************
Mas Vnr Type
['Stone' 'None' 'BrkFace' nan 'BrkCmn']
******************************
...
```

Let's replace the values of missing numerical values with the mean:

```
num_with_missing = ['Lot Frontage', 'Mas Vnr Area', 'BsmtFin SF 1', 'BsmtFin SF 2',
'Bsmt Unf SF', 'Total Bsmt SF', 'Bsmt Full Bath', 'Bsmt Half Bath',
'Garage Yr Blt', 'Garage Cars', 'Garage Area']
for n_col in num_with_missing:
df[n_col] = df[n_col].fillna(df[n_col].mean())
```

Here, we just put them all in a list and assigned new values to them. Next, let's replace missing values for categorical variables:

```
cat_with_missing = [x for x in cols_with_missing_values if x not in num_with_missing]
for c_col in cat_with_missing:
df[c_col] = df[c_col].fillna(df[c_col].mode().to_numpy()[0])
```

After this step, our dataset will have no missing values in it.

#### One-Hot Encoding of Categorical Variables

Keras models, like all machine learning models fundamentally work with numerical data. Categorical data has no meaning to a computer, but it does do us. We need to convert these categorical variables into numerical representations in order for the dataset to be usable.

The technique that we will use to do that conversion is One-Hot Encoding. Pandas provides us with a simple way to automatically perform One-Hot encoding on all categorical variables in the data.

Before that though, we must ensure that no categorical variable in our data is represented as a numerical variable by accident.

#### Checking Variables Data Types

When we read a CSV dataset using Pandas as we did, Pandas automatically tries to determine the type of each variable in the dataset.

Sometimes, Pandas can determine this incorrectly - if a categorical variable is represented with numbers, it can wrongfully infer that it's a numerical variable.

Let's check if there are any data type discrepancies in the `DataFrame`

:

```
data_types = pd.DataFrame({
'Column': df.select_dtypes(exclude='object').columns.values,
'Data type': df.select_dtypes(exclude='object').dtypes.values
})
print(data_types)
```

Column | Data type | |

0 | MS SubClass | int64 |

1 | Lot Frontage | float64 |

2 | Lot Area | int64 |

3 | Overall Qual | int64 |

4 | Overall Cond | int64 |

5 | Year Built | int64 |

6 | Year Remod/Add | int64 |

Based on this table and the variables descriptions from Kaggle, we can notice which variables were falsely considered numerical by Pandas.

For example, `MS SubClass`

was detected as a numerical variable with a data type of `int64`

. However, based on the description of this variable, it specifies the type of the unit being sold.

If we take a look at the unique values of this variable:

```
df['MS SubClass'].unique().tolist()
```

We get this output:

```
[20, 60, 120, 50, 85, 160, 80, 30, 90, 190, 45, 70, 75, 40, 180, 150]
```

## Better understand your data with visualizations

- 30-day no-questions refunds
- Beginner to Advanced
- Updated regularly (update June 2021)
- New bonus resources and guides

This variable represent different unit types as numbers like `20`

(one story dwellings built in 1946 and newer), `60`

(2 story dwellings built in 1946 and newer), etc.

This actually isn't a numerical variable but a categorical one. Let's convert it back into a categorical variable by reassigning it as a string:

```
df['MS SubClass'] = df['MS SubClass'].astype(str)
```

#### Performing One-Hot Encoding

Before performing One-Hot Encoding, we want to select a subset of the features from our data to use from now on. We'll want to do so because our dataset contains 2,930 records and 75 features.

Many of these features are categorical. So if we keep all the features and perform One-Hot Encoding, the resulting number of features will be large and the model might suffer from the curse of dimensionality as a result.

Let's make a list of the variables we want to keep in a subset and trim the `DataFrame`

so we only use these:

```
selected_vars = ['MS SubClass', 'MS Zoning', 'Lot Frontage', 'Lot Area',
'Neighborhood', 'Overall Qual', 'Overall Cond',
'Year Built', 'Total Bsmt SF', '1st Flr SF', '2nd Flr SF',
'Gr Liv Area', 'Full Bath', 'Half Bath', 'Bedroom AbvGr',
'Kitchen AbvGr', 'TotRms AbvGrd', 'Garage Area',
'Pool Area', 'SalePrice']
df = df[selected_vars]
```

Now we can perform One-Hot Encoding easily by using Pandas' `get_dummies()`

function:

```
df = pd.get_dummies(df)
```

After one-hot encoding, the dataset will have 67 variables. Here are the capped first few rows - there are many more variables than this:

Lot Frontage | Lot Area | Overall Qual | Overall Cond | Year Built | Total Bsmt SF | 1st Flr SF | 2nd Flr SF | Gr Liv Area | |

0 | 141 | 31770 | 6 | 5 | 1960 | 1080 | 1656 | 0 | 1656 |

1 | 80 | 11622 | 5 | 6 | 1961 | 882 | 896 | 0 | 896 |

2 | 81 | 14267 | 6 | 6 | 1958 | 1329 | 1329 | 0 | 1329 |

#### Splitting Data into Training and Testing Sets

One of the last steps in data preprocessing is to split it in a training and testing subset. We'll be training the model on the training subset, and evaluating it with an unseen test set.

We will split the data randomly so that the training set will have 80% of the data and the testing set will have 20% of the data. Generally, the training set typically has anywhere between 70-80% of the data, while 20-30% is used for validation.

This is made really simple with Pandas' `sample()`

and `drop()`

functions:

```
train_df = df.sample(frac=0.8, random_state=9)
test_df = df.drop(train_df.index)
```

Now `train_df`

holds our training data and `test_df`

holds our testing data.

Next, we will store the target variable `SalePrice`

separately for each of the training and testing sets:

```
train_labels = train_df.pop('SalePrice')
test_labels = test_df.pop('SalePrice')
```

We're removing the `SalePrice`

value because, well, we want to predict it. There's no point predicting something we already know and have fed to the model. We'll be using the actual values to verify if our predictions are correct.

After this step, `train_df`

will contain the predictor variables of our training data (i.e. all variables excluding the target variable), and `train_labels`

will contain the target variable values for `train_df`

. The same applies to `test_df`

and `test_labels`

.

We perform this operation to prepare for the next step of data scaling.

Note that Pandas'

`pop()`

function will return the specified column (in our case, it is`SalePrice`

) from the dataframe (`train_df`

for example) with removing that column from the dataframe.

At the end of this step, here are the number of records (rows) and features (columns) for each of `train_df`

and `test_df`

:

Set | Number of records | Number of features |

`train_df` | 2344 | 67 |

`test_df` | 586 | 67 |

Moreover, `train_labels`

has 2,344 labels for the 2,344 records of `train_df`

and `test_labels`

has 586 labels for the 586 records in `test_df`

.

Without preprocessing this data, we would have a *much* messier dataset to work with.

#### Data Scaling: Standardization

Finally, we will standardize each variable - except the target variable, of course - in our data.

For training data which is stored now in `train_df`

, we will calculate the mean and standard deviation of each variable. After that, we will subtract the mean from the values of each variable and then divide the resulting values by the standard deviation.

For testing data, we will subtract the training data mean from the values of each variable and then divide the resulting values by the training data standard deviation.

If you'd like to read up on Calculating Mean, Median and Mode in Python or Calculating Variance and Standard Deviation in Python, we've got you covered!

We use values calculated using training data because of the general principle: anything you learn, must be learned from the model's training data. Everything from the test dataset will be completely unknown to the model before testing.

Let's perform the standardization now:

```
predictor_vars = train_df.columns
for col in predictor_vars:
# Calculating variable mean and std from training data
col_mean = train_df[col].mean()
col_std = train_df[col].std()
if col_std == 0:
col_std = 1e-20
train_df[col] = (train_df[col] - col_mean) / col_std
test_df[col] = (test_df[col] - col_mean) / col_std
```

In this code, we first get the names of the predictor variables in our data. These names are the same for training and testing sets because these two sets contain the same variables but different data values.

Then for each predictor variable, we calculate the mean and standard deviation using the training data (`train_df`

), subtract the calculated mean and divide by the calculated standard deviation.

Note that sometimes, the standard deviation is equal to 0 for some variables. In that case, we make the standard deviation equal to an extremely small amount because if we keep it equal to 0, we will get a division-by-zero error when we use it for division later.

This nets us scaled and standardized data in the range of -1 and 1.

With that done, our dataset is ready to be used to train and evaluate a model. We'll be building a deep neural network in the next article.

### Conclusion

Data preprocessing is a crucial step in a Machine Learning pipeline. Without dropping certain variables, dealing with missing values, encoding categorical values and standardization - we'd be feeding a messy (or impossible) dataset into a model.

The model will only be as good as the data we feed it and in this article - we've prepped a dataset to fit a model.