Last week, we looked at how Machine Learning can be useful in industry. Today we will explore the path to machine learning model development and the individual steps that go into that. The above roadmap outlines the four major steps in yellow on the left, and the smaller substeps on the right.
What is Data Preprocessing?
Data Preprocessing is one of the most important steps in tackling a machine learning problem. Depending on your dataset, you may have problems like missing values, useless features, and other types of noise. In the data preprocessing step, you focus on removing features that hold useless data, and addressing the missing values. We will use the Pandas library to work with our data. This is what our raw, unprocessed data looks like.
We can see that there are some features that would provide nothing of value to the machine learning model. These are features like the UDI and Product ID. They are just labels used to name each entry, so they are not actually useful. After removing them, our data is much more model-ready.
In our specific dataset, there were no missing or null values. In the real world this is usually not the case, and you should remove the features(columns) with null values entirely.
What is Exploratory Data Analysis?
Exploratory Data Analysis is a key step that involves the initial investigating of your dataset to find anything unusual or helpful. For example, when performing EDA, you may see that Feature X correlates directly with the output data. This can be helpful when selecting a model, and the inputs that go into the model. Exploratory Data Analysis involves a lot of charting and graphing to gain a better understanding of the summary of your dataset.
Using the .describe() function from the Pandas library is useful to quickly view the stats of your data. You can see the mean for each numerical feature, the standard deviation, min and max, and percentiles.
Another important part of EDA is analyzing the skewness of your data. Using .skew() from the Pandas library, we can see just how skewed each feature is. If the skewness is between -0.5, and 0.5, the data is almost symmetrical. If it’s between -1 and 0.5 (negatively skewed) or between 0.5 and 1 (positively skewed), the data is slightly skewed. And lower than -1 or greater than 1 the data is extremely skewed.
Using Plotly, you can make many different graphs to view your data.
You can mix and match features to see if there is correlation. Here we compare air temperature and failure type.
Keep in mind that Exploratory Data Analysis is meant just that, Exploratory. There is no need to compare every single feature, it is only your initial exploration. EDA is not only useful to you, but it’s good practice so other engineers can look at your notebooks and easily understand it as they read along.
In the next blog post we will return for the third step in our Roadmap To Model Development which includes more data processing. Then, we will finally train and test our model(s).