Category: AI/ML

Posted on

The Roadmap to Model Development Part 3

In last week’s blog post, we finished up the third step (Data Preprocessing for Prediction) and began creating our first models in the final step (Model Development). Today, we will create four more models, and test each of them to compare their accuracy and runtimes.

Creating the models

Last blog post, we created only the Logistic Regression model and the K-Nearest Neighbors model. We will add the following four models today: Support Vector Machine, Decision Trees, Naive Bayes, and Random Forest. We will use the Sci-kit learn library for all of these.

  • Support Vector Machine

In this step, we create the Support Vector Machine model and name it svc. We then append its full name to our ‘classifier’ array, and the instance to our ‘imported_as’ array. The Support Vector Machine model works by separating the data points into ‘hyperplanes’ that have optimal decision boundaries.

  • Decision Tree

Here is the code implemented to create the Decision Tree model. It also appends the name and instance to the ‘classifier’ and ‘imported_as’ arrays, respectively. The Decision Tree model works by creating a ‘tree’ with multiple branches, with a binary question, leading to more branches.

  • Naive Bayes

In this code block, we create the Naive Bayes model and append its name/instance to the ‘classifier’ and ‘imported_as’ arrays. Naive Bayes is a model that attempts to perform classification by assuming all features are independent of each other. 

  • Random Forest Classifier

This is the code to create the Random Forest Classifier Model. We then append it to the ‘classifier’ array and the ‘imported_as’ array. The Random Forest Classifier works by creating multiple decision trees and weighing all their results before a decision.

Testing and measuring accuracy

We can now create our main class, that we will use to create an object with all the models so we can easily compare their accuracies. 

We create a class called ‘Modeling’ in which we have a few functions, ‘fit’ and ‘results’ these two can be called on the ‘Modeling’ class object to fit the models, and to get the results, respectively. Now, we will make an array that holds all the models we want to test. We could also use the ‘imported_as’ array, but for simplicity we will make a new array and name it ‘models_to_test’.

Finally, we can create an object of the ‘Modeling’ class and use our ‘fit’ and ‘results’ functions on it. We will use our previously made arrays of x and y training data, x and y testing data, and our ‘models_to_test’ array as the arguments. 

If it runs properly, you should be returned with ‘name of model’ has been fit for each of the models, and a pandas dataframe that holds the accuracy and runtime of each model. 

The Random Forest Classifier seems to be the most accurate. If you want to test this multiple times to see if the RFC model is ever beat, you can use a simple for loop to get the top model as many times as you’d like.

You can run it as many times as you like, and you’ll see that the Random Forest Classifier is always the most accurate. However, this comes with a price of a significantly higher runtime than the other models (0.4 seconds vs 0.01 and lower). This is why we trained multiple models, so we could compare which one would deliver enough accuracy while still being appropriate for the hardware it will run on. From these models, you can choose which one is most appropriate for your use case.

What’s next?

We now have a few accurate models to choose from for the purpose of predictive maintenance. However, the data used to train and test them is not our own. This project is meant to be easily adaptable to different sets of data, so next we will collect our own data for predictive maintenance and attempt to employ the model(s) on that dataset.

Posted on

The Roadmap to Model Development Continued

Last week we took a look at the first two major steps in the process of model development, Data Preprocessing and Exploratory Data Analysis. In today’s article we will continue on the same roadmap, looking at two more steps that are arguably more interesting. These include the final data preprocessing to prep for model development, and developing the first few models.

Data Preprocessing for Prediction

What is the difference between the first data preprocessing step and the second? In the first step, you will remember that all we did was drop unwanted features. This is solely to prepare the data so we can perform Exploratory Data Analysis. In the second preprocessing step, we do all the other necessary actions for model deployment. 

  1. Encoding categorical features

Label encoding refers to converting the labels into a numeric form so as to convert them into the machine-readable form. We will use it on our training data, and using the “.fit” sci-kit method it will figure out the unique values and assign a value to it, returning the encoded labels.

This is what the code looks like:

  1. Splitting Test and Training Data

Every machine learning model needs test and training data to learn and improve. In this step, we separate the test and training data from the dataset consisting of ten thousand values. We use the “train_test_split” module from sci-kit and “.drop” from the pandas library to get four datasets, x_train, y_train, x_test, and y_test.

This is what the code looks like:

  1. Feature Scaling

Feature Scaling is a technique to normalize/standardize the independent features present in the data in a fixed range. It is done to handle highly varying magnitudes or values/units. Without feature scaling, a machine learning algorithm will weigh greater values higher and almost disregard smaller values, not taking into account the units. Note that we only do this to the x datasets, because the output does not need to be normalized.

This is what the code looks like:

Here is an image that shows how the distribution of the data is changed after feature scaling. 

Model Development

One of the final and most important in any model’s development, the actual creation of the model itself. For the best accuracy possible, we will be creating multiple models and comparing their accuracy to choose the best one. In this week’s post we will look at a couple and next week we will look at the rest. We will first create a place to store our model’s names and their actual instances.

Here is the code I used to do this:

  • The Logistic Regression Model

The Logistic Regression model is a supervised learning algorithm that investigates the relationship between a dependent and independent variable and produces results in a binary format (fail or not fail). This is used to predict failure.

Here is the code for the linear regression model, and for appending it to our arrays.

  • K- Nearest Neighbors Model

K-Nearest Neighbors is also a supervised machine learning algorithm which is mostly used to solve classification problems. It stores all the available cases and classifies new cases based on similarity. “K” is the number of nearest neighbors.

This image shows how k-nearest neighbors uses the location of data distribution to predict.

This is the code to implement the KNN model and append it:

In the next week’s blog post, we will finish the final step by creating more models, including Support Vector Machine, Random Forest, Naive Bayes, and Decision Trees. We will then test the accuracy of all of them to choose the most accurate one.

Posted on

The Roadmap to Model Development

Last week, we looked at how Machine Learning can be useful in industry. Today we will explore the path to machine learning model development and the individual steps that go into that. The above roadmap outlines the four major steps in yellow on the left, and the smaller substeps on the right.

What is Data Preprocessing?

Data Preprocessing is one of the most important steps in tackling a machine learning problem. Depending on your dataset, you may have problems like missing values, useless features, and other types of noise. In the data preprocessing step, you focus on removing features that hold useless data, and addressing the missing values. We will use the Pandas library to work with our data. This is what our raw, unprocessed data looks like.

We can see that there are some features that would provide nothing of value to the machine learning model. These are features like the UDI and Product ID. They are just labels used to name each entry, so they are not actually useful. After removing them, our data is much more model-ready.

In our specific dataset, there were no missing or null values. In the real world this is usually not the case, and you should remove the features(columns) with null values entirely.

What is Exploratory Data Analysis?

Exploratory Data Analysis is a key step that involves the initial investigating of your dataset to find anything unusual or helpful. For example, when performing EDA, you may see that Feature X correlates directly with the output data. This can be helpful when selecting a model, and the inputs that go into the model. Exploratory Data Analysis involves a lot of charting and graphing to gain a better understanding of the summary of your dataset.

Using the .describe()  function from the Pandas library is useful to quickly view the stats of your data. You can see the mean for each numerical feature, the standard deviation, min and max, and percentiles.

Another important part of EDA is analyzing the skewness of your data. Using .skew() from the Pandas library, we can see just how skewed each feature is. If the skewness is between -0.5, and 0.5, the data is almost symmetrical. If it’s between -1 and 0.5 (negatively skewed) or between 0.5 and 1 (positively skewed), the data is slightly skewed. And lower than -1 or greater than 1 the data is extremely skewed.

Using Plotly, you can make many different graphs to view your data.

You can mix and match features to see if there is correlation. Here we compare air temperature and failure type.

Keep in mind that Exploratory Data Analysis is meant just that, Exploratory. There is no need to compare every single feature, it is only your initial exploration. EDA is not only useful to you, but it’s good practice so other engineers can look at your notebooks and easily understand it as they read along.

In the next blog post we will return for the third step in our Roadmap To Model Development which includes more data processing. Then, we will finally train and test our model(s).

Posted on

Machine Learning in Predictive Maintenance

Last week, we looked at the deployment of applications on a Kubernetes cluster. This was one step in preparation of a bigger project, one which involves using machine learning(ML) for predictive maintenance of industrial processes. 

What is Predictive Maintenance?

Predictive maintenance is the process of using historical data to find patterns that can help to predict future failures of industrial processes, AKA machine failure. Traditionally, predictive maintenance has been done by hard-coded rules that say things like, “when x rotations exceeds y threshold, this machine will fail.” The problem with these hard-coded rules is that they all have to be created by patterns that are readily apparent from the data (since they are written by humans).

How Machine Learning can help

Machine learning algorithms can pick up on patterns that would not be visible to the human eye. A ML model can take in a lot of data at a time and use it to “train” itself to find patterns from data that is otherwise unrelated. 

Without ML facilities are prone to performing unnecessary maintenance. By implementing machine learning, you can not only know when things are going to fail, but you can also reduce the wasted money that comes from over-maintenance. This will reduce cost while not risking downtime or the safety of workers. 

There are multiple ways that ML models can be implemented for predictive maintenance purposes. We will explore two common types.

Regression Approach:

This approach is used to predict the Remaining Useful Life (RUL) of an asset. It will tell how many days or cycles are left before a system fails. To implement this, you need static and historical data, with every event labeled. You can only implement this if there is only one type of failure. If there are multiple, you will need multiple models. This is the well-known linear regression approach, which finds a line of best fit according to the data provided.

Classification Approach:

This approach is used to predict if a machine will fail in the next N days or cycles. This can be more useful and accurate than using a regression, because you may only need to know if the system will fail soon. Implementation also requires static and historical labeled data. 

What are we trying to do?

Our approach involves time-series analysis to forecast when machines are due for maintenance.  Alongside that, a Remaining Useful Life model will most likely also be implemented just to have more information. For now, we are using free datasets available at the UCI Machine Learning Repository (UCI Machine Learning Repository) that are specifically made for the purpose of predictive maintenance. Once we have a working model, we can collect our own data and use that instead.

The model will be deployed on the Kubernetes cluster for purposes of redundancy.

Next week we will further explore the creation of our machine learning model. We will discuss cleaning and preparing data, and designing neural networks.

Posted on

Roadblocks to the first Deployment (K3s)

Last week, we looked at setting up a Kubernetes cluster on three Jetson Nanos, to prepare them for application deployment. That’s what we tackled this week, and in today’s blog post we will look at the obstacles we’ve encountered.

For these examples, we have been trying to deploy a test application using the instructions from the Rancher docs. This uses a pre-built container provided by Rancher. To recreate my deployment do the following.

  1. Create a file called testdeploy.yaml and paste the following inside:
apiVersion: apps/v1
kind: Deployment
metadata:
  name: mysite
  labels:
    app: mysite
spec:
  replicas: 1
  selector:
    matchLabels:
      app: mysite
  template:
    metadata:
      labels:
        app: mysite
    spec:
      containers:
        - name: mysite
          image: kellygriffin/hello:v1
          ports:
            - containerPort: 80
  1. Once you’ve saved the file, use the following command to run it. 
kubectl apply -f testdeploy.yaml
  1. View your deployment using:
kubectl get pods

The CrashLoopBackOff error:

If you are running the above steps on a Jetson Nano cluster as well, you may see the common CrashLoopBackOff error.

NAME                      READY   STATUS             RESTARTS      AGE
mysite-57b5b46f97-rfcgx   0/1     CrashLoopBackOff   5 (39s ago)   3m30s

This error is well documented for Kubernetes in general, and is often caused by insufficient resources, trying to access a locked file, or a locked database. However, in the case of the Jetson Nano cluster, I believe the reason for the CrashLoopBackOff error is the ARM_64 architecture it runs on and shares with the new M1 Macbook Pro. A lot of pre-made containers may be designed specifically to be run on x64 architectures or even x86 making them incompatible with the ARM_64 architecture. 

To test this theory, I tried to deploy the Nginx service on the cluster. Nginx should work as it is made to work with ARM_64 architectures. I ran the following, to create a deployment of Nginx using the Nginx image. 

kubectl create deployment nginx --image=nginx

After running kubectl get pods, we can see that the Nginx service is running fine on the cluster, while the testdeploy continues to restart over and over after crashing.

NAME                      READY   STATUS             RESTARTS        AGE
mysite-57b5b46f97-rfcgx   0/1     CrashLoopBackOff   7 (2m31s ago)   13m
nginx-85b98978db-gc6ms    1/1     Running            0               20s

Many premade containers available to use for testing may be designed for different architectures from the ARM_64, which caused the frustrating CrashLoopBackOff error. As long as your Nginx deployment works, your cluster should be working, it will just require learning and much trial and error to deploy properly. 

The ImagePullBackOff error:

The ImagePullBackOff error is a very finicky error. I encountered it the first few times trying to deploy the Nginx pod using the steps outlined above. 

NAME                      READY   STATUS             RESTARTS      AGE
mysite-57b5b46f97-rfcgx   0/1     ImagePullBackOff   0 (9s ago)   4m30s

At first I had no idea how to get around it, and I assumed it was a problem with the place it was pulling the image from. Maybe the image was locked, or corrupted I thought. But after retrying multiple times, I let it run for 8 or so minutes. It retried pulling multiple times, and then one time it just worked. If you’re encountering this error, I recommend letting it run for at least 10 or so minutes before trying something else. 

Those are a few of the more problematic errors I encountered when trying to get to the first deployment. The next step will be to deploy a machine learning model on the cluster. This will require a lot more in terms of containerization, and we will explore that containerization and the creation of the model itself next week.

Posted on

What is Edge AI?

What is Edge AI and How Does It Create a New Paradigm

Edge computing diagram
IEEE Edge Diagram

Artificial intelligence and machine learning (AI/ML) have been around for a while now. Without knowing the intricacies, people still understand that it has powerful uses, such as analyzing data, understanding pictures, or chatting with Siri. The edge is a much newer concept, which I first heard of with the development of 5G.

The edge is data processing with round trip servicing under 20 milliseconds. What does this actually mean? A more practical definition is self-explanatory. The edge is literally the closest point you can put your processing to where you need the result. This leaves a broader concept of edge. If your processing must be done in the cloud, then your edge is simply the closest/fastest servers. If your processing can be done locally, then your edge should be your local processor (smartphone?).

But what do you get when you marry the AI and edge concepts? At one point of time, AI algorithms became practical to implement in the cloud, running on powerful server racks. Since AI was too complex for local devices, the cloud AI was also the edge AI. This is no longer true. Powerhouses such as #arm and #NVIDIA have been making massive breakthroughs in optimizing AI performance. Startup #Hailo has developed their own AI processor. #STM has even put ML cores on individual sensors. AI can now run on embedded processors, sensors, internet gateways, security cameras, etc. The new edge AI is essentially anywhere.

Believe it or not, this new reality is a great thing. Since data is generated and processed locally, the edge AI device does not need to be connected to the internet. This means you can have smart devices that protect your privacy. You can have security systems that cannot be taken down by cutting the internet. AI analytics in remote areas? Got it. Edge enables AI to be anywhere it is useful, when it is useful.

#Millspaw Electronics is launching an edge AI brand, Normal AI. Be sure to connect to learn what we can do for you.