# MLOps Pipeline via Jenkins and Hyperparameter Optimization

**MLOps => Machine Learning + Operations, the easiest way to understand this is to put an MLOps pipeline into practice.**

Let’s set up a pipeline on the AWS cloud using Jenkins and Docker. Additionally, we will employ hyperparameter tuning, which is a parameter whose value is used to regulate the learning process.

MLOps pipeline

> **Contents:**  
> [Jenkins and Its Installation.](#8dc5)  
> [Install Dockers.](#F669)  
> [Job 1.](#7fc5)  
> [Job 2.](#f0ed)  
> [Job 3.](#a821)  
> [Job 4.](#ab58)  
> [Job 5.](#3d97)

### **What is Jenkins?**

The most well-known free automation server is Jenkins. The automation of the software development processes is done with Jenkins.

### **Benefits of Jenkins include:**

* It is written in Java, and thus you can run it on all platforms.
    
* It has lots of plugins, which can help you solve almost any problem you can imagine.
    
* Helps developers to integrate code changes and always produce stable software build.
    

### **Installing Jenkins Manually**

The installation procedure is easy to understand. It includes simple actions like:

* Java installation.
    
* Adding official Jenkins repositories to your package manager.
    
* Package installation.
    
* Initial Jenkins configuration.
    

### **Let’s implement them one by one.**

#### Step 1: Java installation

To be able to run Jenkins, you need to install Java first.

**Step 1: Update the apt package index using this:**

`$ sudo apt update`

**Step 2: Next, install the default Java OpenJDK package using this:**

`$ sudo apt install openjdk-8-jre`

**Step 3: Verify the installation using this command:**

`$ java -version`

> **Note**: you should see the Java version after this command

**Step 4: By the time of this article writing, Jenkins is using the Java 8 version. If you see another version, switch to the correct one:**

`$ sudo update-alternatives — config java`

**Step 5: Add Jenkins repository**

By default, Jenkins is not included in the Ubuntu repositories, and you need to add it.

First, we need to import the repository key:

`$ wget -q -O —` [`https://pkg.jenkins.io/debian-stable/jenkins.io.key`](https://pkg.jenkins.io/debian-stable/jenkins.io.key) `| sudo apt-key add -`

**Step 6: The next step is to append the repository address to the server’s repositories list:**

`$ sudo sh -c ‘echo deb` [`https://pkg.jenkins.io/debian-stable`](https://pkg.jenkins.io/debian-stable) `binary/ > \`

`/etc/apt/sources.list.d/jenkins.list’`

**Step 7: Once that is done, we need to update the information about the software, which we can use from the new repository:**

`$ sudo apt-get update`

**Step 8: Jenkins Package Installation**

Since we have the repository up to date, let’s run the following command:

`$ sudo apt-get -y install jenkins`

**Step 9: Running this command will prompt you to confirm the download and installation.**

By default, Jenkins starts after the installation process. If not, use the following command to do it manually:

`$ sudo systemctl start jenkins`

**Step 10: Let's check the Jenkins service status:**

`$ sudo systemctl status jenkins`

By default, Jenkins runs on port **8080**.

**Step 11: So this can be done by enabling a security group and inbound 8080 port for TCP connection on AWS.**

Then copy your AWS Instance IPV4:

`ubuntu@IP.regrion_name.compute.amazonaws.com:8080`

This command results in opening the Jenkins docker container on port **8080.**

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1703053000920/3d7386b3-dcd1-4029-9c4f-90efa7f96dd7.jpeg align="left")

Getting started page

***The initial password is available here:***

`$ sudo cat /var/lib/jenkins/secrets/initialAdminPassword`

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1703053002555/4eb0669d-6bbd-498e-a24e-d3bb5da20564.png align="left")

Entering the password which was generated by running the above code

**Jenkins Plugins**

At the next screen, you will be required to choose plugins for initial installation. If you do not know which plugins to set up yet, use the “**Install suggested plugins**” option

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1703053004148/39fddf59-65a3-4669-b4c5-c048c152fa44.png align="left")

Select the install suggested plugins

The screen that will appear once you click Install Suggested Plugins is shown below.

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1703053006015/3a9f95d6-8b5e-494b-8e22-d22b52a8b8e8.png align="left")

Thereafter you will be prompted with the below screen.

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1703053007815/1079520e-824a-40c6-bc2d-4f562f89b8b6.png align="left")

We need to create your Jenkins user.

After successfully entering the user details, you will be directed to the instance configuration page to confirm the URL for the Jenkins instance.

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1703053009956/47417380-c313-4907-ba38-5020fd4bacfc.png align="left")

Set your instance configuration or local host

Thereafter clicking on the save and finish buttons. The below screen will appear.

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1703053011798/425025ce-d5ec-418a-a873-526f38b6b874.png align="left")

Jenkins is ready

To access the Jenkins main dashboard, click the “**Start using Jenkins**” button. This action will bring you to the main Jenkins interface.

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1703053013789/c39a9b7f-611e-4251-b7f2-e0228eeb2bec.png align="left")

Jenkins home page

> Because Docker has code dependencies, you must have it installed in order to execute the MLOps pipeline in Jenkins. The official Docker documentation contains a detailed description of the entire installation procedure, so we won’t repeat everything here.

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1703053015154/1d6e1cd7-2a0a-4b86-b4b5-4c9aad4f09ac.png align="left")

Install the docker by running the code in ubuntu

The following commands can be used to launch a Docker container after Docker has been successfully installed. We will create the following configurations to enable Jenkins to function without the root user:

`$ sudo visudio`

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1703053017391/b1e5c4e1-8ad5-442e-884a-f8ad65fe4e8c.png align="left")

visudio

`$ sudo vi /etc/default/jenkins`

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1703053019777/ab329297-d1df-416a-ace8-fa9e1de0417c.png align="left")

make Jenkins root

`$ sudo service jenkins restart`

Now we should get into the EC2 instance and pull the GitHub repository of our Machine Learning project.

`$` [`git clone https://github.com/aniforverizon/MLOPS-Pipeline-Jenkins.git`](https://github.com/aniforverizon/MLOPS-Pipeline-Jenkins.git)

***Then apply the following commands.***

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1703053021206/12da0bc2-9cae-4ba0-a8cf-cae1cf801a70.png align="left")

Using Dockerfile, we will first construct a container image that includes Python as well as all the additional libraries required for ML/DL code to run.

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1703053023439/1878d410-0b66-48d3-b6f8-42c3f44b46a3.png align="left")

docker file

Now build this docker file with a new image name. so we run this command on `ubuntu:18.04 “ docker build -t your_imagename:your_tagname .”`

Thereafter go to the Jenkins dashboard and click on “**New Items”** then the below screen will be prompted.

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1703053025619/e1a75698-dfd8-46cb-8220-629d0572ec81.png align="left")

create job1

Here, type the job’s name and click on Freestyle project before clicking OK. You must now use Jenkins to run a Docker container.

**JOB 1:-**

To deploy code and begin training, Jenkins should immediately launch the appropriate container on preinstalled libraries. The method is outlined below. Enter the command to execute the container by going to the Build Environment and choosing “**Execute shell**.”

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1703053027710/d0b07bd3-36bf-4ca0-8ea2-7bb986fcf7c1.png align="left")

Job1 configuration

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1703053029960/b9bc66fd-3c6b-46bf-8a01-ab5e3ffd1327.png align="left")

job1 executing the command

**JOB 2:**

This task will verify the container’s health and restart it if necessary if it has been halted for any reason. To repeat this process, create a new item called **(MLOPSDocker\_Container\_Setup**) that checks the status of the container from Job 1.

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1703053031945/a97f0778-e6ef-4733-a8c1-7c3cb9984406.png align="left")

Job2

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1703053034129/336ff10f-e876-43e9-a752-c869c2fab9c6.png align="left")

job2 write the below code in the command box

`if sudo docker ps -a | grep job2`

`then`

`exit 0`

`else`

`sudo docker container run -dir -v /mlops/:/root/mlops/--name job2 jeet:v1`

`exit 1`

`fi`

`fi`

**JOB 3 :**

If and only if the docker container is in running mode, this job will begin the machine learning model training.

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1703053036199/be6e0322-64b4-4553-8c1a-cb8cde060393.png align="left")

job3

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1703053038211/1bd039b4-12c6-491f-b245-7db849fe74d5.png align="left")

Execute the command in job3

We can check the output for the training process by the following process.

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1703053040538/d0d9deaf-9f07-4c6b-95e5-0b1b79704bd6.png align="left")

To check the output click on the console output

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1703053042922/1865291e-944b-4887-a881-677280f076b9.png align="left")

console output

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1703053045416/54dee7ee-141c-48be-91cc-2b399b2f795b.png align="left")

Output

Once the training process is done then here comes the mailing service into the picture.

**JOB 4:**

This task will cause the mail regarding the accuracy of your model to be sent. It will assess the model’s accuracy; if it falls below 90%, it will notify the developer via email that the accuracy is lower than expected. Next, it will start the task of tuning the hyperparameters. Additionally, it is based on a prior job.

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1703053047704/28123445-cc00-4411-8458-21077220ef6f.png align="left")

configuration for job4

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1703053049626/7a1b4af6-efdb-4fd5-afb7-803665cba327.png align="left")

Execute the code in job4

**JOB 5 :**

This job will be responsible for hyperparameter tuning of the model and will make changes in the code in order to achieve the required accuracy. Then send the mail to the developer regarding the accuracy.

> Hyper Parameters are those parameters that are assigned before the building of the model like the number of filters, size of the filter, number of neurons, number of epochs, etc.

The same kind of machine learning model can require different constraints, weights, or learning rates to generalize different data patterns. These measures are called hyperparameters and have to be tuned so that the model can optimally solve the machine-learning problem. So to deal with this problem we can automate it where the hyperparameters will change and will make the accuracy of the model to the desire of the developer. So, instead of manually tuning our Hyperparameters, We are giving this job to Jenkins for Automation. Workflow of Process or we say in terms of Jenkins automation jobs

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1703053051850/ac194e46-8b71-443d-9c1d-3abdf8b0dea4.png align="left")

configuration of job5

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1703053053794/b00278cf-eaec-4366-bd91-c090b12f7fba.png align="left")

execute the code in job5

To check the accuracy of the tuned model we can check the below process.

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1703053055890/6e22b344-86c7-4306-b548-ab892c8b7ff2.png align="left")

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1703053058113/82de0e00-0a6e-4859-9afc-38a7b14124f9.png align="left")

Once the pipeline is done then you will get the below-related dashboard.

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1703053060327/b7fcb946-2269-4bb1-8333-c809aa5c6416.png align="left")

***Dashboard of all the jobs***

> I hope this article was informative and provided you with the details you required. If you have any questions while reading the blog, message me on [Instagram](https://www.instagram.com/acanubhav94/) or [LinkedIn](https://www.linkedin.com/in/anubhav-chaturvedi-a7465a72/). **Special credits to my team interns:** [**Jeet**](https://www.linkedin.com/in/jeet-limbachiya-1948281b0/), [**Kishan**](https://www.linkedin.com/in/kishan-patel-68265b14b/) **and** [**Krisha**](https://www.linkedin.com/in/krisha-amlani/)**.**

> Thank You…