In the digital world today, Data is as vital as life. Therefore, the easiness of interacting and extracting data from databases for modeling, visualization, and analytics is important to Data developers in making data-driven business decisions.

Learning Objectives

In this article, we will learn the following concepts :

• How to connect to a database with python using popular PostgreSQL database adapter, Psycopg.

• Perform tasks with DDL and DML statements with psycopg, and gain a high-level understanding of the package.

• Finally, utilize the Python package, ipython-sql to run SQL queries directly in a Jupyter Notebook.

The full codes for this demonstration can be found in the github repository

Prerequisites

The following will be needed to flow along with the demonstration:

• Have a source code editor or Python IDE install in your local system like VSCode, PyCharm, Sublime Text, etc.

• A good understanding of Python programming language.

• A PostgreSQL database installed within your local system or provision with docker.

• A basic understanding of SQL commands.

What is Psycopg?

Psycopg is the most popular PostgreSQL database adapter for the Python programming language. Its main features are the complete implementation of the Python DB API 2.0 specification and the thread safety (several threads can share the same connection).

What is DDL, DML and TCL?

DDL - Data Definition Language:

This is a set of statements that allow users to define or modify data structures and objects such as tables. These include CREATE, ALTER, DROP, RENAME, TRUNCATE.

DML - Data Manipulation Language:

This is a set of statements that allow users to manipulate the objects of a database. These include SELECT, INSERT, UPDATE, and DELETE.

TCL - Transaction Control Language:

This is a set of statements that allow users to save and manage the transactions issued to the database. These include COMMIT, ROLLBACK.

Connect to PostgreSQL with Python

Create a virtual environment:

Here, I will use VSCode terminal to create virtual environment for the project.

You can read up this excellent blog to learn how to create, activate and deactivate python virtual environment.

We might encounter error such as the picture below while activating our environment.

This is because, PowerShell by default restricts running scripts unless we change the execution policies. We can change the execution policies by adding-ExecutionPolicy Bypass args. in VScode. We will create a shell profile with these args by adding below in settings.json (ctrl + shift + p and type "settings.json")

"terminal.integrated.profiles.windows": {
  "PowerShell": {
    "source": "PowerShell",
    "icon": "terminal-powershell",
    "args": ["-ExecutionPolicy", "Bypass"]
  }
},
"terminal.integrated.defaultProfile.windows": "PowerShell",

We can then restart VSCode and the Terminal.

Install the Python package:

Install the required package, Psycopg2 required to connect to the database.

We will install psycopg2-binary, as it is stated in the official website that, the binary package is a practical choice for development and testing.

Below code can be used to install the package in the VSCode terminal:

pip install psycopg2-binary

Set up a PostgreSQL database with docker:

We will set up a database with docker-compose command. If you have a PostgreSQL database installed locally, you can follow along. However, if you wish to start, stop and delete databases without prior installation, you could check out the previous article on how to set up postgresql and padmin wth docker

Below code can be used to start up our database with docker:

docker-compose up

Below image shows the database dashboard with pgAdmin

Connect the database with psycopg2:

Here, we will wrap our code with try and except block to catch any error. We will use a variable conn to implement the connect method to connect to the database. Also, we will make sure to apply the close method to exit the database connection.

The connect method open a session in our database to perform database transactions like DML, DDL, DQL and so on. In order to perform any SQL transaction or statement, we will need to implement the cursor class.

Cursor allows Python code to execute PostgreSQL command in a database session. Cursors are created by the connection.cursor() method: they are bound to the connection for the entire lifetime and all the commands are executed in the context of the database session wrapped by the connection. We will also close the cursor when exit the program.

The best way to handle this exception is to put the closure of the conn object and cursor in the finally block, in case error surface in our connection. Also, if the database is not connected and a close method is applied to the connection object, there will be an error in our finally block. Therefore we declare the variables as None, and create an if statement to execute program.

import os

import psycopg2

user = 'admin'
password = 'password'
host = 'localhost'
db = 'demo_db'
port = 5432

conn = None
curr = None

try:

    conn = psycopg2.connect(
        host = host,
        dbname = db,
        user = user,
        password = password,
        port = port
    )

    cur = conn.cursor()

except Exception as error:
    print(error)

finally:

    if cur is not None:
        cur.close()

    if conn is not None:
        conn.close()

Perform SQL transactions

Perform DDL transaction

After successful connection, we will perform a DDL transaction by creating a table called employee in the database. We will create a script in our python code the same way, we put up the statement in our graphical interface of databases engine and apply the execute command.

After the execution, for the change to persist and save in the database, we have to apply TCL statement commit method on the conn - connection object made.

create_script = """ CREATE TABLE IF NOT EXISTS employee (
                                    id int PRIMARY KEY,
                                    name varchar(50) NOT NULL,
                                    sex varchar(6),
                                    salary int,
                                    dept_id varchar(10)) """
cur.execute(create_script)

conn.commit()

Perform DML transaction

INSERT STATEMENT

We will perform DML transaction with INSERT command to add some data to the employee table. Also, to avoid unnecessary errors during insertion because of some constraint such as PRIMARY and UNIQUE KEY specified in part of the attributes or columns, it will be require of us to DROP the table, re-create it and insert the data.

We will add a list of tuples of data and implement a for loop to insert each row of data into the table.

Below code can be used for the insertion:

cur.execute('DROP TABLE IF EXISTS employee')

insert_script = """ INSERT INTO employee (id, name, sex, salary, dept_id) VALUES (%s, %s, %s, %s, %s)"""

insert_values = [(1, 'Usman', 'M', 100000, 'D2'), (2, 'Esther', 'F', 50000, 'D1'),
     (3, 'Kingsley', 'M', 150000, 'D2'), (4, 'Toyyib', 'M', 103000, 'D1')]

for value in insert_values:
        cur.execute(insert_script, value)

We will refresh our table and query the database with pgadmin interface to confirm the insertion of the data into the table.

SELECT STATEMENT

Also, let's perform a SELECT transaction within our python code to view the data inserted. We can do this with print (cur.fetchall) method within cursor class. But, let's employ a for loop statement to view the data row-wise.

select_script = """ SELECT * FROM employee """
cur.execute(select_script)

 for record in cur.fetchall():
      print(record)

Sometimes, we may want to view part of the attributes of the table. Then, we can employ and import a module inside pyscopg2 extras.DictCursor to view our data in a dict format, a cursor that keeps a list of column name.

from psycopg2.extras import DictCursor
cur = conn.cursor(cursor_factory = DictCursor)

for record in cur.fetchall():
     print(record['name'], record['sex'], record['salary'])

UPDATE STATEMENT

We will also perform an UPDATE transaction within our python script.

update_script = """ UPDATE employee SET salary = 1.50 * salary"""
cur.execute(update_script)

We can also utilize the pgadmin to see the changes in the salary column of the employee table

Finally, we will wrap up the python script with context manager using with statement. A context manager in Python programming sets up a context, runs the code within the context and finally remove the context. The advantage of the context manager in this case will be; ensuring that the cursor object is close automatically as well as the connection is being committed after execution of the code. We will manually close the database connection.

Below code show the implementation of context manager.

# import the libraries
import psycopg2
from psycopg2.extras import DictCursor

# the parameters of the database
user = 'admin'
password = 'password'
host = 'localhost'
db = 'demo_db'
port = 5432

conn = None
try:

    with psycopg2.connect(
        host = host,
        dbname = db,
        user = user,
        password = password,
        port = port
    ) as conn:

        with conn.cursor(cursor_factory=DictCursor) as cur:

            # DDL TRANSACTION
            cur.execute('DROP TABLE IF EXISTS employee')

            create_script = """ CREATE TABLE IF NOT EXISTS employee (
                                            id int PRIMARY KEY,
                                            name varchar(50) NOT NULL,
                                            sex varchar(1),
                                            salary int,
                                            dept_id varchar(10)) """
            cur.execute(create_script)

            # DML STATEMENT OR TRANSACTION
            insert_script = """ INSERT INTO employee (id, name, sex, salary, dept_id) VALUES (%s, %s, %s, %s, %s)"""

            insert_values = [(1, 'Usman', 'M', 100000, 'D2'), (2, 'Esther', 'F', 50000, 'D1'),
            (3, 'Kingsley', 'M', 150000, 'D2'), (4, 'Toyyib', 'M', 103000, 'D1')]

            for value in insert_values:
                cur.execute(insert_script, value)

            update_script = """ UPDATE employee SET salary = 1.50 * salary"""
            cur.execute(update_script)

            select_script = """ SELECT * FROM employee """
            cur.execute(select_script)

            for record in cur.fetchall():
                print(record['name'], record['sex'], record['salary'])

except Exception as error:
    print(error)

finally:

    if conn is not None:
        conn.close()

Load CSV file into database with Python scripts

A python script can be utilize to ingest CSV file(s) into database. The full code for the script can be found in this Github repository.

We must ensure that the CSV file must be in the same directory with the python script with the library installed.

The script will clean the filename and columns to conform to SQL standard by changing the upper case to lower case and remove symbols, change data type to SQL type, create a table with its attributes with the CSV name, convert the date columns to date-time type and finally ingest the file into the database table.

The main function of the Python script is shown below:

def main(file_name : str, date_columns:list = []):
    """
    This runs the functions altogether and print necessary statement

    Parameters
    -----------
    file_name:  str
                Name of CSV file to be ingested 

    date_columns:   list, optional
                    List of date columns to be convert

    Returns
    ----------

    """

    table_name, df = clean_table_and_col_names(file_name, date_columns)
    col_to_str = change_dtype(df)
    conn = connect(parameters_dict)
    copy_from_csv(
        conn = conn,
        df = df,
        table_name = table_name,
        col_to_str = col_to_str
    )

# run this code below
main('employees.csv', ['HIRE_DATE'])

Run SQL in Jupyter Notebooks

This Python package ipython-sql is one of the nifty tools to come across as a data practitioner. This allow easiness of data developer to run jupyter notebook as a SQL IDE.

pip install ipython-sql

After installing the package, we can now execute any SQL query or procedure directly through the Notebook, and also store the result to a variable for further analysis.

Here are some general connection strings for various databases:

• PostgreSQL : postgresql://{user}:{password}@{host}:/{db}
• MySQL: mysql://{user}:{password}@{host}/{db}
• SQLite: sqlite:///name.db
• Oracle: oracle://{user}:{password}@{host}:1521/{db}

Here, we will use the PostgreSQL database connection string to connect to the database.

We will have to start a new Jupyter notebook session with the command below:

%load_ext sql

We will retrieve the data stored in the database's tables using the connection string with the command below:

# parameters
user = 'admin'
password = 'password'
host = 'localhost'
db = 'demo_db'
port = 5432

connection_str = f"postgresql://{user}:{password}@{host}/{db}"

%sql $connection_str

Then, for each query, we will start with %%sql in-line magic to tell the jupyter notebook that we will be running SQL statements.

Let's run some DML command

After the ingestion of the employees CSV file into the database, let's perform some SQL commands on it.

Check the table first five rows:

As we do in pandas, we always check the first five rows with head method. Let's run the SQL query to get the exact result with LIMIT clause.

%%sql

SELECT * FROM employees
LIMIT 5

Ordering with SQL statement:

This works like sort_order method in Pandas. SQL supports the ordering of columns with ORDER BY clause.

Question:

What are the names, date and job id of the employees whose salary is greater than 10,000?

%%sql 
SELECT first_name, last_name, hire_date, job_id, salary
FROM employees
WHERE salary >= 10000
ORDER BY salary DESC;

Grouping with SQL statement:

Transforming a table with aggregate function is one of the importance of groupby method in Pandas. SQL also provides a GROUP BY clause to achieve this functionality.

Question:

What is the count of staffs in each department?

%%sql

SELECT department_id, COUNT(*) as number_in_dept
FROM employees
GROUP BY department_id 
ORDER BY 2 DESC;

Conclusion

In this article, we have learned how to connect PostgreSQL database with python, perform DDL and DML SQL transactions, Ingest CSV file into database with scripts, and finally a Python package to run SQL queries in a Jupyter notebook.

We can now use the result after querying our database to make business decision.

Watch out for my forthcoming articles, where I will be discussing how to automate CSV file ingestion to the database using dockerized Python scripts.

I hope you have managed to get something useful from this article.

Let me know if you have any questions or suggestions, and I will be pleased to respond.

Thanks for reading.

Further Reading

Pyscopg documentation

SQL commands

You have probably heard a lot about docker for easy deployment of applications. It can also make our local development processes very fast and problem-free. Instead of maintaining local installation of databases, we can employ docker and its services.

Learning objectives

This article aims to demonstrate how to set up and run a fully functional platform with PostgreSQL and pgAdmin working on your local machine with the help of docker.

What is Docker?

In simple terms, Docker is a software platform or a tool that help to simplify the building, testing, running, managing and distributing applications along with their dependencies with the help of isolated environment known as containers.

It utilizes a concept called, Containerization to create multiple containers for applications in the same environment.

With docker, users can manage infrastructures in the ways applications are managed.

What is PostgreSQL and pgAdmin?

PostgreSQL is an Open Source Object-Oriented relational database management system (ORDBMS), a database engine, that uses and implement SQL (Structure Query Language) standards with many features that safely store and scale the most complicated data workloads.

pgAdmin is an Open Source development platform, and a graphical user interface administration tool for the PostgreSQL engine. It is one of the PostgreSQL client tools for manipulating schema and data on an instance or multiple instances of existing local or remote PostgreSQL servers.

Benefits of using PostgreSQL with docker for local development.

• Ability to start, stop, and remove containers when done with a project without installations and administration.
• Easy and Optimal way of working on multiple projects, side by side that depend on slightly different database versions.
• There is less confused disorderliness in the development machine.
• Experiment with multiple PostgreSQL projects without having a difficult time managing them.
• Generally, with docker as a service, if it runs on your machine, given its compatibility and version, it will run for your mate.

Prerequisites

Below are some prerequisites to note before diving into PostgreSQL with docker

• First, we will need to install Docker. I’ll use Docker Desktop for Windows installation for the demonstration.

• We will need to understand few terms in docker:

• Docker Images: This is a read-only template that contains a set of instructions for creating a container that can run on the Docker platform. Docker-hub has a variety of images. In this tutorial, we will be fetching PostgreSQL and pgAdmin images.

• Docker Compose: Compose allows defining and running of multi-containers Docker applications. First, we configure Compose’s services by using a YAML file. Afterward, we can start the services after configuration with a single docker-compose command.

• Docker run: The docker run command first creates a write-able container layer over the specified image and then starts it using the specific commands.

• Also, a basic understanding of how relational databases works would be beneficial.

PostgreSQL and pgAdmin with Docker

After successful installation of the docker machine, we can now commence our demonstration. First, As a developer, we make a new directory for every of our project. The following code can be used to create and change directories:

mkdir postgres_docker # make a new directory 
cd postgres_docker # change to the new directory

Now we can open our favorite text editor; in this example, VSCode will be used to demonstrate the tutorial.

Figure 1 :- VScode for code writing and debugging

There are two methods to deploy and run PostgreSQL and pgAdmin with docker.

Method 1 - docker run commands

• Create a bridge network:-

Figure 2 - Two standalone containers not connected.

Containers can only communicate with one another if they share a network. Containers that do not share a network cannot connect. Therefore, we will create a user-defined docker network to ensure communication between the containers while our applications are running in isolation. Once connected to a user-defined network, the containers can communicate easily using only another container’s IP address or name.

docker network create pg_net # This create network named  pg_net to ascertain communication

docker network ls # This lists all the  networks the Engine 'daemon' knows about

Figure 3 : docker network output

• Run the isolated containers with docker-run:-

The below codes can be used to run each container individually, and establish a connection between them with the --network specified command.

 docker run -it \
  -e  POSTGRES_USER="admin"  \
  -e  POSTGRES_PASSWORD="password" \
  -e  POSTGRES_DB="demo_db" \
  -v  {PWD}/demo_data:/var/lib/postgresql/data \
  -p  5431:5432 \
  --restart=always \
  --network=pg_net \
  --name pg_data \
  postgres:13

docker run -d \
  -e  PGADMIN_DEFAULT_EMAIL="admin@admin.com" \
  -e  PGADMIN_DEFAULT_PASSWORD="password" \
  -p  8080:80 \
  --network=pg_net \
  --name pg_admin \
  dpage/pgadmin4

Explanation of the parameters in the docker-run:

• -it:- This ensures the container run in interactive processes (like a shell).
• -e :- This allows users to configure environment variable for Database and pgAdmin superuser authentication.
• -v :- This ensures volume mounting of a local directory inside the container for persisting data/files generated by and used by Docker containers.
• -p :- Publish a container port 5432 to the localhost machine 5431.
• --network :- User-defined network to ensure easy communication of standalone containers using only another container’s IP address or name.
• -d :- This suggests the container run in the background and only the container ID is printed, i.e detached from showing in the cmd prompt.
• --name :- Define a name to ease referencing containers within a Docker network.
• --restart :- A restart policy for how a container should or should not be restarted on exit.
• postgres:13:- Specify the image to start the container. We use postgres:13, where postgres indicate the image and 13 indicates a version/tag of the software from the docker hub.
• -PWD :- This indicates the full pathname of the current working directory.

Figure 4 : The PostgreSQL container with docker run in interactive mode

Also, we can implement a Python library, pgcli, that serves as an interactive command-line interface for the PostgreSQL database server to confirm the establishment of the database engine.

Figure 5 : The pgcli command

Figure 6 : The pgAdmin container with docker-run in detached mode

After a successful run of the two containers, open http://localhost:8080/, the port number is based on the mapping we configured under pgAdmin docker-run code.

• Login with your default mail and password to the pgAdmin
• Click Add New Server
• Insert a name on the General page, Name: db
• Click on Connection:
  • Hostname/Address: pg_data # name of the container
  • Username: admin # username of the database
  • Password: password # password to the database
• Save

Figure 7 : Database dashboard after successful connection.

Stopping Docker Containers

The code below can be used to list active containers, stop and remove the containers

docker ps

docker stop container_id

docker rm container_id

Figure 8 : docker command to stop, remove and list containers

Method 2 - docker-compose commands

Starting a container one at a time with docker run, creating a communication network between the containers, and writing a command intimidatingly long all add up to stress. An alternative approach can alleviate all the stress.

A docker-compose command that codes all runtime configuration data in an aptly named YAML file called docker-compose.yaml can be employed to start multiple containers simultaneously. It also helps with the creation of a default network that connects the services.

First, we will utilize an environment variable .env file, a key-value pair, text file that provides a better way to store, secure, and manage our environment variables defined one per one. The format of a .env file is the same under all operating systems, so make working with environment variables uniform across all platforms. Many times our .gitignore files will have .env files to avoid accidentally sharing the file within the git committed files.

Figure 9 : The environment variables in .env file

Below is an example of the services' configuration file in docker-compose.yaml file

version: '3.5'

services:
  pg_data: # A string that specifies a custom container name, rather than a generated default name
    image: postgres:13  
    environment:       
      - POSTGRES_USER=${POSTGRES_USER} # take its value from .env file
      - POSTGRES_PASSWORD=${POSTGRES_PASSWORD}
      - POSTGRES_DB=${POSTGRES_DB}
    volumes:            
      - ./demo_data:/var/lib/postgresql/data
    ports:             
      - "5432:5432"
    restart: always

  pgadmin:
    image: dpage/pgadmin4 
    environment:         
      - PGADMIN_DEFAULT_EMAIL=${PGADMIN_DEFAULT_EMAIL}
      - PGADMIN_DEFAULT_PASSWORD=${PGADMIN_DEFAULT_PASSWORD}
    volumes:           
      - ./pgadmin_data:/var/lib/pgadmin
    ports:              
      - "8080:80"
    restart: always
    depends_on:       
      - pg_data

volumes:
  demo_data:
  pgadmin_data:

Explanation:

• depends_on: Define the dependencies of services. This allow us to deploy service in order; e.g. postgres -> pgadmin.

• services: this codifies an approach of how the image should be running such as database, port number, executable scripts and so on.

• image: Specify the image to start the container from. In this case, we are going to use dpage/pgadmin4, without tags, we meant latest version from Docker Hub.

Running Docker compose

First, we will check and validate the YAML file by executing below command in our terminal.:

docker-compose config

Figure 10 : The data compose config output.

Next, we will deploy and run the services by executing the command below:

docker-compose up -d

Figure 11 : The data-compose up -d output.

Open pgAdmin

After successful running of the docker-compose services, open http://localhost:8080/ on a browser. Then enter the email and password configured using the environment variable: PGADMIN_DEFAULT_EMAIL and PGADMIN_DEFAULT_PASSWORD.

Figure 12 : pgAdmin login page

Set up Database

Click Add New Server, Insert any Name under General. The value can be anything preferred.

Figure 13 : Server name

• Click on Connection
  • Hostname/address: database service name, in our case is pg_data.
  • Username: environment variable POSTGRES_USER.
  • Password: environment variable POSTGRES_PASSWORD.

Figure 14 : Server connection

And click Save to finish creating. Afterward, we can get access to the database dashboard from the sidebar.

Figure 15 : Database dashboard.

Stopping Data Compose

To stop the Docker Compose service, the below code can be executed:

docker-compose down

Figure 16 : docker-compose down output.

Conclusion

In this guide, we have learned how to use docker run to start PostgreSQL and pgAdmin individually, connect them with a user-defined network for communication, and then use docker-compose to combine them.

We can now use pgAdmin to create and store databases, queries, and tables directly in PostgreSQL, a convenient approach to administering a database. We can also utilize the graphical user interface pgAdmin to import CSV (Comma Separated Values) files into our databases. We can achieve this by writing queries that reference the CSV file path, or by creating a table in pgAdmin and matching the table's column headings to the CSV's columns.

We can also use tools like pgcli, SQLAlchemy, and Psycopg2 to connect to the database from a Python application on our local system. With the Python Pandas package, we'd be able to write queries to tables in our databases and display the results in table format.

Watch out for my forthcoming articles, where I will be discussing how to connect our databases with the libraries aforementioned and use Python scripts to automate CSV file ingestion to the database.

Let me know if you have any questions or suggestions, and I will be pleased to respond.

Thanks for reading.

Connect with me on Twitter || LinkedIn || Github

Further Reading

Containerization

docker-compose

docker network

Environment variable

Docker Tutorial for beginners

Data scientists create models that represent and predict real-world data, yet about 95% of these models get lost in notebooks. To guarantee that these models meet their requirements for real-world applications, like as within a web framework or an Android application, they must be deployed effectively so that they can acquire the necessary data and make accurate predictions.

We'll use the cloud (Amazon Web Services) and a Flask application to deploy our machine learning model for better user engagement in this example.

The full codes for this deployment can be found in the github repository

Step by Step Deployment Processes of ML Models on a Flask Application with AWS EC2 Instance

1. Creating a Machine Learning model:

With Jupyter notebook, creating a machine learning model is simple. We'll use the Life Expectancy Dataset to build a model that can forecast a country's average life expectancy at birth based on a variety of factors. The country, population, GDP, percentage expenditure, total expenditure on health, status, and others are among these characteristics. These datasets were obtained from Kaggle datasets and World bank database.

The problem statement in our case study follows the Regression pattern because it produces a numerical estimate. After completing the ML processes of data cleansing, wrangling, exploratory data analysis, outlier elimination, feature engineering, machine learning pipeline, and model building using several algorithms, the LightGBM model was found to be the best.

# import the necessary libraries
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns

from sklearn.preprocessing import OneHotEncoder, LabelEncoder, StandardScaler
from sklearn.model_selection import train_test_split, StratifiedKFold, cross_val_score, KFold
from sklearn.pipeline import make_pipeline

from sklearn.linear_model import LinearRegression
from sklearn.tree import DecisionTreeRegressor
from sklearn.ensemble import RandomForestRegressor, 

from catboost import CatBoostRegressor
from xgboost import XGBRegressor
from lightgbm import LGBMRegressor

from sklearn.metrics import r2_score, mean_squared_error, mean_absolute_error

#Load the dataset 
countries = pd.read_csv('./Data/Metadata_Country.csv',encoding='latin1') # data from world bank
life_expectancy = pd.read_csv('./Data/Life Expectancy Data.csv') # kaggle dataset

# Machine learning pipeline for algorithm implementation
from collections import OrderedDict
models = OrderedDict([
    ( "Linear Regression",       Pipeline([
                                            ('preprocessor', preprocessor),
                                            ('LRegressor', LinearRegression())])  ),
    ( "Decision Tree Regressor", Pipeline([
                                           ('preprocessor', preprocessor),
                                            ('DTRegressor', DecisionTreeRegressor())])  ),
    ( "Random Forest Regressor", Pipeline([
                                           ('preprocessor', preprocessor),
                                            ('RFRegressor', RandomForestRegressor())])  ),
    ( "Catboost Regressor", Pipeline([
                                           ('preprocessor', preprocessor),
                                            ('CatBoostRegressor', CatBoostRegressor())])  ),
    ( "Xgboost Regressor", Pipeline([
                                           ('preprocessor', preprocessor),
                                            ('XGBoostRegressor', XGBRegressor())])  ),
        ( "Lightgbm Regressor", Pipeline([
                                           ('preprocessor', preprocessor),
                                            ('LightgbmRegressor', LGBMRegressor())])  )

])

# Sorting the algorithm by the coefficient of determination "R²"
score_df = pd.DataFrame.from_dict(scores, columns=['R²'], dtype = "float", orient="index")
score_df.sort_values(by="R²", ascending=False)

A code snippet from VS code that shows the Jupyter notebook, python libraries and the algorithms implementation.

The image above shows the rank of the algorithms

2. Creating a Pickle file to save the model:

Pickle is a standard library in python which is used in serializing machine learning algorithms and saving it to file format .pkl in Python.

import pickle
#save the trained model
pickle.dump(pipeline_LGBM, open('models/pipeline_LGBM.pkl','wb'))

3. Creating a Website templates:

It is necessary to develop a website to collect data from users to make predictions. This folder contains directories, templates - which contains the HTML template and forms that allow users to enter the corresponding variables and see the anticipated average life expectancy - and static - which provides the CSS-styling sheets and JavaScript required for the interface.

<!DOCTYPE html>
<html lang="en">

<head>
    <!-- ========== Meta Tags ========== -->
    <meta charset="utf-8">
    <meta http-equiv="X-UA-Compatible" content="IE=edge">
    <meta name="viewport" content="width=device-width, initial-scale=1">
    <meta name="description" content="ML Prediction Page">

    <!-- ========== Page Title ========== -->
    <title>ML Prediction page</title>


    <!-- ========== Start Stylesheet ========== -->

    <link href="{{ url_for('static', filename ="assets/css/bootstrap.min.css")}}" rel="stylesheet" />
    <link href="{{ url_for('static', filename ="assets/css/flaticon-set.css")}}" rel="stylesheet" />


</head>

<body>


    <!-- Start Banner 
    ============================================= -->
    <div class="banner-area text-center text-normal text-light shadow dark bg-fixed" style="background-image: url('/static/1.jpg');">
        <div class="box-table">
            <div class="box-cell">
                <div class="container">
                    <div class="row">
                        <div class="col-md-8 col-md-offset-2">
                            <div class="content">
                                <h1>This App implements Machine Learning to predict results which had been trained with Life Expectancy datasets.</h1>
                                <p>
                                    The model learns from the data and predicts 96% accuracy.
                                </p>

                                <button class="btn-success btn-sm" style="border-radius: 16px;" onclick="location.href='/predict'">Click Here To Predict</button>
                            </div>
                        </div>
                    </div>
                </div>

            </div>
        </div>
    </div>
    <!-- End Banner -->

    <!-- Start Companies Area 

    <!-- jQuery Frameworks
    ============================================= -->
    <script src="{{ url_for('static', filename ="assets/js/jquery-1.12.4.min.js")}}"></script>
    <script src="{{ url_for('static', filename ="assets/js/bootstrap.min.js")}}"></script>



</body>
</html>

4. Creating a Flask Application:

Flask is a Python-based web application framework. It has a wide range of options and libraries for creating web applications. It makes it easier to call Restful APIs using Python.

# import required libraries
from flask import Flask
from flask import render_template, request, url_for, jsonify
import pickle
import pandas
import os
import numpy 

# Assign parameters such as static folder names- static

STATIC_DIR = os.path.abspath('./static')

#initialize the flask app
app = Flask(__name__,static_folder=STATIC_DIR)

# load the data via pickle file for prediction
model = pickle.load(open('models/pipeline_LGBM.pkl','rb')) 

# state the columns for imputing variables
column =['country', 'year', 'status', 'adult_mortality', 'infant_deaths',
       'alcohol', 'percentage_expenditure', 'hepatitis_b', 'measles', 'bmi',
       'under-five_deaths', 'polio', 'total_expenditure', 'diphtheria',
       'hiv/aids', 'gdp', 'thinness__1-19_years', 'thinness_5-9_years',
       'income_composition_of_resources', 'schooling', 'region', 'incomegroup',
       'Population']

# rendering the homepage template with index function
@app.route("/")
def index():
    return render_template("Home.html")

# rendering the predict.html as it contains the parameters needed for the prediction.
# it requires both the GET for the default value prediction and POST request for prediction if pred.html forms is filled by the User 
@app.route('/predict',methods=['GET','POST'])
def predict():
    '''
    For rendering results on HTML GUI
    '''

    #creates the single function for getting data

    req = request.form
    country = req.get("country",default='Nigeria')
    year = int(req.get("year",default=2021))
    status = req.get("status",default= 'Developing')
    adult_mortality = int(req.get("adult_mortality",default=343))
    infant_deaths = int(req.get("infant_deaths",default=72))
    alcohol = float(req.get("alcohol",default=13))
    percentage_expenditure = float(req.get("percentage_expenditure",default=2300))
    hepatitis_b = float(req.get("hepatitis_b",default=88))
    measles = float(req.get("measles",default=6718))
    bmi = float(req.get("bmi",default=25))
    under_five_deaths = int(req.get("under_five_deaths",default=114))
    polio = float(req.get("polio",default=80))
    total_expenditure = float(req.get("total_expenditure",default=7))
    diphtheria = float(req.get("diphtheria",default=24))
    hiv_aids = float(req.get("hiv_aids",default=100))
    gdp = float(req.get("gdp",default=480000))
    thinness_10_19_years = float(req.get("thinness_10_19_years",default=13.4))
    thinness_5_9_years = float(req.get("thinness_5_9_years",default=13.2))
    income_composition_of_resources = float(req.get("income_composition_of_resources",default=0.52))
    schooling = float(req.get("schooling",default=10.9))
    region = req.get("region",default='Sub-Saharan Africa')
    incomegroup = req.get("incomegroup",default='Lower middle income')
    population = float(req.get("population",default=214140000))


    #storing the data in array format
    array = numpy.array([country,year,status,adult_mortality,infant_deaths, alcohol,percentage_expenditure, hepatitis_b, measles, bmi, under_five_deaths, polio, total_expenditure,diphtheria, hiv_aids, gdp, thinness_10_19_years, thinness_5_9_years, income_composition_of_resources, schooling, region, incomegroup, population]).reshape(1,23)

    #creates a dataframe to hold the data
    data = pandas.DataFrame(data=array,columns=columns)

    #predict over the features gotten from the user with the model
    prediction = model.predict(data)
    output = round(prediction[0])

    #passing value gotten to Predict html template for rendering
    return render_template("Predict.html",prediction_text='The Average life expectancy for {} in year {} is {} years'.format(country, year, output))    

# port values changes as we run on local server and in the cloud

if __name__ == '__main__':
   app.run(host = '0.0.0.0', port = 8080) # cloud

To run this in our local environment, navigate to the directory using CMD- command prompt and type python app.py.

5. Creating an AWS account:

Once our ML models have been deployed successfully locally, we will move on to our main goal: leveraging cloud services to deploy our ML models via AWS. We'll need to register and create an AWS account. We will acquire a year of free tier service by using our credit card information. This will be beneficial in accomplishing the task at hand.

After successfully registering, we must sign in with our user ID to access the AWS Management Console.

6. Downloading the Necessary Softwares:

To accomplish our aim, there is a need to download two software required to ensure our application runs successfully by connecting our local environment with AWS EC2 Instance. These includes:

I. PuTTy:

This is an open-source free terminal emulator, serial interface, and network file transfer program. Its primary advantage is to allow users to run supported protocols, including SSH and Telnet, remotely. This comes with another open-source networking client PuTTygen - a key generator tool for creating pairs of public and private SSH keys. This is used to convert our private SSH key from PEM file to PPK file.

After successfully installation, you can search for PuTTygen in your local machine as it looks like the image below

II. WinSCP:

This is a popular free SFTP and FTP client, a powerful file manager for Windows. It offers an easy-to-use GUI to secure files transfer between a local and remote computer using multiple protocols: Amazon S3, FTP, FTPS, SCP, SFTP, or WebDAV. The application looks like the image below after successful installation.

7. Creating an AWS EC2 Instance:

A search for EC2 Instance can be done through the AWS Management Console. The image below depicts the AWS EC2 Dashboard.

The following steps will ensure a successful launch of an AWS EC2 Instance.

I. Choosing the AMI template:

AMI - Amazon Machine Image is a template that contains the software configuration (OS), application server and applications required to launch EC2 Instance. In this case, we are going to be choosing the free tier template- Ubuntu Server 20.04 LTS (HVM), SSD Volume Type.

II. Choosing an Instance Type:

An instance type that suit our free tier will be chosen to avoid charges from our credit cards. Below is an image that shows a free tier instance type- t2 micro eligible.

III. Configuring the Security Group:

In AWS, security groups are the foundation of network security. They manage the flow of traffic into and out of the EC2 instance. We'll skip to the security group setting because the other default steps are sufficient for our deployment. Because we want our program to operate on several screens with the host-name, we must change the type to All traffic and the source to Anywhere in this situation. The configuration is shown in the image below.

IV. Reviewing and Launching:

We inspect the processes and launch our Instance after carefully configuring the security group. Here, we'll create a new key pair that will allow us to connect to the instance securely. We'll rename it to our liking, then download the key pair and store it in the project's directory folder. The graphic depicts key-pair creation, renaming, and downloading.

V. EC2 Instance launch successfully:

We'll proceed to start the EC2 instance, which may take a few seconds. In the EC2 Dashboard, here's a screenshot of a successful launch of an EC2 Instance.

8. Creating a Private Key PPK File with PuTTygen:

We'll create a private PPK file by converting the PEM file that was downloaded when the instance was launched and loading it with PuTTygen. To retrieve our PEM file from our project folder, load and store the private key generated into our folder, we indicate that all files are presented. The screenshot below depicts the creation and saving of the PuTTygen private key.

9. Securing Transfer of project files via WinSCP:

WinSCP, as previously stated, aids in the secure transfer of files from a local workstation to a remote computer. This will allow us to drag and drop the files/codes we need into the EC2 Instance we've just established. To begin, we'll start WinSCP, which will prompt us to provide a host name.

Second, we'll get the host name from the EC2 dashboard by clicking on the connect tags, then copy and paste the Public DNS from the SSH client. In WinSCP, we'll use this as our host name.

The images below show the EC2 Dashboard's imputation of the host name.

Now we'll go to WinSCP's Advanced Site Settings, find SSH, and click on Authentication to load our private key PPK file generated with PuTTygen.

After that, we select Ok, Create a Username. For my user name, I went with ubuntu. After that, we log in to connect to our Ubuntu server. We also choose our code files by dragging and dropping them into the WinSCP windows on the right side. The amount of time it takes to transfer files is determined by their size. The requirement.txt file is included in these files to allow the installation of the python libraries required to execute the model.

Note: When deploying our model outside of our local system, the IP address and Port number in the Flask app python code must be modified from the defaults of '127.0.0.1' and '5000' to '0.0.0.0' and '8080' to enable smooth execution of the app.py in the cloud using PuTTy terminal.

10. Installing the required libraries with PuTTy:

To begin the configuration of the PuTTy app, we will locate it on our system. We'll start by imputing our hostname, as we did before, and then name the Saved Session. Go to the SSH, locate the Auth and load the saved private PPK file. After that, we go to Session category to save the name, click on it, and open it with PuTTY. The configuration, name, and opening of the saved file are shown in the image below.

We'll communicate with our terminal to confirm that all essential files are uploaded successfully. The screenshot below shows how to use the Git bash to interact with Ubuntu Server.

We must first update and install python with the command below to successfully install our libraries into our Ubuntu cloud server.

sudo apt-get update && sudo apt-get install python3-pip

Also, to install the required python libraries needed to run the model i.e requirement.txt file, we can execute this by running the command.

python install -r requirement.txt

11. Running the Flask application in the ubuntu terminal::

Now that our libraries have been installed successfully, we can execute the program by typing the following command in the terminal.

python3 app.py

The application has now begun to run. Then, to view our application, we'll go to the EC2 dashboard and locate the connect tags, then use the SSH client to copy the Public DNS.

The URL for our application will be Public DNS + port 8080, as seen in the example below:

ec2-34-201-245-241.compute-1.amazonaws.com:8080

The home web page of the application is shown below:

The prediction web page of the application is display as the snapshot below:

12. Final launch of the application:

It was observed that if the terminal closed, and a new page was opened to restart the application, an error occurred. The following error surfaced::

To resolve this problem, we'll need to employ GNU Screen, a full-screen window manager that divides a physical terminal into numerous processes, most commonly interactive shells. Our application will continue to operate even if the window is not visible after disconnection, and our server will remain active by establishing numerous displays even when the git bash terminal is closed.

When the code below is run, it allows the URL to be opened in several browser pages.

screen -R deploy python3 app.py

The code is implemented in the git bash terminal as seen in the image below.

Brief summary:

I learned the following from this writing:

• The steps involved in performing an end-to-end machine learning project.

• Using the pickle python package to save a machine learning model as a ".pkl" file.

• A small-scale web application using the Flask back-end python library.

• The processes for launching an Amazon Web Services EC2 instance.

• Additional software such as PuTTy and WinSCP are introduced for successful file deployment and transfer from a local machine to a remote computer.

• Using the Git bash Ubuntu terminal to install libraries and execute the flask application.

Conclusion:

In this way, we can leverage the cloud services (AWS EC2 Instance) and flask web application to deploy our Machine learning model.

Thanks for reading my article. I would very much appreciate your feedback, suggestions, and critics of my approach to writing. I'm open to collaborations that will further increase my knowledge and growth in ML. Feel free to connect with me on LinkedIn.