How To Add New Columns To A DataFrame Using Python

Introduction

In this article, you will learn how to read Microsoft SQL table, add the results to DataFrame, add new columns to a DataFrame and export DataFrame to excel. I have created a simple Microsoft SQL table for this demo named as Employees which contains the following columns and rows.

I have a requirement to add 2 more columns named as Date and Code to the DataFrame. The values for these columns will be static and same for all the rows.

• Date – today’s date
• Code – value is retrieved from the query string parameter

The output looks like the below:

Topics Covered

This article demonstrates how to build the following:

• Create the project folder
• Install the required packages
• Implement the REST API
• Test the API

Pre-requisites

1. Windows machine and Visual Studio code are used for this article. Complete all the prerequisites mentioned in this article.
2. Install the Microsoft ODBC Driver for SQL Server on Windows.

Tools

1. Visual Studio Code

Task 1: Create the project folder

In this task, you will see how to create the project folder.

Step 1

Open Windows Command Prompt and run the following commands to create the new folder for this project.

mkdir AddcolumnsToDataFrame
cd .\AddcolumnsToDataFrame

Task 2: Install the required packages

In this task, you will see how to install the required packages for this project.

Step 1

Open AddcolumnsToDataFrame folder in Visual Studio Code. Click Terminal -> New Terminal.

Step 2

Run the following command to install the packages required for this project.

pip install flask
pip install flask-restful
pip install pyodbc
pip install openpyxl

• flask is a simple framework for building complex web applications.
• flask-restful is an extension for Flask that adds support for quickly building REST APIs
• pyodbc is an open source Python module that makes accessing ODBC databases simple.
• openpyxl is a Python library to read/write Excel 2010 xlsx/xlsm/xltx/xltm files

Task 3: Implement the REST API

In this task, you will see how to implement the REST API to read the Microsoft SQL table, add the results to DataFrame, add new columns to a DataFrame and export the result to excel.

Step 1

In Visual Studio Code, create a new file app.py under AddcolumnsToDataFrame folder.

Step 2

Copy and paste the below code. Note: Update the connection string.

from flask import Flask, jsonify
from flask_restful import Resource, Api, request
import pyodbc 
from datetime import date
import pandas as pd

# Input Parameters
getdatacmd='SELECT id AS EmployeeId, name AS Name, designation AS Designation, location AS Location FROM Employees'
connstring='DRIVER={ODBC Driver 18 for SQL Server};SERVER=*******\SQLEXPRESS;DATABASE=demo;ENCRYPT=no;Trusted_Connection=yes;'
today=date.today().strftime('%d.%m.%Y')
filename='exported_data'+today+'.xlsx'

# Create the flask app
app = Flask(__name__)
# Create an API object
api = Api(app)

# Class for GetData
class GetData(Resource):
	# GET Request
	def get(self):
		args = request.args
		cnxn = pyodbc.connect(connstring)
		sql_query = pd.read_sql_query( getdatacmd,cnxn)
		df = pd.DataFrame(sql_query)
		df = df.assign(Date = today, Code=args['code'])
		df.to_excel(filename, index = False)
		return filename + ' is exported successfully'
		
# Add the defined resources along with their corresponding urls
api.add_resource(GetData, '/')

# Driver function
if __name__ == '__main__':

	app.run(debug = True)

Task 3: Test the API

In this task, you will see how to test the API that generates the excel file.

Step 1

In Visual Studio Code, run the following command in the Terminal.

python .\app.py 

Step 2

Open the browser and access the below URL. It generates an excel file in the project folder .\AddcolumnsToDataFrame.

http://127.0.0.1:5000/?Code=AX001

References

1. Flask-RESTful documentation
2. Python SQL Driver - pyodbc

Summary

This article describes how to read Microsoft SQL table, add the results to DataFrame, add new columns to a DataFrame and export DataFrame to excel.