Python Pandas Pivot Table

This blog is going to be mostly helpful to folks who work with excel and databases. But, if you work with Excel and find yourself needing to deal with automation of pivot tables, you might also appreciate the technique.

When working with MySQL database, it was not an easy way for me to fetch the data and reshape it to a Pivot Table. I did duckduckgo, found ample of examples but most of them were hard to understand or not yielding the required output.

Python’s Pandas Pivot Table was the savior!! I followed below steps to fetch data from MySQL DB and generate a simple Pivot Table. Later this pivot table exported as an excel file and shared with stakeholders using slack.

  • Create connection string to MySQL DB using PyMySQL
  • Create a SQLAlchemy Engine (ORM)
  • Write a Query statement to fetch data from DB
  • Create a Pandas Dataframe directly from the query using SQLAlchemy
  • Create a Pandas Pivot Table with required Index, Columns and Values
  • Export data to excel or csv file using to_excel and to_csv respectively
  • Share it to stakeholders via slack

Please find below Python code screenshots

I hope this blog triggered a thought to automate the mundane task of generating pivot tables from spreadsheets or databases. Please reach out to me if you want to check out the full code.

Thank you for stopping by this blog and please share your suggestions below under the Comment section

Up Next – Manage Alerts of VMAX3 array using REST API

Image Courtesy:

Matt Whitt
https://ttorial.com/Images/python-automation-automate-mandane-task-python.jpg

Advertisements

phpoffice/phpspreadsheet

phpoffice/phpspreadsheet is a successor of phpExcel. We recently developed an automated solution to pull all the backup success rate reports from Dell EMC Data Protection Advisor (DPA). The portal which displays the report was good but not enough to use that information further like sharing the reports, pivoting etc..

After trying several methods, found this library to be very useful. There aren’t many examples available on the internet to retrieve data from MySQL and write to XLSX file. Hope this blog might be helpful who would like to use this library for creating XLSX files. Fear not, it doesn’t throw errors while opening this file in MS Office Excel

VMAX SRP Utilization Report & Upload to MySQL DB using Python

I had an opportunity to attend Python Basics Training imparted by Mr. Ashish Gulati. Ashish, a technology coach; was really good in explaining basics of language, data types and Pro & Cons of this language. The way he imparted knowledge was an unique experience. He was flexible to explain some of the real time use cases in data analytics, JSON, SSH connectivity etc.. and explained in detail about the various module’s functionality.

Thank much Ashish for the session, it was informative, very simple topics but effective learning experience, hands on coding was a big plus point. I have learned much that will assist me in my workplace. As an outcome, I have already started migrating from Perl to Python.  This blog is about my first attempt to write Python scripts @ work.

Scripts written in Python would run from several VMAX3 Enterprise Storage Management Servers located at various data centers.

Python_vmax_cap

MySQL PHP Generating reports in PDF format

In continuation to my previous blog on charting epoch time series MySQL data using PHP. I started exploring generating better reports from the DB.

Inspired by the EMC SRM (M&R)  storage capacity reports in PDF format, I thought of doing similar kind of reports “in-house” which should be simple yet standard to be shared with customers. I would like to thank Sapan Kumar from EMC who thought me so many things about EMC’s SRM (M&R) custom reporting features.

We have the data in  MySQL DB and wanted to use the PDF module in PHP to generate reports. But which module is suitable and easy to use?

After rigorous testing with FPDF and DOMPDF, I decided to use FPDF (not sure why :p) and it produced nice reports indeed!

Please find sample reports screenshot produced below for reference. You could see the capacity reports of VNX and VMAX storage arrays.

2016-11-30-11_07_12-mremoteng-confcons-xml2016-11-30-11_03_57-mremoteng-confcons-xml

If interested, please reach out to me to get the PHP code. Thanks for stopping by… Please leave your comments / suggestions.

MYSQL PHP Charting epoch Time series data

I was writing PHP code to generate EMC VMAX performance reports / charts stored in MySQL DB. As we know that Unisphere REST API output performance metrics in epoch (13 digits) format (milliseconds). As an example, to draw a line chart for Frontend Director’s Read Response Time with X axis / Abscissa as epoch time. I’ve used below query to generate the datapoints to draw the chart using pChart

SELECT TIME(FROM_UNIXTIME(timstamp/1000)) as TIME, ROUND(AVG(perfval),2) as wrt FROM vmax_perf

where DAYOFMONTH(FROM_UNIXTIME(timstamp/1000)) = DAYOFMONTH(DATE_SUB(curdate(), INTERVAL 1 DAY))

and objname =’fe’ and perfparam =’ReadResponseTime’

group by HOUR(FROM_UNIXTIME(timstamp/1000))

Let me explain the above statement

TIME(FROM_UNIXTIME(timstamp/1000)) :: Extract Time from the epoch (13 digit) format in millisecond

ROUND(AVG(perfval),2) :: Average of Read Response Time per Hour rounded output to 2 digits after decimal point

DAYOFMONTH(FROM_UNIXTIME(timstamp/1000)) = DAYOFMONTH(DATE_SUB(curdate(), INTERVAL 1 DAY)) :: Extract day (ex. ’12’ from 12th Nov 2016 where current date is 13th Nov 2016) and display the data points

group by HOUR(FROM_UNIXTIME(timstamp/1000)) :: Print the datapoints for every hour

2016-11-13-14_27_22-10-76-6-62-remote-desktop-connection

If interested, please reach out to me to get the PHP code.

Stay Tuned for more updates on MySQL queries to generate charts for Daily, Weekly, Monthly and Quarterly performance reports / charts. Thanks for stopping by… Please leave your comments / suggestions.

Thanks for stopping by… Please leave your comments / suggestions.