The other day, I was using pandas to clean some messy Excel data that included several thousand rows of inconsistently formatted currency values. When I tried to clean it up, I realized that it was a little more complicated than I first thought. Coincidentally, a couple of days later, I followed a twitter thread which shed some light on the issue I was experiencing. This article summarizes my experience and describes how to clean up messy currency fields and convert them into a numeric value for further analysis. The concepts illustrated here can also apply to other types of pandas data cleanup tasks.
When dealing with continuous numeric data, it is often helpful to bin the data into
multiple buckets for further analysis. There are several different terms for binning
including bucketing, discrete binning, discretization or quantization. Pandas supports
these approaches using the
This article will briefly describe why you may want to bin your data and how to use the pandas
functions to convert continuous data to a set of discrete buckets. Like many pandas functions,
qcut may seem simple but there is a lot of capability packed into
those functions. Even for more experience users, I think you will learn a couple of tricks
that will be useful for your own analysis.
On September 17th, 2014, I published my first article which means that today is the 5th birthday of Practical Business Python. Thank you to all my readers and all those that have supported me through this process! It has been a great journey and I look forward to seeing what the future holds.
This 5 year anniversary gives me the opportunity to reflect on the blog and what will be coming next. I figured I would use this milestone to walk through a few of the stats and costs associated with running this blog for the past 5 years. This post will not be technical but I am hopeful that my readers as well as current and aspiring bloggers going down this path will find it helpful. Finally, please use the comments to let me know what content you would like to see in the future.
One of the most commonly used pandas functions is
read_excel. This short article shows how you
can read in all the tabs in an Excel workbook and combine them into a single pandas dataframe using
For those of you that want the TLDR, here is the command:
df = pd.concat(pd.read_excel('2018_Sales_Total.xlsx', sheet_name=None), ignore_index=True)
Read on for an explanation of when to use this and how it works.