Practical Business Python

Taking care of business, one python script at a time

Wed 30 December 2015

Getting to the “Plateau of Productivity” with Python

Posted by Chris Moffitt in articles   


As we close out the year, I wanted to take a step back and write a post that will motivate people to learn python and apply it to their daily jobs. Based on some comments I’ve received (and my own personal observations), some people struggle to get started on this journey. They see the potential value of using python in their jobs but are not sure where to start and can not find the time to take the first steps. Closely related to this challenge is finding the perseverance to make it through the inevitable barriers you will encounter. My goal in this article is to provide some items to keep in mind so that you can be successful in your endeavors to learn python and apply it to your job. If you take the time (definitely no easy task) to develop your python skills, you can reap many benefits - outside of the obvious ones you may have started out seeking.

Knowledge is Power

Many people have probably heard of the hype cycle.

Technology Hype Cycle

This graph is generally used to illustrate the hype associated with new technology in the market. Think of ideas such as 3D Printing, Nano-technology or anything “cloud.” All these technologies (and many more) lie somewhere on this continuum. I contend that this hype cycle can apply to your own personal journey as you endeavor to learn a new technology.

Ok, so maybe it can apply to your own technical journey with python (or some other technology), but why should you care? For people of a certain age, do you remember the old GI Joe cartoon and the motto at the end?

GI Joe

I am not sure if knowing is truly half the battle but I do know that more knowledge about what you are getting yourself into will help you prepare for the challenges you will face as you try to learn something new.

Beware the Hype

For this article, we will apply the stages of the hype curve to a hypothetical person who has never typed a line of python code in their life.

Technology Trigger
For some reason, this person has realized that they need to automate some of the tedious processes they do on a daily basis. After some google searching, they learn that python might really be useful.
Peak of Inflated Expectations
Now that they know python is a thing, they start researching it. They become excited about all the big companies using python, the rich library of 3rd party tools, the promise of time savings and the joys of coding. They immediately start to see all the problems they could solve with python. Their excitement builds and this is where they believe so many things are possible!
Trough of disillusionment

As this person starts to delve into the python world and apply it to their problem, it’s not quite as easy as they think. They have a lot to figure out.

  • How do they install python?
  • Which version should they use?
  • How do they even decide how to tackle their problem?
  • How do they get the data in a easy to use format?
  • Where do they find the time to spend on this “side” project?
  • Probably 100’s of other challenges both big and small.

At the end of the day, I think a lot of people dabble with python, get stuck here and never move forward.

Slope of Enlightenment
However, if they persevere, tackle the right problems and have the right goals; they start to climb out of the trough and see how python can help. Slowly but surely, they climb out of this valley and start to apply their newfound knowledge to more and more problems. This is where they start to see the light at the end of the tunnel.
Got That Goin For Me
Plateau of Productivity
At the end of the day, the new pythonista reaches a level where they are more productive with their python tools than they were before. They know which tools to use and when to use them. They start to understand their business problems better and deliver higher quality results with less time and energy. In this model, it’s called a “plateau” but I would contend that someone can continue to grow here for years and years. No one ever “arrives.” There are always options to learn and grow.

If we understand the hype cycle correctly, then it is important to find ways to to push through the Trough of Disillusionment. I believe that if a person new to python or new to the tools discussed in this blog keeps this in mind, they will be more successful in their journey. The rest of this article will talk through some things to keep in mind in order to make it through this process.

Take The Easy First Step

I think the most important item to consider is picking a good starting point for your first python project. Because python is so powerful and you may be climbing towards the Peak of Inflated Expectations, there is a tendency to try to tackle a really challenging problem. Do not start off trying to apply the latest machine learning algorithms to your company’s customer purchase patterns! Rather, take a simple process that you know really well and try to automate it.

Starting with a well-known and simple problem has some advantages:

  • Since you know the problem space, you can focus on learning python.
  • By comparing your python results with your known results, you can ensure you are delivering the right results.
  • Everyone needs to figure out their coding environment and best practices. Take the time to get your environment set up correctly so it’s maintainable in the long term. Review this best practices article for some ideas to keep in mind.

Give Yourself Permission to be Inefficient

The other important item is to give yourself permission to spend time on the problem. Let’s say you choose to automate a fairly simple file manipulation process that takes you 10 minutes a day. As you start automating it, you will probably take several hours to get the project where it needs to be. That is ok! Remember, you may find yourself in the Trough of Disillusionment. Do not stop, keep going!

How many times have you heard people say something like “I have too much work to do” or “I’m too busy to improve this process?” We have all done it. It is easy to keep doing the same old thing and not break out of your comfort zone. If you want to improve a process, you need to realize that the short term time investment will pay off in the end. It is ok to take more time as you start this process.

Keep an Eye Out For Unexpected Wins

As I have climbed the Slope of Enlightenment with my own projects, one of the most surprising items I have found is that many times the benefits I thought I was going to get were not as important as the benefits I did receive. In other words, automation will yield more improvements than just time savings.

Also, I will be the first to admit, that there are times when automation may not be the right thing to do. Not surprisingly, xkcd has a comic about this as well.

Technology Hype Cycle

So, if sometimes the automation does not work out or it is more work than expected, what are some of the unexpected wins you might get?

Offloading the Task to Someone Else

Highly manual tasks fall on someone’s shoulders because “they have always done it.” It may be difficult to hand the task off to a coworker because there is no documentation (or no time to make the documentation). I have found though that once you automate a process, it is much easier to hand it off to someone else. By handing it off, you free up your time. That’s one way you can start climbing the Slope of Enlightenment.

Understanding Your Data Better

When you start automating your processes, you may be forced to deal with some of the ugliness of your data. I don’t think there are very many people that started on a data automation project that did not need some level of data cleaning. You will frequently find that there are opportunities to improve upstream systems to filter out the data before it gets to you. Maybe you can help out others that don’t even know there are problems with their data.

Additionally, as you start to manipulate your data and build solutions, you get a much better feel of what data is available to you and how easy (or difficult) it is to bring multiple data sources together. Overall, I believe that understanding your data at the detailed level can help you understand your business better. If nothing else, it will make you much wiser on your next automation project as you move into the Plateau of Productivity.

Increasing Accuracy

Almost any time you automate a process, you will build increased accuracy or quality in your output. The interesting side effect is that when the data is correct and your output is correct, it is relied upon more and more. Sometimes, people know that reports are “wrong” but choose to suffer the bad results instead of working through the issue and resolving it. Once you iron out those small problems, people will start to appreciate your solution and incorporate it in their daily processes.

Increasing the Frequency of the Task

Let’s say you have to run a manual Excel report once a week and it takes you 1 hour to pull it together and distribute it. This hour is not a huge investment but it is painful enough that you don’t want to run it every day. Once you automate it, you are much more willing to run it as frequently as the business needs.

Minor Changes are Easy

A highly manual process is not very robust. Because it is not very robust, it is difficult to make minor changes to it. I have had many experiences where automation supports incremental changes that were not possible before. Sometimes it is as simple as a minor column name change or breaking reports into multiple sections. Sometimes it is a more complex solution that is difficult to implement in Excel but is trivial in python. Either way, there is more flexibility with a robust code-based solution (stored in version control) than a manual process stored in someone’s head.

Fixing Issues Can Be Easier

A lot of times identifying and fixing bugs are much easier with an automated solution. To illustrate this item, here’s a real life issue I recently experienced. As part of a process to create ~100 Excel files, a user could not find one of the files. I looked through the list of files and realized immediately what the problem was.

The code looked something like this:

default_file_name = "{}-report-{:%m-%d-%Y}.xlsx".format(LName,

Obviously this is going to cause a problem if I have someone named John Smith and Fred Smith. The solution was simple- add their first name into the naming convention.

default_file_name = "{}-{}-report-{:%m-%d-%Y}.xlsx".format(FName, LName,

Granted this is a somewhat simplistic example but it highlights the point that it is easier to troubleshoot this issue than to figure out if I just forgot a name when creating these files by hand. I also know it is fixed for good and I do not need to remember another “manual workaround.”

Major Improvements May Not Require Major Re-work

The python environment will give you options that were not available to your manual process. If you want to integrate with other services or slice and dice multiple files, you’ll have a full python environment available to you. As you build your skills, you will find that there is less need to manually do things. At this point, you are truly moving up the slope of enlightenment.

Do Not Underestimate the Basic Knowledge You Will Learn

Sometimes when you are struggling to figure out some python syntax or get a bug resolved, you may not realize how much you really have learned. The big trick to make it through the Trough of Disillusionment is to recognize how far you have come and how much knowledge you have gained. Even if the automation project is a failure and does not work, you will be smarter for the effort.

To illustrate my point, here are several things that I learned over the past year while working on various projects. Most of these items are related to data analysis topics that I never knew about or understood but are are applicable more broadly:

Even if I never used any of the scripts I developed, I do have this knowledge and can use it to be more effective in my daily work.

Closing Out

This article was a bit of a departure from my normal topics. However, I think it is important to help people think about technology in the context of the bigger picture. I truly believe that many people could learn python and apply it to their jobs to make it better. I think the concept of going through the stages of the hype cycle might frame the process and help set realistic expectations. My hope is that readers will bookmark this post and refer back to it when they get stuck. As always, if you have thoughts on this article, feel free to comment below.

Success Kid