Port Shell Scripts to Python

make is a legendary tool to build software and is equally notorious at the same time. It is really good at what it does and has been forked many times to fix its many flaws and papercuts.

There are alternatives implemented with or without custom domain specific languages (DSL). Some are implemented in Python, such as SCons, pymake, Waf, BuildIt, and Invoke. Some are implemented in other languages like cmake, RAKE, Ninja, and Blur. The question to ask is: which would you use to solve what problems?

My use case had the following properties:

• Build rpm packages through a script

• Use Python instead of shell

I ported a shell script to Python using Invoke in a couple hours. Most of that time was spent trying to understand the semantics of the shell code. During the exercise I learned that Invoke really is a simple and straightforward way to write a build script. It also emerged that I could write pretty much anything in Invoke that needed to have tasks or steps; much like a Makefile.

Invoke excels at writing a standalone executable that needs to perform discrete steps individually or chained in a sequence. It feels very similar to recipes in Chef or playbooks in Ansible.

A Python script that uses the subprocess and argparse modules can do pretty much what Invoke does, only Invoke takes care of the plumbing for you. This nicety is very much appreciated.

Using Invoke means not having to learn a DSL just to do something. It's all Python code optionally interspersed with some shell commands. The learning curve is thus almost non-existent. Sure some things that are one-liners in shell script need more lines to implement but the batteries-included Python library ecosystem makes up for it. The end result being code that's more readable and thus maintainable.

I'm sure at some point in someone's requirements a simpler DSL or config file will be better than writing Python code but here again I think they'll miss something important. I'm all for simple config files to drive software. There's a reason for a plethora of tools for building software: they all solve a particular set of problems well enough to get the job done. They're widely used in different scenarios and have the bug fixes and improvements to show for it.

Recently I have come across a lot of DSLs. Chef, Ansible, Vagrant, Packer, and Terraform all have their own DSLs. They may or may not share syntax but they share a lot of semantics: order of execution, data structures, loops, conditionals, etc. These all integrate well with the problem domain and have valid reasons for their existence.

Nevertheless, having the power of a complete programming language at hand is much more useful. Instead of operating on a blackbox one gets to use libraries to write exactly what needs to be done. As one keeps building on top of these libraries the end result may equal that of another build tool that is driven by config files. The advantage, though, is that it's not a blackbox but an engine that you fully understand and control.

Your engine has its own inspirations and philosophies that fit your use case. It's not hampered by assumptions made by someone else who did not have the exact same problem that you want to solve. For example, I did not need a Makefile that I would have needed to refresh my knowledge on; I needed to copy and edit some files and run rpmbuild. A shell script was already doing the job. Invoke just gave more people in my team an opportunity to modify it or reuse it for their own use cases.

Invoke is the next generation iteration of Fabric. In fact, Fabric 2 is apparently being built on top of Invoke, thus providing remote execution capabilities. The current Fabric version already does both. If you're looking to the future and do not need to use SSH in your scripts then building on Invoke is a better choice.

Finally, I don't intend to encourage you to write an alternative to make. Quite the contrary. I encourage everyone to use make or an alternative blackbox build system when a lot more people will be using it. Standards are good and preferable to custom solutions even if those standard tools and practices have flaws.

Invoke -- and tools like it -- should be used to replace long-maintained shell scripts. Python has a much more readable and understandable syntax than shell. In the long term it will be easier to maintain a Python script. For these use cases Invoke is a great asset. It just so happened that my use case was in the area of build tools. On another day it might not be. But I was able to get the job done in Python, opening up the possibility of using Python and Invoke again for a future project.