Make and makefiles are lost in the past for many developers, its advantages lost in the stream of tools that are constantly reinventing the wheel of building software. It’s time we get off that crazy carousel.
Those who do not understand Unix are condemned to reinvent it, poorly.
Usenet signature, November 1987 — Henry Spencer
What do all these things have in common with each other? At their core, they are each about taking a set of input files and transforming them into a set of output files. And this is exactly what Make is so incredibly good at.
What do makefiles do? Makefiles are simply a declarative way to transform one file (or series of files) into another. That’s it, it’s that simple. It’s not specific to C, C++, or even to programming languages. You could just as easily use a makefile to transform markdown documentation into shipped HTML files, or to pack important files into a zip/tar archive, or do a myriad of other transformations.
Thanks to its declarative nature and implementation, Make is able to use makefiles to only run the bare transforms needed to reach the final destination format. If a source file hasn’t changed since the last transformation, the source file doesn’t have to be processed again. In larger projects, this is a huge win to speed and a boon to the developer experience.
Make first appeared over 40 years ago and a lot of software has been built with it since that time. It’s a battle-tested and stable piece of software that excels at exactly what it was meant to do: transforming files from a source format to a target format, with a very simple and easy-to-understand mechanism for declaring dependencies. It’s all text-based, doesn’t try to solve everything itself, and has a great integration experience through calling out to shell scripts. In other words, it is very Unix-like. This is hardly surprising given it’s birth within the Unix environment.
Some developers have had experiences with very complicated and convoluted makefiles in larger projects. But it doesn’t need to be that way. In fact, it can be quite simple to build an NPM package that is implemented in Typescript (from building the source code to packaging the NPM package):
PATH := node_modules/.bin:$(PATH) source_files := $(wildcard lib/*.ts) build_files := $(source_files:lib/%.ts=dist/%.js) PACKAGE := build/my-package-1.0.0.tgz .PHONY: all all: $(PACKAGE) $(build_files): $(source_files) package.json npm i tsc $(PACKAGE): $(build_files) .npmignore @mkdir -p $(dir $@) @cd $(dir $@) && npm pack $(CURDIR)
Yes, the above example was for building a very small project. But just because the project becomes larger, doesn’t mean the user experience of Make diminishes. Let’s look at an example.
At work, I’m currently working on a larger project that is based on AWS Kinesis and Lambda Functions, a stream-processing system architecture that is serverless. The “service” is based out of one git repository for convenience. But we want easily accessible shared libraries between our different Lambda handlers that can also be independently deployed projects. This makes deploying fixes or new functionality into production much quicker and with much less overhead than deploying the entire service as one large monolith.
Our project structure is inspired by a post by StrongLoop on creating a modular Node.js project structure. Even though we are using TypeScript, this structure still definitely applies to us. So we started with the linking and npm scripts approach outlined in the blog post.
Our project structure ended up looking like this in the abstract:
But as the number of modules grew and the different ordering of dependencies started cropping up (as they do in larger Enterprise software), this approach quickly became unwieldy and painful. We found ourselves with a whole mix of preinstall, postinstall, and prestart scripts. It was very difficult to understand what was happening at build time to bootstrap the service. And integrating new sub-projects was a pain. It was also a “build everything or nothing” type of solution without us putting in a non-trivial amount of extra work.
Before grabbing the latest build hotness like Gulp off the shelf, we decided to take a look at what Make could do for this since it’s an established tool and this is right up its alley. That decision is what kicked off my growing appreciation of Make (and inspired this blog post).
Being a larger and growing project, we were naturally concerned about whether our build solution would scale. I happen to think that using Make, it most definitely does. And other than the Make quirks you get used to after you first use it for a while, I think that a junior developer could integrate their own libraries into this Make process.
Here’s what a potential makefile for the above project would look like:
deps_install := $(CURDIR)/build/last-install-time pkg_lib_foo := $(CURDIR)/build/foo-1.0.0.tgz pkg_lib_bar := $(CURDIR)/build/bar-1.0.0.tgz pkg_lib_baz := $(CURDIR)/build/baz-1.0.0.tgz pkg_alpha := $(CURDIR)/build/alpha-1.0.0.tgz pkg_omega := $(CURDIR)/build/omega-1.0.0.tgz .PHONY: all handlers libs all: libs handlers handlers: $(deps_install) $(MAKE) -C src/handlers/alpha $(MAKE) -C src/handlers/omega libs: $(MAKE) -C src/lib/foo $(MAKE) -C src/lib/bar $(MAKE) -C src/lib/baz $(deps_install): $(pkg_lib_foo) $(pkg_lib_bar) $(pkg_lib_baz) @if [ "$(pkg_lib_foo)" = "$(findstring $(pkg_lib_foo),$?)" ]; then \ cd $(CURDIR)/src/handlers/alpha && npm i $(pkg_lib_foo); \ fi @if [ "$(pkg_lib_bar)" = "$(findstring $(pkg_lib_bar),$?)" ]; then \ cd $(CURDIR)/src/handlers/alpha && npm i $(pkg_lib_bar); \ cd $(CURDIR)/src/handlers/omega && npm i $(pkg_lib_bar); \ fi @if [ "$(pkg_lib_baz)" = "$(findstring $(pkg_lib_baz),$?)" ]; then \ cd $(CURDIR)/src/handlers/omega && npm i $(pkg_lib_baz); \ fi @touch $(deps_install)
As you can see, it doesn’t need to be incredibly complicated. As this is a back-end service, we don’t have browserify, less, or minification. But it should help paint the picture that even with those additions, it should be pretty straightforward.
If you make a change to the baz library, only baz is rebuilt and only baz is re-installed into the omega handler sub-project. Throw a watcher on this process (e.g. an npm script like watch ‘npm run build’ src — wait=5) and your build process becomes more rich and improves the local development experience.
One thing I really like about this is that things only run if they need to. You don’t even need an incremental compiler to make it possible. If source files haven’t been updated, there is no need to regenerate the target files. Make knows this by comparing the last modified times of the source files compared to the target files. You can see the easy integration into existing tools like tsc and npm. I didn’t need to wait until a wrapper was created (or to create my own wrapper) in a code-based build tool.
Another less obvious benefit when comparing Make with code-based build tools like Grunt or Gulp is being declarative vs. imperative. You get to focus on the end result (declaring what needs to be done) instead of focusing on how the actual work is done.
Make is also a standalone tool, so there is no need to bring in a bunch of other code dependencies like code-based task runners do. This not only makes the user experience better, but it also means there are fewer ways the software can break (e.g. a new version of a dependency that breaks functionality in the core tool).
Yes, it’s another tool and language that developers need to learn. But that’s what we get paid to do as developers, right? We always need to be learning new tools and techniques (or re-learning old tools and techniques in this case :P). We accept this forever-learning experience as the latest-and-greatest programming languages or software libraries roll out every month.
But remember, in this case, we are learning a general tool that we will be to leverage in many different ways for a long time. Alton Brown need not worry, this tool is very much a multi-tasker. Make has been around for over 40 years and it’s not going anywhere anytime soon. Can we say the same about Grunt, Gulp, or the next Task Runner du jour?
An accurate concern of using Make historically has been the lack of decent support on Windows. By leveraging Make, you were potentially making life more difficult for all your Windows users. That was a non-starter for many projects. But with the recent addition of Linux support in Windows and the ongoing change of heart under Satya Nadella’s leadership at Microsoft, this concern is hopefully a relic of the past. With all the great stuff to be learned from and used in Linux, I feel this trend is a major boon to software developers.
Now is a great time to learn Make So today is a great time to learn and start leveraging Make and makefiles. They are still very much relevant to our work today as developers. There’s no need for an ever-revolving door of task runners du jour. Don’t succumb to the build tool treadmill and burn yourself out. Learn a powerful tool you will be able to leverage for a long time and isn’t going anywhere any time soon.
Yes, it’s time for Makefiles to make a comeback! Let’s do this!