In 1998, Peter Miller wrote an essay Recursive Make Considered Harmful (PDF). He outlines several convincing arguments for why the traditional practice of recursing through directories and calling make in each directory may not be a good technique.
Mozilla uses recursive make extensively. In fact, during a typical build we traverse each directory in the build tree at least twice, and in some cases three or even four times. So, way back in 2002, Chris Seawood filed a bug to reduce the number of subdirectories we traverse during a typical build. This had the promise of significantly reducing build time on Windows because forking processes on Windows (and especially under cygwin) is really expensive.
Unfortunately, it turned out that a non-recursive make was taking longer than the standard recursive make. I thought, at the time, that this was due to the use of extra $(shell cygpath) subprocesses needed to handle sources from multiple subdirectories correctly. About a month ago, at schrep‘s prompting, I went back to see whether there was a way to get it working. The results are below:
|make -C netwerk libs|
(Nothing to rebuild)
||Standard (recursive) make
It turns out that the problem with non-recursive make, at least as I currently have it coded, is not the cost of forking processes, but the cost of statting files that don’t exist. Every time a makefile adds a directory to
VPATH, make ends up searching through additional directories looking for files. As you can see from the chart, the actual processor time spent by the non-recursive process is a lot less; it takes longer in wall-clock time only because it’s waiting on disk I/O.
I’m hoping to try another technique for non-recursive make that doesn’t set the
VPATH for each source directory. Unfortunately, this is going to involve some changes to the dependency system (which I don’t understand very well), so I don’t know when I’m going to find the time.
My technique involved the use of the
$(eval) makefile function, which is only available in gmake 3.80 or higher. The tests were performed under MSYS, using GNU make 3.81, available from MSYS snapshots. Special thanks to Earnie Boyd for patiently dealing with me and pointing me at the right files.