Depths of the Mozilla Build System
I should really be posting about some important plans for the Mozilla build system that I nailed down during the Firefox summit. But I don’t have time to give that a proper post, so instead I’m going to discuss one of the amazing things I’ve learned about the Mozilla build system over the past few days. Look at this snippet from rules.mk. I’ve been the nominal owner of this code for almost two years, but didn’t really understand it until this week. This little piece of code is one of the things that makes our build system really great and horrible at the same time:
- Whenever people remove or alter the location of header files, this code keeps all the depend builds from going red.
- It causes us to call stat() an extra 10,000+ times per depend build. Probably a lot more than that, actually, but I didn’t instrument it.
We do an end-run around the normal dependency checks done by GNU make: the mddepend.pl script stats and calculates the compiler-generated dependencies in advance. If the dependency is missing or new, it adds a FORCE dependency on the object file. Unfortunately, we do this calculation on each build pass: once for export, once for libs, once for tools, and perhaps another time for check. This causes us to check dependencies many many more times than we actually need to.
What we really want to implement is an “optional dependency”: a directive that if a header has been updated, we should rebuild the object; but if the header doesn’t exist any more, we shouldn’t try to build it (because we don’t have any rules to generate such headers which were removed or relocated intentionally). This is probably not something I’m going to fix any time soon. But I may find time to write it up in detail to propose it as a feature for gmake 3.82.
November 23rd, 2006 at 1:48 am
I think your definition of “optional dependency” isn’t quite right. If the header has been removed, we should also try to rebuild the object (since we want depend builds to break if they still need the header), but we should not try to build the header.
November 27th, 2006 at 2:05 pm
mddepend.pl stats…
My last post mentioned that mddepend.pl causes our build system to do many extra calls to stat(). I’ve done some instrumentation and come up with the following numbers (Linux, Firefox trunk):
Calls to mddepend.pl
mddepend calls to stat()
Ne…
December 1st, 2006 at 4:19 am
Hi Benjamin: Thanks for the interesting post! IMHO, rather than “optional dependencies,” I would describe what you are seeing as two separate concepts:
1. Dynamic dependencies
When it comes time to rebuild a source file, you want to determine the dependencies for it then, not as part of a separate pre-build phase. The scanner can look at the source file and write out a list of headers it depends on to a file. The source file can then depend not only on this file, but also *each* of the files listed within. This eliminates the need for multiple passes and starting over from the beginning. It is also in direct contrast to the way make works—make wants to know about all the dependencies up front. (Well, GNU make allows included Makefile fragments to be dependencies; it will automatically rebuild these fragments first and then start again, but this technique gets messy and has limitations).
2. Dependencies on nonexistent files
Current build tools only notice if existing files are modified or removed; there’s no way to tell them that a target should be considered out of date if a particular file is *created*.
SCons doesn’t really help with these problems because it follows the same general process as make: build up a complete graph of (mostly) static dependencies and then start the actual build using this graph.
The only tool I’m aware of that is designed to build in a purely top-down manner, calculating dependencies on the fly, is D. J. Bernstein’s unpublished “redo” package:
http://cr.yp.to/redo.html
I’m working on a prototype, but ATM it’s definitely not ready for prime-time. Still, feel free to email me if you’re interested.
—Alan