Does the Mozilla build system need to change? The build system is impressively flexible and relatively accurate, but it is not especially easy to use or hack, and it can be slow. There are some common issues that cause major pain for developers. We need to reduce the pain of using our build system, without any major rewrites and without causing major disruption. I have spent some time investigating various options, and I have identified a set of changes that can make a major impact by improving ease of use, speed, and maintenance of the build system.

Pain #1: Setting up a Windows Build Environment

As the Windows Build Prerequisites page suggests, setting up a Windows build environment is very complex and easy to screw up. Developers have to obtain tools from five or six different sources and hook them up with carefully crafted scripts to set environment variables in a magic order. This is one of the major barriers to entry facing developers who want to start hacking Mozilla. This problem is not hard to fix:

• Mozilla will provide an installer for all of the prerequisites for building Mozilla. The only thing that developers will need to install separately is Microsoft Visual C++. The installer will provide scripts and shortcuts that will provide a ready-made build environment. If build requirements change, a new version of the installer will be made available.
• Configure checks will be improved to detect the common setup issues.

As part of this process, I am planning to make the MSYS build environment the official build environment and soon drop support for the cygwin build environment. Cygwin is a known source of performance problems and forces a lot of extra complexity in our build scripts to translate between Windows and unix-style paths.

Pain #2: Lack of Documentation

The Mozilla build system has many features that were added without any documentation whatsoever. Brian Ryner wrote a short summary of the build system last year that I have been slowly expanding. I have moved and expanded the old build glossary of makefile variables and other build-system terms, and provided example makefiles to create certain kinds of output (static library, shared library, component library). I hope to have this reference basically completed before Christmas. Any help others can give to complete and edit this documentation is much appreciated.

Pain #3: Depend Build Speed (Recursive Make Harmful)

The current build system uses a multi-pass recursive make system that, while mostly accurate, can be slow. It is tempting to architect a replacement build system (using either an existing framework such as SCons or WAF, or a homegrown solution), but there are over 2000 build scripts (makefiles, perl scripts, and various build manifests) in the Mozilla tree: none of the new build systems have a facility for porting this makefile logic, and any complete rewrite of the build system that is not incremental would be suicidal.

Instead, I have devised techniques to build large parts of the mozilla tree using fewer invocations of make. This will allow the build system to compute dependencies more accurately, as well as significantly reduce the number
of intermediate static libraries that are created during a build. It will also significantly help build times for people using parallel builds with -j and distributed builds with distcc. These new techniques will require GNU make 3.81, but will otherwise be fairly straightforward and can be implemented gradually, a few directories at a time.

Pain #4: Monolithic configure

As the Mozilla project transitioned from the unified suite to the standalone Firefox and Thunderbird, and now to a multitude of projects/products, the root configure script has become increasingly difficult to manage. It is difficult/impossible to tell which configure options work with which products. All the configure options are thrown into a single autoconf.mk file, which can cause hidden dependencies between modules.

While working on a cross-platform Tamarin build system, I discovered that replacing autoconf configuration scripts with python configuration scripts is relatively easy; it can be done without altering the Makefile-based build system. My first-cut scripts are a bare imitation of autoconf functionality, but fleshing out scriptable compiler and feature tests should be relatively straightforward.

I am hoping to port the main Mozilla configuration scripts to python over the next year. I’ll learn a lot from the Tamarin experience which can be applied to the main tree. I expect that the new scripts will not be ready for Mozilla 1.9, but will be used for Mozilla 2.

Flights of Fancy

If I had unlimited time, I would think about the following things:

• Writing a makefile parser that could be used to read or convert the existing makefiles into a python-based build system with better scriptability and flexibility.
  • Which could detect when rules changed and rebuild automatically
  • Which could detect when JAR members changed and rebuild only those members
• Improving the XUL preprocessor to support “real” #if conditions

Conclusion

Mozilla can and should make incremental improvements to the build system. This will be done gradually and carefully to solve specific pain-points and refactor code without disrupting existing work. The short term projects will reduce the pain for new developers and hackers, while the longer-term projects will reduce the pain of maintaining the configuration/build system.

This entry was posted on 30 Nov, 2006 at 12:13 pm and is tagged with , build-system. Trackback.