Plans for the Mozilla Build System

Does the Mozilla build system need to change? The build system is impressively flexible and relatively accurate, but it is not especially easy to use or hack, and it can be slow. There are some common issues that cause major pain for developers. We need to reduce the pain of using our build system, without any major rewrites and without causing major disruption. I have spent some time investigating various options, and I have identified a set of changes that can make a major impact by improving ease of use, speed, and maintenance of the build system.

Pain #1: Setting up a Windows Build Environment

As the Windows Build Prerequisites page suggests, setting up a Windows build environment is very complex and easy to screw up. Developers have to obtain tools from five or six different sources and hook them up with carefully crafted scripts to set environment variables in a magic order. This is one of the major barriers to entry facing developers who want to start hacking Mozilla. This problem is not hard to fix:

As part of this process, I am planning to make the MSYS build environment the official build environment and soon drop support for the cygwin build environment. Cygwin is a known source of performance problems and forces a lot of extra complexity in our build scripts to translate between Windows and unix-style paths.

Pain #2: Lack of Documentation

The Mozilla build system has many features that were added without any documentation whatsoever. Brian Ryner wrote a short summary of the build system last year that I have been slowly expanding. I have moved and expanded the old build glossary of makefile variables and other build-system terms, and provided example makefiles to create certain kinds of output (static library, shared library, component library). I hope to have this reference basically completed before Christmas. Any help others can give to complete and edit this documentation is much appreciated.

Pain #3: Depend Build Speed (Recursive Make Harmful)

The current build system uses a multi-pass recursive make system that, while mostly accurate, can be slow. It is tempting to architect a replacement build system (using either an existing framework such as SCons or WAF, or a homegrown solution), but there are over 2000 build scripts (makefiles, perl scripts, and various build manifests) in the Mozilla tree: none of the new build systems have a facility for porting this makefile logic, and any complete rewrite of the build system that is not incremental would be suicidal.

Instead, I have devised techniques to build large parts of the mozilla tree using fewer invocations of make. This will allow the build system to compute dependencies more accurately, as well as significantly reduce the number
of intermediate static libraries that are created during a build. It will also significantly help build times for people using parallel builds with -j and distributed builds with distcc. These new techniques will require GNU make 3.81, but will otherwise be fairly straightforward and can be implemented gradually, a few directories at a time.

Pain #4: Monolithic configure

As the Mozilla project transitioned from the unified suite to the standalone Firefox and Thunderbird, and now to a multitude of projects/products, the root configure script has become increasingly difficult to manage. It is difficult/impossible to tell which configure options work with which products. All the configure options are thrown into a single autoconf.mk file, which can cause hidden dependencies between modules.

While working on a cross-platform Tamarin build system, I discovered that replacing autoconf configuration scripts with python configuration scripts is relatively easy; it can be done without altering the Makefile-based build system. My first-cut scripts are a bare imitation of autoconf functionality, but fleshing out scriptable compiler and feature tests should be relatively straightforward.

I am hoping to port the main Mozilla configuration scripts to python over the next year. I’ll learn a lot from the Tamarin experience which can be applied to the main tree. I expect that the new scripts will not be ready for Mozilla 1.9, but will be used for Mozilla 2.

Flights of Fancy

If I had unlimited time, I would think about the following things:

Conclusion

Mozilla can and should make incremental improvements to the build system. This will be done gradually and carefully to solve specific pain-points and refactor code without disrupting existing work. The short term projects will reduce the pain for new developers and hackers, while the longer-term projects will reduce the pain of maintaining the configuration/build system.

Atom Feed for Comments 9 Responses to “Plans for the Mozilla Build System”

  1. Axel Hecht Says:

    Call me in for the #if conditions? I guess I’ll have a hard time with my port of the preprocessor to python if I’m not doing that. This is on my list of things for bug 361583.

  2. Trent Mick Says:

    > Improving the XUL preprocessor to support “real” #if conditions

    Benjamin,

    Would http://trentm.com/projects/preprocess/ be of interest here? It is a multi-language preprocessor that puts the preprocessor statements (#if, #else, #define, …) in the particular file language’s comments so that the unprocessed file can still be valid.

  3. Jeff Walden Says:

    For people not watching it, I have a patch for making #if conditions actually work, in that any #else/#elif/#elifdef/etc. will work the way one would expect it to work, not simply as relative to the immediate past condition.

    Implementing compound conditions (“#if defined(B) || BAR==’foo”‘”) wouldn’t be hard, but it would require making the syntax a little stricter (i.e., “#define BAR foo ” really shouldn’t give BAR a four-character value). If people really care (I don’t think it’s that great a win, to be honest) I could probably write a patch for some type of compound/real conditions, but it wouldn’t be a high priority for me. (I’ve been bitten by #else not working in the past and have seen others be bitten by it, hence why I cared enough about it to make the patch.)

  4. Jeff Walden Says:

    …where by “it” I meant bug 277122:

    https://bugzilla.mozilla.org/show_bug.cgi?id=277122

  5. Nashtiz Huan Says:

    You should take a look at CMake http://www.cmake.org maybe will impress you guys too

  6. Markus Dresch Says:

    what i dream of is a codeblocks project. codeblocks (–> http://www.codeblocks.org) is an ide running on win32, linux and mac os x. as soon as i understand the mozilla build system and find some spare time i’ll give it a shot.

    btw: codeblocks does a wonderful job converting visual studio 6-8 workspaces/project files.

  7. BSBlog » Blog Archive » Mozilla Build System: Now Requiring Python, and new Windows build package Says:

    [...] Mozilla Build System: Now Requiring Python, and new Windows build package Starting next week the Mozilla build system is going to require python on all platforms. This will allow us to gradually convert various build scripts which are currently written in perl, and start hacking on an autoconf replacement written in python. Any python 2.3 or higher will be acceptable. In addition, I have made a production release of MozillaBuild, version 1.0. The MozillaBuild package is now the recommended build environment for all Mozilla developers on Windows. There were a few bugs with the 1.0 releases that have now been corrected in a MozillaBuild 1.1 release candidate. After the Tinderboxes have been upgraded to use the MozillaBuild package on trunk, I am going to be discontinuing support for the cygwin build environment. Please post questions and followups to the mozilla.dev.builds newsgroup. [...]

  8. Gary Johnson Says:

    I think this is fantastic. Getting all the right versions of the right tools in one download is a big time saver. I found using autoconf a real pain. Good luck with this. I have down loaded version 1.1 of Mozilla Build and await your updating the trunk. I grew up with DEC and IBM operating systems and did not see UNIX things until I built Mozilla with BASH. For the folks who are not UNIX proficient, I found customing the start scripts with these minor additions helpful.

    rem gary custom settings
    set MOZCONFIG=c:\moz\mozconfig.txt

    SET CVSROOT=:pserver:anonymous@cvs-mirror.mozilla.org:/cvsroot
    SET CVS_RSH=ssh

    rem changed C:\mozilla-build\msys\etc\profile.d\profile-homedir.sh to CD to C:\moz instead of USERPROFILE
    rem The history file is here by default C:\mozilla-build\msys\home\US\.bash_history

  9. Mike Moening Says:

    After reading above comments and a little review I would like to 2nd the motion to move to something like CMake. http://www.cmake.org
    The idea of using native build tools, like Visual C++ IDE for windows guys, is really appealing.
    From what I can read that tool supports the way Mozilla is built now too.

    If you are looking to attract more quality developers to the Mozilla project this is the way to do it.
    Lowest common denominator build systems don’t cut it.
    Change 1 line of code and wait 10 minutes for a build to finish is no fun (I’m being kind here).
    Please take a look.

Leave a Reply