As I mentioned last week, I’ve been resurrecting a project to report on compiler warnings. A basic form of this buildbot is now operational on the Firefox tinderbox tree (look to the far right for the static-analysis-bsmedberg column). It prints a summary of the total number of warnings on the summary page: in the full tinderbox log, it lists each warning and who can be “blamed” for that warning:

/builds/static-analysis-buildbot/slave/full/build/memory/jemalloc/jemalloc.c:177:1: warning: C++ style comments are not allowed in ISO C90 blamed on Taras Glek  in revision hg:36156fbf817d8a0e2d54a271cf0bff94a1c41c13:memory/jemalloc/jemalloc.c
/builds/static-analysis-buildbot/slave/full/build/js/src/jsdbgapi.cpp:712: warning: ISO C++ forbids casting between pointer-to-function and pointer-to-object blamed on brendan@mozilla.org in revision cvs:3.36:js/src/jsdbgapi.c

Assigning blame can be a tricky process. In order to figure out the blame for a warning, the code uses the following steps:

• Resolve relative paths against the current working directory, using GNU make “Entering/Leaving directory” markers as a guide.
• Dereference symlinks to find the source tree location of an error. For instance, Mozilla headers which produce warnings often do so via paths in dist/include. We have to resolve these to their original source tree location in order to find blame.
• Using mercurial APIs (through python), find the mercurial changeset which introduced the line in question.
• If the code dates back to Mercurial revision 9b2a99adc05e, which is the original import of CVS code to Mercurial, use a database of CVS blame to find the original CVS checking which was responsible for introducing that line of code.

If you’re interested, take a look at the build log parsing code, or see the scripts which save CVS blame to a database (thanks Ted!).

The current reporting system for warnings is very primitive. I’m currently working on a second version of this code which will provide additional features:

• Compare warnings with the previous build and highlight “new” warnings. I do this by recording the error text and the blamed location of the warning. As lines are added and removed from the code, the reported location of the warning changes, but the location of Hg/CVS blame doesn’t. This means it is a stable location which can be used for comparisons across runs. It even works across file renames/moves!
• Web frontend to the warning database to allow users to query warnings by user or directory.
• Classify warnings by “type”. This is not a simple process, because GCC mixes distinctive error text, such as “may be used uninitialized in this function” with variable names; and the granularity of -fdiagnostic-show-option is low enough that it’s not very useful by itself. Oh, I wish GCC had error codes like MSVC does: C1234 is easy to recognize!

At one point, I thought I could implement all of the warning mechanism on the buildbot server by parsing the BuildStep logs. It quickly became clear that I couldn’t do that, because I couldn’t resolve symlinks, and getting Mercurial blame was difficult or impossible. My new version actually uses a hybrid mechanism where the build log is parsed on the buildbot slave: this parses out warnings, resolves symlinks, and looks up blame. It then sends the results back via stdout to the master. A custom build step on the master parses this log, saves the information to a database, and does the checking for new warnings and prints various results to custom build logs.

Long ago, Mozilla had a tinderbox which would collate every warning produced by the Mozilla build and generate statistics and reports about them. I’m trying to re-create this tool.

When building Mozilla or most other large software projects that use GNU make, compiler warnings get sent to stdout (and sometimes stderr). The messages usually look something like this:

../../../dist/include/dom/nsIDOMXULSelectCntrlEl.h:33: warning: ‘virtual nsresult nsIDOMXULSelectControlElement::GetSelectedItem(nsIDOMXULSelectControlItemElement**)’ was hidden
../../../dist/include/dom/nsIDOMXULMultSelectCntrlEl.h:73: warning:   by ‘virtual nsresult nsIDOMXULMultiSelectControlElement::GetSelectedItem(PRInt32, nsIDOMXULSelectControlItemElement**)’

All of the file paths in the warning (or set of warnings in this case) are relative to the current working directory. Because the working directory changes during the build as make recurses through subdirectories, automatic parsers need some way to know what the working directory is at any point in the build log. Make provides the -w option which will print the following output every time it recurses into or leaves a directory:

make[1]: Entering directory ‘/builds/static-analysis-buildbot/slave/full/build’
make[1]: Leaving directory ‘/builds/static-analysis-buildbot/slave/full/build’

This is fine if you are only building in one directory at a time. But with the -j option, it is likely that make will be building in multiple directories at once. This will interleave output from multiple jobs in the same log, making it difficult for an automated parser to make any sense of them.

What I’d like is a tool or technique which will save the build log for each makefile command separately and combine them all at the end of a build.

Pre-emptive snarky comment: “switch Mozilla to use {scons,waf,cmake,ant,…}”

Occasionally, people will come on the #xulrunner or #extdev channel with a question about compiling XPCOM components. The question often goes something like this:

<IRCGuy> I’m following a tutorial on making XPCOM components, but I can’t seem to get them to compile. Can anyone tell me what my problem is?

Hint for asking a good question: IRCGuy needs to tell us some combination of 1) what tutorial he’s following, 2) what the failing command is or 3) what the error message is.

This time, IRCGuy’s compile command and error message are:

IRCGuy@IRCGuy-france:/mnt/data/IRCGuy/public/project/xpcom-test$ make
g++  -I/usr/local/include/xulrunner-1.9/unstable -I/usr/local/include/xulrunner-1.9/stable -L/usr/local/lib/xulrunner-devel-1.9/lib -Wl,-rpath-link,/usr/local/bin  -lxpcomglue_s -lxpcom -lnspr4 -fno-rtti -fno-exceptions -shared -Wl,-z,defs  france2.cpp -o france2.so
/tmp/cceFg2dD.o: In function `NSGetModule':
france2.cpp:(.text+0x38c): undefined reference to `NS_NewGenericModule2(nsModuleInfo const*, nsIModule**)'

IRCGuy’s problem is a problem of link ordering: with most unix-like linkers, it is very important to list object files and libraries in the correct order. The general order you want to follow is as follows:

1. Object files
2. Static libraries – specific to general
3. Dynamic libraries

If an object file needs a symbol, the linker will only resolve that symbol in static libraries that are later in the link line.

The corrected command:

g++  -I/usr/local/include/xulrunner-1.9/unstable -I/usr/local/include/xulrunner-1.9/stable -fno-rtti -fno-exceptions -shared -Wl,-z,defs  france2.cpp -L/usr/local/lib/xulrunner-devel-1.9/lib -Wl,-rpath-link,/usr/local/bin  -lxpcomglue_s -lxpcom -lnspr4 -o france2.so

Bonus tip: correct linker flags for linking XPCOM components can be found on the Mozilla Developer Center article on the XPCOM Glue. As noted in the article, xpcom components want to use the “Dependent Glue” linker strategy.

There are lots of hidden corners in the Mozilla build system. I’m starting a little series about some of the cool and unexpected things you can do with it. Of course, the series is mostly going to be “see the document I just wrote on the Mozilla Developer Center”!

First up: myconfig.mk and myrules.mk. Did you know that you can add arbitrary rules and change arbitrary variables in the Mozilla build system without editing your source tree? You can, by simply dropping a myconfig.mk or myrules.mk file in the objdir/config directory. Let’s say you want to define PL_DHASHMETER in all of Mozilla, so that you can then run Mozilla and collect hashing statistics.

Copy the following file to objdir/config/myconfig.mk:

CFLAGS += -DPL_DHASHMETER
CXXFLAGS += -DPL_DHASHMETER

This is not a very impressive example, because you could achieve the same effect by exporting CFLAGS and CXXFLAGS in your environment before configuring. So check out the Mozilla Developer Center article, which has two additional examples adding a new makefile target and altering build rules to save static analysis output to disk.

Brainstorm of possible improvement: load application/app-config.mk and application/app-rules.mk for application-specific build configuration and build rules.

preed’s post about a new owner for the Build & Release module in Mozilla reminded me to announce more officially that I’m handing over ownership of the Mozilla “Build Config” module to Ted Mielczarek (ted on IRC). Ted has been involved with the Mozilla project for a while, and has done great work with the build system, Breakpad, and other projects.

Being owner of several large modules is tough: it’s hard to just keep track of everything that’s changing in Build Config, “the toolkit”, and XPCOM, let alone provide effective leadership. I’m blessed that I can transfer part of that responsibility without fear. It will allow me to focus my efforts more effectively, particularly to lead some of the architectural changes we’re making for Mozilla 2.

Congratulations Ted!

How hard is it to compile a 32-bit Firefox on an x86-64 Linux OS (preferably FC7, because that’s what I just installed). A quick google search failed me… I’d love directions, hints, or whatever you can provide. Is the correct way to do this to install all the necessary build dependencies in a chroot jail? And if so, how would I go about doing that?

Starting next week the Mozilla build system is going to require python on all platforms. This will allow us to gradually convert various build scripts which are currently written in perl, and start hacking on an autoconf replacement written in python. Any python 2.3 or higher will be acceptable.

In addition, I have made a production release of MozillaBuild, version 1.0. The MozillaBuild package is now the recommended build environment for all Mozilla developers on Windows. There were a few bugs with the 1.0 releases that have now been corrected in a MozillaBuild 1.1 release candidate.

After the Tinderboxes have been upgraded to use the MozillaBuild package on trunk, I am going to be discontinuing support for the cygwin build environment.

Please post questions and followups to the mozilla.dev.builds newsgroup.

Ted Mielczarek (luser on IRC) has been acting as a build-config peer for a while now, and I have been derelict and haven’t updated despot or made an announcement. Ted has been doing some great work integrating Breakpad (formerly airbag) into the toolkit, as well as learning and helping with the tangled web that is our build system. He pretty much single-handedly fixed our build system to support embedding CRT manifests for VC8.

It seems that whenever I challenge him to do something, he comes right back at me with a solution. For instance, I was musing on IRC how nice it would be to have cvsgraph produce SVG instead of a huge image. Four hours later, behold (the code is a wild-ass ugly hack, but it works)!

Belated welcome to Ted!

I have a version of MozillaBuild ready for widespread testing, packaged as an installer. I’m calling this RC1 because I don’t know of any bugs and unless we find some it is fine to put into production. It works with MSVC6, 7, 8, and MSVC 8 Express with an SDK. It builds the trunk, 1.8, and 1.8.0 branches.

Download MozillaBuildSetup.exe now

The installer was built using the patch in bug 366823, which hasn’t been reviewed or landed yet.

Please download and test this package (and start using it as your Windows development environment). If you have issues, please file bugs in Core->Build Config.

Changes From the Previous Release

• Reset CYGWIN and PATH environment variables so as not to pick up any VC paths from the environment.
• Support MSVC8 Express Edition.
• Use the atlthunk_compat code to support the ATL headers provided with the platform SDK.
• Set CVS_RSH=ssh automatically.
• Add an installer.

In addition, I added a script which converts a cygwin source tree with CRLF line endings to an MSYS tree using LF line endings, and a client.mk check to warn users of problems with line endings.

As part of MozillaBuild, I’ve been testing various combinations of compilers and SDKs. It turns out that it is possible to build Mozilla (trunk) using the Free Microsoft Visual C++ Express download with an appropriate SDK and a couple hacks. I have a bug to make MozillaBuild support the free compiler, with a patch that mostly-works.

The ATL headers have a special instruction which requires a library atlthunk.lib, which doesn’t ship with the SDK. The Mozilla Build FAQ has an ingenious solution to this problem which requires editing the SDK headers. Unfortunately, patching other people’s software is not a good idea in general. I’ve found another way around this problem by creating a replacement atlthunk.lib library which contains the two required functions. Source code is currently here. MSVC Express can build the replacement library, as long as you have your INCLUDE, BIN, LIB environment variables set up with SDK paths properly. I’m hoping to include the library as part of MozillaBuild, so that Mozilla code will build “out of the box” with MSVC Express.

The MFCEmbed example will not build with MSVC Express unless you build MFC yourself (which is painful). But MFCEmbed is not part of Firefox builds anyway (and should probably be disabled completely, since it’s an unmaintained bug-ridden mess).