Parsing Compiler Errors
Long ago, Mozilla had a tinderbox which would collate every warning produced by the Mozilla build and generate statistics and reports about them. I’m trying to re-create this tool.
When building Mozilla or most other large software projects that use GNU make, compiler warnings get sent to stdout (and sometimes stderr). The messages usually look something like this:
../../../dist/include/dom/nsIDOMXULSelectCntrlEl.h:33: warning: ‘virtual nsresult nsIDOMXULSelectControlElement::GetSelectedItem(nsIDOMXULSelectControlItemElement**)’ was hidden ../../../dist/include/dom/nsIDOMXULMultSelectCntrlEl.h:73: warning: by ‘virtual nsresult nsIDOMXULMultiSelectControlElement::GetSelectedItem(PRInt32, nsIDOMXULSelectControlItemElement**)’
All of the file paths in the warning (or set of warnings in this case) are relative to the current working directory. Because the working directory changes during the build as make recurses through subdirectories, automatic parsers need some way to know what the working directory is at any point in the build log. Make provides the -w option which will print the following output every time it recurses into or leaves a directory:
make[1]: Entering directory ‘/builds/static-analysis-buildbot/slave/full/build’ make[1]: Leaving directory ‘/builds/static-analysis-buildbot/slave/full/build’
This is fine if you are only building in one directory at a time. But with the -j option, it is likely that make will be building in multiple directories at once. This will interleave output from multiple jobs in the same log, making it difficult for an automated parser to make any sense of them.
What I’d like is a tool or technique which will save the build log for each makefile command separately and combine them all at the end of a build.
Pre-emptive snarky comment: “switch Mozilla to use {scons,waf,cmake,ant,…}”
December 3rd, 2008 at 6:00 pm
Hm, we dealt with this in VS at PGP using perl. There is a header of what is being built by each process, with a number (much like make[1]), but each output line would be prepended with that number. At that point it just became a matter of attaching the name to the number and/or sorting by name.
For this problem I’m thinking maybe a makefile target that calls make on another target with a |tee ?
I’d hope there’s a more elegant solution though.
December 3rd, 2008 at 6:52 pm
What about a way of converting the relative pathnames into absolute pathnames directly, instead (which should be easy to parse later)? I suppose it’s possible the compiler has a switch to display absolute (instead of relative) paths on warnings, but more likely you’d need to do something like wrap the compiler in a small script that looks for warning/error output, extracts the relative paths, and then asks the shell to convert them to absolute paths.
December 3rd, 2008 at 8:39 pm
I don’t know if this is possible with client.mk, but if you can somehow arrange to invoke configure by an absolute pathname, that might be enough to get absolute paths on the compiler command line, and thus in diagnostics…
December 4th, 2008 at 8:53 am
Would setting an absolute –srcdir= help?
December 4th, 2008 at 10:17 am
Zack/Neil: an absolute srcdir won’t help by itself, because most of these warnings come from dist/include, which is relative to the objdir (it uses $(DEPTH) in the makefiles).
In addition, *just* fixing the file paths won’t help, because you still have output from multiple processes interleaved in the same logfile. What you really want is a separate logfile for each build operation.
December 4th, 2008 at 7:32 pm
How hard would it be to add an option to gcc to make it output absolute paths?
December 5th, 2008 at 8:40 pm
an absolute srcdir won’t help by itself, because most of these warnings come from dist/include, which is relative to the objdir (it uses $(DEPTH) in the makefiles).
Hm, so maybe in config.mk change
topsrcdir = $(DEPTH)
to
topsrcdir := $(shell cd $(DEPTH) && pwd)
and then s/$(DEPTH)/$(topsrcdir)/ everywhere relevant? (We already require GNU make, right?)
December 5th, 2008 at 8:47 pm
Also meant to say,
you still have output from multiple processes interleaved in the same logfile.
if all paths are absolute, and you add
-fmessage-length=0to the command line, then the interleaving shouldn’t matter at all. (The “In file/In function” additional hints will be useless, but you don’t need them.)December 9th, 2008 at 6:23 pm
[…] Log in « Parsing Compiler Errors […]