Making pymake faster
Gregory Szorc has an interesting post about the execution time of Mozilla makefiles. In it he describes the mechanism that pymake uses to parse a makefile into an AST, and then evaluate the makefile in a specific context. His data shows that the majority of time spent in pymake occurs during the evaluation phase. This is correct, but this slowness is partly a consequence of how GNU make parses and evaluates variables. I believe that it would not be hard to trade minor incompatibility with GNU make and achieve significant speedup in pymake execution.
When GNU make encounters a “normal” recursively-expanded variable, it does not perform any sort of parsing, it merely stores the text. This means that it is possible to invalid syntax into a variable, or to build up a function from multiple appending assignment, for example:
VALUE = aabbcc TESTVAR = $(subst a,b, TESTVAR += $(VALUE) TESTVAR += ) default: @echo $(TESTVAR)
The GNU make output of this testcase is “bbbbcc”. Early versions of pymake instead threw a warning about an unterminated variable assignment and the result was ” aabbcc )”.
Building up a makefile variable dynamically may be useful in some special cases, but this is not a technique that most projects use. It does mean that pymake cannot parse variables at the time they are assigned: instead it must wait until the variable is evaluated. (This behavior is in the pymake testsuite).
Variable expansion is by far the most expensive part of core pymake execution. The current parsing method which uses python regular expression engine is relatively slow. Other methods we have tried including hand-parsing using string indexes and were even slower. Parsing all of the config.mk and rules.mk variables as part of the cached AST would significantly improve pymake runtime speed on the Mozilla tree, because we could parse each variable once, instead of re-parsing them at each execution.
The original development versions of pymake did do early parsing, and I only changed it to do parsing at evalation-time when the gmake incompatibility was discovered. I don’t think it would be too hard to switch back. I encourage somebody to try it.
