pymake A mostly GNU-compatible python implementation of `make`

About

Purpose

• Improve build speeds, especially on Windows. This can be done by reducing the total number of processes that are launched, especially MSYS shell processes which are expensive.
• Allow writing some complicated build logic directly in Python instead of in shell.
• Allow computing dependencies for special targets, such as members within ZIP files.
• Enable experiments with build system. By writing a makefile parser, we can experiment with converting in-tree makefiles to another build system, such as SCons, waf, ant, ...insert your favorite build tool here. Or we could experiment along the lines of makepp, keeping our existing makefiles, but change the engine to build a global dependency graph.

Known Incompatibilities

• $(eval) is not yet supported
• Parallel builds (-j > 1) are not yet supported
• The vpath directive is not yet supported
• Order-only prerequisites are not yet supported
• Target-specific variables behave differently than in GNU make: in pymake, the target-specific variable only applies to the specific target that is mentioned, and does not apply recursively to all dependencies which are remade. This is an intentional change: the behavior of GNU make is neither deterministic nor intuitive.
• Windows is unlikely to work properly. There are issues to figure out with Windows file paths and shell execution, because Python is not an MSYS binary but we'd like it to use the MSYS shell when appropriate/necessary.
• There is a subtle difference in execution order that can cause unexpected changes in the following circumstance:
  • A file foo.c exists on the VPATH
  • A rule for foo.c exists with a dependency on sometool and no commands
  • sometool is remade for some other reason earlier in the file
  In this case, pymake resets the vpath resolution of foo.c, while GNU make does not. This shouldn't happen in the real world, since a target found on the VPATH without commands is silly. But mozilla/js/src happens to have a rule, which I'm patching.
• pymake does not implement any of the builtin implicit rules or the related variables. Mozilla only cares because pymake doesn't implicitly define $(RM), which I'm also fixing in the Mozilla code.

Issues

• Parsing is currently very slow compared to GNU make, to the point where the overhead is noticeable. I'd love assistance profiling and fixing the parser.

Future Work

Currently, the parser emits data directly to the makefile data model, evaluating ifdefs and other directives during parsing. I'd like to experiment with creating a "parse model" independent of the actual variables: this would allow the results of parsing to be cached (perhaps solving the parse performance issue).

I'd like to be able to run multiple "makes" within the same process, so that recursion did not require launching a new make process. This could also use the parsing cache mentioned above to avoid reparsing.

I'd like to implement a new type of command which is implemented in python. This would allow us to replace the current `nsinstall` binary (and execution costs for the shell and binary) with an in-process python solution.

Author

License