pymake news:<\/p>\n
All measurements are do-nothing depend builds. Full rebuilds aren’t significantly affected because compiler speed overwhelms any time we spend in make.<\/p>\n
Creating Windows processes is more expensive than creating processes on a unix-like operating system. Creating MSYS processes is hugely more expensive. Windows I\/O in general is slow compared to Linux, at least for typical build tasks. Because pymake recurses in a single process, caches parsed makefiles such as rules.mk, and avoids many shell invocations, it can make up for slow parsing times by dramatically reducing time spent elsewhere.<\/p>\n
Don’t use pymake with client.mk on Windows, yet. pymake doesn’t understand MSYS-style paths, which is what configure substitutes for @srcdir@<\/tt> and @topsrcdir@<\/tt> when using client.mk. This will be fixed by the patches available from this bug tree<\/a>.<\/p>\n Configuring manually isn’t hard: to build Firefox in c:\/builds<\/tt>, follow this recipe:<\/p>\n Configure manually as above, or add the following flags to your mozconfig file:<\/p>\n Soon on all platforms this will be as simple as mk_add_options MOZ_ENABLE_PYMAKE=1<\/tt><\/p>\n Special thanks to Arpad Borsos who wrote tests and an implementation of –keep-going for pymake.<\/p>\n Immediate future plans for pymake reduce the process count even further, especially for depend builds:<\/p>\n Currently every invocation of nsinstall is a separate process, and we invoke nsinstall even when all its install targets are up to date. Simple tasks like this will instead be implemented as native python commands. Ted implemented a branch to do this, but the current implementation blocks the only thread. I think we’re going to switch and use shared-nothing threads and message passing to parallelize before making this the default behavior.<\/p>\n Every time Mozilla processes a makefile the build system combines all the compiler-generated dependencies into a single .all.pp file<\/a> using mddepend.pl: this allows developers to move or remove header files without breaking depend builds. Running a perl script for every makefile invocation is silly, especially because all it does is parsing and rewrite makefile syntax. I will have pymake read these dependency files directly and ignore missing files (causing a rebuild without an error) using a syntax includedeps $(INCLUDEFILES)<\/tt><\/p>\n Longer-term work that would make pymake much more useful:<\/p>\n While debugging pymake performance and memory usage I found an interesting fact, which in hind sight should have been obvious: functions which enclose themself in python create reference cycles which have to be cleaned up by the Python garbage collector:<\/p>\n After finding this problem, I refactored (1<\/a>, 2<\/a>, 3<\/a>) the pymake code to use objects instead of closures to save asynchronous state while rebuilding. Also, OptionParser instances create cycles by default. There is a lightly-documented method OptionParser.destroy which can be used to manually break<\/a> these cycles (thanks to Ted for finding it). pymake now runs without creating any reference cycles and I disabled the python garbage collector.<\/p>\n When MSYS goes from an MSYS process to a Windows process, and vice-versa, it munges certain environment variables to account for the path styles. I previously thought that it only munged PATH, but I discovered today that I was wrong: MSYS was munging the MAKEFLAGS environment variable in odd ways.<\/p>\n If MAKEFLAGS in the MSYS process was ‘ -j1 — PATH=e:\/builds\/mozilla-central’ it would be munged into ‘ -j1 — PATH=e;c:\/mozilla-build\/msys\/builds\/mozilla-central’ in a non-MSYS process. Without the leading space the value was not touched. I don’t know why this is, but I altered the pymake code slightly<\/a> so that MAKEFLAGS would never start with a space (and would be more compatible with gmake in the process).<\/p>\n","protected":false},"excerpt":{"rendered":" pymake news: Bad news: pymake is still 5x slower than GNU make on Linux\/Mac. Good news: pymake is 25% faster than msys make (GNU make on Windows)! Best news: there’s a lot of room to make performance better. All measurements are do-nothing depend builds. Full rebuilds aren’t significantly affected because compiler speed overwhelms any time […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[2,6],"tags":[210,32],"class_list":["post-478","post","type-post","status-publish","format-standard","hentry","category-mozilla","category-untagged","tag-pymake","tag-python"],"_links":{"self":[{"href":"http:\/\/benjamin.smedbergs.us\/blog\/wp-json\/wp\/v2\/posts\/478","targetHints":{"allow":["GET"]}}],"collection":[{"href":"http:\/\/benjamin.smedbergs.us\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/benjamin.smedbergs.us\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/benjamin.smedbergs.us\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"http:\/\/benjamin.smedbergs.us\/blog\/wp-json\/wp\/v2\/comments?post=478"}],"version-history":[{"count":10,"href":"http:\/\/benjamin.smedbergs.us\/blog\/wp-json\/wp\/v2\/posts\/478\/revisions"}],"predecessor-version":[{"id":488,"href":"http:\/\/benjamin.smedbergs.us\/blog\/wp-json\/wp\/v2\/posts\/478\/revisions\/488"}],"wp:attachment":[{"href":"http:\/\/benjamin.smedbergs.us\/blog\/wp-json\/wp\/v2\/media?parent=478"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/benjamin.smedbergs.us\/blog\/wp-json\/wp\/v2\/categories?post=478"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/benjamin.smedbergs.us\/blog\/wp-json\/wp\/v2\/tags?post=478"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}$ mkdir \/c\/builds\r\n$ hg clone http:\/\/hg.mozilla.org\/mozilla-central \/c\/builds\/mozilla-central\r\n$ cd \/c\/builds\/mozilla-central\r\n$ autoconf-2.13 && (cd js\/src && autoconf-2.13)\r\n$ mkdir ff-debug\r\n$ cd ff-debug\r\n$ export MAKE='python -O c:\/builds\/mozilla-central\/build\/pymake\/make.py'\r\n$ ..\/configure --enable-application=browser --enable-debug --disable-optimize\r\n$ python -O ..\/build\/pymake\/make.py -j4\r\n<\/pre>\n
How to use pymake on Linux\/Mac<\/h3>\n
export MAKE=\"python -O $topsrcdir\/build\/pymake\/make.py\"\r\nmk_add_options MAKE=\"python -O @TOPSRCDIR@\/build\/pymake\/make.py\"\r\n<\/pre>\n
Thank you!<\/h3>\n
Next plans<\/h3>\n
\n
Python Reference Cycles and Closures<\/h3>\n
def outerFunction<\/span>(outerCallback):\r\n targetsToBuild = [1, 2, 3]\r\n def innerCallback<\/span>():\r\n if len(targetsToBuild):\r\n # innerCallback closes on itself... this creates a reference cycle every time you call outerFunction\r\n # if you call outerFunction 100000 times per build, this can add up really quickly and cause large GC pauses<\/span>\r\n targetsToBuild.pop(0).build(innerCallback<\/span>)\r\n else:\r\n outerCallback()\r\n<\/pre>\nEnvironment Munging in MSYS<\/h3>\n