{"id":433,"date":"2009-02-13T15:57:49","date_gmt":"2009-02-13T19:57:49","guid":{"rendered":"http:\/\/benjamin.smedbergs.us\/blog\/?p=433"},"modified":"2009-02-13T16:29:00","modified_gmt":"2009-02-13T20:29:00","slug":"pymake","status":"publish","type":"post","link":"https:\/\/benjamin.smedbergs.us\/blog\/2009-02-13\/pymake\/","title":{"rendered":"pymake: A Mostly GNU-compatible `make`, Written in Python"},"content":{"rendered":"<p>Mozilla has a lot of Makefiles, and although we&#8217;re not completely happy with our current recursive-make build system, converting to something else <a href=\"http:\/\/blog.mozilla.com\/ted\/2009\/02\/09\/getting-there-from-here\/\">is a very difficult task<\/a>. In order to consider other solutions, we have to have a seamless migration path from our existing infrastructure. Unfortunately for any potential migration, Mozilla uses pretty much every make feature and quirk imaginable. It isn&#8217;t sufficient to get by with some simplistic importer: any migration needs to understand the arcane details of makefile syntax and parsing rules.<\/p>\n<p>So finally, after two years of considering whether it&#8217;s feasible and a good idea, I bit the bullet and wrote a make replacement in python. It was almost as mind-numbingly difficult as I thought it would be. But after two solid weeks, I have a tool which will mostly build Mozilla, and seems to be doing it correctly.<\/p>\n<h3>Why would you do such a thing?<\/h3>\n<p><a href=\"http:\/\/diveintomark.org\/archives\/2007\/01\/21\/wrongroom\">Mark Pilgrim<\/a> was right: only a crazy person says &#8220;I&#8217;m going to build my own build system&#8221;. So rather than creating a whole new build system with a new syntax, I wanted to start out replicating an existing build system exactly. Then we can make incremental changes if appropriate.<\/p>\n<h3>What kind of incremental changes?<\/h3>\n<p>First up are speed increases. Our Windows builds are currently very slow, in part due to the large number of processes that are spawned. There are several ways to attack this:<\/p>\n<ul>\n<li>Each makefile command currently spawns an MSYS shell and <em>then<\/em> (usually) the actual binary. The MSYS shell is expensive and in most cases unnecessary. pymake should be able to skip the shell for simple command lines.\n<li>Mozilla runs `nsinstall` a lot. Axel has already implemented <a href=\"http:\/\/mxr.mozilla.org\/mozilla-central\/source\/config\/nsinstall.py\">nsinstall as a python module<\/a>: our makefile rules should be able to run commands in python and skip the external process.\n<li>We can use recursive make without launching additional make processes by recursing within the same pymake process.\n<\/ul>\n<h3>Where can I see this monstrosity?<\/h3>\n<p><a href=\"http:\/\/benjamin.smedbergs.us\/pymake\/\">The pymake website<\/a>. You can also just <a href=\"http:\/\/hg.mozilla.org\/users\/bsmedberg_mozilla.com\/pymake\">pull or browser the pymake sources<\/a> directly from hg.mozilla.org using Mercurial.<\/p>\n<h3>Why don&#8217;t you just hack GNU make itself?<\/h3>\n<p>There are some improvements we&#8217;re interested in doing which just aren&#8217;t possible within a C codebase:<\/p>\n<ul>\n<li>implementing rules directly in Python\n<li>Condensing multiple invocations of make into a single process<\/ul>\n<p>. Python is also about a zillion times easier to hack quickly.<\/p>\n<h3>pymake is <em>mostly<\/em> GNU-make compatible. What are the differences?<\/h3>\n<p>The known differences are documented in <a href=\"http:\/\/hg.mozilla.org\/users\/bsmedberg_mozilla.com\/pymake\/file\/default\/README\">the README file<\/a>.<\/p>\n<h3>Does it perform well?<\/h3>\n<p>Not yet. pymake currently takes more time than GNU make. Almost all of this time is spent reading and parsing makefiles. Parsing autoconf.mk\/config.mk\/rules.mk from the Mozilla tree takes 0.3 seconds on my Linux machine, where GNU make takes 0.03 seconds. Dependency resolution and execution is as fast or faster than GNU make. I&#8217;d love to have some help profiling and fixing the parsing. Also, pymake does not yet support parallel execution.<\/p>\n<p>Next week sometime, I promise to write about some of the difficulties I encountered writing pymake, and how test-driven development saved me from terrible pain and suffering.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Mozilla has a lot of Makefiles, and although we&#8217;re not completely happy with our current recursive-make build system, converting to something else is a very difficult task. In order to consider other solutions, we have to have a seamless migration path from our existing infrastructure. Unfortunately for any potential migration, Mozilla uses pretty much every [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[2,6],"tags":[182,210,32],"class_list":["post-433","post","type-post","status-publish","format-standard","hentry","category-mozilla","category-untagged","tag-make","tag-pymake","tag-python"],"_links":{"self":[{"href":"https:\/\/benjamin.smedbergs.us\/blog\/wp-json\/wp\/v2\/posts\/433","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/benjamin.smedbergs.us\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/benjamin.smedbergs.us\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/benjamin.smedbergs.us\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/benjamin.smedbergs.us\/blog\/wp-json\/wp\/v2\/comments?post=433"}],"version-history":[{"count":6,"href":"https:\/\/benjamin.smedbergs.us\/blog\/wp-json\/wp\/v2\/posts\/433\/revisions"}],"predecessor-version":[{"id":439,"href":"https:\/\/benjamin.smedbergs.us\/blog\/wp-json\/wp\/v2\/posts\/433\/revisions\/439"}],"wp:attachment":[{"href":"https:\/\/benjamin.smedbergs.us\/blog\/wp-json\/wp\/v2\/media?parent=433"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/benjamin.smedbergs.us\/blog\/wp-json\/wp\/v2\/categories?post=433"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/benjamin.smedbergs.us\/blog\/wp-json\/wp\/v2\/tags?post=433"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}