When a project like Tamarin is released as open-source, there is naturally a lot of interest in the project. I had a meeting with Eric Shepherd, Jeff Dyer, and Steven Johnson to coordinate a documentation plan for the Tamarin project. This is going to be an incremental process, starting with docs that already exist or are being created within Adobe. We have also identified various documents that we would like to create. I have put up notes from the meeting, for future reference.

If you have questions about Tamarin, feel free to ask them in the mozilla.dev.tech.js-engine newsgroup, or the #tamarin channel on irc.mozilla.org. Once you get your answer, please write an article for the Tamarin documentation on the Mozilla Developer Center.

I’ve been looking at python for various build automation. I had what I thought would be a simple problem:

How do I launch a process, collecting stdout/stderr, with a timeout to kill the process if it runs too long?

The python subprocess module gets about 80% there. You can launch a process, and hook up stdout/stderr/stdin. You can poll the process for completion. But subprocess doesn’t have a simple parameter for process timeout. Total time spent: 45 minutes.

So, you use a loop or a thread to wait for the process and kill it if it takes too long, right? Subprocess doesn’t have an instance method to kill the process. Answer according to #python on freenode? os.kill(theprocess.pid, signal.SIGTERM). Except that this apparently doesn’t work on Windows: you have to emulate it. Total time spent: 1.5 hours.

This works, on unixy systems. But it fails miserably on Windows. It turns out that on Windows when you kill a process, any subprocesses that were launched don’t get killed. So I went searching code that I thought must have already solved this problem: BuildBot launches processes and has to kill them, right? Well, it turns out that BuildBot uses Twisted to do the dirty work. Twisted completely ignores the problem, as far as I can tell. It doesn’t use subprocess, but instead has a file called _dumbwin32proc.py which provides the event-driven access to the process pipes and status. This file is uglier than the devil’s rear end. Total time spent: 2.5 hours.

After much pain, I found Windows documentation that might help: Windows 2000+ can put processes into jobs. Instead of killing the parent process, you can kill the entire job. As far as I can tell this should be implementable in Python, but I haven’t found anyone who’s done it yet (even better, abstracted it behind a cross-platform API). If you know of code which has this working properly, please let me know. Otherwise I will be spending another 4 hours tomorrow to get this working (I know only halting python, though I’m getting better quickly). Total time spent: 3.5 hours.

Learning new languages isn’t that hard. Learning new programming worlds, with their bugs and quirks, is really hard.

Update: Solution in my post on killableprocess.py.

As the owner of the Mozilla build system, I get to hear endless moaning about how byzantine the build system is and how it would be nice to use something more modern. I sympathize with the problem: our build system is difficult to set up, has a very steep learning curve, and uses lots of unusual languages. On the other hand, our build system works amazingly well: we have grown a set of rules that build an amazing variety of file types, and works on a wide variety of platforms. Out build system is complex for the very good reason that our needs are complex.

A few of the more aware complainers have suggested alternatives such as SCons. I have avoided making huge changes to our build system because it is overwhelming to contemplate migrating our existing build logic to a new build system. Any new build system would have to compare favorably to our existing system, which has years of bugfixes and features to its credit.

I was excited when I was asked to help implement a cross-platform build system for the Tamarin project. Tamarin was released with platform-specific Visual Studio and XCode project files. I thought that perhaps this was a perfect opportunity to try one of the newer build systems, without paying the cost of porting our entire existing build system over.

I created a requirements list. Unfortunately, I don’t think that any of the existing tools are going to meet our needs without extensive customization. I’m going to continue to examine SCons and WAF more over the next few days, but it looks like the path of easiest success is still going to be an old-school autoconf + make build system. Hopefully at least I’ll be able to fix some issues with recursive make while implementing the new system.

I’ve just posted version 0.6 of the WordPress plugin to generate Atom 1.0 feeds. It incorporates some changes sent to me by Jens Alfke, including making the feed work correctly in Safari. Thanks Jens!

I’d like to welcome Asaf Romano (mano) and Gavin Sharp as full toolkit peers: if you’re not up on the checkin rules, this means that they can review patches in any area of the toolkit. I’m excited that we have people available to replace inactive peers as time goes by and keep a healthy project.

Also, I posted on the Mozilla Developer News blog about a new unit-testing policy that I’m implementing for all new checkins to the toolkit. Please read!

Each year Seneca college (Toronto) hosts FSOSS: the free software & open source symposium. There will be quite a few Mozillians at this year’s conference: Neil Deakin and I are both giving tutorials, on XUL and XULRunner respectively. Eric Shepherd will be speaking on developer documentation, and Mike Shaver will be giving the opening keynote.

If you are in Northern America and have wanted to go to a conference but the big conferences were either too expensive (OSCON) or far away (FOSDEM), I heartily recommend this one: it’s inexpensive ($20-30), Toronto isn’t that hard to get to, and you’ll get a chance to meet some great people.

I’ve prepared a couple new software releases this week:

Firefox Release Repackager (1.2)

Adds support for repacking the NSIS installers used by Firefox 2.

about:config fixer

Fix about:config to display the chrome URI of localized prefs, instead of the localized value.

Is Ubuntu an operating system? Last week at EuroOSCON, Mark Shuttleworth gave the closing keynote outlining what he believes are the major struggles faced by the open-source/free-software community. During his talk, it became clear that Ubuntu is trying to achieve a radical shift in the software world. Ubuntu isn’t trying to be a platform for mass-market application software: it is trying to be the primary provider of both the operating system and all the application software that a typical user would want to run on his machine. Most Linux distributions are like this, and I think it is a dangerous trend that will stifle innovation and usability, or even worse make the desktop irrelevant.
(more…)

For a long time, the Mozilla codebase has contained an API for embedding the rendering engine into GTK applications. This API, called gtkmozembed, is used by projects such as Epiphany. For a long time this API has not been strongly-owned, and ownership fell to me by default. Fortunately, new users of Linux embedding have stepped forward, and I am pleased to announce that Josh Soref (timeless) is taking ownership of the Linux embedding API.

The change of ownership will allow the embedding API to grow according to the needs of embedding clients; these include traditional desktop clients as well as embedders on smaller devices such as PDAs and phones. Josh will soon write a summary of future goals for Linux embedding to the mozilla.dev.embedding newsgroup. Accomplishing these goals may involve additions to the existing gtkmozembed API, a backwards-compatibility layer to support gtkmozembed while promoting a new API, or replacing gtkmozembed outright. I would encourage all current users of gtkmozembed to follow discussions in the mozilla.dev.embedding newsgroup.

In 1998, Peter Miller wrote an essay Recursive Make Considered Harmful (PDF). He outlines several convincing arguments for why the traditional practice of recursing through directories and calling make in each directory may not be a good technique.

Mozilla uses recursive make extensively. In fact, during a typical build we traverse each directory in the build tree at least twice, and in some cases three or even four times. So, way back in 2002, Chris Seawood filed a bug to reduce the number of subdirectories we traverse during a typical build. This had the promise of significantly reducing build time on Windows because forking processes on Windows (and especially under cygwin) is really expensive.

Unfortunately, it turned out that a non-recursive make was taking longer than the standard recursive make. I thought, at the time, that this was due to the use of extra $(shell cygpath) subprocesses needed to handle sources from multiple subdirectories correctly. About a month ago, at schrep‘s prompting, I went back to see whether there was a way to get it working. The results are below:

It turns out that the problem with non-recursive make, at least as I currently have it coded, is not the cost of forking processes, but the cost of statting files that don’t exist. Every time a makefile adds a directory to VPATH, make ends up searching through additional directories looking for files. As you can see from the chart, the actual processor time spent by the non-recursive process is a lot less; it takes longer in wall-clock time only because it’s waiting on disk I/O.

I’m hoping to try another technique for non-recursive make that doesn’t set the VPATH for each source directory. Unfortunately, this is going to involve some changes to the dependency system (which I don’t understand very well), so I don’t know when I’m going to find the time.

My technique involved the use of the $(eval) makefile function, which is only available in gmake 3.80 or higher. The tests were performed under MSYS, using GNU make 3.81, available from MSYS snapshots. Special thanks to Earnie Boyd for patiently dealing with me and pointing me at the right files.