As promised, I have created a python module which can be used to implement a tinderbox client to report arbitrary. I’ve also created a driver which can pull and build a Mozilla-like application. Sources here. I’ve tested it on Windows and Linux, but I fully expect it would work on Mac as well. It requires Python 2.4 and the killableprocess module.

I know it’s not especially obvious how to actually run a build. I use the following command line:

python mozbuild.py --config=/builds/tinderclient/firefox-config.py,/builds/tinderclient/sample-config.py --private-config=/builds/bs-passwords.py

Take a look at the MozillaTest tinderbox logs from Thursday to see the results of my test builds. Please note that anyone is welcome to run a tinderbox that reports to MozillaTest; it’s a place to test tinderbox scripts! If you want to start reporting to an official tree like MozillaExperimental or SeaMonkey-Ports, please ask permission from build@mozilla.org.

For those keeping track of killableprocess, I’ve committed some changes:

• on *nix, create and kill process groups properly;
• on Windows, allow redirecting the standard handles to files (instead of pipes);
• on Windows, allow passing a dictionary for the environment of the new process to create.

Right now this is a technology experiment. We’ll probably use it to drive a Tamarin tinderbox. It’s vaguely possible that Mozilla will switch away from the old-style perl tinderbox client altogether going forward, but that requires replicating a lot of logic, and might not be worth it.

Sometime after the new year, I will be adding a driver script which can perform builds in a loop, perhaps with config updating from CVS the way the current tinderbox scripts do.

I’ve managed, at long last, to solve the problem of launching subprocesses from python. I have created a python module which can launch a subprocess, wait for the process with a timeout, and kill that process and all of its sub-subprocesses correctly, on Windows, Mac, and Linux. Source code is here. It requires python 2.4+ because it subclasses the subprocess module. On Windows, it only works on Win2k+, and it requires the ctypes module, which comes with Python 2.5+, or can be installed into earlier versions of Python.

You will be seeing a python-based tinderbox client appear on the MozillaTest tree shortly. Small projects or projects that don’t want or need the byzantine logic of the existing tinderbox client scripts can use a Python module to do tinderbox reporting using a simple object-oriented API. I’m hoping to use this to get Tamarin builds reporting to a tinderbox tree, as well as do some of the FF+XR build automation (which is significantly different from the existing build process).

I’ve been looking at python for various build automation. I had what I thought would be a simple problem:

How do I launch a process, collecting stdout/stderr, with a timeout to kill the process if it runs too long?

The python subprocess module gets about 80% there. You can launch a process, and hook up stdout/stderr/stdin. You can poll the process for completion. But subprocess doesn’t have a simple parameter for process timeout. Total time spent: 45 minutes.

So, you use a loop or a thread to wait for the process and kill it if it takes too long, right? Subprocess doesn’t have an instance method to kill the process. Answer according to #python on freenode? os.kill(theprocess.pid, signal.SIGTERM). Except that this apparently doesn’t work on Windows: you have to emulate it. Total time spent: 1.5 hours.

This works, on unixy systems. But it fails miserably on Windows. It turns out that on Windows when you kill a process, any subprocesses that were launched don’t get killed. So I went searching code that I thought must have already solved this problem: BuildBot launches processes and has to kill them, right? Well, it turns out that BuildBot uses Twisted to do the dirty work. Twisted completely ignores the problem, as far as I can tell. It doesn’t use subprocess, but instead has a file called _dumbwin32proc.py which provides the event-driven access to the process pipes and status. This file is uglier than the devil’s rear end. Total time spent: 2.5 hours.

After much pain, I found Windows documentation that might help: Windows 2000+ can put processes into jobs. Instead of killing the parent process, you can kill the entire job. As far as I can tell this should be implementable in Python, but I haven’t found anyone who’s done it yet (even better, abstracted it behind a cross-platform API). If you know of code which has this working properly, please let me know. Otherwise I will be spending another 4 hours tomorrow to get this working (I know only halting python, though I’m getting better quickly). Total time spent: 3.5 hours.

Learning new languages isn’t that hard. Learning new programming worlds, with their bugs and quirks, is really hard.

Update: Solution in my post on killableprocess.py.