Adventures in Python: Launching Subprocesses
I’ve been looking at python for various build automation. I had what I thought would be a simple problem:
How do I launch a process, collecting stdout/stderr, with a timeout to kill the process if it runs too long?
The python subprocess module gets about 80% there. You can launch a process, and hook up stdout/stderr/stdin. You can poll the process for completion. But subprocess doesn’t have a simple parameter for process timeout. Total time spent: 45 minutes.
So, you use a loop or a thread to wait for the process and kill it if it takes too long, right? Subprocess doesn’t have an instance method to kill the process. Answer according to #python on freenode? os.kill(theprocess.pid, signal.SIGTERM). Except that this apparently doesn’t work on Windows: you have to emulate it. Total time spent: 1.5 hours.
This works, on unixy systems. But it fails miserably on Windows. It turns out that on Windows when you kill a process, any subprocesses that were launched don’t get killed. So I went searching code that I thought must have already solved this problem: BuildBot launches processes and has to kill them, right? Well, it turns out that BuildBot uses Twisted to do the dirty work. Twisted completely ignores the problem, as far as I can tell. It doesn’t use subprocess, but instead has a file called _dumbwin32proc.py which provides the event-driven access to the process pipes and status. This file is uglier than the devil’s rear end. Total time spent: 2.5 hours.
After much pain, I found Windows documentation that might help: Windows 2000+ can put processes into jobs. Instead of killing the parent process, you can kill the entire job. As far as I can tell this should be implementable in Python, but I haven’t found anyone who’s done it yet (even better, abstracted it behind a cross-platform API). If you know of code which has this working properly, please let me know. Otherwise I will be spending another 4 hours tomorrow to get this working (I know only halting python, though I’m getting better quickly). Total time spent: 3.5 hours.
Learning new languages isn’t that hard. Learning new programming worlds, with their bugs and quirks, is really hard.
Update: Solution in my post on killableprocess.py.
November 9th, 2006 at 4:54 pm
You forgot the “Priceless” punchline, a la “Still being unable to kill a process reliably in Windows after 20 years? Priceless” :-)
November 9th, 2006 at 5:47 pm
If you write the code for managing processes using jobs on Windows, it’d be great if you could contribute it to Twisted. It’s been on the wish list for a while now, but there aren’t many Twisted contributors with Windows experience.
November 9th, 2006 at 6:18 pm
You could probably crib code from Perl: Win32::Job
November 9th, 2006 at 6:49 pm
In my experience getting buildbot to kill a build on Windows before it’s completed is a nightmare, exactly because some of the processes don’t die and conflict with the new build you kick off.
November 9th, 2006 at 7:04 pm
Benjamin, try this one: http://trentm.com/projects/process/
This is the one we wrote for Komodo. I’d be interested in hearing whether it works for you. It’s not perfect but it does have support for waiting for process completion and killing it (on Windows too).
November 9th, 2006 at 7:37 pm
Python 2.5 includes ctypes module which will let you call into Windows DLLs relatively easily, which should enable you to call the real Windows APIs to kill processes etc.
And yeah, we need this at OSAF as well, just haven’t gotten to it yet…
November 10th, 2006 at 12:56 am
I want this too! Are there any open tickets about including a library that does this? Maybe we can vote for it? I’d be happy to send guido some beer to see this included :)
November 22nd, 2006 at 12:26 pm
Try
win32api.TerminateProcess(int(theprocess._handle),-1)
December 11th, 2006 at 10:44 pm
killableprocess.py…
I’ve managed, at long last, to solve the problem of launching subprocesses from python. I have created a python module which can launch a subprocess, wait for the process with a timeout, and kill that process and all of its sub-subprocesses corre…
September 27th, 2007 at 7:00 am
[…] In Windows there are apparently some horrible compromises to be made when doing some simple things. See …adventures-in-python-launching-subprocesses/ […]
April 21st, 2008 at 3:28 am
I’m having the same problem now.
I’m writing a mencoder GUI in Python and I need to launch mencoder from within a GUI and offer a way for the user to abort encoding at any time. This post has given me the shivers now as to how I can achieve it.
I want to try to use standard Python as much as possible and without resorting to external modules or hacks. But is there a way?