Adventures in Python: Launching Subprocesses

I’ve been looking at python for various build automation. I had what I thought would be a simple problem:

How do I launch a process, collecting stdout/stderr, with a timeout to kill the process if it runs too long?

The python subprocess module gets about 80% there. You can launch a process, and hook up stdout/stderr/stdin. You can poll the process for completion. But subprocess doesn’t have a simple parameter for process timeout. Total time spent: 45 minutes.

So, you use a loop or a thread to wait for the process and kill it if it takes too long, right? Subprocess doesn’t have an instance method to kill the process. Answer according to #python on freenode? os.kill(, signal.SIGTERM). Except that this apparently doesn’t work on Windows: you have to emulate it. Total time spent: 1.5 hours.

This works, on unixy systems. But it fails miserably on Windows. It turns out that on Windows when you kill a process, any subprocesses that were launched don’t get killed. So I went searching code that I thought must have already solved this problem: BuildBot launches processes and has to kill them, right? Well, it turns out that BuildBot uses Twisted to do the dirty work. Twisted completely ignores the problem, as far as I can tell. It doesn’t use subprocess, but instead has a file called which provides the event-driven access to the process pipes and status. This file is uglier than the devil’s rear end. Total time spent: 2.5 hours.

After much pain, I found Windows documentation that might help: Windows 2000+ can put processes into jobs. Instead of killing the parent process, you can kill the entire job. As far as I can tell this should be implementable in Python, but I haven’t found anyone who’s done it yet (even better, abstracted it behind a cross-platform API). If you know of code which has this working properly, please let me know. Otherwise I will be spending another 4 hours tomorrow to get this working (I know only halting python, though I’m getting better quickly). Total time spent: 3.5 hours.

Learning new languages isn’t that hard. Learning new programming worlds, with their bugs and quirks, is really hard.

Update: Solution in my post on

Atom Feed for Comments 11 Responses to “Adventures in Python: Launching Subprocesses”

  1. Preed Says:

    You forgot the “Priceless” punchline, a la “Still being unable to kill a process reliably in Windows after 20 years? Priceless” :-)

  2. Jean-Paul Says:

    If you write the code for managing processes using jobs on Windows, it’d be great if you could contribute it to Twisted. It’s been on the wish list for a while now, but there aren’t many Twisted contributors with Windows experience.

  3. Chris Dolan Says:

    You could probably crib code from Perl: Win32::Job

  4. Jonathan Watt Says:

    In my experience getting buildbot to kill a build on Windows before it’s completed is a nightmare, exactly because some of the processes don’t die and conflict with the new build you kick off.

  5. Trent Mick Says:

    Benjamin, try this one:
    This is the one we wrote for Komodo. I’d be interested in hearing whether it works for you. It’s not perfect but it does have support for waiting for process completion and killing it (on Windows too).

  6. Heikki Toivonen Says:

    Python 2.5 includes ctypes module which will let you call into Windows DLLs relatively easily, which should enable you to call the real Windows APIs to kill processes etc.

    And yeah, we need this at OSAF as well, just haven’t gotten to it yet…

  7. Ian Hummel Says:

    I want this too! Are there any open tickets about including a library that does this? Maybe we can vote for it? I’d be happy to send guido some beer to see this included :)

  8. Mike Elkins Says:


  9. BSBlog Says:…

    I’ve managed, at long last, to solve the problem of launching subprocesses from python. I have created a python module which can launch a subprocess, wait for the process with a timeout, and kill that process and all of its sub-subprocesses corre…

  10. pssst … » Running MySQL scripts (.sql files) from python Says:

    […] In Windows there are apparently some horrible compromises to be made when doing some simple things. See …adventures-in-python-launching-subprocesses/ […]

  11. hari Says:

    I’m having the same problem now.

    I’m writing a mencoder GUI in Python and I need to launch mencoder from within a GUI and offer a way for the user to abort encoding at any time. This post has given me the shivers now as to how I can achieve it.

    I want to try to use standard Python as much as possible and without resorting to external modules or hacks. But is there a way?

Leave a Reply