killableprocess.py

I’ve managed, at long last, to solve the problem of launching subprocesses from python. I have created a python module which can launch a subprocess, wait for the process with a timeout, and kill that process and all of its sub-subprocesses correctly, on Windows, Mac, and Linux. Source code is here. It requires python 2.4+ because it subclasses the subprocess module. On Windows, it only works on Win2k+, and it requires the ctypes module, which comes with Python 2.5+, or can be installed into earlier versions of Python.

You will be seeing a python-based tinderbox client appear on the MozillaTest tree shortly. Small projects or projects that don’t want or need the byzantine logic of the existing tinderbox client scripts can use a Python module to do tinderbox reporting using a simple object-oriented API. I’m hoping to use this to get Tamarin builds reporting to a tinderbox tree, as well as do some of the FF+XR build automation (which is significantly different from the existing build process).

Atom Feed for Comments 22 Responses to “killableprocess.py”

  1. Heikki Toivonen Says:

    Wohoo, thanks!

  2. d1223m Says:

    Great! I saw your post describing the problem but missed finding this post for a few days! Great work!

  3. wrybread Says:

    Not that you need it anymore, but I’ve been using the Process module in most of my progs, it does exactly what you describe. I’m surprised this module isn’t more popular:

    http://trentm.com/projects/process/

    import process
    a = process.ProcessOpen(command)
    a.wait(timeout=60)

  4. John Says:

    Nice, useful bit of coding. I like the idea of sleeping and being woken up by the signal in the case where the subprocess returns before the timeout. It’s a bit cleaner than the alternative of creating a separate Timer thread to wake up and kill the process.

    However, due to Python’s limitations, your solution only works in the main threads; which means that I cannot use in my multi-threaded application. Trying to call wait from a thread results in:

    kp.wait(CMD_TIMEOUT) # wait for timeout; kill if runs over
    File “killableprocess.py”, line 207, in wait
    oldsignal = signal.signal(signal.SIGCHLD, DoNothing)
    ValueError: signal only works in main thread

    Due to this python limitation:

    “Some care must be taken if both signals and threads are used in the same program. The fundamental thing to remember in using signals and threads simultaneously is: always perform signal() operations in the main thread of execution. Any thread can perform an alarm(), getsignal(), or pause(); only the main thread can set a new signal handler, and the main thread will be the only one to receive signals (this is enforced by the Python signal module, even if the underlying thread implementation supports sending signals to individual threads). This means that signals can’t be used as a means of inter-thread communication. Use locks instead.”

  5. Dan Fabulich Says:

    Thanks to the tips here, I’ve coded up a killableprocess wrapper in C++ that you can use from any language. When you launch a process with killableprocess.exe, it will automatically associate the child process with a job; when you kill killableprocess.exe, it will kill the job as well, taking all of the children with it.

    http://darkforge.blogspot.com/2007/09/windows-killableprocess-and-case-of.html

  6. Yu Wang Says:

    I’ve been looking for this procedure for quite a loooong time since I missed the python os.killpg() process….
    Thanks to you!

  7. Yu Wang Says:

    I’m here again…I hope not.
    I met the same problem as John met. killableprocess doesn’t work in threads.That limits the usage.
    Mabye I’ll try Dan Fabulich’s c++ program. But, anyone could find a decent way for pure python codes?

  8. Daniel Svensson Says:

    Hi, found your python-modules after trying to tackle this problem. This effectively kills off the subprocess and it’s been a great deal of help for me in my unittesting module for catching timeouts in a platformindependent fashion. However, the version in svn performs a call to winprocess.SW_HIDE, which doesn’t exist in winprocess? I just commented it out, and it works well, but you probably added it for some reason? What value should it be assigned to?

  9. Benjamin Smedberg Says:

    Daniel, it seems I never committed the latest revision of winprocess, but that directory is gone now so you’d have to patch it. The constant for SW_HIDE is 0 (declared in winuser.h).

  10. California Dreams » Blog Archive » Python’s os.system Considered Harmful Says:

    [...] module, but unfortunately even that has some limitations. There is an improvement called the killableprocess [...]

  11. Christian Heimes Says:

    Good work!

    I like to integrate your killable subprocess into the official subprocess module of Python 2.6 and 3.0. I’ve already implemented a much simpler approach but your code goes beyond my implementation. Please contact me as soon as possible.

    Christian

  12. Martin Says:

    Any sugestions on how to use killableprocess to launch multiple processes in parallel and return when they’ve either all completed, or return after a global timer expired, or return when one or more of the processes has a non-zero exit code?

  13. Aviel, Gal Says:

    Hi,

    where can I get the winprocess module from? it is not part of Python 2.5 on windows.
    Thanks !

  14. Benjamin Smedberg Says:

    Gal, the winprocess.py file is in the SVN repository.

  15. Scott Says:

    Hi, thanks a lot for this, works great. Hope it makes it into subprocess! (if it hasn’t already in a newer python)

  16. Jimmy Says:

    so how can we incorporate this script in buildbot for windows? I mean to get rid of the _dumbwin32proc.py.

  17. Mark Says:

    Thanks,

    I spent hours pulling my hair out, trying to figure out why the pid was not matching up to the pid reported by task mangler, er manager. I finally gave up, and ran across this little gem.

    Works great , only a couple minor tweaks to get it to run on python 3.0.1.

    -Mark

  18. Importing fabric into your own scripts | California Dreams Says:

    [...] timeouts for all of the commands. Running external commands that can be killed after a timeout is difficult in [...]

  19. Sergei Says:

    Hi,

    Whenever I run the “killableprocess.Popen(command)” piece of code in my script I get the following error:

    File “killableprocess.py”, line 157, in _execute_child
    winprocess.AssignProcessToJobObject(self._job, hp)
    File “winprocess.py”, line 37, in ErrCheckBool
    raise WinError()
    WindowsError [error 5] Access is denied
    Exception TypeError: “‘NoneType’ object is not callable” in <bound method AutoHANDLE.__del__ of > ignored

    But when I run it from the python interactive console (python 2.6) it works fine.
    That probably means there are permission issues when I run this from the script, but I don’t know how to solve them. I tried running the script from a cmd that I ran as administrator, but it didn’t help.
    Tried looking for similar posts but didn’t find any good solution. Hoping to find help here.

    I’m running on Windows, specifically Windows 7 Ultimate x64, if it’s any help.

    thanks

  20. Tom Says:

    Hi,

    First, thank you very much for the script. I have been able to launch and kill processes at will using your script.

    Windows XP works fine, however, Windows 7 (x64) is giving me the same problem that Sergei described above.

    WindowsError [error 5] Access is denied
    This exception is raised whenever I launch a command through killableprocess.popen(command)

    Has anyone run into this problem and can someone suggest a workaround for this ?

    Thanks in advance,

    Regards,

    Tom

  21. Luke Says:

    Re: the “WindowsError [error 5] Access is denied” issue on Win 7:

    The ErrCheckBool function will always raise WinError() if I start my scripts via a shortcut or by double clicking on them and the subprocesses hang. If run from the command line (or .bat) file, there’s no problem.

    Workarounds are:
    1. running your script from a .bat file or the command line;
    2. commenting out lines 36 & 37 in winprocess.py

  22. Vlad Dovlekaev Says:

    ‘Access denied’ error on Windows may be caused by the fact that you’re trying to run the program that is being monitored by the Program Compatibility Assistant (PCA). In that case such program will be run under special PCA job. This will prevent the killableprocess.py to attach the program to another job object (this job object is used by killableprocess.py to be able to kill all the child processes). In order to overcome this problem, the ‘CREATE_BREAKAWAY_FROM_JOB’ flag must be used while creating the program process and before attaching it to the job object.

    Just add the following lines just before winprocess.CreateProcess call:

    # If the process is being monitored by the Program Compatibility Assistant (PCA),
    # it is placed into a compatibility job.
    # Therefore, the process must be created using CREATE_BREAKAWAY_FROM_JOB before it can be placed in another job
    creationflags |= winprocess.CREATE_BREAKAWAY_FROM_JOB

    Regards,
    Vlad

Leave a Reply