killableprocess.py
I’ve managed, at long last, to solve the problem of launching subprocesses from python. I have created a python module which can launch a subprocess, wait for the process with a timeout, and kill that process and all of its sub-subprocesses correctly, on Windows, Mac, and Linux. Source code is here. It requires python 2.4+ because it subclasses the subprocess module. On Windows, it only works on Win2k+, and it requires the ctypes module, which comes with Python 2.5+, or can be installed into earlier versions of Python.
You will be seeing a python-based tinderbox client appear on the MozillaTest tree shortly. Small projects or projects that don’t want or need the byzantine logic of the existing tinderbox client scripts can use a Python module to do tinderbox reporting using a simple object-oriented API. I’m hoping to use this to get Tamarin builds reporting to a tinderbox tree, as well as do some of the FF+XR build automation (which is significantly different from the existing build process).
December 12th, 2006 at 5:07 pm
Wohoo, thanks!
December 29th, 2006 at 6:23 am
Great! I saw your post describing the problem but missed finding this post for a few days! Great work!
March 1st, 2007 at 7:14 am
Not that you need it anymore, but I’ve been using the Process module in most of my progs, it does exactly what you describe. I’m surprised this module isn’t more popular:
http://trentm.com/projects/process/
import process
a = process.ProcessOpen(command)
a.wait(timeout=60)
March 8th, 2007 at 11:47 pm
Nice, useful bit of coding. I like the idea of sleeping and being woken up by the signal in the case where the subprocess returns before the timeout. It’s a bit cleaner than the alternative of creating a separate Timer thread to wake up and kill the process.
However, due to Python’s limitations, your solution only works in the main threads; which means that I cannot use in my multi-threaded application. Trying to call wait from a thread results in:
kp.wait(CMD_TIMEOUT) # wait for timeout; kill if runs over
File “killableprocess.py”, line 207, in wait
oldsignal = signal.signal(signal.SIGCHLD, DoNothing)
ValueError: signal only works in main thread
Due to this python limitation:
“Some care must be taken if both signals and threads are used in the same program. The fundamental thing to remember in using signals and threads simultaneously is: always perform signal() operations in the main thread of execution. Any thread can perform an alarm(), getsignal(), or pause(); only the main thread can set a new signal handler, and the main thread will be the only one to receive signals (this is enforced by the Python signal module, even if the underlying thread implementation supports sending signals to individual threads). This means that signals can’t be used as a means of inter-thread communication. Use locks instead.”
September 21st, 2007 at 8:30 pm
Thanks to the tips here, I’ve coded up a killableprocess wrapper in C++ that you can use from any language. When you launch a process with killableprocess.exe, it will automatically associate the child process with a job; when you kill killableprocess.exe, it will kill the job as well, taking all of the children with it.
http://darkforge.blogspot.com/2007/09/windows-killableprocess-and-case-of.html
October 25th, 2007 at 2:36 am
I’ve been looking for this procedure for quite a loooong time since I missed the python os.killpg() process….
Thanks to you!
October 31st, 2007 at 2:06 am
I’m here again…I hope not.
I met the same problem as John met. killableprocess doesn’t work in threads.That limits the usage.
Mabye I’ll try Dan Fabulich’s c++ program. But, anyone could find a decent way for pure python codes?
January 10th, 2008 at 9:32 am
Hi, found your python-modules after trying to tackle this problem. This effectively kills off the subprocess and it’s been a great deal of help for me in my unittesting module for catching timeouts in a platformindependent fashion. However, the version in svn performs a call to winprocess.SW_HIDE, which doesn’t exist in winprocess? I just commented it out, and it works well, but you probably added it for some reason? What value should it be assigned to?
January 10th, 2008 at 9:49 pm
Daniel, it seems I never committed the latest revision of winprocess, but that directory is gone now so you’d have to patch it. The constant for SW_HIDE is 0 (declared in winuser.h).
April 16th, 2008 at 8:10 pm
[…] module, but unfortunately even that has some limitations. There is an improvement called the killableprocess […]
April 29th, 2008 at 5:57 pm
Good work!
I like to integrate your killable subprocess into the official subprocess module of Python 2.6 and 3.0. I’ve already implemented a much simpler approach but your code goes beyond my implementation. Please contact me as soon as possible.
Christian
June 4th, 2008 at 11:52 am
Any sugestions on how to use killableprocess to launch multiple processes in parallel and return when they’ve either all completed, or return after a global timer expired, or return when one or more of the processes has a non-zero exit code?
September 8th, 2008 at 7:47 am
Hi,
where can I get the winprocess module from? it is not part of Python 2.5 on windows.
Thanks !
September 8th, 2008 at 1:20 pm
Gal, the winprocess.py file is in the SVN repository.
November 11th, 2008 at 9:23 pm
Hi, thanks a lot for this, works great. Hope it makes it into subprocess! (if it hasn’t already in a newer python)
February 16th, 2009 at 3:21 am
so how can we incorporate this script in buildbot for windows? I mean to get rid of the _dumbwin32proc.py.
March 6th, 2009 at 3:18 am
Thanks,
I spent hours pulling my hair out, trying to figure out why the pid was not matching up to the pid reported by task mangler, er manager. I finally gave up, and ran across this little gem.
Works great , only a couple minor tweaks to get it to run on python 3.0.1.
-Mark
March 6th, 2009 at 10:29 pm
[…] timeouts for all of the commands. Running external commands that can be killed after a timeout is difficult in […]
June 9th, 2010 at 6:57 am
Hi,
Whenever I run the “killableprocess.Popen(command)” piece of code in my script I get the following error:
File “killableprocess.py”, line 157, in _execute_child
winprocess.AssignProcessToJobObject(self._job, hp)
File “winprocess.py”, line 37, in ErrCheckBool
raise WinError()
WindowsError [error 5] Access is denied
Exception TypeError: “‘NoneType’ object is not callable” in <bound method AutoHANDLE.__del__ of > ignored
But when I run it from the python interactive console (python 2.6) it works fine.
That probably means there are permission issues when I run this from the script, but I don’t know how to solve them. I tried running the script from a cmd that I ran as administrator, but it didn’t help.
Tried looking for similar posts but didn’t find any good solution. Hoping to find help here.
I’m running on Windows, specifically Windows 7 Ultimate x64, if it’s any help.
thanks
October 19th, 2010 at 10:03 am
Hi,
First, thank you very much for the script. I have been able to launch and kill processes at will using your script.
Windows XP works fine, however, Windows 7 (x64) is giving me the same problem that Sergei described above.
WindowsError [error 5] Access is denied
This exception is raised whenever I launch a command through killableprocess.popen(command)
Has anyone run into this problem and can someone suggest a workaround for this ?
Thanks in advance,
Regards,
Tom
February 22nd, 2011 at 7:02 pm
Re: the “WindowsError [error 5] Access is denied” issue on Win 7:
The ErrCheckBool function will always raise WinError() if I start my scripts via a shortcut or by double clicking on them and the subprocesses hang. If run from the command line (or .bat) file, there’s no problem.
Workarounds are:
1. running your script from a .bat file or the command line;
2. commenting out lines 36 & 37 in winprocess.py
April 12th, 2012 at 8:26 am
‘Access denied’ error on Windows may be caused by the fact that you’re trying to run the program that is being monitored by the Program Compatibility Assistant (PCA). In that case such program will be run under special PCA job. This will prevent the killableprocess.py to attach the program to another job object (this job object is used by killableprocess.py to be able to kill all the child processes). In order to overcome this problem, the ‘CREATE_BREAKAWAY_FROM_JOB’ flag must be used while creating the program process and before attaching it to the job object.
Just add the following lines just before winprocess.CreateProcess call:
# If the process is being monitored by the Program Compatibility Assistant (PCA),
# it is placed into a compatibility job.
# Therefore, the process must be created using CREATE_BREAKAWAY_FROM_JOB before it can be placed in another job
creationflags |= winprocess.CREATE_BREAKAWAY_FROM_JOB
Regards,
Vlad
January 19th, 2015 at 7:50 pm
[…] used killableprocess successfully on Windows, Linux and Mac. If you are using Cygwin Python, you’ll need […]