{"id":137,"date":"2006-11-09T16:29:08","date_gmt":"2006-11-09T20:29:08","guid":{"rendered":"http:\/\/benjamin.smedbergs.us\/blog\/2006-11-09\/adventures-in-python-launching-subprocesses\/"},"modified":"2006-12-30T20:25:35","modified_gmt":"2006-12-31T00:25:35","slug":"adventures-in-python-launching-subprocesses","status":"publish","type":"post","link":"http:\/\/benjamin.smedbergs.us\/blog\/2006-11-09\/adventures-in-python-launching-subprocesses\/","title":{"rendered":"Adventures in Python: Launching Subprocesses"},"content":{"rendered":"<p>I&#8217;ve been looking at python for various build automation. I had what I thought would be a simple problem:<\/p>\n<h4>How do I launch a process, collecting stdout\/stderr, with a timeout to kill the process if it runs too long?<\/h4>\n<p>The python <a href=\"http:\/\/pydoc.org\/2.4.1\/subprocess.html\">subprocess<\/a> module gets about 80% there. You can launch a process, and hook up stdout\/stderr\/stdin. You can poll the process for completion. But subprocess doesn&#8217;t have a simple parameter for process timeout. Total time spent: 45 minutes.<\/p>\n<p>So, you use a loop or a thread to wait for the process and kill it if it takes too long, right? Subprocess doesn&#8217;t have an instance method to kill the process. Answer according to #python on freenode? <tt>os.kill(theprocess.pid, signal.SIGTERM)<\/tt>. Except that this apparently doesn&#8217;t work on Windows: you have to <a href=\"http:\/\/www.python.org\/infogami-faq\/windows\/how-do-i-emulate-os-kill-in-windows\/\">emulate it<\/a>. Total time spent: 1.5 hours.<\/p>\n<p>This works, on unixy systems. But it fails miserably on Windows. It turns out that on Windows when you kill a process, any subprocesses that were launched don&#8217;t get killed. So I went searching code that I thought must have already solved this problem: <a href=\"http:\/\/buildbot.sourceforge.net\/\">BuildBot<\/a> launches processes and has to kill them, right? Well, it turns out that BuildBot uses <a href=\"http:\/\/twistedmatrix.com\/trac\/\">Twisted<\/a> to do the dirty work. Twisted completely ignores the problem, as far as I can tell. It doesn&#8217;t use subprocess, but instead has a file called <tt>_dumbwin32proc.py<\/tt> which provides the event-driven access to the process pipes and status. This file is uglier than the devil&#8217;s rear end. Total time spent: 2.5 hours.<\/p>\n<p>After much pain, I found <a href=\"http:\/\/www.microsoft.com\/msj\/0698\/win320698.aspx\">Windows documentation<\/a> that might help: Windows 2000+ can put processes into jobs. Instead of killing the parent process, you can kill the entire job. As far as I can tell this should be implementable in Python, but I haven&#8217;t found anyone who&#8217;s done it yet (even better, abstracted it behind a cross-platform API). If you know of code which has this working properly, please let me know. Otherwise I will be spending another 4 hours tomorrow to get this working (I know only halting python, though I&#8217;m getting better quickly). Total time spent: 3.5 hours.<\/p>\n<p>Learning new languages isn&#8217;t that hard. Learning new programming worlds, with their bugs and quirks, is really hard.<\/p>\n<p>Update: Solution in my <a href=\"http:\/\/benjamin.smedbergs.us\/blog\/2006-12-11\/killableprocesspy\/\">post on killableprocess.py<\/a>.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>I&#8217;ve been looking at python for various build automation. I had what I thought would be a simple problem: How do I launch a process, collecting stdout\/stderr, with a timeout to kill the process if it runs too long? The python subprocess module gets about 80% there. You can launch a process, and hook up [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[2],"tags":[30,37,32],"class_list":["post-137","post","type-post","status-publish","format-standard","hentry","category-mozilla","tag-build-system","tag-killableprocess","tag-python"],"_links":{"self":[{"href":"http:\/\/benjamin.smedbergs.us\/blog\/wp-json\/wp\/v2\/posts\/137","targetHints":{"allow":["GET"]}}],"collection":[{"href":"http:\/\/benjamin.smedbergs.us\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/benjamin.smedbergs.us\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/benjamin.smedbergs.us\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"http:\/\/benjamin.smedbergs.us\/blog\/wp-json\/wp\/v2\/comments?post=137"}],"version-history":[{"count":0,"href":"http:\/\/benjamin.smedbergs.us\/blog\/wp-json\/wp\/v2\/posts\/137\/revisions"}],"wp:attachment":[{"href":"http:\/\/benjamin.smedbergs.us\/blog\/wp-json\/wp\/v2\/media?parent=137"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/benjamin.smedbergs.us\/blog\/wp-json\/wp\/v2\/categories?post=137"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/benjamin.smedbergs.us\/blog\/wp-json\/wp\/v2\/tags?post=137"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}