Python多重处理-将输入发送到子进程

Question

I am using the multiprocessing module in python to launch few processes in parallel. These processes are independent of each other. They generate their own output and write out the results in different files. Each process calls an external tool using the subprocess.call method. It was working fine until I discovered an issue in the external tool where due to some error condition it goes into a 'prompt' mode and waits for the user input. Now in my python script I use the join method to wait till all the processes finish their tasks. This is causing the whole thing to wait for this erroneous subprocess call. I can put a timeout for each of the process but I do not know in advance how long each one is going to run and hence this option is ruled out.

How do I figure out if any child process is waiting for an user input and how do I send an 'exit' command to it? Any pointers or suggestions to relevant modules in python will be really appreciated.

My code here:

import subprocess
import sys
import os
import multiprocessing

def write_script(fname,e):
    f = open(fname,'w')
    f.write("Some useful cammnd calling external tool")
    f.close()
    subprocess.call(['chmod','+x',os.path.abspath(fname)])
    return os.path.abspath(fname)

def run_use(mname,script):
    print "ssh "+mname+" "+script
    subprocess.call(['ssh',mname,script])

if __name__ == '__main__':
    dict1 = {}
    dict['mod1'] = ['pp1','ext2','les3','pw4']
    dict['mod2'] = ['aaa','bbb','ccc','ddd']
    machines = ['machine1','machine2','machine3','machine4']
    log_file.write(str(dict1.keys()))
    for key in dict1.keys():
        arr = []
        for mod in dict1[key]:
            d = {}
            arr.append(mod)
            if ((mod == dict1[key][-1]) | (len(arr)%4 == 0)):
                for i in range(0,len(arr)):
                    e = arr.pop()
                    script  = write_script(e+"_temp.sh",e)
                    d[i] = multiprocessing.Process(target=run_use,args=(machines[i],script,))
                    d[i].daemon = True
                for pp in d:
                    d[pp].start()
                for pp in d:
                    d[pp].join()

Answer 1

Since you're writing a shell script to run your subcommands, can you simply tell them to read input from /dev/null ?

#!/bin/bash
# ...
my_other_command -a -b arg1 arg2 < /dev/null
# ...

This may stop them blocking on input and is a really simple solution. If this doesn't work for you, read on for some other options.

The subprocess.call() function is simply shorthand for constructing a subprocess.Popen instance and then calling the wait() method on it. So, your spare processes could instead create their own subprocess.Popen instances and poll them with poll() method on the object instead of wait() (in a loop with a suitable delay). This leaves them free to remain in communication with the main process so you can, for example, allow the main process to tell the child process to terminate the Popen instance with the terminate() or kill() methods and then itself exit.

So, the question is how does the child process tell whether the subprocess is awaiting user input, and that's a trickier question. I would say perhaps the easiest approach is to monitor the output of the subprocess and search for the user input prompt, assuming that it always uses some string that you can look for. Alternatively, if the subprocess is expected to generate output continually then you could simply look for any output and if a configured amount of time goes past without any output then you declare that process dead and terminate it as detailed above.

Since you're reading the output, actually you don't need poll() or wait() - the process closing its output file descriptor is good enough to know that it's terminated in this case.

Here's an example of a modified run_use() method which watches the output of the subprocess:

def run_use(mname,script):
    print "ssh "+mname+" "+script
    proc = subprocess.Popen(['ssh',mname,script], stdout=subprocess.PIPE)
    for line in proc.stdout:
        if "UserPrompt>>>" in line:
            proc.terminate()
            break

In this example we assume that the process either gets hung on on UserPrompt>>> (replace with the appropriate string) or it terminates naturally. If it were to get stuck in an infinite loop, for example, then your script would still not terminate - you can only really address that with an overall timeout, but you didn't seem keen to do that. Hopefully your subprocess won't misbehave in that way, however.

Finally, if you don't know in advance the prompt that will be giving from your process then your job is rather harder. Effectively what you're asking to do is monitor an external process and know when it's blocked reading on a file descriptor, and I don't believe there's a particularly clean solution to this. You could consider running a process under strace or similar, but that's quite an awful hack and I really wouldn't recommend it. Things like strace are great for manual diagnostics, but they really shouldn't be part of a production setup.