Controlling multiple subprocesses in Python?

Question

I want to write a python script that can launch and kill multiple subprocesses (each a python script that implements a little TCP server) as well as retrive output from them while they are running. Is there a specific module for this or should I go ahead and use subprocess?

Answer 1

It seems like you're looking for the multiprocessing module . The interface it similar to the the one of the threading module .

You can launch each process manually or manage them with a Pool .

They can communicate using a Pipe or a Queue , or more easily with a Manager object .

Answer 2

Retrieving output from them can be done by reading a textfile which the subprocesses write.. Or you can use Pickle which is essentially the same..

Example of usage:

child_count=pickle.load( open("child_count.p", "rb"))
child_count-=counter
pickle.dump( child_count, open("child_count.p", "wb"))

For your tcp server.. I think the idea is to have a listener, which then 'launches' a new process and when the process is done it will be killed again.

def reap():
    #Collect child processes
    while 1:
        try:
            result = os.waitpid(-1, os.WNOHANG)
            if not result[0]: break
        except:
            break
        print "Reaped child process %d" % result[0]

host = ''
port = 4200

s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
s.bind((host, port))
s.listen(1)

print "Parent at %d listening for connections" % os.getpid()
while 1:



    try:
        clientsock, clientaddr = s.accept();clientsock.settimeout(90)
        print "Accepting connection"
    except KeyboardInterrupt:
        raise
    except:
        traceback.print_exc()
        continue

    reap()

    #Clone/fork
    try:
        pid = os.fork()
    except:
        print "SOMETHING BAD HAPPENED.. fork failed!"
        clientsock.close()
        continue

    if pid:
        clientsock.close()
        #Cleanup old children
        continue
    else:
        print "New child", os.getpid()
        s.close()

        #try:
        print "Got connection frommm: ", clientsock.getpeername()
        data = clientsock.recv(1024)

        <ENTER your code for the process here>

This can be the core of your server.

What it does, it accepts a connection, then forks itself, continues to listen for other incoming connections while the forked/cloned part will handle the process.

When a new connection will come in while the other processes are done, the processes are recognized as zombie processes and will be reaped.