如何区分Multiprocessing.Pool中的进程？

Question

I'm using python multiprocessing to fork some child processes to run my jobs. There are two demands:

1. I need to know the pid of the child process in case of killing it if I want.
2. I need callback to do some stuffs after job has finished. Because those stuffs use a lock in parent process, it can't be done in child process.

But I get:

1. Process generated by multiprocessing.Process() has an attribute "pid" to get its pid. But I can't add my asynchronous callback, of course I can't wait synchronously neither.
2. Process pool generated by multiprocessing.Pool() provides callback interface. But I can't tell which process in the pool is the one matching my job, since I may need to kill the process according to a specific job.

Task is cheap, here shows the code:

import random, time
import multiprocessing
import os

class Job(object):
    def __init__(self, jobid, jobname, command):
        self.jobid, self.jobname, self.command = jobid, jobname, command

    def __str__(self):
        return "Job <{0:05d}>".format(self.jobid)

    def __repr__(self):
        return self.__str__()

def _run_job(job):
    time.sleep(1)
    print "{} done".format(job)
    return job, random.choice([True, False]) # the second argument indicates whether job has finished successfully

class Test(object):
    def __init__(self):
        self._loc = multiprocessing.Lock()
        self._process_pool = multiprocessing.Pool()

    def submit_job(self, job):
        with self._loc:
            self._process_pool.apply_async(_run_job, (job,), callback=self.job_done)
            print "submitting {} successfully".format(job)

    def job_done(self, result):
        with self._loc:
            # stuffs after job has finished is related to some cleanning work, so it needs the lock of the parent process
            job, success = result
            if success:
                print "{} success".format(job)
            else:
                print "{} failure".format(job)


j1 = Job(1, "test1", "command1")
j2 = Job(2, "test2", "command2")
t = Test()
t.submit_job(j1)
t.submit_job(j2)
time.sleep(3.1) # wait for all jobs finishing

But now I can't get the pid corresponding to each job. For example, I need to kill the job<1>, but I can't find which process in the process pool is related to the job<1>, so I mightn't kill the job whenever I want.

If I use multiprocessing.Process alternatively, I can record pid of every process with its corresponding jobid. But I can't add callback method now.

So is there a way to both get the pid of child process and to add callback method?

Answer 1

Finally I find a solution: use multiprocessing.Event instead.

Since multiprocessing.Pool can't tell me which process is allocated, I can't record it so that I can kill it according to job id whenever I want.

Fortunately, multiprocessing provides Event object as an alternative to callback method. Recall what callback method does: it provides an asynchronous response to child process. Once child process finishes, the parent process can detect it and call the callback method. So the core issue is how parent process detects whether child process has finished or not. That's Event object for.

So the solution is simple: pass a Event object to child process. Once child process finishes, it set the Event object. In parent process, it starts a daemon thread to monitor whether the event is set. If so, it can call the method that does those callback stuffs. Moreover, since I created processes with multiprocessing.Process instead of multiprocessing.Pool , I can easily get its PID, which enables me to kill it.

The solution code:

import time
import multiprocessing
import threading

class Job(object):
    def __init__(self, jobid, jobname, command):
        self.jobid, self.jobname, self.command = jobid, jobname, command
        self.lifetime = 0

    def __str__(self):
        return "Job <{0:05d}>".format(self.jobid)

    def __repr__(self):
        return self.__str__()

def _run_job(job, done_event):
    time.sleep(1)
    print "{} done".format(job)
    done_event.set()

class Test(object):
    def __init__(self):
        self._loc = multiprocessing.Lock()
        self._process_pool = {}
        t = threading.Thread(target=self.scan_jobs)
        t.daemon = True
        t.start()

    def scan_jobs(self):
        while True:
            with self._loc:
                done_jobid = []
                for jobid in self._process_pool:
                    process, event = self._process_pool[jobid]
                    if event.is_set():
                        print "Job<{}> is done in process <{}>".format(jobid, process.pid)
                        done_jobid.append(jobid)
                map(self._process_pool.pop, done_jobid)
            time.sleep(1)


    def submit_job(self, job):
        with self._loc:
            done_event = multiprocessing.Event()
            new_process = multiprocessing.Process(target=_run_host_job, args=(job, done_event))
            new_process.daemon = True
            self._process_pool[job.jobid] = (new_process, done_event)
            new_process.start()
            print "submitting {} successfully".format(job)


j1 = Job(1, "test1", "command1")
j2 = Job(2, "test2", "command2")
t = Test()
t.submit_job(j1)
t.submit_job(j2)
time.sleep(5) # wait for job to finish