Python多處理程序不嵌套子進程？

Question

我有看起來像這樣的代碼：

import multiprocessing
import logging

m = [0,1,2,3]
iter_state = 0

class gener(object):
    def __init__(self, m):
        self.m = m
        self.c = 0

    def __iter__(self):
        return self

    def next(self):
        print "n"
        time.sleep(3)
        ret = self.m[self.c]
        self.c += 1
        return ret 

tt  = gener(m)

itst = multiprocessing.Array('i', 3)

def gen(t):
    itst[t] = t

multiprocessing.log_to_stderr(logging.DEBUG)    

tm = time.time()

job1 = multiprocessing.Process(target=gen, args=(tt.next(),))
job2 = multiprocessing.Process(target=gen, args=(tt.next(),))
job3 = multiprocessing.Process(target=gen, args=(tt.next(),))


job1.start()
job2.start()
job3.start()
job3.join()

for i in itst: 
    print i

tm = time.time() - tm
print tm

具有以下輸出：

OUT：

n
n
n
[INFO/Process-1] child process calling self.run()
[INFO/Process-1] process shutting down
[DEBUG/Process-1] running all "atexit" finalizers with priority >= 0
[DEBUG/Process-1] running the remaining "atexit" finalizers
[INFO/Process-1] process exiting with exitcode 0
[INFO/Process-2] child process calling self.run()
[INFO/Process-2] process shutting down
[DEBUG/Process-2] running all "atexit" finalizers with priority >= 0
[DEBUG/Process-2] running the remaining "atexit" finalizers
[INFO/Process-2] process exiting with exitcode 0
[INFO/Process-3] child process calling self.run()
[INFO/Process-3] process shutting down
[DEBUG/Process-3] running all "atexit" finalizers with priority >= 0
[DEBUG/Process-3] running the remaining "atexit" finalizers
[INFO/Process-3] process exiting with exitcode 0
0
1
2
9.01742887497
[INFO/MainProcess] process shutting down
[DEBUG/MainProcess] running all "atexit" finalizers with priority >= 0
[DEBUG/MainProcess] running the remaining "atexit" finalizers

因此，我們可以看到並行化實際上根本不起作用。

但是，將time.sleep()調用放置在gen()函數內部時，我們看到以下內容：

import multiprocessing
import logging

m = [0,1,2,3]
iter_state = 0

class gener(object):
    def __init__(self, m):
        self.m = m
        self.c = 0

    def __iter__(self):
        return self

    def next(self):
        print "n"
        ret = self.m[self.c]
        self.c += 1
        return ret 

tt  = gener(m)

itst = multiprocessing.Array('i', 3)

def gen(t):
    time.sleep(3)
    itst[t] = t

multiprocessing.log_to_stderr(logging.DEBUG)    

tm = time.time()

job1 = multiprocessing.Process(target=gen, args=(tt.next(),))
job2 = multiprocessing.Process(target=gen, args=(tt.next(),))
job3 = multiprocessing.Process(target=gen, args=(tt.next(),))


job1.start()
job2.start()
job3.start()
job3.join()

for i in itst: 
    print i

tm = time.time() - tm
print tm

OUT：

n
n
n
[INFO/Process-1] child process calling self.run()
[INFO/Process-2] child process calling self.run()
[INFO/Process-3] child process calling self.run()
[INFO/Process-1] process shutting down
[DEBUG/Process-1] running all "atexit" finalizers with priority >= 0
[DEBUG/Process-1] running the remaining "atexit" finalizers
[INFO/Process-1] process exiting with exitcode 0
[INFO/Process-3] process shutting down
[DEBUG/Process-3] running all "atexit" finalizers with priority >= 0
[DEBUG/Process-3] running the remaining "atexit" finalizers
[INFO/Process-3] process exiting with exitcode 0
[INFO/Process-2] process shutting down
[DEBUG/Process-2] running all "atexit" finalizers with priority >= 0
[DEBUG/Process-2] running the remaining "atexit" finalizers
[INFO/Process-2] process exiting with exitcode 0
0
1
2
3.01985812187
[INFO/MainProcess] process shutting down
[DEBUG/MainProcess] running all "atexit" finalizers with priority >= 0
[DEBUG/MainProcess] running the remaining "atexit" finalizers

現在我們可以看到並行化工作正常！

但不幸的是，僅適用於gen()函數。

我的問題：

我們也如何在next()方法內部實現並行化工作？

Answer 1

好吧，您可以只在gen內部實際調用next()來做到這一點。 現在，您正在父進程中調用tt.next() ，並將返回值傳遞給子進程。 相反，您應該將整個tt.next方法傳遞給子級。 但是請注意，通過這樣做，您需要使gener.c成為進程共享的變量，並使用進程安全鎖來保護遞增代碼段的代碼部分。

import multiprocessing
import logging
import time

class gener(object):
    def __init__(self, m):
        self.m = m
        self.c = multiprocessing.Value('i', 0)

    def __iter__(self):
        return self

    def next(self):
        print "n"
        time.sleep(3)
        with self.c.get_lock():
            ret = self.m[self.c.value]
            self.c.value += 1
        return ret 

def gen(func):
    t = func()
    itst[t] = t

if __name__ == "__main__":
    m = [0,1,2,3]
    iter_state = 0

    tt  = gener(m)
    itst = multiprocessing.Array('i', 3)
    multiprocessing.log_to_stderr(logging.DEBUG)    

    tm = time.time()

    jobs = [ multiprocessing.Process(target=gen, args=(tt.next,)) for _ in range(3)]
    for j in jobs:
        j.start()
    for j in jobs:
        j.join()

    for i in itst: 
        print i

    tm = time.time() - tm
    print tm

輸出：

[INFO/Process-3] child process calling self.run()
n
[INFO/Process-1] child process calling self.run()
[INFO/Process-2] child process calling self.run()
n
n
[INFO/Process-2] process shutting down
[INFO/Process-3] process shutting down
[INFO/Process-1] process shutting down
[DEBUG/Process-1] running all "atexit" finalizers with priority >= 0
[DEBUG/Process-1] running the remaining "atexit" finalizers
[DEBUG/Process-3] running all "atexit" finalizers with priority >= 0
[DEBUG/Process-3] running the remaining "atexit" finalizers
[DEBUG/Process-2] running all "atexit" finalizers with priority >= 0
[DEBUG/Process-2] running the remaining "atexit" finalizers
[INFO/Process-2] process exiting with exitcode 0
[INFO/Process-1] process exiting with exitcode 0
[INFO/Process-3] process exiting with exitcode 0
0
1
2
3.02282786369
[INFO/MainProcess] process shutting down
[DEBUG/MainProcess] running all "atexit" finalizers with priority >= 0
[DEBUG/MainProcess] running the remaining "atexit" finalizers