在python中的并行进程中运行类方法

Question

I am trying and failing to run a huge loop in parallel. The loop is exactly one method of a specific class, and inside the loop I call its another method. It does work, but for some reason there is only one process in the list and the output (see code) is always 'Worker 0'. Either the processes are not created or they are not running in parallel. The structure is the following:

main.py

from my_class.py import MyClass

def main():
    class_object = MyClass()
    class_object.method()

if __name__ == '__main__':
    main()

my_class.py

from multiprocessing import Process

MyClass(object):
    def __init__(self):
        # do something

    def _method(self, worker_num, n_workers, amount, job, data):
        for i, val in enumerate(job):
            print('Worker %d' % worker_num)
            self.another_method(val, data)

    def another_method(self):
        # do something to the data

    def method(self):
        # definitions of data and job_size go here

        n_workers = 16
        chunk = job_size // n_workers
        resid = job_size - chunk * n_workers

        workers = []
        for worker_num in range(n_workers):
            st = worker_num * chunk
            amount = chunk if worker_num != n_workers - 1 else chunk + resid
            worker = Process(target=self._method, args=[worker_num, n_workers, amount, job[st:st+amount], data])
            worker.start()
            workers.append(worker)

        for worker in workers:
            worker.join()

        return data

I have read some things about child processes requiring main module to be importable, but I have no idea how to do it in my case.

Answer 1

Question : ... but still only one core is in use. So the question is, can I use multiple cores with Process objects

This does not depend on the Python interpreter which Process is using which CPU.
Relevant: on-what-cpu-cores-are-my-python-processes-running

Extend your def _method(... with the following, to see what actually happens:

Note : getpidcore(pid) is Distribution dependend, could FAIL !

def getpidcore(pid):
    with open('/proc/{}/stat'.format(pid), 'rb') as fh:
        core = int(fh.read().split()[-14])
        return core

class MyClass(object): 
    ...
    def _method(self, worker_num, n_workers, amount, job, data):
        for i, val in enumerate(job):
            core = getpidcore(os.getpid())
            print('core:{} pid:{} Worker({})'.format(core, os.getpid(), (worker_num, n_workers, amount, job)))

Output :

 core:1 pid:7623 Worker((0, 16, 1, [1])) core:1 pid:7625 Worker((2, 16, 1, [3])) core:0 pid:7624 Worker((1, 16, 1, [2])) core:1 pid:7626 Worker((3, 16, 1, [4])) core:1 pid:7628 Worker((5, 16, 1, [6])) core:0 pid:7627 Worker((4, 16, 1, [5])) 

Tested with Python: 3.4.2 on Linux