Python 3.x：多进程无法序列化“ ExFileObject”对象

Question

I try to read txt file from the archive(tar) and process file with function. I have a lot of files and decided to use multiprocessing lib to speed up calculations. But I got error "cannot serialize 'ExFileObject' object". Below is an example of my code. How to serialize 'ExFileObject' object or how to speed up my calculations with multiprocessing lib? I am using Python 3.4

import tarfile
from multiprocessing import Process, Manager

def start():
    result = {}
    file_path = '...somepath'
    tar = tarfile.open(file_path, 'r')
    for file in tar:
        extr_file = tar.extractfile(file)
        p = Process(target=straight_calc, args=(extr_file, result)) #thows error
        p.start()
        p.join()
    print(result)

def straight_calc(file, result):
    content = file.readlines()
    for line in content:
        pass

if __name__ == '__main__':
    start()

Answer 1

Your example code works fine for me. I would only add a NoneType check because NoneType has no readlines attribute and tar.extractfile(file) might return a NoneType value.

import tarfile
from multiprocessing import Process, Manager

def start():
    result = {}
    file_path = '...somepath'
    tar = tarfile.open(file_path, 'r')
    for file in tar:
        extr_file = tar.extractfile(file)
        if extr_file is not None:
            p = Process(target=straight_calc, args=(extr_file, result))
            p.start()
            p.join()
    print(result)

def straight_calc(file, result):
    content = file.readlines()
    for line in content:
        pass

if __name__ == '__main__':
    start()

You might try pickling your extr_file with picklestring = pickle.dumps(extr_file) . Pickling extr_file before executing Process should throw a more useful exception.