在bigdata中使用python代码中的map（）进行多重处理

Question

I'm trying to get some values(which I get using extract function) from urls which are stored in data.file and there are about 3000000 url links in the file. here is my code snippet,

from multiprocessing import Pool
p = Pool(10)
revenuelist = p.map(extract, data.file )

But the problem is, due to internet connection, this is code runs again if there connection problem. How do I add fault tolerance to my code(i:e store intermediate result, to avoid repetition of doing same task).

Answer 1

A very simple solution is using a file to store your current status. Use try...finally to handle fails:

with open(FILENAME) as f:
    current = int(f.read() or 0)

if current:
    skip_lines(current)

try:
    with Pool() as pool:
        results = pool.imap(extract, data.file)
        for result in results:
            do_something(result)
            current += 1
finally:
    with open(FILENAME, "w") as f:
        f.write(str(current))

See also: 'concurrent.futures` (much cooler than multiprocessing.Pool).

A better solution would be using a database to completely track your progress, and/or use a better task queue (for example, celery ) to execute your jobs.