Python中的多处理 <urlopen error ftp error> 下载文件包时

Question

This was a question that was spawned from a previous question (see Downloading a LOT of files using Python ) but was so much more specific than what I originally asked that I thought it deserved its own question.

When running python multiprocessing if I try to download a batch of files at once using threading it throws as error on only some of the files. This is the error, obviously there is an error with urllib2 opening the file but the question is, why?

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib64/anaconda/lib/python2.7/multiprocessing/pool.py", line 250, in map
    return self.map_async(func, iterable, chunksize).get()
  File "/usr/local/lib64/anaconda/lib/python2.7/multiprocessing/pool.py", line 554, in get
    raise self._value
urllib2.URLError: <urlopen error ftp error: >

What is weird is that if I download the files one at a time I do not get this error. And the error is (normally) not consistent. If I run the same process twice it will throw the same error but on different files. This leads me to think that the problem is the interaction of the threads. Maybe 2 threads are trying to ping the site at the same time? Does anyone know what might be causing this?

The machine I am using is a LinuxBox running RedHat with 32 cores.

Here is the code I am using:

from __future__ import division
import pandas as pd
import numpy as np
import urllib2
import os
import linecache
from multiprocessing import Pool
import time

#making our list of urls to download from
data2=pd.read_csv("edgar14A14C.csv")

flist=np.array(data2['filename'])
print len(flist)
print flist

os.chdir(str(os.getcwd())+str('/edgar14A14C'))

###below we have a script to download all of the files in the data2 database
###here you will need to create a new directory named edgar14A14C in your CWD

def job(url):
    print "I'm doing something!"
    file_name = str(url.split('/')[-1])
    u = urllib2.urlopen(url)
    f = open(file_name, 'wb')
    f.write(u.read())
    print file_name
    f.close()


urls = ["ftp://ftp.sec.gov/{0:s}".format(f) for f in flist]
pool = Pool(processes=20)
pool.map(job, urls)

Answer 1

Processes are limited to only having a specific number of file pointers open at a given time. urllib2.urlopen(url) opens a file pointer (the socket). After you are done with the data, be sure to close it: u.close()