Python urllib2 Errors

Question

I have a long list of URLs I need to open for my service. Whenever I attempt to open this entire list, I receive many errors such as this when I initiate the program: [Errno 24] Too many open files . I am using urllib2 and gevent.

Does anyone have any solutions?

Thanks.

Answer 1

使用Gevent.pool一次只打开一定数量的url然后在打开url时重用这些greenlet。

Answer 2

As we still don't know what you exactly do, I can only guess: you are opening many URLs at once, and only then you try to read them. Instead, you should/could open-read-close them and then advance to the next one.

Alternatively, you could create a concurrency capable URL loader: open some urls and try to read them concurrently. After closing one, you can start opening the next. If you limit that to 5 or 10 at once, you shouldn't get a problem any longer.