Scrapy "too many open files" error, caused by too many sockets

Question

I've met a problem that gets error 24: too many open files. After searching I've changed ulimit to 102400...

However my spider still get this error after running for 1 week.

At first I thought it was caused by my pipeline(I changed this), so I checked the /proc/{pid}/fd and found so many socket opened.

BTW, you can find code here: https://github.com/yz21606948/sinaSpider/tree/master/sina

Answer 1

我通过使用增加限制来解决它

ulimit -n unlimited