简体繁体 English

python Ubuntu：打开文件过多[eventpoll]

[英]python Ubuntu: too many open files [eventpoll]

原文 2018-05-11 21:19:50 1 2 python/ ubuntu/ memory-leaks/ python-requests

Basically, it is a multi-threaded crawler program, which uses requests mainly. 基本上，它是一个多线程爬虫程序，主要使用请求。 After running the program for a few hours, I keep getting the error "Too many open files". 运行该程序几个小时后，我不断收到错误消息“打开的文件太多”。

By running: lsof -p pid , I saw a huge number of entries like below: 通过运行： lsof -p pid ，我看到了很多如下所示的条目：

python 75452 xxx 396u a_inode 0,11 0 8121 [eventpoll] python 75452 xxx 396u a_inode 0,11 0 8121 [eventpoll]

I cannot figure out what it is and how to trace back to the problem. 我无法弄清楚它是什么以及如何追溯到问题所在。 Previously, I tried to have it running in Windows and never seen this error. 以前，我尝试使其在Windows中运行，但从未见过此错误。 Any idea how to continue investigating this issue? 知道如何继续调查此问题吗？ thanks. 谢谢。

2 个解决方案

Are you sure that the behavior in Windows really does differ from that of Ubuntu? 您确定Windows的行为确实与Ubuntu的行为不同吗？ Specficially, I think that Windows allows 2048 open files and that the standard ulimit on Ubuntu is 1024. I can imagine a crawler opening 1024 < x < 2048 files. 具体来说，我认为Windows允许2048个打开的文件，而Ubuntu上的标准ulimit是1024。我可以想象一个抓取程序可以打开1024 <x <2048个文件。 That would produce reproduce your situation without meaning that there was a platform specific bug in your python code. 这将重现您的情况，而并不意味着您的python代码中存在特定于平台的错误。

I don't run Windows, so I mght be wrong about the 2048. But Ubuntu's forums can help you get to 2048. See https://askubuntu.com/questions/162229/how-do-i-increase-the-open-files-limit-for-a-non-root-user 我没有运行Windows，所以我对2048可能是错误的。但是Ubuntu的论坛可以帮助您达到2048。请参阅https://askubuntu.com/questions/162229/how-do-i-increase-the-open非root用户的文件限制

If you need >2048, then maybe there is a problem... I hope this helps. 如果您需要> 2048，那么可能有问题...希望对您有所帮助。

I have figured out that it is caused by Gevent. 我发现这是由Gevent引起的。 After replacing gevent with multi-thread, everything is just OK. 用多线程替换gevent之后，一切都OK。

However, I still don't know what's wrong with gevent, which keeps opening new files(eventpoll). 但是，我仍然不知道gevent有什么问题，它会不断打开新文件（eventpoll）。