OSError：[Errno 24]从终端调用脚本时打开的文件太多

Question

Ubuntu 14.04, Pycharm 5.0.1, Python 3.4.3

I have a multi thread intensive script (over 1k) which uses lots of files and sockets it also produces lots of logs. The problem I am facing is very peculiar. Starting the script from pycharm works fine, however invoking the script from terminal produces after a while

OSError: [Errno 24] Too many open files. 

the limits showed by

  ulimit -Hn 

are over 100k, and once again starting from pycharm works fine, starting:

 python3 my_script.py

Produces the error, I even tried using sudo, no luck. Can anybody explain what is happening?

EDIT

Done some more testing and from the looks of if, it seems that starting the script by pycharm it starts as true multhireaded process and starting from bash does not. I base this on results of ps aux:

USER PID  %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
user1   * * * * *            ? Sl   17:38  73:30 /usr/bin/python3.4 /home/user1/new_main.py
user1   * * * * *            ? S+   17:40  71:30 python3.4 /home/user1/new_main.py

For those like me who need a small reference, here is an extract

STAT = multi-character process state
PROCESS STATE CODES
   Here are the different values that the s, stat and state output specifiers (header "STAT" or "S") will display to describe the state of a process:
   S    interruptible sleep (waiting for an event to complete)

   For BSD formats and when the stat keyword is used, additional characters may be displayed:
   l    is multi-threaded (using CLONE_THREAD, like NPTL pthreads do)
   +    is in the foreground process group.

So the system says that the script started by pycharm (Sl) is multithreaded and the other is not.

Answer 1

I believe when you run your script using PyCharm, the IDE adds some options to the command line.

To find the command used by pycharm :

• run your program using pycharm.
• then (while your program runs) type in a terminal : ps aux | grep my_script.py ps aux | grep my_script.py

I had the same issue playing with multi threads and other file descriptors, a way to avoid the problem would probably be by optimizing your code, and / or reducing the maximal number of threads.