change the maximum number of open files in Windows

Question

I would like to know how I can change the maximum number of open files in Windows. Similar to ulimit command under unix. Under my c:\\windows\\system32, I cannot find the CONFIG.NT file, thus changing the file number in CONFIG.NT file won't work for me.

The problem is I wrote a python script, in which I need to call subprocess.Popen(cmd, stdout=sp.PIPE, stdin=sp.PIPE, shell = True) 256 or 512 times. That means I need to launch 256 or 512 background processes. The script works fine whenever the number of launched process is smaller than 255.

Answer 1

This is an old question, but I'll contribute. There is a limit, but it's not a limit of subprocesses per-say. Python opens file descriptors for each pipe on each subprocess. Your code is breaking at 255 because Window has a default limit of 512 open file descriptors (pretty close to the 2 pipes x 255 subprocesses).

See answer 3 in this question: Why python has limit for count of file handles? . This answer shows how to increase Window's default limit. You can change open limit to a max of 2048 per Windows limits.

Answer 2

The general claim is that Windows does not have an upper limit on files; just what you can manage in memory.

I think you will find useful: Russinvich's discussion of Windows handle resource limits useful.

We build parallel programs, that can process thousands of files. In spite of the claims, we seem to run into resource limits when we attempt to open (and hold open simultaneously) several thousand files.

We have found it to be useful to limit the number of opens to a few hundred. This doesn't seem to affect performance much over pushing the limits. This is pretty easily managed with a counting semaphore in your parallel application.