使用子进程的CGI脚本上的python errno 24

Question

I have a python cgi script that runs an application via subprocess over and over again (several thousand times). I keep getting the same error...

Traceback (most recent call last):
  File "/home/linuser/Webpages/cgi/SnpEdit.py", line 413, in <module>
    webpage()
  File "/home/linuser/Webpages/cgi/SnpEdit.py", line 406, in main
    displayOmpResult(form['odfFile'].value)
  File "/home/linuser/Webpages/cgi/SnpEdit.py", line 342, in displayContainerDiv
    makeSection(position,sAoiInput)
  File "/home/linuser/Webpages/cgi/SnpEdit.py", line 360, in displayData
    displayTable(i,j,lAmpAndVars,dOligoSet[key],position)
  File "/home/linuser/Webpages/cgi/SnpEdit.py", line 247, in displayTable
    p = subprocess.Popen(['/usr/bin/pDat',sInputFileLoc,sOutputFileLoc],stdout=fh, stderr=fh)
  File "/usr/lib/python2.6/subprocess.py", line 633, in __init__
    errread, errwrite)
  File "/usr/lib/python2.6/subprocess.py", line 1039, in _execute_child
    errpipe_read, errpipe_write = os.pipe()
OSError: [Errno 24] Too many open files

The function causing it is below.

def displayTable(sData):

    # convert the data to the proper format
    sFormattedData = convertToFormat(sData)

    # write the formatted data to file
    sInputFile = tempfile.mkstemp(prefix='In_')[1]
    fOpen = open(sInputFile,'w')
    fOpen.write(sFormattedData)
    fOpen.close()

    sOutputFileLoc = sInputFile.replace('In_','Out_')

    # run app, requires two files; an input and an output
    # temp file to holds stdout stderr of subprocess
    fh = tempfile.TemporaryFile(mode='w',dir=tempfile.gettempdir())
    p = subprocess.Popen(['/usr/bin/pDat',sInputFileLoc,sOutputFileLoc],stdout=fh, stderr=fh)
    p.communicate()
    fh.close()

    # open output file and print parsed data into a list of dictionaries
    sOutput = open(sOutputFileLoc).read()
    lOutputData = parseOutput(sOutput)

    displayTableHeader(lOutputData)
    displaySimpleTable(lOutputData)

As far as I can tell, I'm closing the files properly. When I run...

import resource
print resource.getrlimit(resource.RLIMIT_NOFILE)

I get...

(1024, 1024)

Do I have to increase this value? I read that subprocess opens several file descriptors. I tried adding "close_fds = True" and I tried using the with statement when creating my file but the result was the same. I suspect the problem may be with the application that I'm subprocessing, pDat, but this program was made by someone else. It requires two inputs; an input file and the location of where you want the output file written to. I suspect it may not be closing the output file that it creates. Aside from this, I can't see what I might be doing wrong. Any suggestions? Thanks.

EDIT: I'm on ubuntu 10.04 running python 2.6.5 and apache 2.2.14

Answer 1

Instead of this...

sInputFile = tempfile.mkstemp(prefix='In_')[1]
fOpen = open(sInputFile,'w')
fOpen.write(sFormattedData)
fOpen.close()

I should have done this...

iFileHandle,sInputFile = tempfile.mkstemp(prefix='In_')
fOpen = open(sInputFile,'w')
fOpen.write(sFormattedData)
fOpen.close()
os.close(iFileHandle)

The mkstemp function makes OS level handles to a file and I wasn't closing them. The solution is described in more detail here... http://www.logilab.org/blogentry/17873

Answer 2

You want to add close_fds=True to the popen call (just in case).

Then, here:

# open output file and print parsed data into a list of dictionaries
    sOutput = open(sOutputFileLoc).read()
    lOutputData = parseOutput(sOutput)

...I might remember wrong, but unless you use the with syntax, I do not think that the output file descriptor has been closed.

UPDATE : the main problem is that you need to know which files are open . On Windows this would require something like Process Explorer. In Linux it's a bit simpler; you just have to invoke the CGI from command line, or be sure that there is only one instance of the CGI running, and fetch its pid with ps command.

Once you have the pid, run a ls -la on the content of the /proc/<PID>/fd directory. All open file descriptors will be there, with the name of the files they point to. Knowing that file so-and-so is opened 377 times, that goes a long way towards finding out where exactly that file is opened (but not closed).