How do I close files in python when I haven't got the file identifiers

Question

I found myself unable to open new files in Python. When I examined with ls -l /proc/PID/fd I saw loads of files open for the Python process. The module I am using is apparently opening lots of files and not closing them.

I was expecting that I could close the files by deleting objects associated with the module that opened the files, but nothing happened.

I was also expecting to see the file objects somewhere in the garbage collection, but I saw nothing that resembled the open files with this:

for obj in gc.get_objects():
    if hasattr(obj, 'read'):
        print(obj)

The files disappear when I quit Python.

Answer 1

The problem is likely that those file descriptors are leaking without being associated to Python objects. Python has no way of seeing actual file descriptors (OS resources) that are not associated with Python objects. If they were associated with Python objects, Python would close them when they are garbage collected. Alternatively, the third-party library does its own tracking of file descriptors.

You can use os.close on plain integers to close the associated file descriptor. If you know which file descriptors you want to keep open (usually, stdin/stdout/stderr, which are 0, 1 and 2, and maybe a few others), you can just close all other integers from 0 to 65535, or simply those in /proc/<pid>/fd :

import os

KEEP_FD = set([0, 1, 2])

for fd in os.listdir(os.path.join("/proc", str(os.getpid()), "fd")):
    if int(fd) not in KEEP_FD:
        try:
            os.close(int(fd))
        except OSError:
            pass

This is a pretty evil hack, though. The better solution would be to fix the third-party library.