JDK 1.7：由于POSIX信号量导致“打开文件太多”？

Question

I've looked through the other similar questions on SO, but they seem to be caused by other issues.

First I made sure I judiciously closed all of my file handles, and then I used lsof -p <pid of java> to look at my list of files.

It stays pretty constant throughout my runtime but then periodically I will get about 10,000 entries listed in lsof like this:

COMMAND   PID USER   FD     TYPE DEVICE  SIZE/OFF     NODE NAME
                                      ...
java    36809  smm *235r  PSXSEM              0t0          kcms00008FC901624000
java    36809  smm *236r  PSXSEM              0t0          kcms00008FC901624000
java    36809  smm *237r  PSXSEM              0t0          kcms00008FC901624000
java    36809  smm *238r  PSXSEM              0t0          kcms00008FC901624000
java    36809  smm *239r  PSXSEM              0t0          kcms00008FC901624000

The man page says PSXSEM type is a POSIX Semaphore. Any clue what JDK uses POSIX Semaphores for? BTW, the app is a single threaded command line app at the moment.

Potentially useful background: I first noticed this after upgrading to JDK 1.7 on Mac OS X 10.7.3:

java version "1.7.0_04"
Java(TM) SE Runtime Environment (build 1.7.0_04-b21)
Java HotSpot(TM) 64-Bit Server VM (build 23.0-b21, mixed mode)

Update: repointing $JAVA_HOME at JDK 1.6 seems to be a workaround for the issue.

java version "1.6.0_31"
Java(TM) SE Runtime Environment (build 1.6.0_31-b04-415-11M3635)
Java HotSpot(TM) 64-Bit Server VM (build 20.6-b01-415, mixed mode)

What is JDK 1.7 doing differently?

Answer 1

I was able to trace it down to this block of code:

BufferedImage image = null;
ImageInputStream stream = null;
try {
    stream = new FileImageInputStream(file);
    image = ImageIO.read(stream);

} catch (Exception ex) {
    log.error("Image could not be read: "+file.getPath());

} finally {
    // ImageIO closes input stream unless null is returned
    // http://docs.oracle.com/javase/7/docs/api/javax/imageio/ImageIO.html#read(javax.imageio.stream.ImageInputStream)
    if (stream != null && image == null) {
        try {
            stream.close();
        } catch (IOException ex) {
            log.error("ERROR closing image input stream: "+ex.getMessage(), ex);
        }
    }
}

The JavaDocs specifically say that this method (unlike the others) closes the stream automatically. In fact, when you attempt to manually close it, it throws an exception saying 'closed'. I was using this overload since the other says it wraps it in a ImageInputStream anyway so I thought I'd save some work.

Changing the block to use a normal FileInputStream fixes the leak:

BufferedImage image = null;
InputStream stream = null;
try {
    stream = new FileInputStream(file);
    image = ImageIO.read(stream);

} catch (Exception ex) {
    log.error("Image could not be read: "+file);

} finally {
    if (stream != null) {
        try {
            stream.close();
        } catch (IOException ex) {
            log.error("ERROR closing image input stream: "+ex.getMessage(), ex);
        }
    }
}

This appears to me to be a bug in JDK 1.7 as 1.6 worked fine here.

Update: I just submitted a bug report to Oracle for this issue.

Answer 2

I found another cause. It seems that the toRGB() method of ColorSpace is leaking semaphores. Running the following code:

import java.awt.color.ColorSpace;
public class Test
{
    public static void main(String[] args) throws Throwable {
        final ColorSpace CIEXYZ = ColorSpace.getInstance(ColorSpace.CS_CIEXYZ);
        for(int i = 0; i < 10000000; i++) {
            CIEXYZ.toRGB(new float[] {80f, 100, 100});
        }
    }
}

With:

java version "1.7.0_04"
Java(TM) SE Runtime Environment (build 1.7.0_04-b21)
Java HotSpot(TM) 64-Bit Server VM (build 23.0-b21, mixed mode)

Will leave you out of system files.

EDIT: already submitted a bug report to Oracle

Answer 3

Update: As other users have said, ImageIO was leaking semaphores in 1.7.0_04 and 1.7.0_05. The bug reports by user juancn and user mckamey have since been marked as fixed and closed (thanks guys!). The explanation:

This fix reports a leak of file handlers on macosx. It mentioned two ways to leak handlers: via ImageIO.read(ImageInputStream), and via semaphores.

I do not observe the first leak: we explicitly close input stream if we found an appropriate reader, and this is enough (at least on 1.7.4) to release file handles.

However, in case of semaphores we leak tons of handles: we perform color conversion for each line of jpeg image, and each time we create a semaphore (because we see 2 or more CPU installed on the system), then we reduce number of separate tasks down to 1 (because we have single scan line to process) and due to this never unlink the semaphore.

The same problem is present on Linux systems, but in less degree because we occupy single file handle per named semaphore, whereas on macosx we always occupy new file handle.

Suggested fix just postpones the creation of named semaphore until we clarify the number of separate tasks, so, now we do not create semaphores for image reading and simple color conversions (like ColorSpace.toRGB()). Beside this, now we use pSem pointer as a trigger for the semaphore destruction.

Even though their reports indicate that the fix is in version 8, a backport report indicates that it was fixed in 1.7.0_06.

So if you're seeing this on 1.7.0_04 or 05, updating to at least 1.7.0_06 will take care of this issue.