Java stream - too many open files

Question

I have a program that goes to a few directories from time to time and and do some kind of processing to the files in those directories.

The problem is from time to time (each two or three days) the program is reaching the OS open files limit.

It is a spring-boot application running in a RHEL 7.

the method that get the files is this:

public File[] getFiles(String dir, int numberOfFiles) throws Exception {
    final Path baseDir = Paths.get(dir);
    List<File> filesFromPath = new ArrayList<File>();
    File[] files = null;

    final BiPredicate<Path, BasicFileAttributes> predicate = (path, attrs) -> attrs.isRegularFile()
            && String.valueOf(path).endsWith(".xml");

    List<Path> result;

    try (Stream<Path> fileStream = Files.find(baseDir, 1, predicate).limit(numberOfFiles).onClose(() -> LOG.debug("Closing file stream."))){
        result = fileStream.collect(Collectors.toList());

        result.forEach(path -> {
            path.toString();
            File file = path.toFile();
            LOG.info("adding {} to process.", file.getName());
            filesFromPath.add(file);
        });

        if (filesFromPath != null && !filesFromPath.isEmpty()) {
            files = filesFromPath.toArray(new File[filesFromPath.size()]);
        }

    } catch (Exception e) {
        LOG.error("Error during file opening/closing", e);
    }

    if (files != null) {
        return files;
    }

    return new File[0];
}

I am using the lsof command to see how many opened files I have, and the list of directories is always growing.

I added a log in the onClise method and this is being called all times I open a stream.

Shouldn't the try with resources, close the stream?

[EDIT]

There is also another peace of code that move processed files to another folder. This code does not use stream, and I couldn't found out what is wrong with it, besides the fact it is ugly.

public void move(File file, String archivePath) throws IOException {
    File backupFile = new File(archivePath);
    if (!backupFile.exists()) {
        backupFile.mkdirs();
    }

    Path source = file.toPath();
    if (file.exists()) {
        Path target = Paths.get(archivePath + File.separator + file.getName());
        Files.move(source, target, StandardCopyOption.REPLACE_EXISTING);
        LOG.info("file {} moved to {}", file, archivePath);
    } else {
        LOG.info("unable to move the file: {} because it was already moved to {}", file, archivePath);
    }
}

[EDIT 2]

and all files are being processed like this:

private void processFile(File[] files) throws Exception {

    DocumentBuilderFactory dbf = DocumentBuilderFactory.newInstance();
    DocumentBuilder db = dbf.newDocumentBuilder();
    XPath xpathParser = XPathFactory.newInstance().newXPath();

    for (int i = 0; i < files.length; i++) {

        File file = files[i];


        Document doc = db.parse(file);

        // DO STUFF
        fileUtils.move(file, processedPath);

    }

}

Thanks.

Answer 1

I hit the same issue. In the case of calls to "parse" there is very much files open and I clearly see the implementation opens an input stream but may not close it in case of a failure.