Java.nio.file.NoSuchFileException but files exits

Question

I'm trying to read files from a folder. There are >1000 files in that folder and i trying read 100 files every 10s by using ScheduledThreadPoolExecutor.scheduleWithFixedDelay()

Files.list(Paths.get(path)).sorted().limit(limit).forEach(file ->{ //limit = 100
                    try{ 
                        String content = new String(Files.readAllBytes(file), "UTF-8");
                        //do business logic
                        doneFile.add(fileName);
                    }catch (Exception e){
                        log.error(e);
                    }
                });

if(doneFile.size() > 0){
                for(String fileName: doneFile){
                    try {
                        Files.delete(Paths.get(path + fileName));
                    } catch (IOException e) {
                        log.error(e);
                    }
                }
            }

About 60-80 files were read successfully and the others throw java.nio.file.NoSuchFileException .
Delete function also throws some java.nio.file.NoSuchFileException .

But finally, all files were read and deleted after some threads in spite of exceptions

What cause exception in this case and how can i fix it? Many thanks! Ps. Sr about my bad English

Answer 1

I commented:

It sounds like something else is adding and removing files. Is it possible that you have two scheduled tasks running? Is it possible that an external process is doing this?

You replied:

I checked log file and got that ScheduledThreadPoolExecutor.scheduleWithFixedDelay() created 10 threads instead of 8 as expected for 4 loops

So ... it sounds like you have two (or more) threads iterating through the same directory, processing and deleting files without any coordination. Combine that with the fact that Files.list is going to buffer part or all of the directory that it is listing, and you will get problems like:

• One thread deletes a file that is in another file's stream before the second thread tries to open it. This leads to a NoSuchFileException when the latter tries to open the file.

• Two threads process the same file at the same time, and one deletes it while the second thread is still processing it. This leads to a NoSuchFileException when the latter tries to delete the file.

Possible solutions:

1. Write the code to ignore these exceptions. (But you could still have two threads processing the same file at the same time, which seems like a bad thing.)

2. Restructure the code so that there is only one thread scanning the directory at any one time.

If you need parallelism when processing within the directory, have the directory processor thread submit a task for each file to an executor with a bounded thread pool.