Java monitor folder for files

Question

I need to monitor a certain folder for new files, which I need to process. I have the following requirements:

1. The filenames of the files are sequence numbers. I need to process each file in order. (Lowest number first, there's no guarantee that each sequence number exists. eg: 1,2,5,8,9
2. If files already exist in the folder during startup, I need to process them directly
3. I need a guarantee that I only process each file once
4. I need to avoid reading incomplete files (which are still being copied)
5. The service should ofcourse be reliable...

What is the most common way to accomplish this?

I'm using Java SE7 and Spring 4.

I already had a look at the WatchService of Java 7 but it seems to have problems with processing already existing files during startup, and avoid processing incomplete files.

Answer 1

Assembling comments into an answer.

Easiest way to parse the files in the correct order is to load the entire directory file listing into an array / list and then sort the list using an appropriate comparator. Eg Load files with File.list() or File.listFiles() .

This is not the most efficient methodology, but for less than 10,000 files should be adequate unless you need faster startup time performance (I can imagine a small lag before processing begins as all of the files are listed).

To avoid reading incomplete files you should acquire an exclusive FileLock (via a FileChannel which you can get from the FileOutputStream or FileInputStream , however you may not be able to get an exclusive lock from the FileInputStream ) on the file. Assuming the OS being used supports file locking (which modern OSes do) and the application writing the file is well behaved and holding a lock (hopefully it is) then as soon as you are able to acquire the lock you know the file is complete.

If for some reason you cannot rely on file locking then you either need to have the writing program first write to a temporary file (perhaps with a different extension) and then atomically move / rename the file (atomic for most OSes if on the same file system / partition), or monitor the file for a period of time to see if further bytes are being written (not the most robust methodology).