Java File watcher with multiple JVM watching single directory for incoming files

Question

I have a situation where there are two java applications are watching a directory for incoming file. Say there is a directory DIR that is being watched by two JVM processes for any files with the extension .SGL.

The problem we face here is that, sometimes both nodes are being notified about the new files and both nodes are trying to process the same file.

Usually we handle these situations using a database that try to insert into a table with unique file name column and only one will succeed and continue processing.

But for this situation, we don't have database.

What is the best way to handle these kind of problems? Can we depend on the file renaming solutions? Is file renaming is atomic operation?

Answer 1

Every watcher (even two in the same JVM) should always be notified of the new File being added.

If you want to divide the work, you can either

• use one JVM to run twice as many threads and divide the work via a queue.
• use an operation which will only succeed for one JVM. eg
  • file rename
  • create a lock file
  • lock the file itself

Is file renaming is atomic operation?

Yes, only one process can successful rename a file, even if both attempt to rename to same name.

Answer 2

For such a situation Spring Integration suggests FileSystemPersistentAcceptOnceFileListFilter : https://docs.spring.io/spring-integration/reference/html/files.html#file-reading

• Stores "seen" files in a MetadataStore to survive application restarts.
• The default key is 'prefix' plus the absolute file name; value is the timestamp of the file.
• Files are deemed as already 'seen' if they exist in the store and have the
• same modified time as the current file.

When you have shared persistent MetadataStore for all your application instances only one of them will process the file. All others will just filter it.