Java: how to synchronize file modification by threads

Question

Only one instance of my Java application can run at a time. It runs on Linux. I need to ensure that one thread doesn't modify the file while the other thread is using it.

I don't know which file locking or synchronization method to use. I have never done file locking in Java and I don't have much Java or programming experience.

I looked into java NIO and I read that "File locks are held on behalf of the entire Java virtual machine. They are not suitable for controlling access to a file by multiple threads within the same virtual machine." Right away I knew that I needed expert help because this is production code and I have almost no idea what I'm doing (and I have to get it done today).

Here's a brief outline of my code to upload some stuff (archive files) to a server. It gets the list of files to upload from a file (call it "listFile") -- and listFile can be modified while this method is reading from it. I minimize the chances of that by copying listFile to a temp file and using that temp file thereafter. But I think I need to lock the file during this copy process (or something like that).

package myPackage;

import java.io.BufferedReader;
import java.io.BufferedWriter;
import java.io.File;
import java.io.FileReader;
import java.io.FileWriter;
import java.io.IOException;
import com.example.my.FileHelper;
import com.example.my.Logger;

public class BatchUploader implements Runnable {

    private int processUploads() {
        File myFileToUpload;
        File copyOfListFile = null;
        try {
            copyOfListFile = new File("/path/to/temp/workfile");
            File origFile = new File("/path/to/listFile"); //"listFile" - the file that contains a list of files to upload
            DataWriter.copyFile(origFile, copyOfListFile);//see code below
        } catch (IOException ex) {
            Logger.log(ex);
        }
        try {
            BufferedReader input = new BufferedReader(new FileReader(copyOfListFile));
            try {
                while (!stopRunning && (fileToUploadName = input.readLine()) != null) {
                    upload(new File(fileToUploadName));
                }
            } finally {
                input.close();
                isUploading = false;
            }

        }
        return filesUploadedCount;
    }
}

Here is the code that modifies the list of files to be uploaded used in the above code:

public class DataWriter {

    public void modifyListOfFilesToUpload(String uploadedFilename) {

        StringBuilder content = new StringBuilder();

        try {
            File listOfFiles = new File("/path/to/listFile"); //file that contains a list of files to upload
            if (!listOfFiles.exists()) {
                //some code
            }

            BufferedReader input = new BufferedReader(new FileReader(listOfFiles));
            try {
                String line = "";
                while ((line = input.readLine()) != null) {
                    if (!line.isEmpty() && line.endsWith(FILE_EXTENSION)) {
                        if (!line.contains(uploadedFilename)) {
                            content.append(String.format("%1$s%n", line));
                        } else {
                            //some code
                        }
                    } else {
                        //some code
                    }
                }
            } finally {
                input.close();
            }
            this.write("/path/to/", "listFile", content.toString(), false, false, false);
        } catch (IOException ex) {
            Logger.debug("Error reading/writing uploads logfile: " + ex.getMessage());
        }
    }

    public static void copyFile(File in, File out) throws IOException {
        FileChannel inChannel = new FileInputStream(in).getChannel();
        FileChannel outChannel = new FileOutputStream(out).getChannel();
        try {
            inChannel.transferTo(0, inChannel.size(), outChannel);
        } catch (IOException e) {
            throw e;
        } finally {
            if (inChannel != null) {
                inChannel.close();
            }
            if (outChannel != null) {
                outChannel.close();
            }
        }
    }

    private void write(String path, String fileName, String data, boolean append, boolean addNewLine, boolean doLog) {
        try {
            File file = FileHelper.getFile(fileName, path);
            BufferedWriter bw = new BufferedWriter(new FileWriter(file, append));
            bw.write(data);
            if (addNewLine) {
                bw.newLine();
            }
            bw.flush();
            bw.close();
            if (doLog) {
                Logger.debug(String.format("Wrote %1$s%2$s", path, fileName));
            }
        } catch (java.lang.Exception ex) {
            Logger.log(ex);
        }

    }
}

Answer 1

My I suggest a slightly different approach. Afair on Linux the file rename (mv) operation is atomic on local disks. No chance for one process to see a 'half written' file.

Let XXX be a sequence number with three (or more) digits. You could let your DataWriter append to a file called listFile-XXX.prepare and write a fixed number N of filenames into it. When N names are written, close the file and rename it (atomic, see above) to listFile-XXX. With the next filename, start writing to listFile-YYY where YYY=XXX+1.

Your BatchUploader may at any time check whether it finds files matching the pattern listFile-XXX, open them, read them upload the named files, close and delete them. There is no chance for the threads to mess up each other's file.

Implementation hints:

1. Make sure to use a polling mechanism in BatchUploader that waits 1 or more seconds if it does not find a file ready for upload (prevent idle wait).
2. You may want to make sure to sort the listFile-XXX according to XXX to make sure the uploading is kept in sequence.
3. Of course you could variate the protocol of when listFile-XXX.prepare is closed. If DataWriter has nothing to do for a longer time, you don't want to have files ready for upload hang around just because there are not yet N ready.

Benefits: no locking (which will be a pain to get right), no copying, easy overview over the work queue and it state in the file system.

Answer 2

Here is a slightly different suggestion. Assuming your file names don't have '\\n' characters in them (it's a big assumption on linux, I know, but you can have your writer look up for that), why not only read complete lines and ignore the incomplete ones? By incomplete lines, I mean lines that end with EOF and not with \\n.

Edit: see more suggestions in comments below.