Multithread: Dynamically object locking based on resource in hava

Question

Need threads expert eyes here...

I am on a poc app where i am uploading files on FTP server

In FTP server have multiple folders. Based on input response I reads files from the folder and move to another folder

The app can access by multiple threads at a time.

So the problem was this:

Suppose FTP have a folders Folder_A and A_A_FOLDER Now Folder_A have 10 files. a thread came and read 10 files from FTP and start some calculation on it, it calculated one by one and then move to A_A_FOLDER it was middle in the process (let suppose it successfully moved 5 files from Folder_A to A_A_FOLDER) then another thread came and it picks remaining 5 files because they were underprocessed by thread 1, so thread 2 also start processing those 5 files

So duplicate files problem here

void m1(String folderName) {
// FTP related code
}

I have solved this problem by using synchronized keyword

Now every thing in sync and all processing working fine

synchronized void m1(String folderName) {
// code
}

folderName decide which folder need to process

Now I have started facing performance issue

because the method is synchronized so all thread going to wait until processing thread not completed its task.

I can improve this by following steps:

(Before going to a solution here is the some story to much dig on the problem)

As I have mentioned folderName parameter of m1 method decide which folder will process, So suppose I have 4 folders (A, B, A_T, B_T) in Ftp server, 2 folders are those where data need to read from (A and B), And 2 folder are those where data will move (A_T and B_T)

A_T and B_T is not the concern here because they are unique for each folder A and B So if the method will read from A then it will move it to A_T same for B (move to B_T)

Now:

Suppose 4 thread comes to m1 method, 3 threads for folder A and 1 for folder B if somehow method synchronized request based on fileName parameter so I can improve the performance, means 1 thread will work on A another 2 threading will block because fileName is same for them so they will wait until first thread not completed it task where thread 4 will parallel work without any locking process because it's file name is different

So how can I achieve this(synchronized on fileName) in code level?

Note: i know i can break this logic using static locking list for resource and then the locks fileName resource eg:

private final Object A = new Object();
private final Object B = new Object();

but the problem with this approach is folder can be dynamically added, so I can't go with this.

Need your help guys.

Answer 1

One approach would to be maintain a lock per directory:

public class DirectoryTaskManager {
    public static void main(String[] args) throws IOException {
        DirectoryTaskManager manager = new DirectoryTaskManager();
        manager.withDirLock(new File("Folder_A"), () -> System.out.println("Doing something..."));
    }

    public void withDirLock(File dir, Runnable task) throws IOException {
        ReentrantLock lock = getDirLock(dir);
        lock.lock();
        try {
            task.run();
        } finally {
            lock.unlock();
        }
    }

    private Map<File, ReentrantLock> dirLocks = Collections.synchronizedMap(new HashMap<>());

    public ReentrantLock getDirLock(File dir) throws IOException {
        // Resolve the canonical file here so that different paths 
        // to the same file use the same lock
        File canonicalDir = dir.getCanonicalFile();
        if (!canonicalDir.exists() || !canonicalDir.isDirectory()) {
            throw new FileNotFoundException(canonicalDir.getName());
        }
        return dirLocks.computeIfAbsent(canonicalDir, d -> new ReentrantLock());
    }
}

Answer 2

Thanks @teppic and @OlegSklyar for your direction Finally here is full working example,

FolderImpl -> have method name call which can accessed by many threads

I have used ConcurrentHashMap(Reads can happen very fast while write is done with a lock.) which is lil faster then synchronizedMap which will hold the folder name and a ReentrantLock, so lock will work on folder name

public class FolderImpl {
    private FolderImpl(){
        System.out.println("init................");
    }

    private ConcurrentHashMap<String, ReentrantLock> concurrentHashMap= new ConcurrentHashMap();
    private static final FolderImpl singleTon = new FolderImpl();
    public static FolderImpl getSingleTon() {
        return singleTon;
    }

    public void call(String name) throws Exception{
        ReentrantLock getDirLock = getDirLock(name);
        getDirLock.lock();
        try {
        for (int i = 0; i < 100; i ++) {
            System.out.println(i+":"+name+":"+Thread.currentThread().getName());
            try {
                Thread.sleep(30);
            } catch (Exception e) {
                e.printStackTrace();
            }
        }}finally {
            getDirLock.unlock();
        }

    }

    public ReentrantLock getDirLock(String site)  {
        return concurrentHashMap.computeIfAbsent(site, d -> new ReentrantLock());
    }
}

TaskCaller thread call the call method, here is sleep flavor so another thred can git execution time

public class TaskCaller extends Thread{
    public FolderImpl singleTon = FolderImpl.getSingleTon();
    public TaskCaller(String name) {
        super();
        this.name = name;
    }

    private String name;
    @Override
    public void run() {
        for (int i = 0; i < 5; i++) {
            System.out.println("Name:"+Thread.currentThread().getName());

            try {
                singleTon.call(name);
                sleep(10);
            } catch (Exception e) {
                e.printStackTrace();
            }
        }
    }
}

TestExecution class will execute 10 number of thread for testing

public class TestExecution {

    public static void main(String[] args) {
        TaskCaller testThreadCC = new TaskCaller("A_FOLDER");
        TaskCaller testThreadCC2 = new TaskCaller("A_FOLDER");
        TaskCaller testThreadCC3 = new TaskCaller("B_FOLDER");
        TaskCaller testThreadCC4 = new TaskCaller("C_FOLDER");
        TaskCaller testThreadCC5 = new TaskCaller("C_FOLDER");
        TaskCaller testThreadCC6 = new TaskCaller("C_FOLDER");
        TaskCaller testThreadCC7 = new TaskCaller("A_FOLDER");
        TaskCaller testThreadCC8 = new TaskCaller("A_FOLDER");
        TaskCaller testThreadCC9 = new TaskCaller("B_FOLDER");
        TaskCaller testThreadCC10 = new TaskCaller("B_FOLDER");

        testThreadCC.start();
        testThreadCC2.start();
        testThreadCC3.start();
        testThreadCC4.start();
        testThreadCC5.start();
        testThreadCC6.start();
        testThreadCC7.start();
        testThreadCC8.start();
        testThreadCC9.start();
        testThreadCC10.start();

    }

}