[英]Multithread: Dynamically object locking based on resource in hava
Need threads expert eyes here... 需要线程专家的眼睛在这里...
I am on a poc app where i am uploading files on FTP server 我在POC应用程序中,在FTP服务器上上传文件
In FTP server have multiple folders. 在FTP服务器中有多个文件夹。 Based on input response I reads files from the folder and move to another folder 根据输入响应,我从该文件夹中读取文件并移至另一个文件夹
The app can access by multiple threads at a time. 该应用程序一次可以访问多个线程。
So the problem was this: 所以问题是这样的:
Suppose FTP have a folders Folder_A and A_A_FOLDER Now Folder_A have 10 files. 假设FTP有一个文件夹Folder_A和A_A_FOLDER现在Folder_A有10个文件。 a thread came and read 10 files from FTP and start some calculation on it, it calculated one by one and then move to A_A_FOLDER it was middle in the process (let suppose it successfully moved 5 files from Folder_A to A_A_FOLDER) then another thread came and it picks remaining 5 files because they were underprocessed by thread 1, so thread 2 also start processing those 5 files 一个线程从FTP中读取了10个文件,并开始进行一些计算,它一个接一个地计算,然后移到A_A_FOLDER,它在进程中间(假设它已成功地将5个文件从Folder_A移到A_A_FOLDER),然后另一个线程进入了它选择剩余的5个文件,因为它们被线程1处理不足,因此线程2也开始处理这5个文件
So duplicate files problem here 所以这里有重复文件的问题
void m1(String folderName) {
// FTP related code
}
I have solved this problem by using synchronized keyword 我已经通过使用synced关键字解决了这个问题
Now every thing in sync and all processing working fine 现在,所有事物都已同步,所有处理工作正常
synchronized void m1(String folderName) {
// code
}
folderName decide which folder need to process folderName决定需要处理的文件夹
Now I have started facing performance issue 现在我开始面临性能问题
because the method is synchronized so all thread going to wait until processing thread not completed its task. 因为该方法是同步的,所以所有线程都将等待,直到处理线程未完成其任务。
I can improve this by following steps: 我可以按照以下步骤进行改进:
(Before going to a solution here is the some story to much dig on the problem) (在找到解决方案之前,这里是一些深入探讨该问题的故事)
As I have mentioned folderName parameter of m1 method decide which folder will process, So suppose I have 4 folders (A, B, A_T, B_T) in Ftp server, 2 folders are those where data need to read from (A and B), And 2 folder are those where data will move (A_T and B_T) 正如我提到的m1方法的folderName参数决定要处理的文件夹,所以假设我在Ftp服务器中有4个文件夹(A,B,A_T,B_T),其中2个文件夹是需要从(A和B)读取数据的文件夹, 2个文件夹是数据将移动的文件夹(A_T和B_T)
A_T and B_T is not the concern here because they are unique for each folder A and B So if the method will read from A then it will move it to A_T same for B (move to B_T) 这里不考虑A_T和B_T,因为它们对于每个文件夹A和B都是唯一的,因此,如果该方法将从A中读取,则它将把它移到与B相同的A_T中(移至B_T)
Now: 现在:
Suppose 4 thread comes to m1 method, 3 threads for folder A and 1 for folder B if somehow method synchronized request based on fileName parameter so I can improve the performance, means 1 thread will work on A another 2 threading will block because fileName is same for them so they will wait until first thread not completed it task where thread 4 will parallel work without any locking process because it's file name is different 假设m1方法有4个线程,文件夹A有3个线程,文件夹B有1个线程(如果以某种方式基于fileName参数同步请求,以便我提高性能,则意味着1个线程将在A上工作,另外2个线程将阻塞,因为fileName相同)为他们服务,所以他们将等到第一个线程未完成任务时,线程4将并行工作而无需任何锁定过程,因为文件名不同
So how can I achieve this(synchronized on fileName) in code level? 那么,如何在代码级别实现这一目标(与fileName同步)?
Note: i know i can break this logic using static locking list for resource and then the locks fileName resource eg: 注意:我知道我可以使用资源的静态锁定列表然后锁定fileName资源来打破此逻辑,例如:
private final Object A = new Object();
private final Object B = new Object();
but the problem with this approach is folder can be dynamically added, so I can't go with this. 但是这种方法的问题是可以动态添加文件夹,所以我不能这样做。
Need your help guys. 需要你们的帮助。
One approach would to be maintain a lock per directory: 一种方法是保持每个目录的锁定:
public class DirectoryTaskManager {
public static void main(String[] args) throws IOException {
DirectoryTaskManager manager = new DirectoryTaskManager();
manager.withDirLock(new File("Folder_A"), () -> System.out.println("Doing something..."));
}
public void withDirLock(File dir, Runnable task) throws IOException {
ReentrantLock lock = getDirLock(dir);
lock.lock();
try {
task.run();
} finally {
lock.unlock();
}
}
private Map<File, ReentrantLock> dirLocks = Collections.synchronizedMap(new HashMap<>());
public ReentrantLock getDirLock(File dir) throws IOException {
// Resolve the canonical file here so that different paths
// to the same file use the same lock
File canonicalDir = dir.getCanonicalFile();
if (!canonicalDir.exists() || !canonicalDir.isDirectory()) {
throw new FileNotFoundException(canonicalDir.getName());
}
return dirLocks.computeIfAbsent(canonicalDir, d -> new ReentrantLock());
}
}
Thanks @teppic and @OlegSklyar for your direction Finally here is full working example, 感谢@teppic和@OlegSklyar的指导最后,这里是完整的工作示例,
FolderImpl -> have method name call which can accessed by many threads FolderImpl->具有可以被许多线程访问的方法名称调用
I have used ConcurrentHashMap(Reads can happen very fast while write is done with a lock.) which is lil faster then synchronizedMap which will hold the folder name and a ReentrantLock, so lock will work on folder name 我用过ConcurrentHashMap(用锁完成写操作时读操作可能会非常快。)这要快于syncedMap,后者将保存文件夹名称和ReentrantLock,因此锁定将对文件夹名称起作用
public class FolderImpl {
private FolderImpl(){
System.out.println("init................");
}
private ConcurrentHashMap<String, ReentrantLock> concurrentHashMap= new ConcurrentHashMap();
private static final FolderImpl singleTon = new FolderImpl();
public static FolderImpl getSingleTon() {
return singleTon;
}
public void call(String name) throws Exception{
ReentrantLock getDirLock = getDirLock(name);
getDirLock.lock();
try {
for (int i = 0; i < 100; i ++) {
System.out.println(i+":"+name+":"+Thread.currentThread().getName());
try {
Thread.sleep(30);
} catch (Exception e) {
e.printStackTrace();
}
}}finally {
getDirLock.unlock();
}
}
public ReentrantLock getDirLock(String site) {
return concurrentHashMap.computeIfAbsent(site, d -> new ReentrantLock());
}
}
TaskCaller thread call the call method, here is sleep flavor so another thred can git execution time TaskCaller线程调用call方法,这是睡眠的味道,因此另一个线程可以git执行时间
public class TaskCaller extends Thread{
public FolderImpl singleTon = FolderImpl.getSingleTon();
public TaskCaller(String name) {
super();
this.name = name;
}
private String name;
@Override
public void run() {
for (int i = 0; i < 5; i++) {
System.out.println("Name:"+Thread.currentThread().getName());
try {
singleTon.call(name);
sleep(10);
} catch (Exception e) {
e.printStackTrace();
}
}
}
}
TestExecution class will execute 10 number of thread for testing TestExecution类将执行10个线程进行测试
public class TestExecution {
public static void main(String[] args) {
TaskCaller testThreadCC = new TaskCaller("A_FOLDER");
TaskCaller testThreadCC2 = new TaskCaller("A_FOLDER");
TaskCaller testThreadCC3 = new TaskCaller("B_FOLDER");
TaskCaller testThreadCC4 = new TaskCaller("C_FOLDER");
TaskCaller testThreadCC5 = new TaskCaller("C_FOLDER");
TaskCaller testThreadCC6 = new TaskCaller("C_FOLDER");
TaskCaller testThreadCC7 = new TaskCaller("A_FOLDER");
TaskCaller testThreadCC8 = new TaskCaller("A_FOLDER");
TaskCaller testThreadCC9 = new TaskCaller("B_FOLDER");
TaskCaller testThreadCC10 = new TaskCaller("B_FOLDER");
testThreadCC.start();
testThreadCC2.start();
testThreadCC3.start();
testThreadCC4.start();
testThreadCC5.start();
testThreadCC6.start();
testThreadCC7.start();
testThreadCC8.start();
testThreadCC9.start();
testThreadCC10.start();
}
}
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.