[英]How thread lock should be handled to keep other threads waiting until downloading a file and then allow all threads to read the file in one go
我正在使用 ExecutorService fixedThreadPool() 來運行TASK 。
此處的任務定義為從特定的 URL 下載文件，如果不存在則將其保存到數據庫中，否則僅從數據庫中讀取文件。 所以它更像是一個讀寫器問題，執行器線程池的任何一個線程都可以充當一次寫入器，而其他線程將成為后續請求的讀取器。
我正在使用 Semaphore 來執行此操作，但這種方法的問題是后續讀取請求是按順序發生的。
如果 4 個TASK旨在命中相同的 URL 我需要同步，直到文件下載並釋放信號量，即在 4 個線程中任何人都可以獲得鎖並且 rest 3 正在等待。 下載完成后，所有剩余的 3 個線程應同時讀取下載的文件。 但在我的案例中，最后一步是按順序進行的，這也會對項目績效產生影響。
說了上面的用例，下面是我的示例代碼：
以下 Runnable 被傳遞給 ExecutorService 以在 SharedObject class 上執行任務。
class DownloadRunnable(SharedObjectMT sharedObject, String url) implement Runnable {
void run() {
sharedObject.loadFile(url);
}
}
class SharedObjectMT {
// This Hashmap acts ConcurrentHashMap with URL and semaphore mapping. So
// multiple threads requesting for the same URL will only be synchronized on their
// corresponding semaphore. And threads requesting for different URLs
// will run concurrently.
private static HashMap<String, Semaphore> syncMap = new HashMap<>();
.....
void loadFile(String url) {
// Let all threads enter sequentially and try to assign a Semaphore for their url in the
// hashmap. If the url has never been requested, then only a Semaphore(say S1) will be
// assigned to that url. And for all the other threads with *same request url*, this
// Semaphore(S1) will be used to handle concurrency.
synchronized(syncMap) {
if(syncMap[url] == null) {
syncMap[url] = new Semaphore(1);
}
}
Semaphore semaphore = syncMap[url];
synchronized(semaphore) {
............
............
semaphore.acquire();
String filePath = findInDatabase(url);
if(filePath != null) {
semaphore.release(); // no need to hold semaphore since file already downloaded.
printStatus("Already downloaded file from url = "+url);
} else {
// This DownloadThread is actually a mock of my real project where a third-party
// library uses a thread to download the file.
DownloadThread(() -> {
printStatus("Download completed for url= "+ url +". Releasing semaphore.");
semaphore.release();
}).start();
.............
.............
}
}
}
}
我知道單個信號量無法幫助我。 也許我們可以再使用 1 個 Semaphore 來區分讀寫鎖或任何其他鎖定機制。 所以需要一些關於如何使用這種一次性同步的幫助。
注意：如果您在上述代碼中發現任何語法錯誤，請忽略，因為實際項目在 Kotlin 但這是一個基本的 Java 多線程問題，所以我將其發布為 ZD52387880E1EA228381972D375921 代碼。
我不確定 Kotlin，但我可以在 Java 中演示：
import java.io.IOException;
import java.util.HashMap;
import java.util.Objects;
import java.util.Random;
import java.util.concurrent.Callable;
import java.util.concurrent.ExecutionException;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.Future;
import java.util.concurrent.TimeUnit;
public class DownloadOrRead {
//Utility method, which just generates a random String instance...
private static String randomString(final int length) {
String alphabet = "abcdefghijklmnopqrstuvwxyz";
alphabet += alphabet.toUpperCase();
alphabet += "0123456789";
final int alphabetSize = alphabet.length();
final char[] chars = new char[length];
final Random rand = new Random();
for (int i = 0; i < chars.length; ++i)
chars[i] = alphabet.charAt(rand.nextInt(alphabetSize));
return String.valueOf(chars);
}
public static class DownLoadCallable implements Callable<String> {
private final String url;
public DownLoadCallable(final String url) {
this.url = Objects.requireNonNull(url);
}
@Override
public String call() throws IOException, InterruptedException {
/*Utilize url property here to download the file...
In our case, just simulate a download delay supposedly...*/
Thread.sleep(5000L + (long) (Math.random() * 10000L));
//Return the file's local path...
return randomString(20); //In our case, a random String of 20 characters.
}
}
//This is the method you are looking for:
public static String loadPath(final ExecutorService executorService, //Can be shared between calls of loadPath...
final HashMap<String, Future<String>> urlToFuture, //MUST be shared between calls of loadPath!
final String url) //The URL. Can be the same as a URL in a previous call of loadPath.
throws InterruptedException, ExecutionException {
final Future<String> future;
synchronized (urlToFuture) {
if (!urlToFuture.containsKey(url)) //If nowhere to be seen...
urlToFuture.put(url, executorService.submit(new DownLoadCallable(url))); //Create a Future...
future = urlToFuture.get(url); //Obtain the Future (new or old).
}
return future.get(); //Outside the synchronized block!
}
public static void main(final String[] args) {
System.out.println("Creating ExecutorService...");
final ExecutorService executorService = Executors.newFixedThreadPool(10);
System.out.println("Creating shared map...");
final HashMap<String, Future<String>> urlToFuture = new HashMap<>();
System.out.println("Creating random URLs...");
final String[] urls = new String[]{randomString(10), randomString(20), randomString(15)};
try {
System.out.println("Downloading files sequencially...");
final Random rand = new Random();
for (int i = 0; i < 100; ++i)
System.out.println(loadPath(executorService, urlToFuture, urls[rand.nextInt(urls.length)]));
executorService.shutdown();
executorService.awaitTermination(10, TimeUnit.MINUTES);
}
catch (final InterruptedException | ExecutionException x) {
System.err.println(x);
}
}
}
整個想法是將Callable提交給處理下載的ExecutorService 。 我們還利用submit方法返回的Future來獲取所需的結果路徑/文件/任何內容。 只需調用get的Future object 即可。 您唯一需要同步的是指向Future的 URL 的Map 。
You will notice, when running this test program, that the first file is blocking until downloaded, then subsequent calls for the same URL are finished immediately (because the URL is already downloaded) and we only block for each new URL (which is not downloaded然而）。 在這種情況下，我只使用 3 個隨機 URL，每個 URL 需要 5 到 15 秒才能完成，這給了我們大約 15 到 45 秒的正常運行時間，因為我們按順序下載它們。
loadPath方法到此結束。 但是在上面的示例代碼中，文件是按順序下載的。 如果您還需要多個Thread來下載，您可以從許多Thread調用loadPath （不需要在共享Map之外的其他地方進一步同步）。
正如在此答案中所讀到的那樣，似乎在操作完成后調用相同Future的get方法，總是會產生相同的 object 或如果失敗則拋出相同的Exception 。 這是我們在本文中提供的代碼中使用的優勢。
或者甚至更好，正如@drekbour 在評論中指出的那樣，使用computeIfAbsent和ConcurrentHashMap來完成這項工作，如下所示：
import java.util.Objects;
import java.util.Random;
import java.util.concurrent.Callable;
import java.util.concurrent.ConcurrentHashMap;
import java.util.concurrent.ExecutionException;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.Future;
import java.util.concurrent.TimeUnit;
public class DownloadOrRead1 {
//Utility method, which just generates a random String instance...
private static String randomString(final int length) {
String alphabet = "abcdefghijklmnopqrstuvwxyz";
alphabet += alphabet.toUpperCase();
alphabet += "0123456789";
final int alphabetSize = alphabet.length();
final char[] chars = new char[length];
final Random rand = new Random();
for (int i = 0; i < chars.length; ++i)
chars[i] = alphabet.charAt(rand.nextInt(alphabetSize));
return String.valueOf(chars);
}
public static class DownLoadCallable implements Callable<String> {
private final String url;
public DownLoadCallable(final String url) {
this.url = Objects.requireNonNull(url);
}
@Override
public String call() throws InterruptedException {
System.out.println("Downloading " + url + "...");
/*Utilize url property here to download the file...
In our case, just simulate a download delay supposedly...*/
Thread.sleep(5000L + (long) (Math.random() * 10000L));
System.out.println("Downloaded " + url + '.');
//Return the file's local path...
return randomString(20); //In our case, a random String of 20 characters.
}
}
//This is the method you are looking for:
public static String loadPath(final ExecutorService executorService, //Can be shared between calls of loadPath...
final ConcurrentHashMap<String, Future<String>> urlToFuture, //MUST be shared between calls of loadPath!
final String url) //The URL. Can be the same as a URL in a previous call of loadPath.
throws InterruptedException, ExecutionException {
return urlToFuture.computeIfAbsent(url, url2 -> executorService.submit(new DownLoadCallable(url2))).get();
}
public static void main(final String[] args) {
System.out.println("Creating ExecutorService...");
final ExecutorService executorService = Executors.newFixedThreadPool(10);
System.out.println("Creating shared Map...");
final ConcurrentHashMap<String, Future<String>> urlToFuture = new ConcurrentHashMap<>();
System.out.println("Creating random URLs...");
final String[] urls = new String[]{randomString(10), randomString(10), randomString(10)};
try {
System.out.println("Downloading files sequencially...");
final Random rand = new Random();
for (int i = 0; i < 100; ++i) {
final String url = urls[rand.nextInt(urls.length)];
System.out.println("Path for " + url + ": " + loadPath(executorService, urlToFuture, url));
}
executorService.shutdown();
executorService.awaitTermination(10, TimeUnit.MINUTES);
}
catch (final InterruptedException | ExecutionException x) {
System.err.println(x);
}
}
}
為什么“通知”會喚醒所有等待的線程，盡管只有一個線程應該受到影響？
[英]Why does 'notify' wake up all waiting threads although only one thread should be affected?
如何啟動並發線程以獲取文件鎖並互相等待
[英]How to start concurrent threads that acquire a file lock and wait on each other
我的線程如何將文件標記為正在使用中，以便其他線程無法讀取該文件？
[英]How can my thread mark a file as in use so other threads won't read it?
許多線程等待鎖定一個對象，但沒有線程持有該鎖
[英]Many threads waiting to lock an object, but no thread is holding that lock
聲明:本站的技術帖子網頁，遵循CC BY-SA 4.0協議，如果您需要轉載，請注明本站網址或者原文地址。任何問題請咨詢:yoyou2525@163.com.