[英]write from multiple threads on same file without locking, Java
I am making a download manager and I want multiple threads downloading different segments of file to write to file at different places at a time. 我正在制作一个下载管理器,并且我希望有多个线程下载文件的不同部分,以便一次写入不同位置的文件。 Just for every one's clarification I dont want file to lock because it will kill the purpose of different threads writing at a time.
只是为了每个人的澄清,我都不希望文件被锁定,因为它会杀死一次写不同线程的目的。 I am using Apache HttpClient library and FileChannel transferFrom().
我正在使用Apache HttpClient库和FileChannel transferFrom()。 Current code only downloads the first segment and simply ignores other segments.
当前代码仅下载第一段,而忽略其他段。
Code Explanation: The startDownload method creates a new file and checks if link support partial content, If it does it starts threads for each segment otherwise a single thread will download the whole file.The getFileName is function for extracting file name from URI. 代码说明:startDownload方法将创建一个新文件并检查链接是否支持部分内容;如果这样做,它将为每个段启动线程,否则,一个线程将下载整个文件。getFileName是用于从URI提取文件名的函数。 The Download method contains the code which actually downloads the file using HttpClient and then writes it using transferFrom.
Download方法包含实际使用HttpClient下载文件,然后使用transferFrom写入文件的代码。
public void startDownload() {
Thread thread = new Thread(() -> {
try {
String downloadDirectory = "/home/muhammad/";
URI uri = new URI("http://94.23.204.158/JDownloader.zip");
int segments = 2;
// Create a HttpClient for checking file for segmentation.
CloseableHttpClient Checkingclient = HttpClients.createDefault();
// get request for checking size of file.
HttpGet checkingGet = new HttpGet(uri);
CloseableHttpResponse checkingResponse = Checkingclient.execute(checkingGet);
long sizeofFile = checkingResponse.getEntity().getContentLength();
// Create a new file in downloadDirectory with name extracted from uri.
File file = new File(downloadDirectory + getFileName(uri));
if (!file.exists()) {
file.createNewFile();
}
// set range header for checking server support for partial content.
checkingGet.setHeader("Range", "bytes=" + 0 + "-" + 1);
checkingResponse = Checkingclient.execute(checkingGet);
// Check if response code is 206 (partial content response code).
if (checkingResponse.getStatusLine().getStatusCode() == 206) {
//find size of each segment.
final long sizeOfEachSegment = sizeofFile / segments;
//Download each segment independently.
for (int i = 0; i < segments; i++) {
Download(i * sizeOfEachSegment, (i + 1) * sizeOfEachSegment, sizeOfEachSegment, file, uri);
}
// Thread used for last few Bytes and EOF.
Download(sizeOfEachSegment * segments, sizeofFile, Long.MAX_VALUE, file, uri);
} else {
System.err.println("server dont support partial content");
System.out.println(checkingResponse.getStatusLine().getStatusCode());
// Download complete file using single thread.
Download(0, 0, Long.MAX_VALUE, file, uri);
}
} catch (IOException | URISyntaxException ex) {
Logger.getLogger(Downloader.class.getName()).log(Level.SEVERE, null, ex);
}
});
thread.start();
}
public void Download(long start, long end, long sizeOfEachSegment, File file, URI uri) {
Thread thread = new Thread(() -> {
try {
FileChannel fileChannel = new FileOutputStream(file).getChannel();
CloseableHttpClient client = HttpClients.createDefault();
HttpGet get = new HttpGet(uri);
// Range header for defining which segment of file we want to receive.
if (end != 0) {
String byteRange = start + "-" + end;
get.setHeader("Range", "bytes=" + byteRange);
}
CloseableHttpResponse response = client.execute(get);
ReadableByteChannel inputChannel = Channels.newChannel(response.getEntity().getContent());
fileChannel.transferFrom(inputChannel, start, sizeOfEachSegment);
response.close();
client.close();
fileChannel.close();
} catch (IOException | IllegalStateException exception) {
Logger.getLogger(Downloader.class.getName()).log(Level.SEVERE, null, exception);
}
});
thread.start();
}
Some fix to existing code that can make multiple threads to write to same file at same time without waiting will be nice but I am also interested in studying other more efficient techniques if they can do the above task. 可以使多个线程无需等待就可以同时写入同一文件的现有代码的某些修补程序会很不错,但是如果它们可以完成上述任务,我也有兴趣研究其他更有效的技术。 If in any case writing to a file without waiting is impossible then any other efficient solution is more then welcome.
如果在任何情况下都不可能不等待写入文件,那么欢迎使用任何其他有效的解决方案。 Thanks in advance :)
提前致谢 :)
您可以让一个线程写入文件,而不是将多个线程写入同一个文件,而是由多个线程生成数据,但将其存储在文件编写器线程的某种缓冲区中。
Writing to the same file from different threads is not going to help you at all - it would probably even dramatically harm throughput. 从不同的线程写入相同的文件根本无法帮助您-甚至可能会严重损害吞吐量。
You should use one thread to write to the file and feed it from a queue. 您应该使用一个线程来写入文件并从队列中进行馈送。
Something like: 就像是:
class WriteBlock {
long offset;
byte[] data;
}
BlockingQueue<WriteBlock> writeQueue = new LinkedBlockingQueue<>();
Now each downloading thread should read a block from the download, create a WriteBlock
and post it into the queue. 现在,每个下载线程都应从下载中读取一个块,创建一个
WriteBlock
并将其发布到队列中。
Meanwhile the writing thread sucks WriteBlock
s out of the queue and writes them as fast as it can. 同时,写入线程将
WriteBlock
s从队列中吸出,并尽可能快地写入它们。
There may be optimizations to resequence the blocks while in the queue (perhaps with a PriorityBlockingQueue ) but do it the simple way first. 可能有一些优化措施,可以在队列中(也许使用PriorityBlockingQueue )对块进行重新排序,但首先要采用简单的方法。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.