I am trying to find out the optimal way to upload a large file to the server. On my local, I tried both methods:
But here the merging time is way more as compared to upload time, atleast on my local. Hence, the parallel upload is always a lot slower than the sequential upload. How can I improve my upload time?
Here is the Sequential Upload Code :
[HttpPost]
[RequestFormLimits(MultipartBodyLengthLimit = 209715200)]
[RequestSizeLimit(209715200)]
public async Task<IActionResult> UploadSeq()
{
string filename = "./Uploads/seqfile";
using(Stream file = Request.Body)
using (FileStream stream = new FileStream(filename, FileMode.Append))
{
await file.CopyToAsync(stream);
await stream.FlushAsync();
}
return Ok();
}
And here is the parallel code :
[HttpPost("/ParUpload/{id}/{size}")]
[RequestFormLimits(MultipartBodyLengthLimit = 209715200)]
[RequestSizeLimit(209715200)]
public async Task<IActionResult> UploadParallel(int id, int size)
{
string filename = "./Uploads/parfilecomplete.mkv";
using (Stream file = Request.Body)
using (FileStream stream = new FileStream(filename, FileMode.OpenOrCreate, FileAccess.Write, FileShare.Write))
{
stream.Seek(id * size, SeekOrigin.Begin);
await file.CopyToAsync(stream);
await stream.FlushAsync();
}
return Ok();
}
For large files, the parallel upload is preferred. But there are few things to consider
Parallel for the win, but you have to do extra work if you want to develop such system.
You should not create more than 6 connections per domain (FQDN) since there are connection limits in place on client side. You can increase the limit by using same ip but different domain names like upload1.domain.com
, upload2.domain.com
, etc.. but then you will probably fill the bandwidth.
You should not open and close file every time a part is received, it is a very expensive operation. Instead create a writer class that open the file and you queue received part and the location to write, and writer will write when it can, and closes the file when it writes all parts. You would trade disk latency with memory.
If you can enable TCP fast start. This will let you send data to your server faster
Choose your buffer size wisely. Try with different sizes where it requires minimal disk IO and even less network IO. Buffer size must a multiple of disk sector size.
These are what comes to my mind from my previous experience. I do not remember exact details but that items should help.
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.