Downloading file parts using HttpWebRequest C#

Question

I am trying to download a 100GB file using HttpWebRequest. The download will be split into parts depending on a preset part size. Below is the code I use to download the file:

private static void AddRangeHeaderHack(WebHeaderCollection headers, long start, long end)
        {
            // Original workaround by Eric Cadwell, code taken from
            // https://connect.microsoft.com/VisualStudio/feedback/ViewFeedback.aspx?FeedbackID=93714
            Type type = headers.GetType();

            System.Reflection.MethodInfo setAddVerified = type.GetMethod("SetAddVerified",
                System.Reflection.BindingFlags.NonPublic | System.Reflection.BindingFlags.Instance | System.Reflection.BindingFlags.FlattenHierarchy
                );

            string rangeHeaderValue = String.Format("bytes={0}-{1}", start, end);
            if (setAddVerified != null)
                setAddVerified.Invoke(headers, new object[] { "Range", rangeHeaderValue });
        }

        private ulong GetRemoteFileSize(string URI)
        {
            ulong size = 0;
            HttpWebRequest req = null;
            try
            {
                req = (HttpWebRequest)WebRequest.Create(URI);
                using (var res = (HttpWebResponse)req.GetResponse())
                {
                    size = (ulong)res.ContentLength;
                    res.Close();
                }
            }
            catch (Exception ex)
            {

            }

            if (req != null)
            {
                try
                {
                    req.Abort();
                    req = null;
                }
                catch (Exception)
                {

                }
            }

            return size;
        }

        private int DownloadFromLink(string sSource, string sDestination)
        {
            int nRetryCount = 0;
            int nMaxRetry = 5;
            var lastProgress = DateTime.Now;
            ulong offset = 0;
            var bRetrying = false;
            var bResumable = false;

            var fileSize = GetRemoteFileSize(sSource);
            if (fileSize > 0)
                bResumable = true;

            while (true)
            {
                HttpWebRequest webRequest = null;
                try
                {
                    try
                    {
                        bRetrying = false;
                        do
                        {
                            try
                            {
                                if (bDownloadAbort)
                                {
                                    return -1;
                                }

                                webRequest = (HttpWebRequest)WebRequest.Create(sSource);
                                webRequest.Timeout = 3600000;

                                if (offset > 0)
                                {
                                    AddRangeHeaderHack(webRequest.Headers, (long)offset, (long)fileSize);
                                }

                                // Retrieve the response from the server
                                using (var webResponse = (HttpWebResponse)webRequest.GetResponse())
                                {

                                    var acceptRanges = String.Compare(webResponse.Headers["Accept-Ranges"], "bytes", true) == 0;

                                    // Open the URL for download 
                                    using (var streamResponse = webResponse.GetResponseStream())
                                    {
                                        if (streamResponse != null)
                                        {
                                            // Create a new file stream where we will be saving the data (local drive)
                                            using (var streamLocal = new FileStream(sDestination, offset>0?FileMode.Append:FileMode.Create, FileAccess.Write, FileShare.ReadWrite))
                                            {
                                                // It will store the current number of bytes we retrieved from the server
                                                int bytesSize = 0;
                                                // A buffer for storing and writing the data retrieved from the server
                                                byte[] downBuffer = new byte[/*16384*/ 1024 * 1024];
                                                bool binitialtry = true;
                                                int nRetries = 0;

                                                if (offset > 0)
                                                {
                                                    streamLocal.Seek((long)offset, SeekOrigin.Begin);
                                                }

                                                // Loop through the buffer until the buffer is empty
                                                while ((bytesSize = streamResponse.Read(downBuffer, 0, downBuffer.Length)) > 0 
                                                    || (File.Exists(sDestination) && (offset < (ulong)fileSize) && nRetries < 5 && bResumable))
                                                {
                                                    if (binitialtry && bytesSize == 0)
                                                    {
                                                        binitialtry = false;
                                                    }

                                                    if (!binitialtry && bytesSize == 0)
                                                    {
                                                        nRetries++;
                                                        bRetrying = nRetries<5;
                                                        break;
                                                    }

                                                    if (bDownloadAbort)
                                                    {
                                                        try { streamLocal.Close(); }
                                                        catch { }

                                                        return;
                                                    }

                                                    try
                                                    {
                                                        // Write the data from the buffer to the local hard drive
                                                        streamLocal.Write(downBuffer, 0, bytesSize);
                                                        offset += (ulong)bytesSize;
                                                    }
                                                    catch (IOException ex)
                                                    {
                                                        if (streamResponse != null)
                                                            streamResponse.Close();

                                                        if (streamLocal != null)
                                                            streamLocal.Close();

                                                        if (webRequest != null)
                                                            webRequest.Abort();

                                                        return -1;
                                                    }

                                                    Interlocked.Add(ref actualDownloaded, bytesSize);
                                                }

                                                // When the above code has ended, close the streams
                                                if (streamResponse != null)
                                                    streamResponse.Close();

                                                if (streamLocal != null)
                                                    try { streamLocal.Close(); }
                                                    catch { }

                                                if (webRequest != null)
                                                    webRequest.Abort();

                                                if (webRequest != null)
                                                    wcDownload.Dispose();

                                                streamLocal.Close();
                                            }
                                            streamResponse.Close();
                                        }
                                    }
                                    webResponse.Close();
                                }

                                if(!bRetrying)
                                    break;
                            }
                            catch (IOException ex)
                            {
                                if (webRequest != null)
                                    webRequest.Abort();

                                if (wcDownload != null)
                                    wcDownload.Dispose();

                                if (nRetryCount <= nMaxRetry)
                                {
                                    Thread.Sleep(10000);
                                    nRetryCount++;
                                    bRetrying = true;
                                }
                                else
                                {
                                    break;
                                }
                            }
                            catch (UnauthorizedAccessException ex)
                            {
                                if (webRequest != null)
                                    webRequest.Abort();

                                break;
                            }
                            catch (WebException ex)
                            {
                                if (webRequest != null)
                                    webRequest.Abort();

                                if (wcDownload != null)
                                    wcDownload.Dispose();

                                if (nRetryCount <= nMaxRetry)
                                {
                                    Thread.Sleep(10000);
                                    nRetryCount++;
                                    bRetrying = true;
                                }
                                else
                                {
                                    break;
                                }
                            }
                            finally
                            {


                            }

                        } while (bRetrying);


                    }
                    catch (Exception ex)
                    {
                        break;
                    }
                }
                catch
                {
                    break;
                }

                if(!bRetrying)
                    break;
            }
        }

If I try to download the file in 1 part, with out adding the range header, the code runs smoothly and the file downloads normally. When I add a range header, say from 10GB to 15GB or frankly any value, the code reaches streamResponse.Read and hangs there for several minutes then it throws an exception:

Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host

When the code retries the connection after the exception, the download resumes normally and the client is able to read data from the stream.

Can someone help me determine why such thing is happening?

Just to clear the matter about the server, the file is currently hosted on an Amazon S3 server, and the download is done from a generated direct link.

Answer 1

It could be a server setting, according to http://www.w3.org/Protocols/rfc2616/rfc2616-sec8.html#sec8.1.4

Clients that use persistent connections SHOULD limit the number of simultaneous connections that they maintain to a given server. A single-user client SHOULD NOT maintain more than 2 connections with any server or proxy. A proxy SHOULD use up to 2*N connections to another server or proxy, where N is the number of simultaneously active users. These guidelines are intended to improve HTTP response times and avoid congestion.

Try FDM, and see if it has a problem. http://www.freedownloadmanager.org/

Answer 2

I don't know how to download a file in parts using HttpWebRequest, but I found this example online to build an own implementation. The article is about the HttpClient in C#. There is also complete code and project you can find in the download section of this page.

The Problem is that not all server support partial download. So NuGet packages can be used that handle the exceptions eg: https://www.nuget.org/packages/downloader or https://www.nuget.org/packages/Shard.DonwloadLibrary . These Libraries will handle the chunks and convert them back into a readable file or stream.

Downloader:

var downloader = new DownloadService(new()
    {
        ChunkCount = 8,
    });
    string file = @"Your_Path\fileName.zip";
    string url = @"https://file-examples.com/fileName.zip";
    await downloader.DownloadFileTaskAsync(url, file);

or Download Library:

    string file = "Your_Path";
    string url = "https://file-examples.com/fileName.zip";
    var downloader = new LoadRequest(url,new()
    {
        Chunks = 8,
        DestinationPath= file,
    });

    await downloader.Task;

I hope I could help!