大量下载文件

Question

I'm writing a piece of code that is supposed to download a single file in 1 MB chunks. I used the same code as in this question: Download large file in small chunks in C# and converted it to VB.NET.

The code, same as the previous question works fine and write the file to disk. But it seems that something goes wrong in the second web request. For the sake of testing, I'm downloading a PNG file from https://d13yacurqjgara.cloudfront.net/users/22/screenshots/631004/attachments/53012/wallpaper-retina-cinemadisplay.png using my VB.NET code below.

The downloaded PNG seems to be fine half the way and the remaining image is scrambled.

I increased the defaultSize and chunk to 10 MB and the file downloads perfectly, but it seems that something in the For/Next loop truncates or pollutes the data.

Any thoughts what might be causing this to happen?

Private Const defaultSize As Long = 1048576
Private chunk As Long = 1048576
Private offset As Long = 0

Private Function downloadFile(ByVal url As String, ByVal filename As String) As Boolean
    Dim size As Long = getSize(url)
    Dim blockSize As Integer = Convert.ToInt32(size / defaultSize)
    Dim remainder As Integer = Convert.ToInt32(size Mod defaultSize)

    If remainder > 0 Then
        blockSize += 1
    End If

    Dim fileStream As FileStream = File.Create("C:\mydirectory\" & filename)

    For i As Integer = 0 To blockSize - 1
        If i = blockSize - 1 Then
            chunk = remainder
        End If

        Dim req As HttpWebRequest = HttpWebRequest.Create(url)
        req.Method = WebRequestMethods.Http.Get
        req.AddRange(Convert.ToInt32(offset), Convert.ToInt32(chunk + offset))
        Dim resp As HttpWebResponse = req.GetResponse()

        Using respStream As Stream = resp.GetResponseStream
            Dim buffer(4096) As Byte
            Dim bytesRead As Integer
            Do
                bytesRead = respStream.Read(buffer, 0, 4096)
                If bytesRead > 0 Then fileStream.Write(buffer, 0, bytesRead)
            Loop While bytesRead > 0
        End Using

        offset += chunk

        resp.Close()
        resp.Dispose()
    Next

    fileStream.Close()

    Return True
End Function

Private Function getSize(ByVal url As String) As Long
    Dim req As WebRequest = WebRequest.Create(url)
    req.Method = WebRequestMethods.Http.Head
    Dim resp As WebResponse = req.GetResponse
    Return Long.Parse(resp.ContentLength)
End Function

Answer 1

Aside from tidying up a bit of your code (which I presume you will be doing anyway, since you converted this from C#), making one small change gets it work perfectly.

The line

offset += chunk

should be

offset += chunk + 1

This solves the issue because offset starts at 0, and you're adding the chunk size to it on the next loop, therefore you're adding a megabyte and starting there. You want to be continuing from the byte following the last megabyte! I did begin to explain how it works, but the example I came up with doesn't work... so I can't explain it! Maybe someone else can; it's to do with req.AddRange - you're specifying a range, not a total.

Answer 2

One issue on the code conversion.

Convert.ToInt32(size / defaultSize)

This is the same code in the C# code and VB ignoring the semicolon. However it does not consider the nuance that the "/" operator can return a different value with integer division in the two languages.

The variable Size and defaultSize are both long integers. When dividing two integers in C# the result is an integer, decimal chopped and not rounded. In VB dividing two integers the result is coerced to decimal type then rounded.

/ Operator (C# Reference)

/ Operator (VB Reference)

For example if the values for size and defaultSize are 68 and 10.

VB

Dim result As Integer = Convert.ToInt32(size / defaultSize) 'VB result 7

C#

int result = Convert.ToInt32(size / defaultSize); // C# result 6

How this may affect the algorithm I have not pursued, given the original code was claimed to still not be working.