Multiple Threading

Question

I am developing an application (in C#). Which is basically a file uploading/downloading application (just like filezila).

But there is one special feature on that (on file Downloading) :-

We can divide a file in various segments(up to 100). And can repeat this process by various Cycles (means we can divide a file in various segments and can repeat this process more then one time(by Cycles),at same time).

I am using Threading concept for that.In the application there are some progress bars and labels that are showing the uploading/downloading status.

My problem is that :-

If i do this process (downloading) for more numbers for example :-

100 Cycles and (the cycles can be more then 100)

100 Segments

Means we are diving a file into 100 parts (segment) and the same process will work 100 times(Cycles).

Means 100 * 100 =10,000 files will download at the time by same process.

But i am facing problem in that if i take big numbers for this task then there should be more threads will opens.

And application will hangs if there are 100 TCP Ports/Threads are running simultaneously.

I tried so many things for that but i did not got success yet.Like -

Asynchronous Delegates,ThreadPool etc.

I am facing a major problem in updating a progress bars and labels (because the current functionality is working with Asynchronous Delegates).

Please suggest me some way to resolve this issue.

Answer 1

Because all these threads want to pass data to the UI thread to show the progress, the UI thread is becoming the bottleneck of your application.

You could add a timer that every now and then causes the UI thread to inspect the progress of the jobs.

This way the UI thread is not driven by the huge number of threads but by time.

Post some code if you need more details.

Answer 2

Hello Erno i am sending some code for better understanding :-

this is the Downloading function :

Public Sub FileDownload()

    txtBytesToDownload.Text = totalBytesToDownload.ToString("N", nfi)  'set textbox with total file size

    del = New MyDelegate(AddressOf DownloadDataInRange)

    callback = New AsyncCallback(AddressOf CalcCallback)


    TotalBytesToDownloadForEachCycle = totalBytesToDownload
    downloadedBytes = 0
    dlStopWatch = Stopwatch.StartNew()

    Dim flag As Integer = 0

    Dim segment As Integer = nuSegment.Value          'total segments
    Dim cyl = cycleNumericUpDown.Value                'total cycles

    'ServicePointManager.DefaultConnectionLimit = segment

    If UtilityFunctions.CheckRangeSupported(dlUrlHttpTextBox.Text) Then

        Dim segmentSize As Integer = 0
        segmentSize = TotalBytesToDownloadForEachCycle / segment

        ' For cy As Integer = 0 To (cyl - 1)

        For ctr As Integer = 0 To (segment - 1)        'loop for total Segments

            GC.Collect()
            flag += 1

            Dim fileValues As New FileValues()

            If ctr = 0 Then
                fileValues.StartPoint = 0               'starting point of each segment (in bytes)
            Else
                fileValues.StartPoint = segmentSize * ctr + 1
            End If

            fileValues.EndPoint = fileValues.StartPoint + segmentSize   'end point of each segment (in bytes)

            If (ctr = (segment - 1)) Then
                fileValues.EndPoint += TotalBytesToDownloadForEachCycle Mod segment
            End If

            fileValues.URL = dlUrlHttpTextBox.Text                         'downloading file url
            Dim str As String = ctr.ToString() + "_" + flag.ToString()

            ' del.BeginInvoke(fileValues, callback, Nothing)


            newThread = New System.Threading.Thread(AddressOf DownloadBytes)      'Thread starts here
            newThread.Priority = ThreadPriority.Highest
            newThread.IsBackground = True
            newThread.Start(fileValues)

            'ThreadPool.QueueUserWorkItem(AddressOf DownloadBytes, fileValues)

            fileValues = Nothing

        Next

        ' ProgressBarTotal.Value += 1

        ' Next

    Else
        Dim fileValues As New FileValues()
        fileValues.StartPoint = 0
        fileValues.EndPoint = TotalBytesToDownloadForEachCycle
        fileValues.URL = dlUrlHttpTextBox.Text
        Dim newThread As New System.Threading.Thread(AddressOf DownloadBytes)
        newThread.Name = "Thread1"
        newThread.Start(fileValues)
        'ThreadPool.QueueUserWorkItem(AddressOf DownloadBytes, fileValues)
        fileValues = Nothing

    End If

    Console.WriteLine("Http Downloload End")
    Debug.Print("thread List 1 count =" + _threadsList1.Count.ToString() + ", thread list 2 count =" + _threadsList2.Count.ToString() + ", thread list 3 count =" + _threadsList3.Count.ToString())

    'MessageBox.Show("finished - flag=" + flag.ToString())

    'After all Cycles Complete 
    'startButton.Enabled = True
    'abortButton.Enabled = False
    'skipButton.Enabled = False
    'DataGridViewDLOrPing.Enabled = True
    'DataGridViewUL.Enabled = True

    'protocolComboBox.Enabled = True
    'modelComboBox.Enabled = True
    'testTypeComboBox.Enabled = True
    'measurementComboBox.Enabled = True
    'cycleNumericUpDown.Enabled = True
    'DelayNumericUpDown.Enabled = True
    'InputBoxPrivilege()

End Sub

This is the function that Thread will call to read the bytes :

Public Sub DownloadBytes(ByVal p_fileValues As FileValues)

    Dim httpWebRequest As HttpWebRequest
    Dim httpWebResponse As HttpWebResponse
    Dim responseStream As Stream

    Dim threadName = threadID

    Try
        Console.WriteLine("Start " + Thread.CurrentThread.Name)
        httpWebRequest = DirectCast(WebRequest.Create(p_fileValues.URL), HttpWebRequest)
        'httpWebRequest.ProtocolVersion = HttpVersion.Version11
        httpWebRequest.KeepAlive = False
        httpWebRequest.AddRange(p_fileValues.StartPoint, p_fileValues.EndPoint)
        httpWebRequest.Credentials = CredentialCache.DefaultCredentials
        httpWebRequest.Proxy = WebRequest.DefaultWebProxy
        httpWebResponse = CType(httpWebRequest.GetResponse(), HttpWebResponse)
        responseStream = httpWebResponse.GetResponseStream()

        Dim bytesSize As Integer = 0
        ' A buffer for storing and writing the data retrieved from the server
        Dim downBuffer As Byte() = New Byte(2047) {}
        Dim bytesAlreadyDownloaded As Int64 = p_fileValues.StartPoint

        ' Loop through the buffer until the buffer is empty
        While (True)

            'end while loop if the Abort button is clicked
            If (isActionAborted = True) Then
                Exit While
            End If

            'currentCycle is > total Cycles ,then end while
            If (currentCycleDownload > cycleNumericUpDown.Value) Then
                Exit While
            End If

            bytesSize = responseStream.Read(downBuffer, 0, downBuffer.Length)

            bytesAlreadyDownloaded += bytesSize

            'speedtimer.Start()

            If (bytesSize <= 0) Then
                If (downloadedBytes < totalBytesToDownload) Then
                    Me.Invoke(New UploadProgressCallback(AddressOf Me.UpdateDownloadProgress), New Object() {(totalBytesToDownload - downloadedBytes), totalBytesToDownload})
                End If

                Exit While
            End If

            ' Invoke the method that updates the form's label and progress bar
            Me.Invoke(New DownloadProgressCallback(AddressOf Me.UpdateDownloadProgress), New Object() {bytesSize, TotalBytesToDownloadForEachCycle})

            If (bytesAlreadyDownloaded > p_fileValues.EndPoint) Then
                'Console.WriteLine("Downloading part files Exit " + p_fileValues.StartPoint.ToString() + "," + p_fileValues.EndPoint.ToString())
                Exit While
            End If

        End While

        responseStream.Flush()
        responseStream.Close()
        httpWebResponse.Close()
        responseStream.Dispose()
        responseStream = Nothing
        httpWebResponse = Nothing
        Console.WriteLine("End " + Thread.CurrentThread.Name)

        'ProgressDownload.Value = Convert.ToInt32((downloadedBytes * 100) / totalBytesToDownload)
        ' downloadedBytesTextBox.Text = downloadedBytes.ToString("N", nfi)

    Catch ex As Exception

        Console.WriteLine("Error " + Thread.CurrentThread.Name)
        Console.WriteLine(ex)

    Finally
        Console.WriteLine("Finally " + Thread.CurrentThread.Name)

        GC.Collect()

        ' GC.WaitForPendingFinalizers()
        'newThread.Abort()


        'Try
        'Thread.CurrentThread.Abort()

        'Catch ex1 As Exception

        'End Try
    End Try

    'Return 1

End Sub

This function will updates the progress bars and the other labels on the form :

Private Sub UpdateDownloadProgress(ByVal BytesRead As Int64, ByVal TotalBytes As Int64)
    If Not swDL Is Nothing AndAlso swDL.IsRunning Then
        swDL.Stop()
        If swDL.ElapsedMilliseconds > 0 Then
            resultGrid.Rows.Item(HandoverGridCounter - 1).Cells(4).Value &= "DL: " & swDL.ElapsedMilliseconds & " ms"
        End If
        swDL = Nothing
    End If


    If (dlCurrentspeed > 0) Then
        'txtCurrentSpeedDL.Text = Math.Round((dlCurrentspeed / 1024), 0) & " KB/s"
    End If


    downloadedBytes += BytesRead
    If downloadedBytes >= totalBytesToDownload Then
        downloadedBytes = totalBytesToDownload
    End If

    ProgressDownload.Value = Convert.ToInt32((downloadedBytes * 100) / TotalBytes)
    downloadedBytesTextBox.Text = downloadedBytes.ToString("N", nfi)  ' & " bytes"

    If totalBytesToDownload = 0 Then
        totalBytesToDownload = TotalBytes
        txtBytesToDownload.Text = totalBytesToDownload.ToString("N", nfi)
    End If

    If downloadedBytes >= totalBytesToDownload Then
        dlCurrentspeed = 0
        dlStopWatch.Stop()

        testEndTickDownload = My.Computer.Clock.TickCount
        testDeltaDownload = (testEndTickDownload - testStartTickDownload) / 1000
        If DLtimedOut = True Then
            DownloadCompleted(TestStatus.Cancelled.ToString(), "")
        Else
            DownloadCompleted(TestStatus.Completed.ToString(), "")
        End If
        If currentCycleDownload <= cycleNumericUpDown.Value Then
            If protocolComboBox.Text = "HTTP" Then
                StartHttpDownload()
            Else
                StartFtpDownload()
            End If
        End If
    End If
End Sub

Answer 3

To asynchronously update the UI from a worker thread (whether explicitly created, a work item queued to the thread pool or the callback from an asynchronous API) use Control.BeginInvoke .

This will not block the worker thread while waiting for the UI thread to process the action.

But remember the UI thread cannot assume a specific ordering of asynchronous operations completing (two downloads of equal side started in a particular order are unlikely to complete in that order: too many external factors affect the speed of downloads).

Finally most BeginABC methods require a call to the corresponding EndABC , but (helpfully) Control.BeginInvoke does not.