Powershell limiting BITS transfers to 4 at a time

Question

I'm working on a script to transfer Office 365 ProPlus binaries to 12 separate file shares using BITS. I want to also display progress of these transfers on screen. My issue is, I want to limit the number of simultaneous BITS transfers to no more than 4 at a time. As one job completes, I then want to start the next job in queue and continue to receive progress until all jobs complete.

This is what I have so far. I first start with this function to create all my BITS jobs for all network locations in a suspended state.

function Start-BinaryTransfer
{
[CmdletBinding()]
param
(
    [array]$DistrictsToUpdate
)

$Source = "$BaseSource\$UpdateChannel"

if ("All" -in $DistrictsToUpdate.Split(','))
{
    foreach ($Destination in $ReposList)
    {
        Copy-Item -Path $Source -Destination $($Destination.Location + '\') -Filter { $_.PSisContainer } -Recurse -ErrorAction SilentlyContinue
        Get-ChildItem -Path $Source -Recurse | Where-Object{ $_.PSisContainer } | ForEach-Object {
            $spath = $_.FullName.Remove(0, $Source.Length + 1)
            $BITSJobs += Start-BitsTransfer -Source $Source\$spath\*.* `
                                            -Destination "$($Destination.Location)\$UpdateChannel\$spath" `
                                            -DisplayName "$($Destination.District) File Transfer" `
                                            -Description "Transferring from [$Source] to [$($Destination.Location)\$UpdateChannel]" `
                                            -Suspended
        }
    }
}

Once all jobs have been created, I then try to use this While loop to start 4 jobs at a time, and display progress as I go. Unfortunately, the actual behavior is that it will attempt to start all 12 jobs at once, which then bogs down network resources.

While (Get-BitsTransfer | Where JobState -EQ "Suspended")
{
Get-BitsTransfer | Where JobState -EQ "Suspended" | ForEach-Object {
    for ($JobsCount = 0; $JobsCount -le 4; $JobsCount++)
    {
        if ($JobsCount -lt 4)
        {
            Resume-BitsTransfer -BitsJob $_ -Asynchronous
            Get-BitsTransfer | Where JobState -EQ "Transferring" | ForEach-Object {
                Write-Progress `
                               -Id $([math]::Abs($_.DisplayName.GetHashCode())) `
                               -Activity "$($_.DisplayName)" `
                               -Status "$($_.Description)" `
                               -CurrentOperation "$([math]::Floor($_.BytesTransferred / $_.BytesTotal * 100)) % Complete" `
                               -PercentComplete $([math]::Floor($_.BytesTransferred / $_.BytesTotal * 100))
            }
        }
      }
    }

    if (Get-BitsTransfer | Where JobState -EQ "Transferred")
    {
        Get-BitsTransfer | Where JobState -EQ "Transferred" | Complete- BitsTransfer
        $JobsCount--
    }
}

Answer 1

With the for -loop, you are resetting $JobsCount to 0 every run

for ($JobsCount = 0; $JobsCount -le 4; $JobsCount++)

Something like this should work (with minimal amount of changes to your current code):

$JobsCount = 0
While (Get-BitsTransfer | Where JobState -EQ "Suspended")
{
Get-BitsTransfer | Where JobState -EQ "Suspended" | ForEach-Object {
    # remove for-loop
        if ($JobsCount -lt 4)
        {
            $JobsCount++
            ...

You also have to modify your while -loop, currently it will exit when there're no more jobs with JobState "Suspended" , but you still have to wait for jobs with JobState "Transferring" and complete them when they are "Transferred" . Suggestion: While ((Get-BitsTransfer) -ne $null) or even more simple While (Get-BitsTransfer)

Answer 2

Thanks for suggestion, that did the trick. I plan to further refine my code, but here's what I have so far that currently works.

function Update-JobProgress
{
    [CmdletBinding()]
    param ()

    Get-BitsTransfer | Where-Object { ($_.JobState -EQ "Transferring") -and ($_.DisplayName -Like "*File Transfer") } | ForEach-Object {
        Write-Progress -Id $([math]::Abs($_.DisplayName.GetHashCode())) `
                       -Activity "$($_.DisplayName)" `
                       -Status "$($_.Description)" `
                       -CurrentOperation "$([math]::Floor($_.BytesTransferred / $_.BytesTotal * 100)) % Complete" `
                       #-PercentComplete $([math]::Floor($_.BytesTransferred / $_.BytesTotal * 100))
    }
}


While (Get-BitsTransfer)
{
    $JobsCount = (Get-BitsTransfer | Where-Object { ($_.JobState -ne "Suspended") -and ($_.DisplayName -Like "*File Transfer") }).Count

    Get-BitsTransfer | Where-Object JobState -EQ "Suspended" | ForEach-Object {
        if ($JobsCount -lt 4)
        {
            Resume-BitsTransfer -BitsJob $_ -Asynchronous
            $JobsCount++
        }
    }

    Update-JobProgress

    if (Get-BitsTransfer | Where-Object { ($_.JobState -EQ "Transferred") -and ($_.DisplayName -Like "*File Transfer") })
    {
        Get-BitsTransfer | Where-Object { ($_.JobState -EQ "Transferred") -and ($_.DisplayName -Like "*File Transfer") } | ForEach-Object {
            Write-Progress `
                           -Id $([math]::Abs($_.DisplayName.GetHashCode())) `
                           -Activity "$($_.DisplayName)" `
                           -Completed
            $_ | Complete-BitsTransfer
            $JobsCount--
        }
    }
}

In my testing, I've found inconsistent results with using the -PercentComplete switch to actually display the scrolling progress bars in a console window; the first job that completes tends to linger and causes odd visual glitches. Omitting the switch and forgoing the progress bars seems to resolve the issue.