在 powershell 中運行 N 個並行作業

Question

我有以下 powershell 腳本

$list = invoke-sqlcmd 'exec getOneMillionRows' -Server...
$list | % {
    GetData $_ > $_.txt
    ZipTheFile $_.txt $_.txt.zip
    ...
}

如何以有限的最大作業數量並行運行腳本塊（ { GetDatta $_ > $_.txt....} ），例如一次最多可以生成 8 個文件？

Answer 1

與用戶“Start-Automating”發布的想法相同，但糾正了在他的示例中點擊 else 子句時忘記啟動被阻止的作業的錯誤：

$servers = @('a', 'b', 'c', 'd', 'e', 'f', 'g', 'h', 'i', 'j', 'k', 'l', 'm', 'n')

foreach ($server in $servers) {
    $running = @(Get-Job | Where-Object { $_.State -eq 'Running' })
    if ($running.Count -ge 4) {
        $running | Wait-Job -Any | Out-Null
    }

    Write-Host "Starting job for $server"
    Start-Job {
        # do something with $using:server. Just sleeping for this example.
        Start-Sleep 5
        return "result from $using:server"
    } | Out-Null
}

# Wait for all jobs to complete and results ready to be received
Wait-Job * | Out-Null

# Process the results
foreach($job in Get-Job)
{
    $result = Receive-Job $job
    Write-Host $result
}

Remove-Job -State Completed

Answer 2

Start-Job cmdlet 允許您在后台運行代碼。 要按照您的要求進行操作，類似於下面的代碼應該可以工作。

foreach ($server in $servers) {
    $running = @(Get-Job | Where-Object { $_.State -eq 'Running' })
    if ($running.Count -le 8) {
        Start-Job {
             Add-PSSnapin SQL
             $list = invoke-sqlcmd 'exec getOneMillionRows' -Server...
             ...
        }
    } else {
         $running | Wait-Job
    }
    Get-Job | Receive-Job
}

希望這可以幫助。

Answer 3

使用SplitPipeline模塊的Split-Pipeline cmdlet 應該很容易。 代碼看起來很簡單：

Import-Module SplitPipeline
$list = invoke-sqlcmd 'exec getOneMillionRows' -Server...
$list | Split-Pipeline -Count 8 {process{
    GetData $_ > $_.txt
    ZipTheFile $_.txt $_.txt.zip
    ...
}}

Answer 4

后台工作就是答案。 您還可以使用 [System.Collection.Queue] 限制運行隊列中的作業。 PowerShell 團隊有一篇關於此主題的博客文章： https : //devblogs.microsoft.com/powershell/scaling-and-queuing-powershell-background-jobs/

使用排隊方法可能是限制后台作業的最佳答案。

Answer 5

舊線程，但我認為這可能會有所幫助：

$List = C:\List.txt
$Jobs = 8

Foreach ($PC in Get-Content $List)
{
Do
    {
    $Job = (Get-Job -State Running | measure).count
    } Until ($Job -le $Jobs)

Start-Job -Name $PC -ScriptBlock { "Your command here $Using:PC" }
Get-Job -State Completed | Remove-Job
}

Wait-Job -State Running
Get-Job -State Completed | Remove-Job
Get-Job

當“運行”作業的數量超過允許運行的“$jobs”數量時，“Do”循環暫停“foreach”。 比等待剩余的完成並顯示失敗的作業......

Answer 6

我使用並改進了一個多線程函數，你可以像這樣使用它：

$Script = {
    param($Computername)
    get-process -Computername $Computername
}

@('Srv1','Srv2') | Run-Parallel -ScriptBlock $Script

將此代碼包含在您的腳本中

function Run-Parallel {
    <#
        .Synopsis
            This is a quick and open-ended script multi-threader searcher
            http://www.get-blog.com/?p=189#comment-28834
            Improove by Alban LOPEZ 2016

        .Description
            This script will allow any general, external script to be multithreaded by providing a single
            argument to that script and opening it in a seperate thread.  It works as a filter in the
            pipeline, or as a standalone script.  It will read the argument either from the pipeline
            or from a filename provided.  It will send the results of the child script down the pipeline,
            so it is best to use a script that returns some sort of object.

        .PARAMETER ScriptBlock
            This is where you provide the PowerShell ScriptBlock that you want to multithread.

        .PARAMETER ItemObj
            The ItemObj represents the arguments that are provided to the child script.  This is an open ended
            argument and can take a single object from the pipeline, an array, a collection, or a file name.  The
            multithreading script does it's best to find out which you have provided and handle it as such.
            If you would like to provide a file, then the file is read with one object on each line and will
            be provided as is to the script you are running as a string.  If this is not desired, then use an array.

        .PARAMETER InputParam
            This allows you to specify the parameter for which your input objects are to be evaluated.  As an example,
            if you were to provide a computer name to the Get-Process cmdlet as just an argument, it would attempt to
            find all processes where the name was the provided computername and fail.  You need to specify that the
            parameter that you are providing is the "ComputerName".

        .PARAMETER AddParam
            This allows you to specify additional parameters to the running command.  For instance, if you are trying
            to find the status of the "BITS" service on all servers in your list, you will need to specify the "Name"
            parameter.  This command takes a hash pair formatted as follows:

            @{"key" = "Value"}
            @{"key1" = "Value"; "key2" = 321; "key3" = 1..9}

        .PARAMETER AddSwitch
            This allows you to add additional switches to the command you are running.  For instance, you may want
            to include "RequiredServices" to the "Get-Service" cmdlet.  This parameter will take a single string, or
            an aray of strings as follows:

            "RequiredServices"
            @("RequiredServices", "DependentServices")

        .PARAMETER MaxThreads
            This is the maximum number of threads to run at any given time.  If ressources are too congested try lowering
            this number.  The default value is 20.

        .PARAMETER SleepTimer_ms
            This is the time between cycles of the child process detection cycle.  The default value is 200ms.  If CPU
            utilization is high then you can consider increasing this delay.  If the child script takes a long time to
            run, then you might increase this value to around 1000 (or 1 second in the detection cycle).

        .PARAMETER TimeOutGlobal
            this is the TimeOut in second for listen the last thread, after this timeOut All thread are closed, only each other are returned

        .PARAMETER TimeOutThread
            this is the TimeOut in second for each thread, the thread are aborted at this time

        .PARAMETER PSModules
            List of PSModule name to include for use in ScriptBlock

        .PARAMETER PSSapins
            List of PSSapin name to include for use in ScriptBlock

        .EXAMPLE
            1..20 | Run-Parallel -ScriptBlock {param($i) Start-Sleep $i; "> $i sec <"} -TimeOutGlobal 15 -TimeOutThread 5
        .EXAMPLE
            Both of these will execute the scriptBlock and provide each of the server names in AllServers.txt
            while providing the results to GridView.  The results will be the output of the child script.

            gc AllServers.txt | Run-Parallel $ScriptBlock_GetTSUsers -MaxThreads $findOut_AD.ActiveDirectory.Servers.count -PSModules 'PSTerminalServices' | out-gridview
    #>
    Param(
        [Parameter(ValueFromPipeline=$true,ValueFromPipelineByPropertyName=$true)]
            $ItemObj,
        [ScriptBlock]$ScriptBlock = $null,
        $InputParam = $Null,
        [HashTable] $AddParam = @{},
        [Array] $AddSwitch = @(),
        $MaxThreads = 20,
        $SleepTimer_ms = 100,
        $TimeOutGlobal = 300,
        $TimeOutThread = 100,
        [string[]]$PSSapins = $null,
        [string[]]$PSModules = $null,
        $Modedebug = $true
    )
    Begin{
        $ISS = [system.management.automation.runspaces.initialsessionstate]::CreateDefault()
        ForEach ($Snapin in $PSSapins){
            [void]$ISS.ImportPSSnapIn($Snapin, [ref]$null)
        }
        ForEach ($Module in $PSModules){
            [void]$ISS.ImportPSModule($Module)
        }
        $RunspacePool = [runspacefactory]::CreateRunspacePool(1, $MaxThreads, $ISS, $Host)
        $RunspacePool.CleanupInterval=1000
        $RunspacePool.Open()

        $Jobs = @()
    }
    Process{
        #ForEach ($Object in $ItemObj){
            if ($ItemObj){
                Write-Host $ItemObj -ForegroundColor Yellow
                $PowershellThread = [powershell]::Create().AddScript($ScriptBlock)

                If ($InputParam -ne $Null){
                    $PowershellThread.AddParameter($InputParam, $ItemObj.ToString()) | out-null
                }Else{
                    $PowershellThread.AddArgument($ItemObj.ToString()) | out-null
                }
                ForEach($Key in $AddParam.Keys){
                    $PowershellThread.AddParameter($Key, $AddParam.$key) | out-null
                }
                ForEach($Switch in $AddSwitch){
                    $PowershellThread.AddParameter($Switch) | out-null
                }
                $PowershellThread.RunspacePool = $RunspacePool
                $Handle = $PowershellThread.BeginInvoke()
                $Job =  [pscustomobject][ordered]@{
                    Handle = $Handle
                    Thread = $PowershellThread
                    object = $ItemObj.ToString()
                    Started = Get-Date
                }
                $Jobs += $Job
            }
        #}
    }
    End{
        $GlobalStartTime = Get-Date
        $continue = $true
        While (@($Jobs | Where-Object {$_.Handle -ne $Null}).count -gt 0 -and $continue)  {
            ForEach ($Job in $($Jobs | Where-Object {$_.Handle.IsCompleted -eq $True})){
                $out = $Job.Thread.EndInvoke($Job.Handle)
                $out # return vers la sortie srandard
                #Write-Host $out -ForegroundColor green
                $Job.Thread.Dispose() | Out-Null
                $Job.Thread = $Null
                $Job.Handle = $Null
            }
            foreach ($InProgress in $($Jobs | Where-Object {$_.Handle})) {
                if ($TimeOutGlobal -and (($(Get-Date) - $GlobalStartTime).totalseconds -gt $TimeOutGlobal)){
                    $Continue = $false
                    #Write-Host $InProgress -ForegroundColor magenta
                }
                if (!$Continue -or ($TimeOutThread -and (($(Get-Date) - $InProgress.Started).totalseconds -gt $TimeOutThread))) {
                    $InProgress.thread.Stop() | Out-Null
                    $InProgress.thread.Dispose() | Out-Null
                    $InProgress.Thread = $Null
                    $InProgress.Handle = $Null
                    #Write-Host $InProgress -ForegroundColor red
                }
            }
            Start-Sleep -Milliseconds $SleepTimer_ms
        }
        $RunspacePool.Close() | Out-Null
        $RunspacePool.Dispose() | Out-Null
    }
}

Answer 7

舊線程，但我對它的貢獻，是您計算正在運行的作業的部分。 上面的一些答案不適用於 0 或 1 個正在運行的作業。 我使用的一個小技巧是將結果放入強制數組中，然后對其進行計數：

[array]$JobCount = 獲取作業狀態運行

$JobCount.Count