Running several python scripts in endless loops and in parallel (with PowerShell)

Question

I have seven python scripts that all manipulate some files in my folder system based on some information on an MSSQL server. The code is written in a way that a script should just restart whenever it has finished. Additionally, the scripts should run in parallel. The order does not matter as long as they are all executed every now and again (that is, it would be bad if Script 1 (reading a file) runs endlessly while Script 7 (deleting all the files that are already read) is never executed. However, it wouldn't matter if Script 1 is run several times before Script 7 is run once).

So far, I've found a solution with PowerShell. I have 7 separate PowerShell Scripts (process1.ps1, process2.ps1, ..., process7.ps1) that all look as follows:

while($true)
{
    $i++
    Start-Process -NoNewWindow -Wait python F:\somewhere\something.py
    Start-Sleep -Seconds 10
}

This works if I open 7 different PowerShell consoles and start one .ps1 in each like this:

& "F:\PowerShellScripts\process1.ps1"

However, opening and monitoring seven sessions every time is cumbersome. Is there a way to start all these processes in one go but still ensure that they are parallelized correctly?

Answer 1

You are able to parallelize command in powershell via:

• Jobs
• Runspaces
• Workflows

For easiest use, take jobs (my recommendation for your needs). For best performance, use runspaces. I have not tried workflows yet.

A starter for jobs:

$scriptpaths = "C:\temp\1.py", "C:\temp\2.py"
foreach ($path in $scriptpaths){
    start-job -ScriptBlock {
        Param($path)
        while($true)
        {
            Start-Process -NoNewWindow -Wait python $path
            Start-Sleep -Seconds 10
        }
    } -ArgumentList $path
}

Please do read the documentations though, this code is far from ideal. Also, this code does not synchronize your scripts. If one runs faster than the others, they will desynchronize.