将 Powershell Start-Process 的结果添加到文件中，而不是用 -RedirectStandardOutput 替换它

Question

我在 Powershell 中使用以下命令在后台转换文件，但想将结果全部记录在一个文件中。 现在 -RedirectStandardOutput 每次运行都会替换文件。

foreach ($l in gc ./files.txt) {Start-Process -FilePath "c:\Program Files (x86)\calibre2\ebook-convert.exe" -Argumentlist "'$l' '$l.epub'" -Wait -WindowStyle Hidden -RedirectStandardOutput log.txt}

我尝试了重定向，但日志为空。 如果可能的话，我想把它保持为单线。

foreach ($l in gc ./files.txt) {Start-Process -FilePath "c:\Program Files (x86)\calibre2\ebook-convert.exe" -Argumentlist "`"$l`" `"$l.epub`"" -Wait -WindowStyle Hidden *> log.txt}

Answer 1

如果顺序、同步执行是可接受的，您可以简化命令以使用单个输出重定向（假设ebook-convert.exe是控制台子系统应用程序，因此 PowerShell同步执行（以阻塞方式）。：

Get-Content ./files.txt | ForEach-Object {
  & 'c:\Program Files (x86)\calibre2\ebook-convert.exe' $_ "$_.epub" 
} *> log.txt

^{在>之前放置*告诉 PowerShell 重定向所有输出流，对于外部程序，这意味着 stdout 和 stderr。}

如果要控制字符编码，请使用Out-File - >实际上是 - 的别名，并带有-Encoding参数； 或者，最好使用文本输出 - 外部程序输出始终在 PowerShell - Set-Content中。 要同时捕获stderr输出，请在Out-File / Set-Content调用之前将*>&1附加到管道段中的命令。

请注意，PowerShell 永远不会将来自外部程序的原始输出传递给文件——它们总是首先根据存储在[Console]::OutputEncoding （系统的活动旧版 OEM 代码页）中的编码解码为 .NET 字符串，然后使用文件写入 cmdlet 自己的默认值在保存到文件时重新编码，除非被-Encoding覆盖 - 请参阅此答案以获取更多信息。

---

如果您想要异步、并行执行（例如通过默认异步的Start-Process ），您最好的选择是：

• 写入单独的（临时）文件：

  • 在每次调用中将不同的输出文件传递给-RedirectStandardOutput / -RedirectStandardError 。

  • 请注意，如果您想合并stdout 和 stderr 输出并将其捕获到同一个文件中，您必须通过 shell （可能是另一个 PowerShell 实例）调用您的.exe文件并使用其重定向功能； 对于 PowerShell，它将是*>log.txt ； 对于cmd.exe （如下图），应该是> log.txt 2>&1

• 等待所有启动的进程完成：

  • 将-PassThru传递给Start-Process并收集返回的进程信息对象。

  • 然后使用Wait-Process等待所有进程终止； 根据需要使用-Timeout参数。

• 然后将它们合并到一个日志文件中。

这是一个实现：

$procsAndLogFiles = 
  Get-Content ./files.txt | ForEach-Object -Begin { $i = 0 } {
    # Create a distinct log file for each process,
    # and return its name along with a process-information object representing
    # each process as a custom object.
    $logFile = 'log{0:000}.txt' -f ++$i
    [pscustomobject] @{
      LogFile = $logFile
      Process = Start-Process -PassThru -WindowStyle Hidden `
                  -FilePath 'cmd.exe' `
                  -Argumentlist "/c `"`"c:\Program Files (x86)\calibre2\ebook-convert.exe`" `"$_`" `"$_.epub`" >`"$logFile`" 2>&1`"" 
    }
  }

# Wait for all processes to terminate.
# Add -Timeout and error handling as needed.
$procsAndLogFiles.Process | Wait-Process

# Merge all log files.
Get-Content -LiteralPath $procsAndLogFiles.LogFile > log.txt

# Clean up.
Remove-Item -LiteralPath $procsAndLogFiles.LogFile

---

如果要限制并行执行，以限制一次可以运行多少个后台进程：

# Limit how many background processes may run in parallel at most.
$maxParallelProcesses = 10

# Initialize the log file.
# Use -Force to unconditionally replace an existing file.
New-Item log.txt  

# Initialize the list in which those input files whose conversion
# failed due to timing out are recorded.
$allTimedOutFiles = [System.Collections.Generic.List[string]]::new()

# Process the input files in batches of $maxParallelProcesses
Get-Content -ReadCount $maxParallelProcesses ./files.txt |
  ForEach-Object {

    $i = 0
    $launchInfos = foreach ($file in $_) {
      # Create a distinct log file for each process,
      # and return its name along with the input file name / path, and 
      # a process-information object representing each process, as a custom object.
      $logFile = 'log{0:000}.txt' -f ++$i
      [pscustomobject] @{
        InputFile = $file
        LogFile = $logFile
        Process = Start-Process -PassThru -WindowStyle Hidden `
          -FilePath 'cmd.exe' `
          -ArgumentList "/c `"`"c:\Program Files (x86)\calibre2\ebook-convert.exe`" `"$file`" `"$_.epub`" >`"$file`" 2>&1`"" 
      }
    }

    # Wait for the processes to terminate, with a timeout.
    $launchInfos.Process | Wait-Process -Timeout 30 -ErrorAction SilentlyContinue -ErrorVariable errs

    # If not all processes terminated within the timeout period,
    # forcefully terminate those that didn't.
    if ($errs) {
      $timedOut = $launchInfos | Where-Object { -not $_.Process.HasExited }
      Write-Warning "Conversion of the following input files timed out; the processes will killed:`n$($timedOut.InputFile)"
      $timedOut.Process | Stop-Process -Force
      $allTimedOutFiles.AddRange(@($timedOut.InputFile))
    }

    # Merge all temp. log files and append to the overall log file.
    $tempLogFiles = Get-Content -ErrorAction Ignore -LiteralPath ($launchInfos.LogFile | Sort-Object)
    $tempLogFiles | Get-Content >> log.txt

    # Clean up.
    $tempLogFiles | Remove-Item

  }

# * log.txt now contains all combined logs
# * $allTimedOutFiles now contains all input file names / paths 
#   whose conversion was aborted due to timing out.

请注意，上述限制技术不是最佳的，因为每批输入都一起等待，此时下一批开始。 更好的方法是在可用的并行“插槽”之一启动后立即启动一个新进程，如下一节所示； 但是，请注意，需要PowerShell (Core) 7+ 。

---

PowerShell (Core) 7+：使用ForEach-Object -Parallel有效地限制并行执行：

PowerShell (Core) 7+ 通过-Parallel参数向ForEach-Object cmdlet 引入了基于线程的并行性，该参数具有内置限制，默认情况下最多默认为 5 个线程，但可以通过-ThrottleLimit显式控制范围。

这可以实现有效的节流，因为一旦有可用插槽打开，就会启动一个新线程。

以下是演示该技术的独立示例； 它适用于 Windows 和类 Unix 平台：

• 输入是 9 个整数，转换过程是简单地通过在 1 到 9 之间休眠一个随机秒数来模拟，然后回显输入数字。

• 对每个子进程应用 6 秒的超时，这意味着随机数量的子进程将超时并被杀死。

#requires -Version 7

# Use ForEach-Object -Parallel to launch child processes in parallel,
# limiting the number of parallel threads (from which the child processes are 
# launched) via -ThrottleLimit.
# -AsJob returns a single job whose child jobs track the threads created.
$job = 
 1..9 | ForEach-Object -ThrottleLimit 3 -AsJob -Parallel {
  # Determine a temporary, thread-specific log file name.
  $logFile = 'log_{0:000}.txt' -f $_
  # Pick a radom sleep time that may or may not be smaller than the timeout period.
  $sleepTime = Get-Random -Minimum 1 -Maximum 9
  # Launch the external program asynchronously and save information about
  # the newly launched child process.
  if ($env:OS -eq 'Windows_NT') {
    $ps = Start-Process -PassThru -WindowStyle Hidden cmd.exe "/c `"timeout $sleepTime >NUL & echo $_ >$logFile 2>&1`""
  }
  else { # macOS, Linux
    $ps = Start-Process -PassThru sh "-c `"{ sleep $sleepTime; echo $_; } >$logFile 2>&1`""
  }
  # Wait for the child process to exit within a given timeout period.
  $ps | Wait-Process -Timeout 6 -ErrorAction SilentlyContinue
  # Check if a timout has occurred (implied by the process not having exited yet)
  $timedOut = -not $ps.HasExited
  if ($timedOut) {
    # Note: Only [Console]::WriteLine produces immediate output, directly to the display.
    [Console]::WriteLine("Warning: Conversion timed out for: $_")
    # Kill the timed-out process.
    $ps | Stop-Process -Force
  }
  # Construct and output a custom object that indicates the input at hand,
  # the associated log file, and whether a timeout occurred.
  [pscustomobject] @{
    InputFile = $_
    LogFile = $logFile
    TimedOut = $timedOut
  }
 }

# Wait for all child processes to exit or be killed
$processInfos = $job | Receive-Job -Wait -AutoRemoveJob

# Merge all temporary log files into an overall log file.
$tempLogFiles = Get-Item -ErrorAction Ignore -LiteralPath ($processInfos.LogFile | Sort-Object)
$tempLogFiles | Get-Content > log.txt

# Clean up the temporary log files.
$tempLogFiles | Remove-Item

# To illustrate the results, show the overall log file's content
# and which inputs caused timeouts.
[pscustomobject] @{
  CombinedLogContent = Get-Content -Raw log.txt
  InputsThatFailed = ($processInfos | Where-Object TimedOut).InputFile
} | Format-List

# Clean up the overall log file.
Remove-Item log.txt

Answer 2

如果您不使用Start-Process而是直接调用，则可以使用重定向和附加到文件：

foreach ($l in gc ./files.txt) {& 'C:\Program Files (x86)\calibre2\ebook-convert.exe' "$l" "$l.epub" *>> log.txt}

Answer 3

目前，我正在对 mklement0 的答案进行改编。 ebook-convert.exe 经常挂起，因此如果该过程花费的时间超过指定时间，我需要将其关闭。 这需要异步运行，因为文件数量和处理器时间占用（5 到 25% 取决于转换）。 超时需要针对每个文件，而不是针对整个作业。

$procsAndLogFiles = 
  Get-Content ./files.txt | ForEach-Object -Begin { $i = 0 } {
    # Create a distinct log file for each process,
    # and return its name along with a process-information object representing
    # each process as a custom object.
    $logFile = 'd:\temp\log{0:000}.txt' -f ++$i
    Write-Host "$(Get-Date) $_"
    [pscustomobject] @{
      LogFile = $logFile
      Process = Start-Process `
        -PassThru `
        -FilePath "c:\Program Files (x86)\calibre2\ebook-convert.exe" `
        -Argumentlist "`"$_`" `"$_.epub`"" `
        -WindowStyle Hidden `
        -RedirectStandardOutput $logFile `
        | Wait-Process -Timeout 30
    }
  }

# Wait for all processes to terminate.
# Add -Timeout and error handling as needed.
$procsAndLogFiles.Process

# Merge all log files.
Get-Content -LiteralPath $procsAndLogFiles.LogFile > log.txt

# Clean up.
Remove-Item -LiteralPath $procsAndLogFiles.LogFile

Answer 4

由于我的其他答案中的问题没有完全解决（没有杀死所有花费超过超时限制的进程）我用 Ruby 重写了它。 它不是 powershell，但如果你遇到这个问题并且也知道 Ruby（或不知道）它可以帮助你。 我相信这是解决杀戮问题的线程的使用。

require 'logger'

LOG        = Logger.new("log.txt")
PROGRAM    = 'c:\Program Files (x86)\calibre2\ebook-convert.exe'
LIST       = 'E:\ebooks\english\_convert\mobi\files.txt'
TIMEOUT    = 30
MAXTHREADS = 6

def run file, log: nil
  output = ""
  command  = %Q{"#{PROGRAM}" "#{file}" "#{file}.epub"  2>&1}
  IO.popen(command+" 2>&1") do |io|
    begin
      while (line=io.gets) do
        output += line
        log.info line.chomp if log
      end
    rescue => ex
        log.error ex.message
      system("taskkill /f /pid #{io.pid}") rescue log.error $@
    end
  end
  if File.exist? "#{file}.epub"
    puts "converted   #{file}.epub" 
    File.delete(file)
  else
    puts "error       #{file}" 
  end
  output
end

threads = []

File.readlines(LIST).each do |file|
    file.chomp! # remove line feed
  # some checks
    if !File.exist? file
        puts "not found   #{file}"
        next
    end
    if File.exist? "#{file}.epub"
        puts "skipping    #{file}"
        File.delete(file) if File.exist? file
        next
    end

    # go on with the conversion
    thread = Thread.new {run(file, log: LOG)}
    threads << thread
    next if threads.length < MAXTHREADS
    threads.each do |t|
        t.join(TIMEOUT)
        unless t.alive?
            t.kill
            threads.delete(t)
        end
    end
end