bash 腳本並行執行

Question

我正在嘗試在 bash 腳本中使用parallel ，以驗證 s3 路徑是否存在，並且我正在嘗試通過計算路徑中的對象來驗證多個 s3 路徑。 如果 object 的計數為零，它將繼續到 for 循環中的下一個日期，並行它不會按預期工作。

對於我在for循環中提供的日期范圍，我們實際上在 s3bucket 中沒有這些文件夾，並且在 function checkS3Path中，如果 s3 路徑不存在，我正在創建一個 0KB 文件，但我沒有看到那些 0KB 文件是在之后創建的腳本被執行。 從腳本的 output 中，我看到S3 Path Consists CSV Files, Proceeding to next step folder1:+2019-10-03 ，而不是S3 Path Doesnt Exists folder1:+2019-10-03 。 請參閱下面的 output。

請讓我可能是什么問題。

這是示例代碼。

#!/bin/bash
#set -x
s3Bucket=testbucket
version=v20
Array=(folder1 folder2 folder3)

checkS3Path() {
  fldName=$1
  date=$2
  objectNum=$(aws s3 ls s3://${s3Bucket}/${version}/${fldName}/date=${date}/ | wc -l)
  echo $objectNum
  if [ "$objectNum" -eq  0 ]
  then
    echo "S3 Path Doesnt Exists ${fldName}:${date}" >> /app/${fldName}.log
    touch /home/ubuntu/${fldName}_${date}.txt
    continue
  else
    echo "S3 Path Consists csv Files, Proceeding to next step ${fldName}:${date}"
  fi
}

final() {
  fldName=$1
  date=$2
  checkS3Path $fldName $date
  function2 $fldName $date
  function3 $fldName $date
}

export -f final checkS3Path

for date in 2019-10-{01..03}
do
#  finalstep folder1 $date
  parallel --jobs 4 --eta finalstep ::: "${Array[@]}" ::: +"$date"
done

這是我看到的 output。

$ ./test.sh
Academic tradition requires you to cite works you base your article on.
When using programs that use GNU Parallel to process data for publication
please cite:

  O. Tange (2011): GNU Parallel - The Command-Line Power Tool,
  ;login: The USENIX Magazine, February 2011:42-47.

This helps funding further development; AND IT WON'T COST YOU A CENT.
If you pay 10000 EUR you should feel free to use GNU Parallel without citing.

To silence this citation notice: run 'parallel --citation'.


Computers / CPU cores / Max jobs to run
1:local / 4 / 4

Computer:jobs running/jobs completed/%of started jobs/Average seconds to complete
ETA: 0s Left: 14 AVG: 0.00s  local:4/0/100%/0.0s 202
S3 Path Consists CSV Files, Proceeding to next step folder1:+2019-10-01
ETA: 0s Left: 13 AVG: 0.00s  local:4/1/100%/2.0s 202
S3 Path Consists CSV Files, Proceeding to next step folder2:+2019-10-01
ETA: 0s Left: 12 AVG: 0.00s  local:4/2/100%/1.0s 202
S3 Path Consists CSV Files, Proceeding to next step folder3:+2019-10-01
Academic tradition requires you to cite works you base your article on.
When using programs that use GNU Parallel to process data for publication
please cite:

  O. Tange (2011): GNU Parallel - The Command-Line Power Tool,
  ;login: The USENIX Magazine, February 2011:42-47.

This helps funding further development; AND IT WON'T COST YOU A CENT.
If you pay 10000 EUR you should feel free to use GNU Parallel without citing.

To silence this citation notice: run 'parallel --citation'.


Computers / CPU cores / Max jobs to run
1:local / 4 / 4

Computer:jobs running/jobs completed/%of started jobs/Average seconds to complete
ETA: 0s Left: 14 AVG: 0.00s  local:4/0/100%/0.0s 202
S3 Path Consists CSV Files, Proceeding to next step folder1:+2019-10-02
ETA: 0s Left: 13 AVG: 0.00s  local:4/1/100%/0.0s 202
S3 Path Consists CSV Files, Proceeding to next step folder2:+2019-10-02
ETA: 6s Left: 12 AVG: 0.50s  local:4/2/100%/0.5s 202
S3 Path Consists CSV Files, Proceeding to next step folder3:+2019-10-02
ETA: 3s Left: 11 AVG: 0.33s  local:4/3/100%/0.3s 202
Academic tradition requires you to cite works you base your article on.
When using programs that use GNU Parallel to process data for publication
please cite:

  O. Tange (2011): GNU Parallel - The Command-Line Power Tool,
  ;login: The USENIX Magazine, February 2011:42-47.

This helps funding further development; AND IT WON'T COST YOU A CENT.
If you pay 10000 EUR you should feel free to use GNU Parallel without citing.

To silence this citation notice: run 'parallel --citation'.


Computers / CPU cores / Max jobs to run
1:local / 4 / 4

Computer:jobs running/jobs completed/%of started jobs/Average seconds to complete
ETA: 0s Left: 14 AVG: 0.00s  local:4/0/100%/0.0s 202
S3 Path Consists CSV Files, Proceeding to next step folder1:+2019-10-03
ETA: 0s Left: 13 AVG: 0.00s  local:4/1/100%/1.0s 202
S3 Path Consists CSV Files, Proceeding to next step folder2:+2019-10-03
ETA: 0s Left: 12 AVG: 0.00s  local:4/2/100%/0.5s 202
S3 Path Consists CSV Files, Proceeding to next step folder3:+2019-10-03
ETA: 0s Left: 11 AVG: 0.00s  local:4/3/100%/0.3s 202

$

謝謝

Answer 1

如果checkS3Path在手動運行時有效，那么您可能只需要：

export s3Bucket=testbucket
export version=v20

每個 GNU Parallel 作業在其自己的 shell（從 Perl 開始）中運行，這就是您需要導出變量的原因，如果您希望它們對作業可見。

另請查看env_parallel以自動執行此操作。