在 AWS EC2 上运行三个“连接”脚本

Question

I have 3 scripts: the 1st and the 3rd are written in R, and the 2nd in Python.

The output of the 1st script is the input of the 2nd script, and its output is the input of the 3rd one.

The inputs and outputs are search keywords or phrases.

For example, the output of the 1st script is Hello , then the 2nd turns the word to olleH , and the 3rd one converts the letters to uppercase: OLLEH .

My question is how can I connect those scripts and let them run automatically, without my intervention, on AWS. What will be the commands? How can the output of the 1st script be saved, and play a role as the input of the 2nd one, etc.?

Answer 1

I would start an sh Script (or bat on a Windows machine). Then use the return values for the scripts as input for the next. So something like:

SET var1 = Rscript script1.R
SET var2 = py script2.py $var1
SET var3 = Rscript script3.R $ $var2
echo $var3

Of course you need to change your scripts to using the inputs you submitted.

Answer 2

I have never used AWS so I'm unfamiliar with that, but this seems like a workflow management system would solve these issues. Take a look into snakemake or nextflow. With these tools you can easily (after you get used to it) do exactly what you describe. Run scripts/tools that depend on each other sequentially (and also in parallel).

Answer 3

You can use AWS Step Functions to achieve your goal. For Python parts you can use AWS Lambda tasks, for R parts - AWS ECS tasks, and orchestrate data flow accordingly.

https://docs.aws.amazon.com/step-functions/latest/dg/welcome.html

For commands, I wouldn't count on receiving a comprehensive response - workflows are complex and very individual in each case, but I would recommend defining them via some sort of IaC solution like CloudFormation or AWS CDK and keeping them under git.

https://docs.aws.amazon.com/cdk/api/latest/docs/aws-stepfunctions-readme.html