如何在 EMR 上运行的 Bash 脚本中“获取”另一个文件

Question

Is there a way when running a step in AWS EMR to 'source' another file in a bash script?

For example, this is a script, let's call it 'run.sh':

#!/bin/sh
source $1
echo $VAR1" "$VAR2
exit0

Then have another file, call it 'parms.sh':

VAR1=this
VAR2=that

Then run from AWS EMR step:

aws emr add-steps \
--cluster-id 123 \
--steps Type=CUSTOM_JAR,\
Name="Script Test",\
ActionOnFailure=CONTINUE,\
Jar=s3://us-east-1.elasticmapreduce/libs/script-runner/script-runner.jar,\
Args=[s3://mybucket/run.sh,"s3://mybucket/parms.sh"]

This error exits as soon as it hits the 'source' statement. I assume it can't 'source' from S3 for some reason?

Answer 1

So sourcing from S3 can't work, which I suppose make sense on some level. Looked a little closer at EMR docs, and it seems the solution is to copy both files to somewhere on the cluster as part of the step then execute:

aws emr add-steps \
--cluster-id 123 \
--steps Type=CUSTOM_JAR,\
Name="Script Test",\
ActionOnFailure=CONTINUE,\
Jar=s3://us-east-1.elasticmapreduce/libs/script-runner/script-runner.jar,\
Args=[/usr/bin/bash,-c,"aws s3 cp s3://mybucket/run.sh /home/hadoop; \
aws s3 cp s3://mybucket/parms.sh /home/hadoop; 
/usr/bin/chmod u+x /home/hadoop/run.sh; \
/usr/bin/chmod u+x /home/hadoop/parms.sh; \
cd /home/hadoop; 
./run.sh ./parms.sh"]