AWS EMR HadoopJarStepConfig step function arguments not working

Question

I am working on adding step functions to AWS EMR which will add an extra line in "crontab -e". As I don't want to append directly to the crontab file as I might have duplicate cron jobs, I am fetching all the existing jobs, adding the new job and removing the duplicate like so:

HadoopJarStepConfig turnOnCronJob1 = new HadoopJarStepConfig()
                .withJar("command-runner.jar")
                .withArgs("bash", "-c", "\"(crontab -l; echo '*/5 * * * * /mnt/" + scriptName + "')", "|", "sort -u", "|", "crontab -\"");

But, the function is failing with an exit code 1. I have ran this command directly on my EMR cluster as:

hadoop jar /var/lib/aws/emr/step-runner/hadoop-jars/command-runner.jar bash -c "(crontab -l; echo '*/5 * * * * /mnt/reademrpushmetrics.sh') | sort -u | crontab -"

And, this is working fine with no errors. I have tried following HadoopJarStepConfig, but they are not working either:

HadoopJarStepConfig turnOnCronJob2 = new HadoopJarStepConfig()
                .withJar("command-runner.jar")
                .withArgs("(crontab -l; echo '*/5 * * * * /mnt/" + scriptName + "')", "|", "sort -u", "|", "crontab -");
HadoopJarStepConfig copyOldCronTabToNew1 = new HadoopJarStepConfig()
                .withJar("command-runner.jar")
                .withArgs("crontab -l", ">", "/mnt/newCronTab");
HadoopJarStepConfig copyOldCronTabToNew2 = new HadoopJarStepConfig()
                .withJar("command-runner.jar")
                .withArgs("crontab", "-l", ">", "/mnt/newCronTab");
HadoopJarStepConfig copyOldCronTabToNew3 = new HadoopJarStepConfig()
                .withJar("command-runner.jar")
                .withArgs("bash", "-c", "\"crontab -l > /mnt/newCronTab\"");

The last three code config file was an alternative method that I was trying by creating a new "newCronTab" file and storing my new crontab jobs before adding all of them to the "crontab -e". The last config actually gives a different exitCode error 127, and not 1 like others as shown here: All of these command seems to work if I run them directly on EMR cluster with hadoop command. I am 99% sure that the problem is in the arguments that I am passing with not escaping certain characters properly. If anyone has encountered this, how did you fix it? Thank you!

Answer 1

The problem was that in the arguments of HadoopJarStepConfig , you are not supposed to add an extra double quotes. So, the correct format for turnOnCronJob1 and copyOldCronTabToNew3 looks like:

HadoopJarStepConfig turnOnCronJob1 = new HadoopJarStepConfig()
                    .withJar("command-runner.jar")
                    .withArgs("bash", "-c", "(crontab -l; echo '*/5 * * * * /mnt/" + scriptName + "') | sort -u | crontab -");
HadoopJarStepConfig copyOldCronTabToNew3 = new HadoopJarStepConfig()
                    .withJar("command-runner.jar")
                    .withArgs("bash", "-c", "crontab -l > /mnt/newCronTab");

Also, things to note is that turnOnCronJob2 and copyOldCronTabToNew1 config arguments are invalid ways of passing arguments. But, copyOldCronTabToNew2 config arguments are correct.