Spark（纱线）远程应用程序提交bash

Question

I'm trying to deploy spark apps through jenkins. Manually I can do something like :

ssh -T -i ~/.ssh/keys/me.pem hadoop@....compute.amazonaws.com spark-submit --master yarn --deploy-mode cluster --class myclass.myobject s3://mybucket/myjar1.0.jar

The problem I have is that spark-submit doesn't detach once the job is submitted to yarn. I have not found a command line option to detach after deploy. Any ideas on deploying through jenkins?

Answer 1

添加的nohup在你的命令的begening并与结尾。

ssh -T -i ~/.ssh/keys/me.pem hadoop@....compute.amazonaws.com nohup spark-submit --master yarn --deploy-mode cluster --class myclass.myobject s3://mybucket/myjar1.0.jar &

Answer 2

只有带有--conf spark.yarn.submit.waitAppCompletion = false arg的简单ssh命令才能实际工作。