使用linux“ at”功能运行docker命令

Question

I am running a PySpark script that runs a while loop for 2 hours. I'd love to be able to schedule this to run overnight.

I type in these two lines in order to run the code. After ssh into the server I type:

docker exec -it r-hadoop-spark-yarn-spark-master-1-085ae410 bash

This takes me to a working directory in Docker, I then type:

nohup /opt/spark-2.0.2/bin/spark-submit --master spark://hadoop-spark-yarn-spark-master-1:7077 --executor-memory 8G bm_plog_hdfs.py &> SparkPLOG.out &

to run the script (if all goes well).

I have tried inserting these two lines, without the 'bash' into a script and scheduling this with 'at' but it doesn't appear to run.

Answer 1

通过运行以下命令，使用-d尝试分离模式：

  docker exec -d r-hadoop-spark-yarn-spark-master-1-085ae410 bash -c "nohup /opt/spark-2.0.2/bin/spark-submit --master spark://hadoop-spark-yarn-spark-master-1:7077 --executor-memory 8G bm_plog_hdfs.py &> SparkPLOG.out &"