How to choose the queue for Spark job using spark-submit?
Question
有没有办法提供参数或设置来选择我想要运行spark_submit作业的队列？
1 answers
solution1
11 ACCPTED 2017-04-21 20:53:11
By using --queue So an example of a spark-submit job would be:-
spark-submit --master yarn --conf spark.executor.memory=48G --conf spark.driver.memory=6G --packages [packages separated by ,] --queue [queue_name] --class [class_name] [jar_file] [arguments]
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.
Related Question
How to submit PySpark job on Kubernetes (minikube) using spark-submit
How to submit spark job from within java program to standalone spark cluster without using spark-submit?
How to get application Id/Job Id of job submitted to Spark cluster using Spark-submit command ?
Spark-submit job performance
set HBase properties for Spark Job using spark-submit
Error while submitting a spark scala job using spark-submit
How much should one choose the number of executors and cores on a spark-submit job?
Adding jars to a Spark Job - spark-submit
how to launch spark-job.py using spark-submit after kafka producer started
How to trigger a spark job without using “spark-submit”? real-time instead of batch
Related Tags