spark-submit 在退出程序之前不等待 state FINISHED

Question

I am submitting a spark job to our yarn service via spark-submit. From my understanding spark-submit should continue running until there is a state status of FINISHED before moving on. However once submitted through bamboo, spark-submit is exiting and going straight to the wait which then the sql query is going to run. But the sql query shouldnt run until the spark job is 100% finished. Not sure why my spark-submit is not waiting. Any help is appreciated, thanks

nohup spark-submit --name "${APP_NAME}" \
                    --class "${SPARK_CLASS_NAME}" \
                    --files jaas.conf,kafka.properties,distributed.properties,${KEYTAB},pools.xml \
                    --principal ${PRINCIPAL} \
                    --keytab ${KEYTAB_ALT} \
                    --conf "spark.driver.extraJavaOptions=${JVM_ARGS}" \
                    --conf "spark.executor.extraJavaOptions=${JVM_ARGS}" \
                    --conf spark.haplogic.env=${ENV} \
                    --conf spark.scheduler.allocation.file=${POOL_SCHEDULER_FILE} \
                    --conf spark.master=yarn \
                    --conf spark.submit.deployMode=cluster \
                    --conf spark.yarn.submit.waitAppCompletion=true \
                    --conf spark.driver.memory=$(getProperty "spark.driver.memory") \
                    --conf spark.executor.memory=$(getProperty "spark.executor.memory") \
                    --conf spark.executor.instances=$(getProperty "spark.executor.instances") \
                    --conf spark.executor.cores=$(getProperty "spark.executor.cores") \
                    --conf spark.yarn.maxAppAttempts=$(getProperty "spark.yarn.maxAppAttempts") \
                    --conf spark.dynamicAllocation.enabled=$(getProperty "spark.dynamicAllocation.enabled") \
                    --conf spark.yarn.queue=$(getProperty "spark.yarn.queue") \
                    --conf spark.memory.fraction=$(getProperty "spark.memory.fraction") \
                    --conf spark.memory.storageFraction=$(getProperty "spark.memory.storageFraction") \
                    --conf spark.eventLog.enabled=$(getProperty "spark.eventLog.enabled") \
                    --conf spark.serializer=org.apache.spark.serializer.JavaSerializer \
                    --conf spark.acls.enable=true \
                    --conf spark.admin.acls.groups=${USER_GROUPS} \
                    --conf spark.acls.enable.groups=${USER_GROUPS} \
                    --conf spark.ui.view.acls.groups=${USER_GROUPS} \
                    --conf spark.serializer=org.apache.spark.serializer.JavaSerializer \
                    --conf spark.yarn.appMasterEnv.SECRETS_LIB_MASTER_KEY=${SECRETS_LIB_MASTER_KEY} \
                    ${JARFILE_NAME} >> ${LOG_FILE} 2>&1 &
sleep 90

Answer 1

The issue was a bash knowledge issue, & commands runs spark-submit on the background so bamboo will act as if it was complete after the sleep.