How to submit a pyspark job by using spark submit?

Question

I am using Spark 2.4.3 Version. is this command enough to submit a job?

spark-submit accum.py /home/karthi/accm.txt

where to submit this command?

Answer 1

Yes, if you want to submit a Spark job with a Python module, you have to run spark-submit module.py .

Spark is a distributed framework so when you submit a job, it means that you 'send' the job in a cluster. But, you can also easily run it in your machine, with the same command (standalone mode).

You can find examples in Spark official documentation: https://spark.apache.org/docs/2.4.3/submitting-applications.html

NOTE : In order to run the spark-submit, you have two choices:

Go to /path/to/spark/bin and run the the spark-submit /path/to/module.py

Or add the followingn in .bashrc and use run-submit anywhere

export SPARK_HOME=/usr/local/spark 
export PATH=$PATH:$SPARK_HOME/bin