Pyspark: How to use --files tag for multiple files while running job on Yarn cluster

Question

I am new to Spark and using python to write jobs using pyspark. I wanted to run my script on a yarn cluster and remove the verbose logging by sending a log4j.properties for setting logging level to WARN using --files tag. I have a local csv file that the script uses and i need to include this as well. How do I use --files tag to include both the files?

I am using the following command:

/opt/spark/bin/spark-submit --master yarn --deploy-mode cluster --num-executors 50 --executor-cores 2 --executor-memory 2G --files /opt/spark/conf/log4j.properties ./list.csv ./read_parquet.py

But I get the following error: Error: Cannot load main class from JAR file:/opt/spark/conf/./list.csv `

Answer 1

You could be remove the "." infront of / for second file...Here i removed this is working.

/opt/spark/bin/spark-submit --master yarn --deploy-mode cluster --num-executors 50 --executor-cores 2 --executor-memory 2G --files /opt/spark/conf/log4j.properties /list.csv  /read_parquet.py

Answer 2

you can send comma separated files in a string like this through file paths :

--files "filepath1,filepath2,filepath3" \\

worked for me!!