Difference between yarn-client mode and yarn-cluster mode

Question

I am having a little problem while running similar code on the yarn-client mode as well as the yarn-cluster mode. My code executes perfectly when I run it in the client mode, but fails when made to run on the yarn-cluster node.

It throws a file not file exception, stating that pyspark.zip file could not be found. Any insight into this would be helpful.

Answer 1

In yarn-cluster mode, the driver runs in the Application Master (inside a YARN container). In yarn-client mode, it runs in the client.

In yarn-cluster mode, the spark-shell is not supported.

Coming back to your problem: which version of Spark are you using ? In version below 1.4, running pyspark in yarn is currently limited to yarn-client mode (see SPARK-5162 )