running spark 2.3 with python 3.x on yarn

Question

I am trying to run example pi.py using spark-submit but I am getting following error,

Python 3.6.5
[GCC 4.3.4 [gcc-4_3-branch revision 152973]] on linux
Type "help", "copyright", "credits" or "license" for more information.
Traceback (most recent call last):
  File "/var/lib/spark/python/pyspark/shell.py", line 31, in <module>
    from pyspark import SparkConf
  File "/var/lib/spark/python/pyspark/__init__.py", line 110, in <module>
    from pyspark.sql import SQLContext, HiveContext, Row
  File "/var/lib/spark/python/pyspark/sql/__init__.py", line 45, in <module>
    from pyspark.sql.types import Row
  File "/var/lib/spark/python/pyspark/sql/types.py", line 27, in <module>
    import ctypes
  File "Python-3.6.5_suse/lib/python3.6/ctypes/__init__.py", line 7, in <module>
    from _ctypes import Union, Structure, Array
ImportError: libffi.so.4: cannot open shared object file: No such file or directory

I am new to python and spark but when I set PYSPARK_PYTHON path in spark-defaults.sh to some older version of python like 3.3.x then it works perfectly fine.

am I setting anything wrong or I do need any other library? This looks like libraries issue.

Thanks!

Answer 1

I found what the problem was! My small yarn cluster has different OS hosts some suse's some centos's and when I set the PYSPARK_PYTHON in the spark-env.sh that configuration was having a central python path so the libraries weren't matching and it was throwing the libffi.so error. So, checking the type of host OS against the lib python path was helpful. Once I set the correct path and run,

./bin/spark-submit --deploy-mode client examples/src/main/python/pi.py

then I could verify the local libraries were set properly. I didn't need to install any additional python libraries such as pyspark or py4j as suggested in comments or other answers.