How to install PySpark and Pytest on Mac with Java8

Question

I am trying to set up a PySpark project with Pytest on Mac, in a Pycharm project. Currently I have a PyCharm project with venv environment, using Pyenv Python 3.6 interpreter. However, my problem now is when I try to do the following, I got an error. I saw some posts says Spark runs on Java 8 but my Java version is 11.

So I downloaded Java 8 and in Pycharm I did export JAVA_HOME=/usr/libexec/java_home -v 1.8 , java -version shows 1.8.0, but it still has the same error. What else should I do? Many thanks for your help.

  log4jLogger = spark.sparkContext._jvm.org.apache.log4j
  log = log4jLogger.logManager.getLogger(__name__)
 E       TypeError: 'JavaPackage' object is not callable

Answer 1

To set JAVA_HOME, use:

export JAVA_HOME=$(/usr/libexec/java_home -v 1.8)

(Note the addition of " $(...) ".)