I have installed pyspark on macOS using brew but I'm getting the error when I type pyspark in zsh:
Traceback (most recent call last):
File "/opt/homebrew/bin/find_spark_home.py", line 86, in <module>
print(_find_spark_home())
File "/opt/homebrew/bin/find_spark_home.py", line 52, in _find_spark_home
module_home = os.path.dirname(find_spec("pyspark").origin)
AttributeError: 'NoneType' object has no attribute 'origin'
I've tried setting the path inside the pyspark script but then got
/opt//homebrew/Cellar/apache-spark/3.3.1/bin/load-spark-env.sh: line 2: /opt/homebrew/Cellar/apache-spark/3.3.1/libexec/bin/load-spark-env.sh: Permission denied
/opt//homebrew/Cellar/apache-spark/3.3.1/bin/load-spark-env.sh: line 2: exec: /opt/homebrew/Cellar/apache-spark/3.3.1/libexec/bin/load-spark-env.sh: cannot execute: Undefined error: 0
How do I resolve this error?
I first had to locate and copy the apache spark directory to usr/local:
sudo cp -r /opt/homebrew/Cellar/apache-spark /usr/local/Cellar/
I found the spark directory with sudo find /opt/ -name find_spark_home.py
then I set the environment variables:
SPARK_HOME=/usr/local/Cellar/apache-spark/3.3.1/libexec
export PATH=/usr/local/Cellar/apache-spark/3.3.1/bin:$PATH
after that typing pyspark gives:
Python 3.9.6 (default, Oct 18 2022, 12:41:40)
[Clang 14.0.0 (clang-1400.0.29.202)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
23/01/23 13:31:11 WARN Utils: Your hostname, Reggies-MacBook-Pro.local resolves to a loopback address: 127.0.0.1; using 192.168.0.20 instead (on interface en0)
23/01/23 13:31:11 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
23/01/23 13:31:12 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Welcome to
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/__ / .__/\_,_/_/ /_/\_\ version 3.3.1
/_/
Using Python version 3.9.6 (default, Oct 18 2022 12:41:40)
Spark context Web UI available at http://192.168.0.20:4040
Spark context available as 'sc' (master = local[*], app id = local-1674498672860).
SparkSession available as 'spark'.
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.