帶有spyder的Anaconda：ImportError：無法導入名稱“ SparkConf”

Question

我在anaconda的pyspark安裝了pyspark（通過使用： conda install -c conda-forge pyspark ），它在這里（我認為）

/Users/myuser/anaconda3/envs/testenv1/lib/python3.6/site-packages/pyspark/python/pyspark

此路徑存在，接下來我啟動spyder ：

(testenv1) ➜ ~ spyder

並且此代碼產生以下錯誤，我認為site-packeges會自動“包含”，還是一個不同的問題？

import os
os.environ['SPARK_HOME'] = "/Users/myuser/anaconda3/envs/testenv1/lib/python3.6/site-packages/pyspark" # Not working but also not sure why I need to add this line at all pyspark appears to be in `site-packages`

from pyspark import SparkConf, SparkContext

conf = SparkConf().setMaster("local").setAppName("WordCount")
sc = SparkContext(conf = conf)

我得到以下錯誤：

runfile('/Users/myuser/dev/projects/python-snippets/pyspark.py', wdir='/Users/myuser/dev/projects/python-snippets')
Traceback (most recent call last):

  File "<ipython-input-1-969f4e596614>", line 1, in <module>
    runfile('/Users/myuser/dev/projects/python-snippets/pyspark.py', wdir='/Users/myuser/dev/projects/python-snippets')

  File "/Users/myuser/anaconda3/envs/testenv1/lib/python3.6/site-packages/spyder/utils/site/sitecustomize.py", line 705, in runfile
    execfile(filename, namespace)

  File "/Users/myuser/anaconda3/envs/testenv1/lib/python3.6/site-packages/spyder/utils/site/sitecustomize.py", line 102, in execfile
    exec(compile(f.read(), filename, 'exec'), namespace)

  File "/Users/myuser/dev/projects/python-snippets/pyspark.py", line 13, in <module>
    from pyspark import SparkConf, SparkContext

  File "/Users/myuser/dev/projects/python-snippets/pyspark.py", line 13, in <module>
    from pyspark import SparkConf, SparkContext

ImportError: cannot import name 'SparkConf'

請注意，我已嘗試將spyder中的python解釋器更新為： /Users/myuser/anaconda3/envs/testenv1/bin/python3.6但我得到了相同的確切錯誤。

Answer 1

python-snippets/pyspark.py您的文件嗎？ 如果是，則不應使用名稱pyspark.py因為它會與原始pyspark軟件包沖突。

請將該文件重命名為其他文件，它應該可以工作