Apache Spark：啟動PySpark時出錯

Question

在Centos機器上，Python v2.6.6和Apache Spark v1.2.1

嘗試運行./pyspark時出現以下錯誤

似乎python的一些問題，但無法弄清楚

15/06/18 08:11:16 INFO spark.SparkContext: Successfully stopped SparkContext
Traceback (most recent call last):
  File "/usr/lib/spark_1.2.1/spark-1.2.1-bin-hadoop2.4/python/pyspark/shell.py", line 45, in <module>
    sc = SparkContext(appName="PySparkShell", pyFiles=add_files)
  File "/usr/lib/spark_1.2.1/spark-1.2.1-bin-hadoop2.4/python/pyspark/context.py", line 105, in __init__
    conf, jsc)
  File "/usr/lib/spark_1.2.1/spark-1.2.1-bin-hadoop2.4/python/pyspark/context.py", line 157, in _do_init
    self._accumulatorServer = accumulators._start_update_server()
  File "/usr/lib/spark_1.2.1/spark-1.2.1-bin-hadoop2.4/python/pyspark/accumulators.py", line 269, in _start_update_server
    server = AccumulatorServer(("localhost", 0), _UpdateRequestHandler)
  File "/usr/lib64/python2.6/SocketServer.py", line 402, in __init__
    self.server_bind()
  File "/usr/lib64/python2.6/SocketServer.py", line 413, in server_bind
    self.socket.bind(self.server_address)
  File "<string>", line 1, in bind
socket.gaierror: [Errno -2] Name or service not known
>>> 15/06/18 08:11:16 INFO remote.RemoteActorRefProvider$RemotingTerminator: Shutting down remote daemon.
15/06/18 08:11:16 INFO remote.RemoteActorRefProvider$RemotingTerminator: Remote daemon shut down; proceeding with flushing remote transports.

Answer 1

從日志中看起來pyspark無法理解主機localhost 。請檢查您的/ etc / hosts文件，如果localhost不可用，請添加一個應該解決此問題的條目。

例如：

[IP] [主機名] localhost

如果您無法更改服務器的主機條目，請編輯/python/pyspark/accumulators.py第269行，如下所示

server = AccumulatorServer（（“[ 來自hosts文件的服務器主機名 ]”，0），_ UpdatedquestHandler）