[英]Why does Spark not run on Eclipse?
I have installed pysark2.1 with hadoop2.6 on Eclipse (Eclipse plugins: PyDev) using Python 3.7, JRE 8, JDK 1.8. 我已经使用Python 3.7,JRE 8,JDK 1.8在Eclipse(Eclipse插件:PyDev)上安装了带有hadoop2.6的pysark2.1。
I am trying to run a simple test code: 我正在尝试运行一个简单的测试代码:
from pyspark.sql import SparkSession
spark = SparkSession.builder.getOrCreate()
But I get the following error: 但是我收到以下错误:
Using Spark's default log4j profile: 使用Spark的默认log4j配置文件:
org/apache/spark/log4j-defaults.properties Setting default log level to "WARN". org / apache / spark / log4j-defaults.properties将默认日志级别设置为“ WARN”。 To adjust logging level use sc.setLogLevel(newLevel). 要调整日志记录级别,请使用sc.setLogLevel(newLevel)。 For SparkR, use setLogLevel(newLevel). 对于SparkR,请使用setLogLevel(newLevel)。 18/12/30 17:04:33 ERROR SparkUncaughtExceptionHandler: Uncaught exception in thread Thread[main,5,main] java.util.NoSuchElementException: key not found: _PYSPARK_DRIVER_CALLBACK_HOST 18/12/30 17:04:33错误SparkUncaughtExceptionHandler:线程Thread [main,5,main] java.util.NoSuchElementException中未捕获的异常:找不到键:_PYSPARK_DRIVER_CALLBACK_HOSTat scala.collection.MapLike$class.default(MapLike.scala:228) 在scala.collection.MapLike $ class.default(MapLike.scala:228)
at scala.collection.AbstractMap.default(Map.scala:59) 在scala.collection.AbstractMap.default(Map.scala:59)
at scala.collection.MapLike$class.apply(MapLike.scala:141) 在scala.collection.MapLike $ class.apply(MapLike.scala:141)
at scala.collection.AbstractMap.apply(Map.scala:59) 在scala.collection.AbstractMap.apply(Map.scala:59)
at org.apache.spark.api.python.PythonGatewayServer$$anonfun$main$1.apply$mcV$sp(PythonGatewayServer.scala:50) at org.apache.spark.util.Utils$.tryOrExit(Utils.scala:1228) 在org.apache.spark.api.python.Python.PythonGatewayServer $$ anonfun $ main $ 1.apply $ mcV $ sp(PythonGatewayServer.scala:50)在org.apache.spark.util.Utils $ .tryOrExit(Utils.scala:1228) )
at org.apache.spark.api.python.PythonGatewayServer$.main(PythonGatewayServer.scala:37) at org.apache.spark.api.python.PythonGatewayServer.main(PythonGatewayServer.scala) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 在org.apache.spark.api.python.PythonGatewayServer.main(PythonGatewayServer.scala)在org.apache.spark.api.python.PythonGatewayServer $ .main(PythonGatewayServer.scala:37)在sun.reflect.NativeMethodAccessorImpl.invoke0(本机方法)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) 在sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)在sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)在java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:738) at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:187) at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:212) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:126) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) 在org.apache.spark.deploy.SparkSubmit $ .doRunMain $ 1(SparkSubmit.scala:org.apache.spark.deploy.SparkSubmit $ .org $ apache $ spark $ deploy $ SparkSubmit $$ runMain(SparkSubmit.scala:738) 187)在org.apache.spark.deploy.SparkSubmit $ .main(SparkSubmit.scala:126)在org.apache.spark.deploy.SparkSubmit $ .submit(SparkSubmit.scala:212)在org.apache.spark.deploy .SparkSubmit.main(SparkSubmit.scala)Traceback (most recent call last): 追溯(最近一次通话):
File "C:\\Users\\charfoush\\eclipse-workspace\\sample2\\test2.py", line 7, in 在第7行的文件“ C:\\ Users \\ charfoush \\ eclipse-workspace \\ sample2 \\ test2.py”
spark = SparkSession.builder.getOrCreate()
File "C:\\Users\\charfoush\\AppData\\Local\\Programs\\Python\\Python37-32\\lib\\site-packages\\pyspark\\sql\\session.py", line 173, in getOrCreate getOrCreate中的第173行的文件“ C:\\ Users \\ charfoush \\ AppData \\ Local \\ Programs \\ Python \\ Python37-32 \\ lib \\ site-packages \\ pyspark \\ sql \\ session.py”
sc = SparkContext.getOrCreate(sparkConf)
File "C:\\Users\\charfoush\\AppData\\Local\\Programs\\Python\\Python37-32\\lib\\site-packages\\pyspark\\context.py", line 351, in getOrCreate getOrCreate中的文件“ C:\\ Users \\ charfoush \\ AppData \\ Local \\ Programs \\ Python \\ Python37-32 \\ lib \\ site-packages \\ pyspark \\ context.py”,第351行
SparkContext(conf=conf or SparkConf())
File "C:\\Users\\charfoush\\AppData\\Local\\Programs\\Python\\Python37-32\\lib\\site-packages\\pyspark\\context.py", line 115, in init 文件“C:\\用户\\ charfoush \\应用程序数据\\本地\\程序\\ Python的\\ Python37-32 \\ LIB \\站点包\\ pyspark \\ context.py”,线路115,在初始化
SparkContext._ensure_initialized(self, gateway=gateway, conf=conf)
File "C:\\Users\\charfoush\\AppData\\Local\\Programs\\Python\\Python37-32\\lib\\site-packages\\pyspark\\context.py", line 300, in _ensure_initialized _ensure_initialized中的文件“ C:\\ Users \\ charfoush \\ AppData \\ Local \\ Programs \\ Python \\ Python37-32 \\ lib \\ site-packages \\ pyspark \\ context.py”,第300行
SparkContext._gateway = gateway or launch_gateway(conf)
File "C:\\Users\\charfoush\\AppData\\Local\\Programs\\Python\\Python37-32\\lib\\site-packages\\pyspark\\java_gateway.py", line 93, in launch_gateway 文件“ C:\\ Users \\ charfoush \\ AppData \\ Local \\ Programs \\ Python \\ Python37-32 \\ lib \\ site-packages \\ pyspark \\ java_gateway.py”,第93行,位于launch_gateway中
raise Exception("Java gateway process exited before sending its port number") Exception: Java gateway process exited before sending
its port number 它的端口号
This issue can occur for instance: 例如,可能会发生此问题:
SPARK_HOME
AND PYTHONPATH
environment variables (ensure none of them is targetting an older version) 或者如果您没有正确定义SPARK_HOME
和PYTHONPATH
环境变量(请确保它们均未针对较旧的版本) 使用eclipse neon 3
为我工作https://blog.puneethabm.com/pyspark-dev-set-up-eclipse-windows/ 在此处输入链接描述
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.