![](/img/trans.png)
[英]ImportError: cannot import name 'VarcharType' from 'pyspark.sql.types'
[英]Pyspark - ImportError: cannot import name 'SparkContext' from 'pyspark'
当我使用 python 和 pyspark 开发数据管道时,我面临以下错误。
PS C:\\Users\\folder\\Documents\\folder\\projects\\code\\etl-gd\\src\\jobs\\greater-data> python test.py Traceback(最近一次调用):文件“test.py”,第1行,在 from pyspark.conf import SparkConf File "C:\\Users\\folder\\AppData\\Local\\Programs\\Python\\Python37\\lib\\site-packages\\pyspark__init__.py", line 51, in from pyspark.context import SparkContext File "C :\\Users\\folder\\AppData\\Local\\Programs\\Python\\Python37\\lib\\site-packages\\pyspark\\context.py", line 43, in from pyspark.profiler import ProfilerCollector, BasicProfiler File "C:\\Users\\folder\\ AppData\\Local\\Programs\\Python\\Python37\\lib\\site-packages\\pyspark\\profiler.py”,第 18 行,导入 cProfile 文件“C:\\Users\\folder\\AppData\\Local\\Programs\\Python\\Python37\\lib\\” cProfile.py”,第 10 行,在导入配置文件中作为 _pyprofile 文件“C:\\Users\\folder\\Documents\\folder\\projects\\code\\etl-gd\\src\\jobs\\greater-data\\profile.py”,第 2 行,从 awsglue.context import GlueContext File "C:\\Users\\folder\\Documents\\folder\\projects\\code\\etl-gd\\src\\jobs\\greater-data\\awsg lue__init__.py", line 13, in from .dynamicframe import DynamicFrame File "C:\\Users\\folder\\Documents\\folder\\projects\\code\\etl-gd\\src\\jobs\\greater-data\\awsglue\\dynamicframe.py",第 20 行,从 pyspark.sql.dataframe 导入数据帧文件“C:\\Users\\folder\\AppData\\Local\\Programs\\Python\\Python37\\lib\\site-packages\\pyspark\\sql__init__.py”,第 45 行,来自 pyspark .sql.types import Row File "C:\\Users\\folder\\AppData\\Local\\Programs\\Python\\Python37\\lib\\site-packages\\pyspark\\sql\\types.py", line 36, in from pyspark import SparkContext ImportError:无法从“pyspark”导入名称“SparkContext”(C:\\Users\\folder\\AppData\\Local\\Programs\\Python\\Python37\\lib\\site-packages\\pyspark__init__.py)
该代码非常简单,只是尝试一下:
from pyspark.conf import SparkConf
print("hello world")
Java、spark、python 和 pyspark 正确安装如下:
> PS
> C:\Users\folder\Documents\folder\projects\code\etl-gd\src\jobs\greater-data>
> java -version java version "1.8.0_241" Java(TM) SE Runtime Environment
> (build 1.8.0_241-b07) Java HotSpot(TM) 64-Bit Server VM (build
> 25.241-b07, mixed mode) PS C:\Users\folder\Documents\folder\projects\code\etl-gd\src\jobs\greater-data>
> PS
> C:\Users\folder\Documents\folder\projects\code\etl-gd\src\jobs\greater-data> python --version
> Python 3.7.6
> PS
> C:\Users\folder\Documents\folder\projects\code\etl-gd\src\jobs\greater-data>
> spark-shell --version Welcome to
> ____ __
> / __/__ ___ _____/ /__
> _\ \/ _ \/ _ `/ __/ '_/ /___/ .__/\_,_/_/ /_/\_\ version 2.4.3
> /_/
>
> Using Scala version 2.11.12, Java HotSpot(TM) 64-Bit Server VM,
> 1.8.0_231 Branch heads/v2.4.3 Compiled by user vaviliv on 2019-09-17T17:31:05Z Revision c3e32bf06c35ba2580d46150923abfa795b4446a
> Url https://github.com/apache/spark Type --help for more information.
> PS
> C:\Users\folder\Documents\folder\projects\code\etl-gd\src\jobs\greater-data>
> pyspark --version
> Welcome to
> ____ __
> / __/__ ___ _____/ /__
> _\ \/ _ \/ _ `/ __/ '_/
> /___/ .__/\_,_/_/ /_/\_\ version 2.4.3
> /_/
>
> Using Scala version 2.11.12, Java HotSpot(TM) 64-Bit Server VM, 1.8.0_231
> Branch heads/v2.4.3
> Compiled by user vaviliv on 2019-09-17T17:31:05Z
> Revision c3e32bf06c35ba2580d46150923abfa795b4446a
> Url https://github.com/apache/spark
> Type --help for more information.
预先感谢您的帮助。
我知道了。 我创建了一个单独的虚拟环境,因为我的电脑中有一些版本的 python 和 spark。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.