[英]Method showString([class java.lang.Integer, class java.lang.Integer, class java.lang.Boolean]) does not exist in PySpark
This is the snippet: 这是代码段:
from pyspark import SparkContext
from pyspark.sql.session import SparkSession
sc = SparkContext()
spark = SparkSession(sc)
d = spark.read.format("csv").option("header", True).option("inferSchema", True).load('file.csv')
d.show()
After this runs into the error: 之后遇到错误:
An error occurred while calling o163.showString. Trace:
py4j.Py4JException: Method showString([class java.lang.Integer, class java.lang.Integer, class java.lang.Boolean]) does not exist
All the other methods work well. 所有其他方法都可以正常工作。 Tried researching alot but in vain.
试图进行大量研究但徒劳无功。 Any lead will be highly appreciated
任何线索将不胜感激
This is an indicator of a Spark version mismatch. 这表明Spark版本不匹配。 Before Spark 2.3
show
method took only two arguments: 在Spark 2.3之前,
show
方法仅接受两个参数:
def show(self, n=20, truncate=True):
since 2.3 it takes three arguments: 从2.3开始,它需要三个参数:
def show(self, n=20, truncate=True, vertical=False):
In your case Python client seems to invoke the latter one, while the JVM backend uses the older version. 在您的情况下,Python客户端似乎调用了后者,而JVM后端使用了较旧的版本。
Since SparkContext
initialization undergone significant changes in 2.4, which would cause failure on SparkContext.__init__
, you're likely using: 由于
SparkContext
初始化在2.4中进行了重大更改,这将导致SparkContext.__init__
失败,因此您可能使用:
You can confirm that by checking versions directly from your session, Python: 您可以通过直接从会话中检查版本来确认这一点:Python:
sc.version
vs. JVM: 与JVM:
sc._jsc.version()
Problems like this, are usually a result of misconfigured PYTHONPATH
(either directly, or by using pip
installed PySpark
on top per-existing Spark binaries) or SPARK_HOME
. 这样的问题通常是由于
PYTHONPATH
配置错误(直接导致的,或者是通过在现有的Spark二进制文件的顶部使用pip
安装的PySpark
)或SPARK_HOME
。
On spark-shell console, enter the variable name and see the data type. 在spark-shell控制台上,输入变量名称并查看数据类型。 As an alternative, you can tab twice after variable named.
另外,您也可以在变量名之后两次制表。 and it will show necessary function which could be applied.
它将显示可以应用的必要功能。 Example of a DataFrame object.
DataFrame对象的示例。
res23: org.apache.spark.sql.DataFrame = [order_id: string, book_name: string ... 1 more field]
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.