使用 spark-shell 安裝 package Graphframes

Question

我正在嘗試使用spark-shell安裝 PySpark package Graphframes ：

pyspark --packages graphframes:graphframes:0.8.1-spark3.0-s_2.12

但是，終端中有這樣的錯誤：

root@hpcc:~# pyspark --packages graphframes:graphframes:0.8.1-spark3.0-s_2.12
Python 3.6.9 (default, Jan 26 2021, 15:33:00) 
[GCC 8.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.apache.spark.unsafe.Platform (file:/root/spark-3.0.2-bin-hadoop3.2/jars/spark-unsafe_2.12-3.0.2.jar) to constructor java.nio.DirectByteBuffer(long,int)
WARNING: Please consider reporting this to the maintainers of org.apache.spark.unsafe.Platform
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
Ivy Default Cache set to: /root/.ivy2/cache
The jars for the packages stored in: /root/.ivy2/jars
:: loading settings :: url = jar:file:/root/spark-3.0.2-bin-hadoop3.2/jars/ivy-2.4.0.jar!/org/apache/ivy/core/settings/ivysettings.xml
graphframes#graphframes added as a dependency
:: resolving dependencies :: org.apache.spark#spark-submit-parent-bb0fc7e9-5af7-4189-98e4-7ac76a8d97a9;1.0
    confs: [default]
:: resolution report :: resolve 2691ms :: artifacts dl 1ms
    :: modules in use:
    ---------------------------------------------------------------------
    |                  |            modules            ||   artifacts   |
    |       conf       | number| search|dwnlded|evicted|| number|dwnlded|
    ---------------------------------------------------------------------
    |      default     |   1   |   0   |   0   |   0   ||   0   |   0   |
    ---------------------------------------------------------------------

:: problems summary ::
:::: WARNINGS
        module not found: graphframes#graphframes;0.8.1-spark3.0-s_2.12

    ==== local-m2-cache: tried

      file:/root/.m2/repository/graphframes/graphframes/0.8.1-spark3.0-s_2.12/graphframes-0.8.1-spark3.0-s_2.12.pom

      -- artifact graphframes#graphframes;0.8.1-spark3.0-s_2.12!graphframes.jar:

      file:/root/.m2/repository/graphframes/graphframes/0.8.1-spark3.0-s_2.12/graphframes-0.8.1-spark3.0-s_2.12.jar

    ==== local-ivy-cache: tried

      /root/.ivy2/local/graphframes/graphframes/0.8.1-spark3.0-s_2.12/ivys/ivy.xml

      -- artifact graphframes#graphframes;0.8.1-spark3.0-s_2.12!graphframes.jar:

      /root/.ivy2/local/graphframes/graphframes/0.8.1-spark3.0-s_2.12/jars/graphframes.jar

    ==== central: tried

      https://repo1.maven.org/maven2/graphframes/graphframes/0.8.1-spark3.0-s_2.12/graphframes-0.8.1-spark3.0-s_2.12.pom

      -- artifact graphframes#graphframes;0.8.1-spark3.0-s_2.12!graphframes.jar:

      https://repo1.maven.org/maven2/graphframes/graphframes/0.8.1-spark3.0-s_2.12/graphframes-0.8.1-spark3.0-s_2.12.jar

    ==== spark-packages: tried

      https://dl.bintray.com/spark-packages/maven/graphframes/graphframes/0.8.1-spark3.0-s_2.12/graphframes-0.8.1-spark3.0-s_2.12.pom

      -- artifact graphframes#graphframes;0.8.1-spark3.0-s_2.12!graphframes.jar:

      https://dl.bintray.com/spark-packages/maven/graphframes/graphframes/0.8.1-spark3.0-s_2.12/graphframes-0.8.1-spark3.0-s_2.12.jar

        ::::::::::::::::::::::::::::::::::::::::::::::

        ::          UNRESOLVED DEPENDENCIES         ::

        ::::::::::::::::::::::::::::::::::::::::::::::

        :: graphframes#graphframes;0.8.1-spark3.0-s_2.12: not found

        ::::::::::::::::::::::::::::::::::::::::::::::



:: USE VERBOSE OR DEBUG MESSAGE LEVEL FOR MORE DETAILS
Exception in thread "main" java.lang.RuntimeException: [unresolved dependency: graphframes#graphframes;0.8.1-spark3.0-s_2.12: not found]
    at org.apache.spark.deploy.SparkSubmitUtils$.resolveMavenCoordinates(SparkSubmit.scala:1389)
    at org.apache.spark.deploy.DependencyUtils$.resolveMavenDependencies(DependencyUtils.scala:54)
    at org.apache.spark.deploy.SparkSubmit.prepareSubmitEnvironment(SparkSubmit.scala:308)
    at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:871)
    at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180)
    at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)
    at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)
    at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1007)
    at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1016)
    at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Traceback (most recent call last):
  File "/root/spark-3.0.2-bin-hadoop3.2/python/pyspark/shell.py", line 38, in <module>
    SparkContext._ensure_initialized()
  File "/root/spark-3.0.2-bin-hadoop3.2/python/pyspark/context.py", line 327, in _ensure_initialized
    SparkContext._gateway = gateway or launch_gateway(conf)
  File "/root/spark-3.0.2-bin-hadoop3.2/python/pyspark/java_gateway.py", line 105, in launch_gateway
    raise Exception("Java gateway process exited before sending its port number")
Exception: Java gateway process exited before sending its port number
>>> quit()
root@hpcc:~# 

我正在使用 Ubuntu OS 18.04.5 LTS

JDK 版本為 11.0.11

Scala 版本為 2.12.13

Spark-shel 版本是 3.0.2

我想知道是什么問題？ 我該如何克服這個問題？

Answer 1

jar 必須從repos.spark-packages.org下載。 不幸的是，在使用pyspark參數時， --packages沒有檢查這個 repo。 如果您的機器有正在運行的 Maven 安裝可用，解決問題的最簡單方法是手動將 jar 下載到本地 Maven 存儲庫：

mvn org.apache.maven.plugins:maven-dependency-plugin:2.1:get
 -Dartifact=graphframes:graphframes:0.8.1-spark3.0-s_2.12 
 -DrepoUrl=https://repos.spark-packages.org

此命令會將 jar（以及所有必需的依賴項，如果有）下載到位於/root/.m2/repository的本地 Maven 存儲庫。 從這個位置pyspark可以拿起 jar。