通过 spark-submit 运行应用程序时出现错误 java.lang.NullPointerException

Question

例外

2020-10-31 18:00:40,904 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Exception in thread "main" java.lang.NullPointerException
        at org.apache.hadoop.fs.Path.getName(Path.java:414)
        at org.apache.spark.deploy.DependencyUtils$.downloadFile(DependencyUtils.scala:136)
        at org.apache.spark.deploy.SparkSubmit.$anonfun$prepareSubmitEnvironment$8(SparkSubmit.scala:376)
        at scala.Option.map(Option.scala:230)
        at org.apache.spark.deploy.SparkSubmit.prepareSubmitEnvironment(SparkSubmit.scala:376)
        at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:871)
        at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180)
        at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)
        at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)
        at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1007)
        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1016)
        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

我的 build.sbt 文件

    name := "StockPredictionStructuredStreaming"
    version := "0.1"
    scalaVersion := "2.12.10"
    libraryDependencies += "org.apache.spark" %% "spark-core" % "3.0.0"
    libraryDependencies += "org.apache.spark" %% "spark-sql" % "3.0.0"
    // https://mvnrepository.com/artifact/org.apache.spark/spark-streaming
    libraryDependencies += "org.apache.spark" %% "spark-streaming" % "3.0.0"
    libraryDependencies += "org.apache.spark" %% "spark-mllib" % "3.0.0"
    // https://mvnrepository.com/artifact/org.apache.spark/spark-streaming-kafka-0-10
    libraryDependencies += "org.apache.spark" %% "spark-streaming-kafka-0-10" % "2.4.0"
    // https://mvnrepository.com/artifact/org.apache.spark/spark-sql-kafka-0-10
    libraryDependencies += "org.apache.spark" %% "spark-sql-kafka-0-10" % "3.0.0"

Answer 1

spark-submit --class SparkStructuredStreaming.StockPredictionKafkaStructuredStreaming --packages org.apache.spark:spark-sql-kafka-0-10_2.12:3.0.0,org.apache.spark-streaming-kafka-0-10_2.12:2.4.0 /home/niraj/IdeaProjects/StockPredictionStructuredStreaming/target/scala-2.12/stockpredictionstructuredstreaming_2.12-0.1.jar "localhost:9092" "kafkatutorial" "./SavedOutput" "./pythonFiles/StockPricePrediction.py"