堆棧內存溢出
  簡體   English   中英

使用 Akka HTTP 的 Spark 即服務

[英]Spark as A Service Using Akka HTTP

 原文  2019-07-10 03:12:01  5   2    scala/ apache-spark/ akka/ akka-http

問題描述

我關注了一個使用 Akka HTTP 將 Spark 公開為 WebService 的博客。

該代碼在我獨立運行時運行良好，但在我嘗試將其連接到 Spark Standalone 集群時失敗。 我做了一些分析，許多人建議正確設置主機條目和 spark-env.sh。 我已經完成了所有這些，但仍然無法正常工作。

注意：我的 Spark 版本和我的 pom 中的 Spark 版本也是一樣的，我可以使用“spark-submit”將作業提交到同一個集群。

主機文件

10.93.125.98 master
10.93.125.97 cluster-node-2
10.93.125.99 cluster-node-3

火花奴隸文件

master
cluster-node-2
cluster-node-3

火花環境文件

export SPARK_MASTER_HOST=master

斯卡拉代碼

object SparkFactory {
  val spark: SparkSession = SparkSession.builder
    .master(AppConfig.sparkMaster)
    .appName(AppConfig.sparkAppName)
    .getOrCreate

  val sc = spark.sparkContext
  val sparkConf = sc.getConf
}

完整代碼可在以下 GitHub 存儲庫中找到

https://github.com/spoddutur/spark-as-service-using-embedded-server

當我嘗試連接到 spark 集群時出現以下錯誤。 任何指針將不勝感激。

[service-user@10.93.125.98 cloud]$ java -jar spark-with-scala-0.0.1-SNAPSHOT-allinone.jar -m spark://master:7077
Press RETURN to stop...
[INFO] [07/10/2019 15:53:49.205] [WebServer-akka.actor.default-dispatcher-4] [akka.actor.ActorSystemImpl(WebServer)] Server online at http://master:8001/
^C[service-user@10.93.125.98 cloud]$ java -jar spark-with-scala-0.0.1-SNAPSHOT-allinone.jar -m spark://master:7077
Press RETURN to stop...
[INFO] [07/10/2019 15:56:15.096] [WebServer-akka.actor.default-dispatcher-2] [akka.actor.ActorSystemImpl(WebServer)] Server online at http://master:8001/
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
19/07/10 15:56:27 INFO SparkContext: Running Spark version 2.4.3
19/07/10 15:56:28 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
19/07/10 15:56:28 INFO SparkContext: Submitted application: spark-with-scala
19/07/10 15:56:28 INFO SecurityManager: Changing view acls to: service-user
19/07/10 15:56:28 INFO SecurityManager: Changing modify acls to: service-user
19/07/10 15:56:28 INFO SecurityManager: Changing view acls groups to:
19/07/10 15:56:28 INFO SecurityManager: Changing modify acls groups to:
19/07/10 15:56:28 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(service-user); groups with view permissions: Set(); users  with modify permissions: Set(service-user); groups with modify permissions: Set()
19/07/10 15:56:29 INFO Utils: Successfully started service 'sparkDriver' on port 39006.
19/07/10 15:56:29 INFO SparkEnv: Registering MapOutputTracker
19/07/10 15:56:29 INFO SparkEnv: Registering BlockManagerMaster
19/07/10 15:56:29 INFO BlockManagerMasterEndpoint: Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information
19/07/10 15:56:29 INFO BlockManagerMasterEndpoint: BlockManagerMasterEndpoint up
19/07/10 15:56:29 INFO DiskBlockManager: Created local directory at /tmp/blockmgr-f77fe9f4-dbb8-49db-bbf2-9fca39e18e11
19/07/10 15:56:30 INFO MemoryStore: MemoryStore started with capacity 127.2 MB
19/07/10 15:56:30 INFO SparkEnv: Registering OutputCommitCoordinator
19/07/10 15:56:30 INFO Utils: Successfully started service 'SparkUI' on port 4040.
19/07/10 15:56:30 INFO SparkUI: Bound SparkUI to 0.0.0.0, and started at http://master:4040
19/07/10 15:56:30 INFO StandaloneAppClient$ClientEndpoint: Connecting to master spark://master:7077...
19/07/10 15:56:31 INFO TransportClientFactory: Successfully created connection to master/10.93.125.98:7077 after 79 ms (0 ms spent in bootstraps)
[INFO] [07/10/2019 15:56:47.256] [WebServer-akka.actor.default-dispatcher-4] [akka.actor.ActorSystemImpl(WebServer)] Request timeout encountered for request [GET /count Empty]
19/07/10 15:56:50 INFO StandaloneAppClient$ClientEndpoint: Connecting to master spark://master:7077...
19/07/10 15:57:10 INFO StandaloneAppClient$ClientEndpoint: Connecting to master spark://master:7077...
19/07/10 15:57:30 ERROR StandaloneSchedulerBackend: Application has been killed. Reason: All masters are unresponsive! Giving up.
19/07/10 15:57:30 WARN StandaloneSchedulerBackend: Application ID is not initialized yet.
19/07/10 15:57:30 INFO SparkUI: Stopped Spark web UI at http://master:4040
19/07/10 15:57:30 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 40701.
19/07/10 15:57:30 INFO NettyBlockTransferService: Server created on master:40701
19/07/10 15:57:31 INFO BlockManager: Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy
19/07/10 15:57:31 INFO StandaloneSchedulerBackend: Shutting down all executors
19/07/10 15:57:31 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Asking each executor to shut down
19/07/10 15:57:31 WARN StandaloneAppClient$ClientEndpoint: Drop UnregisterApplication(null) because has not yet connected to master
19/07/10 15:57:31 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
19/07/10 15:57:31 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(driver, master, 40701, None)
19/07/10 15:57:31 INFO MemoryStore: MemoryStore cleared
19/07/10 15:57:31 INFO BlockManager: BlockManager stopped
19/07/10 15:57:31 INFO BlockManagerMasterEndpoint: Registering block manager master:40701 with 127.2 MB RAM, BlockManagerId(driver, master, 40701, None)
19/07/10 15:57:31 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver, master, 40701, None)
19/07/10 15:57:31 INFO BlockManager: Initialized BlockManager: BlockManagerId(driver, master, 40701, None)
19/07/10 15:57:31 INFO BlockManagerMaster: BlockManagerMaster stopped
19/07/10 15:57:31 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
19/07/10 15:57:31 ERROR SparkContext: Error initializing SparkContext.
java.lang.IllegalArgumentException: requirement failed: Can only call getServletHandlers on a running MetricsSystem
        at scala.Predef$.require(Predef.scala:281)
        at org.apache.spark.metrics.MetricsSystem.getServletHandlers(MetricsSystem.scala:91)
        at org.apache.spark.SparkContext.<init>(SparkContext.scala:516)
        at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2520)
        at org.apache.spark.sql.SparkSession$Builder.$anonfun$getOrCreate$5(SparkSession.scala:935)
        at org.apache.spark.sql.SparkSession$Builder$$Lambda$650.00000000641966D0.apply(Unknown Source)
        at scala.Option.getOrElse(Option.scala:138)
        at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:926)
        at com.learning.spark.SparkFactory$.<init>(SparkFactory.scala:19)
        at com.learning.spark.SparkFactory$.<clinit>(SparkFactory.scala)
        at com.learning.spark.web.HttpService$.<init>(HttpService.scala:13)
        at com.learning.spark.web.HttpService$.<clinit>(HttpService.scala)
        at com.learning.spark.web.WebServer$.$anonfun$routes$12(WebServer.scala:47)
        at com.learning.spark.web.WebServer$$$Lambda$579.000000006405B250.apply(Unknown Source)
        at akka.http.scaladsl.server.directives.RouteDirectives.$anonfun$complete$1(RouteDirectives.scala:47)
        at akka.http.scaladsl.server.directives.RouteDirectives$$Lambda$580.000000006405B700.apply(Unknown Source)
        at akka.http.scaladsl.server.StandardRoute$$anon$1.apply(StandardRoute.scala:19)
        at akka.http.scaladsl.server.StandardRoute$$anon$1.apply(StandardRoute.scala:19)
        at akka.http.scaladsl.server.directives.BasicDirectives.$anonfun$mapRouteResult$2(BasicDirectives.scala:66)
        at akka.http.scaladsl.server.directives.BasicDirectives$$Lambda$578.000000006405A3C0.apply(Unknown Source)
        at akka.http.scaladsl.server.directives.BasicDirectives.$anonfun$textract$2(BasicDirectives.scala:159)
        at akka.http.scaladsl.server.directives.BasicDirectives$$Lambda$394.00000000ED20F650.apply(Unknown Source)
        at akka.http.scaladsl.server.directives.BasicDirectives.$anonfun$mapRequestContext$2(BasicDirectives.scala:43)
        at akka.http.scaladsl.server.directives.BasicDirectives$$Lambda$577.0000000064059D40.apply(Unknown Source)
        at akka.http.scaladsl.server.directives.BasicDirectives.$anonfun$textract$2(BasicDirectives.scala:159)
        at akka.http.scaladsl.server.directives.BasicDirectives$$Lambda$394.00000000ED20F650.apply(Unknown Source)
        at akka.http.scaladsl.server.RouteConcatenation$RouteWithConcatenation.$anonfun$$tilde$2(RouteConcatenation.scala:47)
        at akka.http.scaladsl.server.RouteConcatenation$RouteWithConcatenation$$Lambda$563.00000000640532B0.apply(Unknown Source)
        at akka.http.scaladsl.util.FastFuture$.strictTransform$1(FastFuture.scala:41)
        at akka.http.scaladsl.util.FastFuture$.transformWith$extension1(FastFuture.scala:45)
        at akka.http.scaladsl.util.FastFuture$.flatMap$extension(FastFuture.scala:26)
        at akka.http.scaladsl.server.RouteConcatenation$RouteWithConcatenation.$anonfun$$tilde$1(RouteConcatenation.scala:44)
        at akka.http.scaladsl.server.RouteConcatenation$RouteWithConcatenation$$Lambda$396.00000000ED210970.apply(Unknown Source)
        at akka.http.scaladsl.server.RouteConcatenation$RouteWithConcatenation.$anonfun$$tilde$1(RouteConcatenation.scala:44)
        at akka.http.scaladsl.server.RouteConcatenation$RouteWithConcatenation$$Lambda$396.00000000ED210970.apply(Unknown Source)
        at akka.http.scaladsl.server.RouteConcatenation$RouteWithConcatenation.$anonfun$$tilde$1(RouteConcatenation.scala:44)
        at akka.http.scaladsl.server.RouteConcatenation$RouteWithConcatenation$$Lambda$396.00000000ED210970.apply(Unknown Source)
        at akka.http.scaladsl.server.directives.BasicDirectives.$anonfun$mapRouteResultWith$2(BasicDirectives.scala:72)
        at akka.http.scaladsl.server.directives.BasicDirectives$$Lambda$558.0000000064050AA0.apply(Unknown Source)
        at akka.http.scaladsl.server.directives.BasicDirectives.$anonfun$textract$2(BasicDirectives.scala:159)
        at akka.http.scaladsl.server.directives.BasicDirectives$$Lambda$394.00000000ED20F650.apply(Unknown Source)
        at akka.http.scaladsl.server.directives.ExecutionDirectives.$anonfun$handleExceptions$2(ExecutionDirectives.scala:32)
        at akka.http.scaladsl.server.directives.ExecutionDirectives$$Lambda$553.000000006404E0B0.apply(Unknown Source)
        at akka.http.scaladsl.server.directives.BasicDirectives.$anonfun$textract$2(BasicDirectives.scala:159)
        at akka.http.scaladsl.server.directives.BasicDirectives$$Lambda$394.00000000ED20F650.apply(Unknown Source)
        at akka.http.scaladsl.server.Route$.$anonfun$asyncHandler$1(Route.scala:86)
        at akka.http.scaladsl.server.Route$$$Lambda$419.00000000ED29C4C0.apply(Unknown Source)
        at akka.stream.impl.fusing.MapAsyncUnordered$$anon$31.onPush(Ops.scala:1375)
        at akka.stream.impl.fusing.GraphInterpreter.processPush(GraphInterpreter.scala:523)
        at akka.stream.impl.fusing.GraphInterpreter.execute(GraphInterpreter.scala:409)
        at akka.stream.impl.fusing.GraphInterpreterShell.runBatch(ActorGraphInterpreter.scala:606)
        at akka.stream.impl.fusing.GraphInterpreterShell$AsyncInput.execute(ActorGraphInterpreter.scala:485)
        at akka.stream.impl.fusing.GraphInterpreterShell.processEvent(ActorGraphInterpreter.scala:581)
        at akka.stream.impl.fusing.ActorGraphInterpreter.akka$stream$impl$fusing$ActorGraphInterpreter$$processEvent(ActorGraphInterpreter.scala:749)
        at akka.stream.impl.fusing.ActorGraphInterpreter$$anonfun$receive$1.applyOrElse(ActorGraphInterpreter.scala:764)
        at akka.actor.Actor.aroundReceive(Actor.scala:539)
        at akka.actor.Actor.aroundReceive$(Actor.scala:537)
        at akka.stream.impl.fusing.ActorGraphInterpreter.aroundReceive(ActorGraphInterpreter.scala:671)
        at akka.actor.ActorCell.receiveMessage(ActorCell.scala:612)
        at akka.actor.ActorCell.invoke(ActorCell.scala:581)
        at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:268)
        at akka.dispatch.Mailbox.run(Mailbox.scala:229)
        at akka.dispatch.Mailbox.exec(Mailbox.scala:241)
        at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
        at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
        at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
        at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
19/07/10 15:57:31 INFO SparkContext: SparkContext already stopped.
19/07/10 15:57:31 INFO SparkContext: Successfully stopped SparkContext
Uncaught error from thread [WebServer-akka.actor.default-dispatcher-5]: null, shutting down JVM since 'akka.jvm-exit-on-fatal-error' is enabled for ActorSystem[WebServer]
java.lang.ExceptionInInitializerError
        at java.lang.J9VMInternals.ensureError(J9VMInternals.java:146)
        at java.lang.J9VMInternals.recordInitializationFailure(J9VMInternals.java:135)
        at com.learning.spark.web.HttpService$.<init>(HttpService.scala:13)
        at com.learning.spark.web.HttpService$.<clinit>(HttpService.scala)
        at com.learning.spark.web.WebServer$.$anonfun$routes$12(WebServer.scala:47)
        at com.learning.spark.web.WebServer$$$Lambda$579.000000006405B250.apply(Unknown Source)
        at akka.http.scaladsl.server.directives.RouteDirectives.$anonfun$complete$1(RouteDirectives.scala:47)
        at akka.http.scaladsl.server.directives.RouteDirectives$$Lambda$580.000000006405B700.apply(Unknown Source)
        at akka.http.scaladsl.server.StandardRoute$$anon$1.apply(StandardRoute.scala:19)
        at akka.http.scaladsl.server.StandardRoute$$anon$1.apply(StandardRoute.scala:19)
        at akka.http.scaladsl.server.directives.BasicDirectives.$anonfun$mapRouteResult$2(BasicDirectives.scala:66)
        at akka.http.scaladsl.server.directives.BasicDirectives$$Lambda$578.000000006405A3C0.apply(Unknown Source)
        at akka.http.scaladsl.server.directives.BasicDirectives.$anonfun$textract$2(BasicDirectives.scala:159)
        at akka.http.scaladsl.server.directives.BasicDirectives$$Lambda$394.00000000ED20F650.apply(Unknown Source)
        at akka.http.scaladsl.server.directives.BasicDirectives.$anonfun$mapRequestContext$2(BasicDirectives.scala:43)
        at akka.http.scaladsl.server.directives.BasicDirectives$$Lambda$577.0000000064059D40.apply(Unknown Source)
        at akka.http.scaladsl.server.directives.BasicDirectives.$anonfun$textract$2(BasicDirectives.scala:159)
        at akka.http.scaladsl.server.directives.BasicDirectives$$Lambda$394.00000000ED20F650.apply(Unknown Source)
        at akka.http.scaladsl.server.RouteConcatenation$RouteWithConcatenation.$anonfun$$tilde$2(RouteConcatenation.scala:47)
        at akka.http.scaladsl.server.RouteConcatenation$RouteWithConcatenation$$Lambda$563.00000000640532B0.apply(Unknown Source)
        at akka.http.scaladsl.util.FastFuture$.strictTransform$1(FastFuture.scala:41)
        at akka.http.scaladsl.util.FastFuture$.transformWith$extension1(FastFuture.scala:45)
        at akka.http.scaladsl.util.FastFuture$.flatMap$extension(FastFuture.scala:26)
        at akka.http.scaladsl.server.RouteConcatenation$RouteWithConcatenation.$anonfun$$tilde$1(RouteConcatenation.scala:44)
        at akka.http.scaladsl.server.RouteConcatenation$RouteWithConcatenation$$Lambda$396.00000000ED210970.apply(Unknown Source)
        at akka.http.scaladsl.server.RouteConcatenation$RouteWithConcatenation.$anonfun$$tilde$1(RouteConcatenation.scala:44)
        at akka.http.scaladsl.server.RouteConcatenation$RouteWithConcatenation$$Lambda$396.00000000ED210970.apply(Unknown Source)
        at akka.http.scaladsl.server.RouteConcatenation$RouteWithConcatenation.$anonfun$$tilde$1(RouteConcatenation.scala:44)
        at akka.http.scaladsl.server.RouteConcatenation$RouteWithConcatenation$$Lambda$396.00000000ED210970.apply(Unknown Source)
        at akka.http.scaladsl.server.directives.BasicDirectives.$anonfun$mapRouteResultWith$2(BasicDirectives.scala:72)
        at akka.http.scaladsl.server.directives.BasicDirectives$$Lambda$558.0000000064050AA0.apply(Unknown Source)
        at akka.http.scaladsl.server.directives.BasicDirectives.$anonfun$textract$2(BasicDirectives.scala:159)
        at akka.http.scaladsl.server.directives.BasicDirectives$$Lambda$394.00000000ED20F650.apply(Unknown Source)
        at akka.http.scaladsl.server.directives.ExecutionDirectives.$anonfun$handleExceptions$2(ExecutionDirectives.scala:32)
        at akka.http.scaladsl.server.directives.ExecutionDirectives$$Lambda$553.000000006404E0B0.apply(Unknown Source)
        at akka.http.scaladsl.server.directives.BasicDirectives.$anonfun$textract$2(BasicDirectives.scala:159)
        at akka.http.scaladsl.server.directives.BasicDirectives$$Lambda$394.00000000ED20F650.apply(Unknown Source)
        at akka.http.scaladsl.server.Route$.$anonfun$asyncHandler$1(Route.scala:86)
        at akka.http.scaladsl.server.Route$$$Lambda$419.00000000ED29C4C0.apply(Unknown Source)
        at akka.stream.impl.fusing.MapAsyncUnordered$$anon$31.onPush(Ops.scala:1375)
        at akka.stream.impl.fusing.GraphInterpreter.processPush(GraphInterpreter.scala:523)
        at akka.stream.impl.fusing.GraphInterpreter.execute(GraphInterpreter.scala:409)
        at akka.stream.impl.fusing.GraphInterpreterShell.runBatch(ActorGraphInterpreter.scala:606)
        at akka.stream.impl.fusing.GraphInterpreterShell$AsyncInput.execute(ActorGraphInterpreter.scala:485)
        at akka.stream.impl.fusing.GraphInterpreterShell.processEvent(ActorGraphInterpreter.scala:581)
        at akka.stream.impl.fusing.ActorGraphInterpreter.akka$stream$impl$fusing$ActorGraphInterpreter$$processEvent(ActorGraphInterpreter.scala:749)
        at akka.stream.impl.fusing.ActorGraphInterpreter$$anonfun$receive$1.applyOrElse(ActorGraphInterpreter.scala:764)
        at akka.actor.Actor.aroundReceive(Actor.scala:539)
        at akka.actor.Actor.aroundReceive$(Actor.scala:537)
        at akka.stream.impl.fusing.ActorGraphInterpreter.aroundReceive(ActorGraphInterpreter.scala:671)
        at akka.actor.ActorCell.receiveMessage(ActorCell.scala:612)
        at akka.actor.ActorCell.invoke(ActorCell.scala:581)
        at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:268)
        at akka.dispatch.Mailbox.run(Mailbox.scala:229)
        at akka.dispatch.Mailbox.exec(Mailbox.scala:241)
        at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
        at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
        at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
        at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
Caused by: java.lang.IllegalArgumentException: requirement failed: Can only call getServletHandlers on a running MetricsSystem
        at scala.Predef$.require(Predef.scala:281)
        at org.apache.spark.metrics.MetricsSystem.getServletHandlers(MetricsSystem.scala:91)
        at org.apache.spark.SparkContext.<init>(SparkContext.scala:516)
        at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2520)
        at org.apache.spark.sql.SparkSession$Builder.$anonfun$getOrCreate$5(SparkSession.scala:935)
        at org.apache.spark.sql.SparkSession$Builder$$Lambda$650.00000000641966D0.apply(Unknown Source)
        at scala.Option.getOrElse(Option.scala:138)
        at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:926)
        at com.learning.spark.SparkFactory$.<init>(SparkFactory.scala:19)
        at com.learning.spark.SparkFactory$.<clinit>(SparkFactory.scala)
        ... 57 more
19/07/10 15:57:31 INFO ShutdownHookManager: Shutdown hook called
[ERROR] [SECURITY][07/10/2019 15:57:31.461] [WebServer-akka.actor.default-dispatcher-5] [akka.actor.ActorSystemImpl(WebServer)] Uncaught error from thread [WebServer-akka.actor.default-dispatcher-5]: null, shutting down JVM since 'akka.jvm-exit-on-fatal-error' is enabled for ActorSystem[WebServer]
java.lang.ExceptionInInitializerError
        at java.lang.J9VMInternals.ensureError(J9VMInternals.java:146)
        at java.lang.J9VMInternals.recordInitializationFailure(J9VMInternals.java:135)
        at com.learning.spark.web.HttpService$.<init>(HttpService.scala:13)
        at com.learning.spark.web.HttpService$.<clinit>(HttpService.scala)
        at com.learning.spark.web.WebServer$.$anonfun$routes$12(WebServer.scala:47)
        at com.learning.spark.web.WebServer$$$Lambda$579.000000006405B250.apply(Unknown Source)
        at akka.http.scaladsl.server.directives.RouteDirectives.$anonfun$complete$1(RouteDirectives.scala:47)
        at akka.http.scaladsl.server.directives.RouteDirectives$$Lambda$580.000000006405B700.apply(Unknown Source)
        at akka.http.scaladsl.server.StandardRoute$$anon$1.apply(StandardRoute.scala:19)
        at akka.http.scaladsl.server.StandardRoute$$anon$1.apply(StandardRoute.scala:19)
        at akka.http.scaladsl.server.directives.BasicDirectives.$anonfun$mapRouteResult$2(BasicDirectives.scala:66)
        at akka.http.scaladsl.server.directives.BasicDirectives$$Lambda$578.000000006405A3C0.apply(Unknown Source)
        at akka.http.scaladsl.server.directives.BasicDirectives.$anonfun$textract$2(BasicDirectives.scala:159)
        at akka.http.scaladsl.server.directives.BasicDirectives$$Lambda$394.00000000ED20F650.apply(Unknown Source)
        at akka.http.scaladsl.server.directives.BasicDirectives.$anonfun$mapRequestContext$2(BasicDirectives.scala:43)
        at akka.http.scaladsl.server.directives.BasicDirectives$$Lambda$577.0000000064059D40.apply(Unknown Source)
        at akka.http.scaladsl.server.directives.BasicDirectives.$anonfun$textract$2(BasicDirectives.scala:159)
        at akka.http.scaladsl.server.directives.BasicDirectives$$Lambda$394.00000000ED20F650.apply(Unknown Source)
        at akka.http.scaladsl.server.RouteConcatenation$RouteWithConcatenation.$anonfun$$tilde$2(RouteConcatenation.scala:47)
        at akka.http.scaladsl.server.RouteConcatenation$RouteWithConcatenation$$Lambda$563.00000000640532B0.apply(Unknown Source)
        at akka.http.scaladsl.util.FastFuture$.strictTransform$1(FastFuture.scala:41)
        at akka.http.scaladsl.util.FastFuture$.transformWith$extension1(FastFuture.scala:45)
        at akka.http.scaladsl.util.FastFuture$.flatMap$extension(FastFuture.scala:26)
        at akka.http.scaladsl.server.RouteConcatenation$RouteWithConcatenation.$anonfun$$tilde$1(RouteConcatenation.scala:44)
        at akka.http.scaladsl.server.RouteConcatenation$RouteWithConcatenation$$Lambda$396.00000000ED210970.apply(Unknown Source)
        at akka.http.scaladsl.server.RouteConcatenation$RouteWithConcatenation.$anonfun$$tilde$1(RouteConcatenation.scala:44)
        at akka.http.scaladsl.server.RouteConcatenation$RouteWithConcatenation$$Lambda$396.00000000ED210970.apply(Unknown Source)
        at akka.http.scaladsl.server.RouteConcatenation$RouteWithConcatenation.$anonfun$$tilde$1(RouteConcatenation.scala:44)
        at akka.http.scaladsl.server.RouteConcatenation$RouteWithConcatenation$$Lambda$396.00000000ED210970.apply(Unknown Source)
        at akka.http.scaladsl.server.directives.BasicDirectives.$anonfun$mapRouteResultWith$2(BasicDirectives.scala:72)
        at akka.http.scaladsl.server.directives.BasicDirectives$$Lambda$558.0000000064050AA0.apply(Unknown Source)
        at akka.http.scaladsl.server.directives.BasicDirectives.$anonfun$textract$2(BasicDirectives.scala:159)
        at akka.http.scaladsl.server.directives.BasicDirectives$$Lambda$394.00000000ED20F650.apply(Unknown Source)
        at akka.http.scaladsl.server.directives.ExecutionDirectives.$anonfun$handleExceptions$2(ExecutionDirectives.scala:32)
        at akka.http.scaladsl.server.directives.ExecutionDirectives$$Lambda$553.000000006404E0B0.apply(Unknown Source)
        at akka.http.scaladsl.server.directives.BasicDirectives.$anonfun$textract$2(BasicDirectives.scala:159)
        at akka.http.scaladsl.server.directives.BasicDirectives$$Lambda$394.00000000ED20F650.apply(Unknown Source)
        at akka.http.scaladsl.server.Route$.$anonfun$asyncHandler$1(Route.scala:86)
        at akka.http.scaladsl.server.Route$$$Lambda$419.00000000ED29C4C0.apply(Unknown Source)
        at akka.stream.impl.fusing.MapAsyncUnordered$$anon$31.onPush(Ops.scala:1375)
        at akka.stream.impl.fusing.GraphInterpreter.processPush(GraphInterpreter.scala:523)
        at akka.stream.impl.fusing.GraphInterpreter.execute(GraphInterpreter.scala:409)
        at akka.stream.impl.fusing.GraphInterpreterShell.runBatch(ActorGraphInterpreter.scala:606)
        at akka.stream.impl.fusing.GraphInterpreterShell$AsyncInput.execute(ActorGraphInterpreter.scala:485)
        at akka.stream.impl.fusing.GraphInterpreterShell.processEvent(ActorGraphInterpreter.scala:581)
        at akka.stream.impl.fusing.ActorGraphInterpreter.akka$stream$impl$fusing$ActorGraphInterpreter$$processEvent(ActorGraphInterpreter.scala:749)
        at akka.stream.impl.fusing.ActorGraphInterpreter$$anonfun$receive$1.applyOrElse(ActorGraphInterpreter.scala:764)
        at akka.actor.Actor.aroundReceive(Actor.scala:539)
        at akka.actor.Actor.aroundReceive$(Actor.scala:537)
        at akka.stream.impl.fusing.ActorGraphInterpreter.aroundReceive(ActorGraphInterpreter.scala:671)
        at akka.actor.ActorCell.receiveMessage(ActorCell.scala:612)
        at akka.actor.ActorCell.invoke(ActorCell.scala:581)
        at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:268)
        at akka.dispatch.Mailbox.run(Mailbox.scala:229)
        at akka.dispatch.Mailbox.exec(Mailbox.scala:241)
        at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
        at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
        at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
        at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
Caused by: java.lang.IllegalArgumentException: requirement failed: Can only call getServletHandlers on a running MetricsSystem
        at scala.Predef$.require(Predef.scala:281)
        at org.apache.spark.metrics.MetricsSystem.getServletHandlers(MetricsSystem.scala:91)
        at org.apache.spark.SparkContext.<init>(SparkContext.scala:516)
        at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2520)
        at org.apache.spark.sql.SparkSession$Builder.$anonfun$getOrCreate$5(SparkSession.scala:935)
        at org.apache.spark.sql.SparkSession$Builder$$Lambda$650.00000000641966D0.apply(Unknown Source)
        at scala.Option.getOrElse(Option.scala:138)
        at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:926)
        at com.learning.spark.SparkFactory$.<init>(SparkFactory.scala:19)
        at com.learning.spark.SparkFactory$.<clinit>(SparkFactory.scala)
        ... 57 more

2 個解決方案

解決方案1
 1 已采納  2019-07-14 15:09:12

我解決了上述問題。 分享根本原因供他人參考並節省時間。

根本原因是，我使用的是 Spark 2.4.3，它在內部使用 Scala 2.11，而我正在使用的 Scala 項目使用的是 Scala 2.12。 這一直在制造所有問題。

我已經在我的項目中降級了 Scala 版本以使其正常工作。

解決方案2
 0  2019-07-10 09:02:06

您的問題缺少最小的可重復示例、堆棧跟蹤的尾部，甚至沒有指向您所指文章的鏈接。

但據我所知，您沒有在應用程序中實例化SparkContext的實例。

暫無
暫無

聲明:本站的技術帖子網頁，遵循CC BY-SA 4.0協議，如果您需要轉載，請注明本站網址或者原文地址。任何問題請咨詢:yoyou2525@163.com.

相關問題 使用Akka / Akka Streams / Akka HTTP時沖突的Akka版本 akka http：Akka溪流與演員建立休息服務 使用Akka HTTP上傳文件 在Akka-http中使用handleWebsocketMessage 使用Akka Http進行多個綁定 允許對 AKKA-HTTP 服務的所有請求 如何在 Akka Actor 中模擬 HTTP 服務 如何使用Akka HTTP進行身份驗證 使用Akka http通過請求傳遞數據 在名稱中使用破折號的Akka HTTP參數
相關標簽
 
粵ICP備18138465號  © 2020-2024 STACKOOM.COM