堆栈内存溢出
  简体   繁体   English

使用 Akka HTTP 的 Spark 即服务

[英]Spark as A Service Using Akka HTTP

 原文  2019-07-10 03:12:01  7   2    scala/ apache-spark/ akka/ akka-http

问题描述

I've followed a blog for exposing Spark as a WebService using Akka HTTP.我关注了一个使用 Akka HTTP 将 Spark 公开为 WebService 的博客。

The code works fine when I'm running it independently but fails when I tried to connect it to a Spark Standalone cluster.该代码在我独立运行时运行良好，但在我尝试将其连接到 Spark Standalone 集群时失败。 I did some analysis and many suggested to set host entry and spark-env.sh correctly.我做了一些分析，许多人建议正确设置主机条目和 spark-env.sh。 I have done all those and still, it's not working.我已经完成了所有这些，但仍然无法正常工作。

Note : My Spark Version and the Spark version in my pom is also the same and I can submit a job to the same cluster using "spark-submit".注意：我的 Spark 版本和我的 pom 中的 Spark 版本也是一样的，我可以使用“spark-submit”将作业提交到同一个集群。

Host File主机文件

10.93.125.98 master
10.93.125.97 cluster-node-2
10.93.125.99 cluster-node-3

Spark Slaves File火花奴隶文件

master
cluster-node-2
cluster-node-3

Spark Env File火花环境文件

export SPARK_MASTER_HOST=master

Scala Code斯卡拉代码

object SparkFactory {
  val spark: SparkSession = SparkSession.builder
    .master(AppConfig.sparkMaster)
    .appName(AppConfig.sparkAppName)
    .getOrCreate

  val sc = spark.sparkContext
  val sparkConf = sc.getConf
}

The full code is available in the below GitHub repo完整代码可在以下 GitHub 存储库中找到

https://github.com/spoddutur/spark-as-service-using-embedded-server https://github.com/spoddutur/spark-as-service-using-embedded-server

Getting the below error when I'm trying to connect to the spark cluster.当我尝试连接到 spark 集群时出现以下错误。 Any pointers would be much appreciated.任何指针将不胜感激。

[service-user@10.93.125.98 cloud]$ java -jar spark-with-scala-0.0.1-SNAPSHOT-allinone.jar -m spark://master:7077
Press RETURN to stop...
[INFO] [07/10/2019 15:53:49.205] [WebServer-akka.actor.default-dispatcher-4] [akka.actor.ActorSystemImpl(WebServer)] Server online at http://master:8001/
^C[service-user@10.93.125.98 cloud]$ java -jar spark-with-scala-0.0.1-SNAPSHOT-allinone.jar -m spark://master:7077
Press RETURN to stop...
[INFO] [07/10/2019 15:56:15.096] [WebServer-akka.actor.default-dispatcher-2] [akka.actor.ActorSystemImpl(WebServer)] Server online at http://master:8001/
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
19/07/10 15:56:27 INFO SparkContext: Running Spark version 2.4.3
19/07/10 15:56:28 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
19/07/10 15:56:28 INFO SparkContext: Submitted application: spark-with-scala
19/07/10 15:56:28 INFO SecurityManager: Changing view acls to: service-user
19/07/10 15:56:28 INFO SecurityManager: Changing modify acls to: service-user
19/07/10 15:56:28 INFO SecurityManager: Changing view acls groups to:
19/07/10 15:56:28 INFO SecurityManager: Changing modify acls groups to:
19/07/10 15:56:28 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(service-user); groups with view permissions: Set(); users  with modify permissions: Set(service-user); groups with modify permissions: Set()
19/07/10 15:56:29 INFO Utils: Successfully started service 'sparkDriver' on port 39006.
19/07/10 15:56:29 INFO SparkEnv: Registering MapOutputTracker
19/07/10 15:56:29 INFO SparkEnv: Registering BlockManagerMaster
19/07/10 15:56:29 INFO BlockManagerMasterEndpoint: Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information
19/07/10 15:56:29 INFO BlockManagerMasterEndpoint: BlockManagerMasterEndpoint up
19/07/10 15:56:29 INFO DiskBlockManager: Created local directory at /tmp/blockmgr-f77fe9f4-dbb8-49db-bbf2-9fca39e18e11
19/07/10 15:56:30 INFO MemoryStore: MemoryStore started with capacity 127.2 MB
19/07/10 15:56:30 INFO SparkEnv: Registering OutputCommitCoordinator
19/07/10 15:56:30 INFO Utils: Successfully started service 'SparkUI' on port 4040.
19/07/10 15:56:30 INFO SparkUI: Bound SparkUI to 0.0.0.0, and started at http://master:4040
19/07/10 15:56:30 INFO StandaloneAppClient$ClientEndpoint: Connecting to master spark://master:7077...
19/07/10 15:56:31 INFO TransportClientFactory: Successfully created connection to master/10.93.125.98:7077 after 79 ms (0 ms spent in bootstraps)
[INFO] [07/10/2019 15:56:47.256] [WebServer-akka.actor.default-dispatcher-4] [akka.actor.ActorSystemImpl(WebServer)] Request timeout encountered for request [GET /count Empty]
19/07/10 15:56:50 INFO StandaloneAppClient$ClientEndpoint: Connecting to master spark://master:7077...
19/07/10 15:57:10 INFO StandaloneAppClient$ClientEndpoint: Connecting to master spark://master:7077...
19/07/10 15:57:30 ERROR StandaloneSchedulerBackend: Application has been killed. Reason: All masters are unresponsive! Giving up.
19/07/10 15:57:30 WARN StandaloneSchedulerBackend: Application ID is not initialized yet.
19/07/10 15:57:30 INFO SparkUI: Stopped Spark web UI at http://master:4040
19/07/10 15:57:30 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 40701.
19/07/10 15:57:30 INFO NettyBlockTransferService: Server created on master:40701
19/07/10 15:57:31 INFO BlockManager: Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy
19/07/10 15:57:31 INFO StandaloneSchedulerBackend: Shutting down all executors
19/07/10 15:57:31 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Asking each executor to shut down
19/07/10 15:57:31 WARN StandaloneAppClient$ClientEndpoint: Drop UnregisterApplication(null) because has not yet connected to master
19/07/10 15:57:31 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
19/07/10 15:57:31 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(driver, master, 40701, None)
19/07/10 15:57:31 INFO MemoryStore: MemoryStore cleared
19/07/10 15:57:31 INFO BlockManager: BlockManager stopped
19/07/10 15:57:31 INFO BlockManagerMasterEndpoint: Registering block manager master:40701 with 127.2 MB RAM, BlockManagerId(driver, master, 40701, None)
19/07/10 15:57:31 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver, master, 40701, None)
19/07/10 15:57:31 INFO BlockManager: Initialized BlockManager: BlockManagerId(driver, master, 40701, None)
19/07/10 15:57:31 INFO BlockManagerMaster: BlockManagerMaster stopped
19/07/10 15:57:31 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
19/07/10 15:57:31 ERROR SparkContext: Error initializing SparkContext.
java.lang.IllegalArgumentException: requirement failed: Can only call getServletHandlers on a running MetricsSystem
        at scala.Predef$.require(Predef.scala:281)
        at org.apache.spark.metrics.MetricsSystem.getServletHandlers(MetricsSystem.scala:91)
        at org.apache.spark.SparkContext.<init>(SparkContext.scala:516)
        at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2520)
        at org.apache.spark.sql.SparkSession$Builder.$anonfun$getOrCreate$5(SparkSession.scala:935)
        at org.apache.spark.sql.SparkSession$Builder$$Lambda$650.00000000641966D0.apply(Unknown Source)
        at scala.Option.getOrElse(Option.scala:138)
        at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:926)
        at com.learning.spark.SparkFactory$.<init>(SparkFactory.scala:19)
        at com.learning.spark.SparkFactory$.<clinit>(SparkFactory.scala)
        at com.learning.spark.web.HttpService$.<init>(HttpService.scala:13)
        at com.learning.spark.web.HttpService$.<clinit>(HttpService.scala)
        at com.learning.spark.web.WebServer$.$anonfun$routes$12(WebServer.scala:47)
        at com.learning.spark.web.WebServer$$$Lambda$579.000000006405B250.apply(Unknown Source)
        at akka.http.scaladsl.server.directives.RouteDirectives.$anonfun$complete$1(RouteDirectives.scala:47)
        at akka.http.scaladsl.server.directives.RouteDirectives$$Lambda$580.000000006405B700.apply(Unknown Source)
        at akka.http.scaladsl.server.StandardRoute$$anon$1.apply(StandardRoute.scala:19)
        at akka.http.scaladsl.server.StandardRoute$$anon$1.apply(StandardRoute.scala:19)
        at akka.http.scaladsl.server.directives.BasicDirectives.$anonfun$mapRouteResult$2(BasicDirectives.scala:66)
        at akka.http.scaladsl.server.directives.BasicDirectives$$Lambda$578.000000006405A3C0.apply(Unknown Source)
        at akka.http.scaladsl.server.directives.BasicDirectives.$anonfun$textract$2(BasicDirectives.scala:159)
        at akka.http.scaladsl.server.directives.BasicDirectives$$Lambda$394.00000000ED20F650.apply(Unknown Source)
        at akka.http.scaladsl.server.directives.BasicDirectives.$anonfun$mapRequestContext$2(BasicDirectives.scala:43)
        at akka.http.scaladsl.server.directives.BasicDirectives$$Lambda$577.0000000064059D40.apply(Unknown Source)
        at akka.http.scaladsl.server.directives.BasicDirectives.$anonfun$textract$2(BasicDirectives.scala:159)
        at akka.http.scaladsl.server.directives.BasicDirectives$$Lambda$394.00000000ED20F650.apply(Unknown Source)
        at akka.http.scaladsl.server.RouteConcatenation$RouteWithConcatenation.$anonfun$$tilde$2(RouteConcatenation.scala:47)
        at akka.http.scaladsl.server.RouteConcatenation$RouteWithConcatenation$$Lambda$563.00000000640532B0.apply(Unknown Source)
        at akka.http.scaladsl.util.FastFuture$.strictTransform$1(FastFuture.scala:41)
        at akka.http.scaladsl.util.FastFuture$.transformWith$extension1(FastFuture.scala:45)
        at akka.http.scaladsl.util.FastFuture$.flatMap$extension(FastFuture.scala:26)
        at akka.http.scaladsl.server.RouteConcatenation$RouteWithConcatenation.$anonfun$$tilde$1(RouteConcatenation.scala:44)
        at akka.http.scaladsl.server.RouteConcatenation$RouteWithConcatenation$$Lambda$396.00000000ED210970.apply(Unknown Source)
        at akka.http.scaladsl.server.RouteConcatenation$RouteWithConcatenation.$anonfun$$tilde$1(RouteConcatenation.scala:44)
        at akka.http.scaladsl.server.RouteConcatenation$RouteWithConcatenation$$Lambda$396.00000000ED210970.apply(Unknown Source)
        at akka.http.scaladsl.server.RouteConcatenation$RouteWithConcatenation.$anonfun$$tilde$1(RouteConcatenation.scala:44)
        at akka.http.scaladsl.server.RouteConcatenation$RouteWithConcatenation$$Lambda$396.00000000ED210970.apply(Unknown Source)
        at akka.http.scaladsl.server.directives.BasicDirectives.$anonfun$mapRouteResultWith$2(BasicDirectives.scala:72)
        at akka.http.scaladsl.server.directives.BasicDirectives$$Lambda$558.0000000064050AA0.apply(Unknown Source)
        at akka.http.scaladsl.server.directives.BasicDirectives.$anonfun$textract$2(BasicDirectives.scala:159)
        at akka.http.scaladsl.server.directives.BasicDirectives$$Lambda$394.00000000ED20F650.apply(Unknown Source)
        at akka.http.scaladsl.server.directives.ExecutionDirectives.$anonfun$handleExceptions$2(ExecutionDirectives.scala:32)
        at akka.http.scaladsl.server.directives.ExecutionDirectives$$Lambda$553.000000006404E0B0.apply(Unknown Source)
        at akka.http.scaladsl.server.directives.BasicDirectives.$anonfun$textract$2(BasicDirectives.scala:159)
        at akka.http.scaladsl.server.directives.BasicDirectives$$Lambda$394.00000000ED20F650.apply(Unknown Source)
        at akka.http.scaladsl.server.Route$.$anonfun$asyncHandler$1(Route.scala:86)
        at akka.http.scaladsl.server.Route$$$Lambda$419.00000000ED29C4C0.apply(Unknown Source)
        at akka.stream.impl.fusing.MapAsyncUnordered$$anon$31.onPush(Ops.scala:1375)
        at akka.stream.impl.fusing.GraphInterpreter.processPush(GraphInterpreter.scala:523)
        at akka.stream.impl.fusing.GraphInterpreter.execute(GraphInterpreter.scala:409)
        at akka.stream.impl.fusing.GraphInterpreterShell.runBatch(ActorGraphInterpreter.scala:606)
        at akka.stream.impl.fusing.GraphInterpreterShell$AsyncInput.execute(ActorGraphInterpreter.scala:485)
        at akka.stream.impl.fusing.GraphInterpreterShell.processEvent(ActorGraphInterpreter.scala:581)
        at akka.stream.impl.fusing.ActorGraphInterpreter.akka$stream$impl$fusing$ActorGraphInterpreter$$processEvent(ActorGraphInterpreter.scala:749)
        at akka.stream.impl.fusing.ActorGraphInterpreter$$anonfun$receive$1.applyOrElse(ActorGraphInterpreter.scala:764)
        at akka.actor.Actor.aroundReceive(Actor.scala:539)
        at akka.actor.Actor.aroundReceive$(Actor.scala:537)
        at akka.stream.impl.fusing.ActorGraphInterpreter.aroundReceive(ActorGraphInterpreter.scala:671)
        at akka.actor.ActorCell.receiveMessage(ActorCell.scala:612)
        at akka.actor.ActorCell.invoke(ActorCell.scala:581)
        at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:268)
        at akka.dispatch.Mailbox.run(Mailbox.scala:229)
        at akka.dispatch.Mailbox.exec(Mailbox.scala:241)
        at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
        at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
        at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
        at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
19/07/10 15:57:31 INFO SparkContext: SparkContext already stopped.
19/07/10 15:57:31 INFO SparkContext: Successfully stopped SparkContext
Uncaught error from thread [WebServer-akka.actor.default-dispatcher-5]: null, shutting down JVM since 'akka.jvm-exit-on-fatal-error' is enabled for ActorSystem[WebServer]
java.lang.ExceptionInInitializerError
        at java.lang.J9VMInternals.ensureError(J9VMInternals.java:146)
        at java.lang.J9VMInternals.recordInitializationFailure(J9VMInternals.java:135)
        at com.learning.spark.web.HttpService$.<init>(HttpService.scala:13)
        at com.learning.spark.web.HttpService$.<clinit>(HttpService.scala)
        at com.learning.spark.web.WebServer$.$anonfun$routes$12(WebServer.scala:47)
        at com.learning.spark.web.WebServer$$$Lambda$579.000000006405B250.apply(Unknown Source)
        at akka.http.scaladsl.server.directives.RouteDirectives.$anonfun$complete$1(RouteDirectives.scala:47)
        at akka.http.scaladsl.server.directives.RouteDirectives$$Lambda$580.000000006405B700.apply(Unknown Source)
        at akka.http.scaladsl.server.StandardRoute$$anon$1.apply(StandardRoute.scala:19)
        at akka.http.scaladsl.server.StandardRoute$$anon$1.apply(StandardRoute.scala:19)
        at akka.http.scaladsl.server.directives.BasicDirectives.$anonfun$mapRouteResult$2(BasicDirectives.scala:66)
        at akka.http.scaladsl.server.directives.BasicDirectives$$Lambda$578.000000006405A3C0.apply(Unknown Source)
        at akka.http.scaladsl.server.directives.BasicDirectives.$anonfun$textract$2(BasicDirectives.scala:159)
        at akka.http.scaladsl.server.directives.BasicDirectives$$Lambda$394.00000000ED20F650.apply(Unknown Source)
        at akka.http.scaladsl.server.directives.BasicDirectives.$anonfun$mapRequestContext$2(BasicDirectives.scala:43)
        at akka.http.scaladsl.server.directives.BasicDirectives$$Lambda$577.0000000064059D40.apply(Unknown Source)
        at akka.http.scaladsl.server.directives.BasicDirectives.$anonfun$textract$2(BasicDirectives.scala:159)
        at akka.http.scaladsl.server.directives.BasicDirectives$$Lambda$394.00000000ED20F650.apply(Unknown Source)
        at akka.http.scaladsl.server.RouteConcatenation$RouteWithConcatenation.$anonfun$$tilde$2(RouteConcatenation.scala:47)
        at akka.http.scaladsl.server.RouteConcatenation$RouteWithConcatenation$$Lambda$563.00000000640532B0.apply(Unknown Source)
        at akka.http.scaladsl.util.FastFuture$.strictTransform$1(FastFuture.scala:41)
        at akka.http.scaladsl.util.FastFuture$.transformWith$extension1(FastFuture.scala:45)
        at akka.http.scaladsl.util.FastFuture$.flatMap$extension(FastFuture.scala:26)
        at akka.http.scaladsl.server.RouteConcatenation$RouteWithConcatenation.$anonfun$$tilde$1(RouteConcatenation.scala:44)
        at akka.http.scaladsl.server.RouteConcatenation$RouteWithConcatenation$$Lambda$396.00000000ED210970.apply(Unknown Source)
        at akka.http.scaladsl.server.RouteConcatenation$RouteWithConcatenation.$anonfun$$tilde$1(RouteConcatenation.scala:44)
        at akka.http.scaladsl.server.RouteConcatenation$RouteWithConcatenation$$Lambda$396.00000000ED210970.apply(Unknown Source)
        at akka.http.scaladsl.server.RouteConcatenation$RouteWithConcatenation.$anonfun$$tilde$1(RouteConcatenation.scala:44)
        at akka.http.scaladsl.server.RouteConcatenation$RouteWithConcatenation$$Lambda$396.00000000ED210970.apply(Unknown Source)
        at akka.http.scaladsl.server.directives.BasicDirectives.$anonfun$mapRouteResultWith$2(BasicDirectives.scala:72)
        at akka.http.scaladsl.server.directives.BasicDirectives$$Lambda$558.0000000064050AA0.apply(Unknown Source)
        at akka.http.scaladsl.server.directives.BasicDirectives.$anonfun$textract$2(BasicDirectives.scala:159)
        at akka.http.scaladsl.server.directives.BasicDirectives$$Lambda$394.00000000ED20F650.apply(Unknown Source)
        at akka.http.scaladsl.server.directives.ExecutionDirectives.$anonfun$handleExceptions$2(ExecutionDirectives.scala:32)
        at akka.http.scaladsl.server.directives.ExecutionDirectives$$Lambda$553.000000006404E0B0.apply(Unknown Source)
        at akka.http.scaladsl.server.directives.BasicDirectives.$anonfun$textract$2(BasicDirectives.scala:159)
        at akka.http.scaladsl.server.directives.BasicDirectives$$Lambda$394.00000000ED20F650.apply(Unknown Source)
        at akka.http.scaladsl.server.Route$.$anonfun$asyncHandler$1(Route.scala:86)
        at akka.http.scaladsl.server.Route$$$Lambda$419.00000000ED29C4C0.apply(Unknown Source)
        at akka.stream.impl.fusing.MapAsyncUnordered$$anon$31.onPush(Ops.scala:1375)
        at akka.stream.impl.fusing.GraphInterpreter.processPush(GraphInterpreter.scala:523)
        at akka.stream.impl.fusing.GraphInterpreter.execute(GraphInterpreter.scala:409)
        at akka.stream.impl.fusing.GraphInterpreterShell.runBatch(ActorGraphInterpreter.scala:606)
        at akka.stream.impl.fusing.GraphInterpreterShell$AsyncInput.execute(ActorGraphInterpreter.scala:485)
        at akka.stream.impl.fusing.GraphInterpreterShell.processEvent(ActorGraphInterpreter.scala:581)
        at akka.stream.impl.fusing.ActorGraphInterpreter.akka$stream$impl$fusing$ActorGraphInterpreter$$processEvent(ActorGraphInterpreter.scala:749)
        at akka.stream.impl.fusing.ActorGraphInterpreter$$anonfun$receive$1.applyOrElse(ActorGraphInterpreter.scala:764)
        at akka.actor.Actor.aroundReceive(Actor.scala:539)
        at akka.actor.Actor.aroundReceive$(Actor.scala:537)
        at akka.stream.impl.fusing.ActorGraphInterpreter.aroundReceive(ActorGraphInterpreter.scala:671)
        at akka.actor.ActorCell.receiveMessage(ActorCell.scala:612)
        at akka.actor.ActorCell.invoke(ActorCell.scala:581)
        at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:268)
        at akka.dispatch.Mailbox.run(Mailbox.scala:229)
        at akka.dispatch.Mailbox.exec(Mailbox.scala:241)
        at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
        at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
        at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
        at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
Caused by: java.lang.IllegalArgumentException: requirement failed: Can only call getServletHandlers on a running MetricsSystem
        at scala.Predef$.require(Predef.scala:281)
        at org.apache.spark.metrics.MetricsSystem.getServletHandlers(MetricsSystem.scala:91)
        at org.apache.spark.SparkContext.<init>(SparkContext.scala:516)
        at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2520)
        at org.apache.spark.sql.SparkSession$Builder.$anonfun$getOrCreate$5(SparkSession.scala:935)
        at org.apache.spark.sql.SparkSession$Builder$$Lambda$650.00000000641966D0.apply(Unknown Source)
        at scala.Option.getOrElse(Option.scala:138)
        at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:926)
        at com.learning.spark.SparkFactory$.<init>(SparkFactory.scala:19)
        at com.learning.spark.SparkFactory$.<clinit>(SparkFactory.scala)
        ... 57 more
19/07/10 15:57:31 INFO ShutdownHookManager: Shutdown hook called
[ERROR] [SECURITY][07/10/2019 15:57:31.461] [WebServer-akka.actor.default-dispatcher-5] [akka.actor.ActorSystemImpl(WebServer)] Uncaught error from thread [WebServer-akka.actor.default-dispatcher-5]: null, shutting down JVM since 'akka.jvm-exit-on-fatal-error' is enabled for ActorSystem[WebServer]
java.lang.ExceptionInInitializerError
        at java.lang.J9VMInternals.ensureError(J9VMInternals.java:146)
        at java.lang.J9VMInternals.recordInitializationFailure(J9VMInternals.java:135)
        at com.learning.spark.web.HttpService$.<init>(HttpService.scala:13)
        at com.learning.spark.web.HttpService$.<clinit>(HttpService.scala)
        at com.learning.spark.web.WebServer$.$anonfun$routes$12(WebServer.scala:47)
        at com.learning.spark.web.WebServer$$$Lambda$579.000000006405B250.apply(Unknown Source)
        at akka.http.scaladsl.server.directives.RouteDirectives.$anonfun$complete$1(RouteDirectives.scala:47)
        at akka.http.scaladsl.server.directives.RouteDirectives$$Lambda$580.000000006405B700.apply(Unknown Source)
        at akka.http.scaladsl.server.StandardRoute$$anon$1.apply(StandardRoute.scala:19)
        at akka.http.scaladsl.server.StandardRoute$$anon$1.apply(StandardRoute.scala:19)
        at akka.http.scaladsl.server.directives.BasicDirectives.$anonfun$mapRouteResult$2(BasicDirectives.scala:66)
        at akka.http.scaladsl.server.directives.BasicDirectives$$Lambda$578.000000006405A3C0.apply(Unknown Source)
        at akka.http.scaladsl.server.directives.BasicDirectives.$anonfun$textract$2(BasicDirectives.scala:159)
        at akka.http.scaladsl.server.directives.BasicDirectives$$Lambda$394.00000000ED20F650.apply(Unknown Source)
        at akka.http.scaladsl.server.directives.BasicDirectives.$anonfun$mapRequestContext$2(BasicDirectives.scala:43)
        at akka.http.scaladsl.server.directives.BasicDirectives$$Lambda$577.0000000064059D40.apply(Unknown Source)
        at akka.http.scaladsl.server.directives.BasicDirectives.$anonfun$textract$2(BasicDirectives.scala:159)
        at akka.http.scaladsl.server.directives.BasicDirectives$$Lambda$394.00000000ED20F650.apply(Unknown Source)
        at akka.http.scaladsl.server.RouteConcatenation$RouteWithConcatenation.$anonfun$$tilde$2(RouteConcatenation.scala:47)
        at akka.http.scaladsl.server.RouteConcatenation$RouteWithConcatenation$$Lambda$563.00000000640532B0.apply(Unknown Source)
        at akka.http.scaladsl.util.FastFuture$.strictTransform$1(FastFuture.scala:41)
        at akka.http.scaladsl.util.FastFuture$.transformWith$extension1(FastFuture.scala:45)
        at akka.http.scaladsl.util.FastFuture$.flatMap$extension(FastFuture.scala:26)
        at akka.http.scaladsl.server.RouteConcatenation$RouteWithConcatenation.$anonfun$$tilde$1(RouteConcatenation.scala:44)
        at akka.http.scaladsl.server.RouteConcatenation$RouteWithConcatenation$$Lambda$396.00000000ED210970.apply(Unknown Source)
        at akka.http.scaladsl.server.RouteConcatenation$RouteWithConcatenation.$anonfun$$tilde$1(RouteConcatenation.scala:44)
        at akka.http.scaladsl.server.RouteConcatenation$RouteWithConcatenation$$Lambda$396.00000000ED210970.apply(Unknown Source)
        at akka.http.scaladsl.server.RouteConcatenation$RouteWithConcatenation.$anonfun$$tilde$1(RouteConcatenation.scala:44)
        at akka.http.scaladsl.server.RouteConcatenation$RouteWithConcatenation$$Lambda$396.00000000ED210970.apply(Unknown Source)
        at akka.http.scaladsl.server.directives.BasicDirectives.$anonfun$mapRouteResultWith$2(BasicDirectives.scala:72)
        at akka.http.scaladsl.server.directives.BasicDirectives$$Lambda$558.0000000064050AA0.apply(Unknown Source)
        at akka.http.scaladsl.server.directives.BasicDirectives.$anonfun$textract$2(BasicDirectives.scala:159)
        at akka.http.scaladsl.server.directives.BasicDirectives$$Lambda$394.00000000ED20F650.apply(Unknown Source)
        at akka.http.scaladsl.server.directives.ExecutionDirectives.$anonfun$handleExceptions$2(ExecutionDirectives.scala:32)
        at akka.http.scaladsl.server.directives.ExecutionDirectives$$Lambda$553.000000006404E0B0.apply(Unknown Source)
        at akka.http.scaladsl.server.directives.BasicDirectives.$anonfun$textract$2(BasicDirectives.scala:159)
        at akka.http.scaladsl.server.directives.BasicDirectives$$Lambda$394.00000000ED20F650.apply(Unknown Source)
        at akka.http.scaladsl.server.Route$.$anonfun$asyncHandler$1(Route.scala:86)
        at akka.http.scaladsl.server.Route$$$Lambda$419.00000000ED29C4C0.apply(Unknown Source)
        at akka.stream.impl.fusing.MapAsyncUnordered$$anon$31.onPush(Ops.scala:1375)
        at akka.stream.impl.fusing.GraphInterpreter.processPush(GraphInterpreter.scala:523)
        at akka.stream.impl.fusing.GraphInterpreter.execute(GraphInterpreter.scala:409)
        at akka.stream.impl.fusing.GraphInterpreterShell.runBatch(ActorGraphInterpreter.scala:606)
        at akka.stream.impl.fusing.GraphInterpreterShell$AsyncInput.execute(ActorGraphInterpreter.scala:485)
        at akka.stream.impl.fusing.GraphInterpreterShell.processEvent(ActorGraphInterpreter.scala:581)
        at akka.stream.impl.fusing.ActorGraphInterpreter.akka$stream$impl$fusing$ActorGraphInterpreter$$processEvent(ActorGraphInterpreter.scala:749)
        at akka.stream.impl.fusing.ActorGraphInterpreter$$anonfun$receive$1.applyOrElse(ActorGraphInterpreter.scala:764)
        at akka.actor.Actor.aroundReceive(Actor.scala:539)
        at akka.actor.Actor.aroundReceive$(Actor.scala:537)
        at akka.stream.impl.fusing.ActorGraphInterpreter.aroundReceive(ActorGraphInterpreter.scala:671)
        at akka.actor.ActorCell.receiveMessage(ActorCell.scala:612)
        at akka.actor.ActorCell.invoke(ActorCell.scala:581)
        at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:268)
        at akka.dispatch.Mailbox.run(Mailbox.scala:229)
        at akka.dispatch.Mailbox.exec(Mailbox.scala:241)
        at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
        at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
        at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
        at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
Caused by: java.lang.IllegalArgumentException: requirement failed: Can only call getServletHandlers on a running MetricsSystem
        at scala.Predef$.require(Predef.scala:281)
        at org.apache.spark.metrics.MetricsSystem.getServletHandlers(MetricsSystem.scala:91)
        at org.apache.spark.SparkContext.<init>(SparkContext.scala:516)
        at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2520)
        at org.apache.spark.sql.SparkSession$Builder.$anonfun$getOrCreate$5(SparkSession.scala:935)
        at org.apache.spark.sql.SparkSession$Builder$$Lambda$650.00000000641966D0.apply(Unknown Source)
        at scala.Option.getOrElse(Option.scala:138)
        at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:926)
        at com.learning.spark.SparkFactory$.<init>(SparkFactory.scala:19)
        at com.learning.spark.SparkFactory$.<clinit>(SparkFactory.scala)
        ... 57 more

2 个解决方案

解决方案1
 1 已采纳  2019-07-14 15:09:12

I got the above issue fixed.我解决了上述问题。 Sharing the root cause for others reference and to save time.分享根本原因供他人参考并节省时间。

The root cause is, I am using Spark 2.4.3 and it's internally using Scala 2.11 and the Scala project I'm using is having Scala 2.12.根本原因是，我使用的是 Spark 2.4.3，它在内部使用 Scala 2.11，而我正在使用的 Scala 项目使用的是 Scala 2.12。 This was creating all the problems the whole time.这一直在制造所有问题。

I have downgraded the Scala version in my project to get it to work.我已经在我的项目中降级了 Scala 版本以使其正常工作。

解决方案2
 0  2019-07-10 09:02:06

Your question lacks minimal reproducible example, tail of the stacktrace and even a link to the article you are referring to.您的问题缺少最小的可重复示例、堆栈跟踪的尾部，甚至没有指向您所指文章的链接。

But from what I can see you don't instantiate an instance of SparkContext in your application.但据我所知，您没有在应用程序中实例化SparkContext的实例。

暂无
暂无

声明:本站的技术帖子网页，遵循CC BY-SA 4.0协议，如果您需要转载，请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.

相关问题 使用Akka / Akka Streams / Akka HTTP时冲突的Akka版本 - Conflicting Akka versions when using Akka / Akka Streams / Akka HTTP akka http：Akka溪流与演员建立休息服务 - akka http: Akka streams vs actors to build a rest service 使用Akka HTTP上传文件 - File upload using Akka HTTP 在Akka-http中使用handleWebsocketMessage - using handleWebsocketMessage in akka-http 使用Akka Http进行多个绑定 - Using Akka Http for Multiple Bindings 允许对 AKKA-HTTP 服务的所有请求 - Allow all requests to the AKKA-HTTP service 如何在 Akka Actor 中模拟 HTTP 服务 - How to mock HTTP service within Akka Actor 如何使用Akka HTTP进行身份验证 - How to do authentication using Akka HTTP 使用Akka http通过请求传递数据 - Passing data with request using Akka http 在名称中使用破折号的Akka HTTP参数 - Using Akka HTTP parameters with dashes in their names
相关标签
 
粤ICP备18138465号  © 2020-2024 STACKOOM.COM