[英]Spark in HA: java.lang.NoSuchMethodError: org.apache.curator.utils.PathUtils.validatePath(Ljava/lang/String;)Ljava/lang/String;
I'm trying to setup HA for Spark 1.5.2 in standalone mode (via zookeeper).我正在尝试在独立模式下(通过 zookeeper)为 Spark 1.5.2 设置 HA。 When starting spark, I'm getting this exception.
启动火花时,我收到此异常。 Looks like an incompatibility issue.
看起来像是不兼容的问题。 However I'm not sure how to proceed.
但是我不确定如何继续。
15/12/14 10:42:27 ERROR ErrorMonitor: Uncaught fatal error from thread [sparkMaster-akka.actor.default-dispatcher-3] shutting down ActorSystem [sparkMaster]
java.lang.NoSuchMethodError: org.apache.curator.utils.PathUtils.validatePath(Ljava/lang/String;)Ljava/lang/String;
at org.apache.curator.framework.recipes.leader.LeaderLatch.<init>(LeaderLatch.java:147)
at org.apache.curator.framework.recipes.leader.LeaderLatch.<init>(LeaderLatch.java:125)
at org.apache.spark.deploy.master.ZooKeeperLeaderElectionAgent.start(ZooKeeperLeaderElectionAgent.scala:39)
at org.apache.spark.deploy.master.ZooKeeperLeaderElectionAgent.<init>(ZooKeeperLeaderElectionAgent.scala:34)
at org.apache.spark.deploy.master.ZooKeeperRecoveryModeFactory.createLeaderElectionAgent(RecoveryModeFactory.scala:74)
at org.apache.spark.deploy.master.Master.onStart(Master.scala:165)
at org.apache.spark.rpc.akka.AkkaRpcEnv$$anonfun$actorRef$lzycompute$1$1$$anon$1$$anonfun$preStart$1.apply$mcV$sp(AkkaRpcEnv.scala:100)
at org.apache.spark.rpc.akka.AkkaRpcEnv.org$apache$spark$rpc$akka$AkkaRpcEnv$$safelyCall(AkkaRpcEnv.scala:197)
at org.apache.spark.rpc.akka.AkkaRpcEnv$$anonfun$actorRef$lzycompute$1$1$$anon$1.preStart(AkkaRpcEnv.scala:99)
at akka.actor.Actor$class.aroundPreStart(Actor.scala:472)
at org.apache.spark.rpc.akka.AkkaRpcEnv$$anonfun$actorRef$lzycompute$1$1$$anon$1.aroundPreStart(AkkaRpcEnv.scala:92)
at akka.actor.ActorCell.create(ActorCell.scala:580)
at akka.actor.ActorCell.invokeAll$1(ActorCell.scala:456)
at akka.actor.ActorCell.systemInvoke(ActorCell.scala:478)
at akka.dispatch.Mailbox.processAllSystemMessages(Mailbox.scala:263)
at akka.dispatch.Mailbox.run(Mailbox.scala:219)
at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:397)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
15/12/14 10:42:27 ERROR ActorSystemImpl: Uncaught fatal error from thread [sparkMaster-akka.actor.default-dispatcher-3] shutting down ActorSystem [sparkMaster]
java.lang.NoSuchMethodError: org.apache.curator.utils.PathUtils.validatePath(Ljava/lang/String;)Ljava/lang/String;
at org.apache.curator.framework.recipes.leader.LeaderLatch.<init>(LeaderLatch.java:147)
at org.apache.curator.framework.recipes.leader.LeaderLatch.<init>(LeaderLatch.java:125)
at org.apache.spark.deploy.master.ZooKeeperLeaderElectionAgent.start(ZooKeeperLeaderElectionAgent.scala:39)
at org.apache.spark.deploy.master.ZooKeeperLeaderElectionAgent.<init>(ZooKeeperLeaderElectionAgent.scala:34)
at org.apache.spark.deploy.master.ZooKeeperRecoveryModeFactory.createLeaderElectionAgent(RecoveryModeFactory.scala:74)
at org.apache.spark.deploy.master.Master.onStart(Master.scala:165)
at org.apache.spark.rpc.akka.AkkaRpcEnv$$anonfun$actorRef$lzycompute$1$1$$anon$1$$anonfun$preStart$1.apply$mcV$sp(AkkaRpcEnv.scala:100)
at org.apache.spark.rpc.akka.AkkaRpcEnv.org$apache$spark$rpc$akka$AkkaRpcEnv$$safelyCall(AkkaRpcEnv.scala:197)
at org.apache.spark.rpc.akka.AkkaRpcEnv$$anonfun$actorRef$lzycompute$1$1$$anon$1.preStart(AkkaRpcEnv.scala:99)
at akka.actor.Actor$class.aroundPreStart(Actor.scala:472)
at org.apache.spark.rpc.akka.AkkaRpcEnv$$anonfun$actorRef$lzycompute$1$1$$anon$1.aroundPreStart(AkkaRpcEnv.scala:92)
at akka.actor.ActorCell.create(ActorCell.scala:580)
at akka.actor.ActorCell.invokeAll$1(ActorCell.scala:456)
at akka.actor.ActorCell.systemInvoke(ActorCell.scala:478)
at akka.dispatch.Mailbox.processAllSystemMessages(Mailbox.scala:263)
at akka.dispatch.Mailbox.run(Mailbox.scala:219)
at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:397)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
Use same curator version as of spark使用与 spark 相同的策展人版本
<dependency>
<groupId>org.apache.curator</groupId>
<artifactId>curator-client</artifactId>
<version>1.5.2</version>
</dependency>
I came across with this error when trying to setup highly available Spark 2.4.7 with Hadoop 3.3.0 .我在尝试使用Hadoop 3.3.0设置高度可用的Spark 2.4.7时遇到了这个错误。 Downgrading Hadoop version to 3.2.2 solved the problem.
将 Hadoop 版本降级到3.2.2解决了这个问题。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.