[英]Playframework Spark provisioning exception on reload
play and spark are both awesome. 游戏和火花都很棒。 However I have some troubles combining them.
但是我把它们结合起来有些麻烦。 Play offers the nice re-compilation mechanism.
Play提供了很好的重新编译机制。 However it is not possible to re-instantiate a spark context.
但是,无法重新实例化火花上下文。
If I had some errors in my code / changed some code and play re-compiles I unfortunately receive the following error: 如果我的代码中有一些错误/更改了一些代码并重新编译,我很遗憾地收到以下错误:
ProvisionException: Unable to provision, see the following errors:
1) Error injecting constructor, org.apache.spark.SparkException: Only one SparkContext may be running in this JVM (see SPARK-2243). To ignore this error, set spark.driver.allowMultipleContexts = true. The currently running SparkContext was created at:
org.apache.spark.SparkContext.<init>(SparkContext.scala:82)
controllers.Application.createSparkContext(Application.scala:38)
controllers.Application.<init>(Application.scala:35)
controllers.Application$$FastClassByGuice$$b5b6aa19.newInstance(<generated>)
One workaround is to manually kill the play application and then re-run it. 一种解决方法是手动终止播放应用程序,然后重新运行它。 But this does not seem to be good.
但这似乎并不好。 Any better ideas?
有更好的想法吗?
I had the same problem. 我有同样的问题。 There is a solution which worked for me:
有一个解决方案对我有用:
Define LocalSparkProvider.scala object: 定义LocalSparkProvider.scala对象:
object LocalSparkProvider {
val sparkContext = new SparkContext(new SparkConf().setAppName("myApplication").setMaster("local"))
val sqlContext = new SQLContext = new SQLContext(sparkContext)
}
Now create Global.scala object under the root package (in Play application it's "app" directory) 现在在根包下创建Global.scala对象(在Play应用程序中它是“app”目录)
import play.api.GlobalSettings
object Global extends GlobalSettings {
override def onStop(app: play.api.Application): Unit = { // Use an explicit definition of the package!
LocalSparkProvider.sparkContext.stop()
}
}
When the application is reloaded then Play triggers onStop method. 重新加载应用程序后,Play触发onStop方法。 Which can be used to stop spark context.
哪个可以用来阻止火花环境。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.