[英]spark-shell “error: type mismatch” using :paste to define class/companion object
Using spark-shell v1.6. 使用spark-shell v1.6。
Big differences when I load a class def and its companion object depending on how it's done. 当我加载类def及其伴随对象时,取决于它是如何完成的。
:paste
- PROBLEM :paste
- 问题 First things first... since the repl finds it ambiguous to load a companion object with the same name, I give it an altered name. 首先要做的事情......因为repl发现加载具有相同名称的伴随对象是不明确的,所以我给它一个更改的名称。 No problem there.
没问题。 Setup lookes like this...
设置看起来像这样......
import scala.util.Try
class Foo5(val i: Int)
object Foo5Companion {
def apply(i: Int): Foo5 = new Foo5(i)
def apply(d: Double): Foo5 = new Foo5(d.toInt)
def applyTry(i: Int): Try[Foo5] = Try { apply(i) }
def applyTry(d: Double): Try[Foo5] = Try { apply(d) }
}
Now let's do something simple with this class. 现在让我们用这个类做一些简单的事情。
val ls_i: List[Int] = List(1,2,3)
val ls_foo: List[Foo5] = ls_i.map(Foo5Companion.apply)
If I've loaded the class def and companion object with :paste
I get this error... to be clear, Foo5 has only been defined once in a new session. 如果我用以下方法加载了类def和companion对象
:paste
我得到了这个错误......要清楚,Foo5只在新会话中定义了一次。 This is not an instance of the issue described here: "error: type mismatch" in Spark with same found and required datatypes 这不是此处描述的问题的实例:Spark中的“错误:类型不匹配”具有相同的已找到和必需的数据类型
<console>:42: error: type mismatch;
found : List[Foo5]
required: List[Foo5]
val ls_foo: List[Foo5] = ls_i.map(Foo5Companion.apply)
BUT... 但...
if i load the same defs in line-by-line (without using :paste
)... it works fine... 如果我逐行加载相同的defs(不使用
:paste
)...它工作正常...
ls_foo: List[Foo5] = List($iwC$$iwC$Foo5@66f1a93a, $iwC$$iwC$Foo5@39d53a3, $iwC$$iwC$Foo5@4dddf42f)
My question is... what's the difference? 我的问题是......有什么区别? Why is
:paste
causing a problem and making the repl think Foo5
is ambiguous? 为什么
:paste
导致问题并使repl认为Foo5
含糊不清?
Edit: this bug was fixed in 2.11.9 , so latest 2.11.12 works. 编辑:此错误已在2.11.9中修复 ,因此最新的2.11.12工作。
Edit: since you're on Spark 1.6, I supposed you're stuck on 2.10. 编辑:因为你在Spark 1.6,我认为你被困在2.10。 That's OK, I just picked up Homer again the other day, and in a very old version (Lattimore, 1951).
没关系,我前几天再次选择荷马,而且是一个非常古老的版本(Lattimore,1951)。
It looks like an old bug with how the Spark shell is handling imports from history under -Yrepl-class-based
. 它看起来像一个旧的错误,Spark shell如何处理
-Yrepl-class-based
历史记录中的导入。
With -Xprint:typer
: 使用
-Xprint:typer
:
import scala.util.Try;
import $line3.$read.INSTANCE.$iw.$iw.Foo5;
private[this] val $line3$read: $line3.$read = $line3.$read.INSTANCE;
<stable> <accessor> def $line3$read: $line3.$read = $iw.this.$line3$read;
import $iw.this.$line3$read.$iw.$iw.Foo5Companion;
One import is via an aliased member, so the paths to the two Foo5
differ. 一个导入是通过别名成员,因此两个
Foo5
的路径不同。
You can use :load
instead of :paste
in this case. 你可以使用
:load
而不是:paste
在这种情况下:paste
。 Normally, you'd :paste
companions. 通常,您会
:paste
随播广告。
This is no consolation, but it will be fixed when Spark upgrades to 2.12: 这不是安慰,但是当Spark升级到2.12时它将被修复:
$ ~/scala-2.11.8/bin/scala -Yrepl-class-based
Welcome to Scala 2.11.8 (OpenJDK 64-Bit Server VM, Java 1.8.0_171).
Type in expressions for evaluation. Or try :help.
scala> :paste foo5.scala
Pasting file foo5.scala...
import scala.util.Try
defined class Foo5
defined object Foo5Companion
scala> val ls_i: List[Int] = List(1,2,3)
ls_i: List[Int] = List(1, 2, 3)
scala> val ls_foo: List[Foo5] = List(1,2,3).map(Foo5Companion.apply)
<console>:15: error: type mismatch;
found : List[Foo5]
required: List[Foo5]
val ls_foo: List[Foo5] = List(1,2,3).map(Foo5Companion.apply)
^
scala> :quit
$ scala -Yrepl-class-based
________ ___ / / ___
/ __/ __// _ | / / / _ |
__\ \/ /__/ __ |/ /__/ __ |
/____/\___/_/ |_/____/_/ | |
|/ version 2.12.6
scala> :paste foo5.scala
Pasting file foo5.scala...
import scala.util.Try
defined class Foo5
defined object Foo5Companion
scala> val ls_i: List[Int] = List(1,2,3)
ls_i: List[Int] = List(1, 2, 3)
scala> val ls_foo: List[Foo5] = List(1,2,3).map(Foo5Companion.apply)
ls_foo: List[Foo5] = List(Foo5@52354202, Foo5@6b1321b7, Foo5@342ee097)
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.