[英]udf No TypeTag available for type string
I don't understand a behavior of spark. 我不了解火花的行为。
I create an udf wich return an Integer like below 我创建一个udf,返回一个整数,如下所示
import org.apache.spark.sql.SQLContext
import org.apache.spark.{SparkConf, SparkContext}
object Show {
def main(args: Array[String]): Unit = {
val (sc,sqlContext) = iniSparkConf("test")
val testInt_udf = sqlContext.udf.register("testInt_udf", testInt _)
}
def iniSparkConf(appName: String): (SparkContext, SQLContext) = {
val conf = new SparkConf().setAppName(appName)//.setExecutorEnv("spark.ui.port", "4046")
val sc = new SparkContext(conf)
sc.setLogLevel("WARN")
val sqlContext = new SQLContext(sc)
(sc, sqlContext)
}
def testInt() : Int= {
return 2
}
}
I work perfectly but if I change the return type of method test from Int to String 我工作得很好,但是如果我将方法测试的返回类型从Int更改为String
val testString_udf = sqlContext.udf.register("testString_udf", testString _)
def testString() : String = {
return "myString"
}
I get the following error 我收到以下错误
Error:(34, 43) No TypeTag available for String
val testString_udf = sqlContext.udf.register("testString_udf", testString _)
Error:(34, 43) not enough arguments for method register: (implicit evidence$1: reflect.runtime.universe.TypeTag[String])org.apache.spark.sql.UserDefinedFunction.
Unspecified value parameter evidence$1.
val testString_udf = sqlContext.udf.register("testString_udf", testString _)
here are my embedded jars: 这是我的嵌入式罐子:
datanucleus-api-jdo-3.2.6
datanucleus-core-3.2.10
datanucleus-rdbms-3.2.9
spark-1.6.1-yarn-shuffle
spark-assembly-1.6.1-hadoop2.6.0
spark-examples-1.6.1-hadoop2.6.0
I am a little bit lost... Do you have any idea? 我有点迷路了...你有什么主意吗?
Since I can't reproduce the issue copy-pasting just your example code into a new file, I bet that in your real code String
is actually shadowed by something else. 由于无法复制仅将示例代码复制粘贴到新文件中的问题,所以我敢打赌,在您的真实代码中,
String
实际上被其他东西遮盖了。 To verify this theory you can try to change you signature to 为了验证这一理论,您可以尝试将您的签名更改为
def testString() : scala.Predef.String = {
return "myString"
}
or 要么
def testString() : java.lang.String = {
return "myString"
}
If this one compiles, search for "String" to see how you shadowed the standard type. 如果可以编译,请搜索“ String”以查看如何隐藏标准类型。 If you use IntelliJ Idea, you can try to use "Ctrl+B" (GoTo) to find it out.
如果使用IntelliJ Idea,则可以尝试使用“ Ctrl + B”(转到)找出来。 The most obvious candidate is that you used
String
as a name of generic type parameter but there might be some other choices. 最明显的候选人是您使用
String
作为泛型类型参数的名称,但可能还有其他选择。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.