[英]Encoders.product[of a scala trait ].schema in spark
How to create a schema for spark from a trait?如何从特征创建火花模式? Considering a trait:
考虑一个特征:
trait A{
val name:String
val size:String
}
As:作为:
Encoders.product[A].schema
gives:给出:
Error:type arguments do not conform to method product's type parameter bounds [T <: Product]
Also the number of fields will be more then the limit of case class parameters > 200此外,字段数将超过 case class parameters > 200 的限制
Case class do supports more than 22 columns, try creating outside all other class/object.案例 class 确实支持超过 22 列,请尝试在所有其他类/对象之外创建。 If your need is to create a dataframe schema with large number of fields, this should work.
如果您需要创建一个包含大量字段的 dataframe 架构,这应该可行。
val schema: StructType = StructType(
Array(
StructField(name = "name", StringType),
StructField(name = "size", StringType)
)
)
val data = Seq(Row("Ramanan","29"))
spark.createDataFrame(spark.sparkContext.parallelize(data),schema).show()
I cannot give you all the details why this is not working but I am proposing a slightly alternative solution that we frequently use in our Scala Spark projects.我不能告诉你为什么这不起作用的所有细节,但我提出了一个我们在 Scala Spark 项目中经常使用的稍微替代的解决方案。
The signature of Encoders.product
looks like Encoders.product
的签名看起来像
product[T <: scala.Product](implicit evidence$5 : scala.reflect.runtime.universe.TypeTag[T])
which means tt expects a class that extends Product
trait and an implicit TypeTag.这意味着 tt 期望 class 扩展
Product
特征和隐式 TypeTag。
Instead of a trait, you could create a case class
as case classes are extending Product
(and Serializable
) automatically.您可以创建
case class
而不是特征,因为案例类会自动扩展Product
(和Serializable
)。
In order to get a schema you could do:为了获得一个模式,你可以这样做:
case class A (
val name: String,
val size: String
)
def createSchema[T <: Product]()(implicit tag: scala.reflect.runtime.universe.TypeTag[T]) = Encoders.product[T].schema
val schema = createSchema[A]()
schema.printTreeString()
/*
root
|-- name: string (nullable = true)
|-- size: string (nullable = true)
*/
As said in the beginning, I can't explain all the details, just provide a working solution and hoping it fit your needs.正如一开始所说,我无法解释所有细节,只是提供一个可行的解决方案并希望它能满足您的需求。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.