[英]Spark: can I convert a Scala set to a DataType in spark SQL?
For example, Scala array corresponds to ArrayType in Spark SQL, which can be used in schema definitions. 例如,Scala数组对应于Spark SQL中的ArrayType,可在架构定义中使用。 For Map there's MapType. 对于Map,有MapType。
How about Set ? Set怎么样?
A set (no pun intended) of supported types is limited a not extensible. 一组受支持的类型(无双关语)受限制且不可扩展。 You'll find a full list of supported types in the Spark SQL, DataFrames and Datasets Guide - as you can check there is no type for Set
. 您可以在《 Spark SQL,数据框架和数据集指南》中找到受支持类型的完整列表,因为您可以检查Set
是否没有类型。
The best you can do is to use ArrayType
which maps to scala.collection.Seq
and handle set specific operations yourself. 最好的办法是使用ArrayType
映射到scala.collection.Seq
并自己处理特定于集合的操作。
It is possible to use binary Encoders
( How to store custom objects in Dataset? ) but these are intended for strongly typed datasets, and have limited applications when used with DataFrames
. 可以使用二进制Encoders
( 如何在数据集中存储自定义对象? ),但这些二进制Encoders
用于强类型数据集,并且与DataFrames
使用时具有有限的应用程序。
There is none. 空无一人。 Exhaustive list is here: http://spark.apache.org/docs/latest/sql-programming-guide.html#data-types 详尽的清单在这里: http : //spark.apache.org/docs/latest/sql-programming-guide.html#data-types
Find full example code at " examples/src/main/scala/org/apache/spark/examples/sql/SparkSQLExample.scala
" in the Spark repo. 在Spark存储库中的“ examples/src/main/scala/org/apache/spark/examples/sql/SparkSQLExample.scala
”中找到完整的示例代码。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.