Spark中的Sort Merge Join中的键何时不可排序？

Question

当我阅读有关排序合并连接的文章时，它说这是广播连接之后 Spark 中最受欢迎的一种，但前提是连接键是可排序的。 我的问题是什么时候加入键是不可排序的？ 任何数据类型都可以排序。 你能帮我理解一个键可能无法排序的场景吗？

Answer 1

请参阅https://www.waitingforcode.com/apache-spark-sql/sort-merge-join-spark-sql/read 。 优秀的网站。

并非所有类型都可以排序。 例如日历间隔类型。

报价：

"for not sortable keys the sort merge join" should "not be used" in {
import sparkSession.implicits._
// Here we explicitly define the schema. Thanks to that we can show
// the case when sort-merge join won't be used, i.e. when the key is not sortable
// (there are other cases - when broadcast or shuffle joins can be chosen over sort-merge
//  but it's not shown here).
// Globally, a "sortable" data type is:
// - NullType, one of AtomicType
// - StructType having all fields sortable
// - ArrayType typed to sortable field
// - User Defined DataType backed by a sortable field
// The method checking sortability is   org.apache.spark.sql.catalyst.expressions.RowOrdering.isOrderable
// As  you see, CalendarIntervalType is not included in any of above points,
// so even if the data structure is the same (id + login for customers, id + customer id + amount for orders)
// with exactly the same number of rows, the sort-merge join won't be applied here.