在Spark Scala中的joinWith之后返回组合的数据集

Question

Given the below two Spark Datasets , flights and capitals , what would be the most efficient way to return combined (ie "joined") result without converting first to a DataFrame or writing out all the columns out by name in a .select() method? I know, for example, that I can access either tuple with (eg .map(x => x._1 ) or use the * operator with:

result.select("_1.*","_2.*")

But the latter may result in duplicate column names and I'm hoping for a cleaner solution.

Thank you for your help.

case class Flights(tripNumber: Int, destination: String)

case class Capitals(state: String, capital: String)

val flights = Seq(
  (55, "New York"),
  (3, "Georgia"),
  (12, "Oregon")
).toDF("tripNumber","destination").as[Flights]

val capitals = Seq(
  ("New York", "Albany"),
  ("Georgia", "Atlanta"),
  ("Oregon", "Salem")
).toDF("state","capital").as[Capitals]

val result = flights.joinWith(capitals,flights.col("destination")===capitals.col("state"))

Answer 1

There are 2 options, but you will have to use join instead of joinWith :

1. That is the best part of the Dataset API, is to drop one of the join columns , thus no need to repeat projection columns in a select: val result = flights.join(capitals,flights("destination")===capitals("state")).drop(capitals("state"))
2. rename join column to be the same in both datasets and use a slightly different way of specifying the join: val result = flights.join(capitals.withColumnRenamed("state", "destination"), Seq("destination"))

Output:

result.show
+-----------+----------+-------+
|destination|tripNumber|capital|
+-----------+----------+-------+
|   New York|        55| Albany|
|    Georgia|         3|Atlanta|
|     Oregon|        12|  Salem|
+-----------+----------+-------+