如何仅使用数据集API联接两个数据集

Question

I am struggling to flatten the dataset resulting from joining two other datasets. Below is my code :

  val family = Seq(
      Person(0, "Agata", 0),
      Person(1, "Iweta", 0),
      Person(2, "Patryk", 2),
      Person(3, "Maksym", 0)).toDS
    val cities = Seq(
      City(0, "Warsaw"),
      City(1, "Washington"),
      City(2, "Sopot")).toDS

then the join :

val joined = family.joinWith(cities, family("cityId") ===cities("id"),"crossjoin")

The obtained result is :

joined: org.apache.spark.sql.Dataset[(Person, City)]
 _1| _2|

[0,Agata,0]|[0,Warsaw]| | 
[1,Iweta,0]|[0,Warsaw]| |
[2,Patryk,2]| [2,Sopot]| |
[3,Maksym,0]|[0,Warsaw] |

I want to flatten this and to get the following dataset :

val output: Dataset= 
[0,Agata,0,Warsaw]|
[1,Iweta,0,Warsaw]|
[2,Patryk,2,Sopot]| 
[3,Maksym,0,Warsaw]

Any idea how to do this without using dataframe API , I want it to be totally done with Dataset API. Thanks a lot for your help. Best Regards

Answer 1

Using join itself, You will get the same output.

family.join(cities, family("cityId")===cities("id")).drop("id")

Sample Output:

+--------+------+--------+
|cityName|cityId|cityName|
+--------+------+--------+
|   Agata|     0|  Warsaw|
|   Iweta|     0|  Warsaw|
|  Patryk|     2|   Sopot|
|  Maksym|     0|  Warsaw|
+--------+------+--------+

Answer 2

...
val joined = family.join(cities).where(family("cityid") === cities("id")).drop("id") // adding this means use the DF 
joined.show

The experimental aspect means not yet usage of joinWith should be formally considered.

DF and DS are growing towards eachother, so it should not matter. What you want is not actually possible I believe.

Also, they say Dataset[Row] aka DataFrame.