在 Spark 中联合两个不同类型的 DataFrame

Question

In my recent project, I need to union two dataframes of different sizes. For example:

Here is my sample data:

df1:

name    number    address
kevin   101        NZ
gevin   102        CA

here all the fields are of type String.

df2:

name    number    address
kevin   [101,102]    NZ
gevin   [102,103]    CA

Here name and address are type string and number is of type array<string> .

Now I need to union these two dataframes. My expexcted outcome is like:

name    number    address
kevin   101         NZ
gevin   102         CA
kevin   [101,102]   NZ
gevin   [102,103]   CA

final df types should be same as the df2(string, array, string).

Answer 1

You can convert to array for the first dataframe as well and union both dataframe.

import org.apache.spark.sql.functions._

df1.withColumn("number", array($"number"))
   .union(df2)

Output:

+-----+----------+-------+
|name |number    |address|
+-----+----------+-------+
|kevin|[101]     |NZ     |
|gevin|[102]     |CZ     |
|kevin|[101, 102]|NZ     |
|gevin|[102, 103]|CZ     |
+-----+----------+-------+

Hope this helps!