使用 Scala 和 Spark 将数组列的行与另一个数据帧的标题进行比较

Question

I am using Scala and Spark. I have two data frames.

The first one is like following:

+------+------+-----------+
| num1 | num2 |    arr    |
+------+------+-----------+
|   25 |   10 | [a,c]     |
|   35 |   15 | [a,b,d]   |
+------+------+-----------+

In the second one the data frame headers are

num1, num2, a, b, c, d

I have created a case class by adding all the possible header columns.

Now what I want is, by matching the columns num1 and num2, I have to check whether the array in arr column contains the headers of the second data frame. If it so the value should be 1, else 0.

So the required output is:

+------+------+---+---+---+---+
| num1 | num2 | a | b | c | d |
+------+------+---+---+---+---+
|   25 |   10 | 1 | 0 | 1 | 0 |
|   35 |   15 | 1 | 1 | 0 | 1 |
+------+------+---+---+---+---+

Answer 1

If I understand correctly, you want to transform the array column arr into one column per possible value, that would contain whether or not the array contains that value.

If so, you can use the array_contains function like this:

val df = Seq((25, 10, Seq("a","c")), (35, 15, Seq("a","b","d")))
             .toDF("num1", "num2", "arr")

val values = Seq("a", "b", "c", "d")
df
    .select(Seq("num1", "num2").map(col) ++
            values.map(x => array_contains('arr, x) as x) : _*)
    .show

+----+----+---+---+---+---+
|num1|num2|  a|  b|  c|  d|
+----+----+---+---+---+---+
|  25|  10|  1|  0|  1|  0|
|  35|  15|  1|  1|  0|  1|
+----+----+---+---+---+---+