Spark Scala JOIN 后所有类似 dataframe 列的动态比较

Question

I am trying to compare columns in a JOINed table / relation that have similar names as a result of the JOIN - which can be qualified with l. or r.

The code via a val to be used with withColumn to compare all columns lx with r.x does not work - but it does run without run-time error as shown below. Basically I want to compare all lx col names with all r.x col names.

FULL LISTING

import spark.implicits._
import org.apache.spark.sql.functions._
import org.apache.spark.sql.Column

val dfCur = sc.parallelize(Seq( 
    (1,"2019-01-01","2018-01-01",1,2),
    (7,"2019-01-01","2019-01-01",100,200),
    (3,"2019-01-01","2019-01-03",5,6)
)).toDF("customer_id", "report_date", "date", "value_1", "value_2")

val dfRaw = sc.parallelize(Seq( 
   (2,"2019-01-01","2019-01-01",1,2),
   (7,"2019-01-01","2019-01-01",100,300),
   (3,"2019-01-01","2019-01-03",5,6)
)).toDF("customer_id", "report_date", "date", "value_1", "value_2")

// contrived and not necessarily correct data, but that is not the focus
val difference = dfCur.union(dfRaw).except(dfCur.intersect(dfRaw))

val prefixL = "l."
val prefixR = "r."

//val diffCols = difference.columns
//    .map(c=> difference(c).as(s"$prefixL$c"))
//    .map(x => when(x =!= s"$prefixR$x", concat(lit(","), lit(x))).otherwise(","))
//    .reduce(concat(_, _))

// Updated attempt, to no avail
val diffCols = difference.columns
.map(c=> (c,difference(c).as(s"$prefixL$c")))
.map(x => when(x._2 =!= s"${prefixR}${x._1}", concat(lit(","), lit(x._1))).otherwise(","))
.reduce(concat(_,_))

val result = difference.as("l")
    .join(dfRaw.as("r"), $"l.customer_id" === $"r.customer_id","inner")
    .withColumn("XYZ2", diffCols)

Output something like this below.

+-----------+-----------+----------+-------+-------+-----------+-----------+----------+-------+-------+-------------------------+
|customer_id|report_date|date      |value_1|value_2|customer_id|report_date|date      |value_1|value_2|XYZ2                     |
+-----------+-----------+----------+-------+-------+-----------+-----------+----------+-------+-------+-------------------------+
|2          |2019-01-01 |2019-01-01|1      |2      |7          |2019-01-01 |2019-01-01|100    |300    |,,date_1,value_1,,|

UPDATE

Aspects applied as per suggestion in comments. Ran, but did not produce correct results.

Answer 1

Thanks for the help as it was partially correct, but the final clue was to add col(...) like this:

val diffCols2 = difference
    .columns
    .map(c=> (c,difference(c).as(s"${prefixL}${c}")))
    .map(x => when(x._2 =!= col(s"${prefixR}${x._1}"), concat(lit(","), lit(x._1)))
                                            .otherwise(","))
    .reduce(concat(_,_)) 

col(s"${prefixR}${x._1}") is the point.

I noted that mapping successively can change things. It's not always obvious. In any event a way of getting the differences over a JOINed table with similar names issues.