按列比较两个 spark 数据帧并获取不匹配的记录

Question

I want to iterate and compare the columns between two spark dataframes and store the mismatch records.

I am getting the mismatch records in dataframe format so i want to store in any variable as dataframe is immutable. Please suggest how to store dataframe output as rows and columns in variable or collection.

Var mismatchValues = new ArrayBuffer[String]()

Val columns1 = srcTable_colMismatch.schema.fields.map(_.name.tostring)

Val selectiveDifference = columns1.map(c=> srcTable_colMismatch.select(c, "hash_key","row_num"). exceptAll(tgtTable_colMismatch.select(c, "hash_key","row_num").as(c)))

selectiveDifference.zipWithIndex.foreach{ case (e,i) => if(e.count>0) mismatchValues += sortedMismatchRecords.select("*").as("SRC").join(e.as("dif"), $"SRC.hash_key" === $"dif.hash_key" && "SRC.columns1(i) != e.schema.fields.map(_.name)).select("SRC.*").collect.mkstring(",") }

Val convertedDF = mismatchValues.map(a=> a.toString).toDF()
ConvertedDF show()

Answer 1

Spark can do this for you:

df1.union(df2).subtract(df1.intersect(df2))

I strongly discourage you from using a variable to compare unless you have it on good authority that it will fit in memory.

df1.union(df2)\ # create one data set
 .subtract(\ # remove items that match this data frame
  df1.intersect(df2)\ # all items that are in both dataframes
 )