比较spark中两个数据框中的列

Question

I have two dataframes, both of them contain different number of columns. I need to compare three fields between them to check if those are equal.

I tried following approach but its not working.

if(df_table_stats("rec_cnt").equals(df_aud("REC_CNT")) || df_table_stats("hashcount").equals(df_aud("HASH_CNT")) || round(df_table_stats("hashsum"),0).equals(round(df_aud("HASH_TTL"),0)))
    {
        println("Job executed succefully")
    }

df_table_stats("rec_cnt") , this returns Column rather than actual value hence condition becoming false.

Also, please explain difference between df_table_stats.select("rec_cnt") and df_table_stats("rec_cnt") .

Thanks.

Answer 1

根据您的条件使用 sql 和 inner join df 。

Answer 2

Per my comment, the syntax you're using are simple column references, they don't actually return data. Assuming you MUST use Spark for this, you'd want a method that actually returns the data, known in Spark as an action . For this case you can use take to return the first Row of data and extract the desired columns:

val tableStatsRow: Row = df_table_stats.take(1).head
val audRow: Row = df_aud.take(1).head

val tableStatsRecCount = tableStatsRow.getAs[Int]("rec_cnt")
val audRecCount = audRow.getAs[Int]("REC_CNT")

//repeat for the other values you need to capture

However, Spark definitely is overkill if this is all you're using it for. You could use a simple JDBC library for Scala like ScalikeJDBC to do these queries and capture the primitives in the results.