加入spark后有效统计记录

Question

This is what I am doing. I need to get number of records present in one dataset and not the other and then again join with a third dataset to get some other columns.

val tooCompare = dw
        .select(
          "loc",
          "id",
          "country",
          "region"
        ).dropDuplicates()

val previous = dw
        .select(
          "loc",
          "id",
          "country",
          "region"
        ).dropDuplicates()

val delta = tooCompare.exceptAll(previous).cache()
 
val records = delta
        .join(
          dw,//another dataset
          delta
            .col("loc").equalTo(dw.col("loc"))
            .and(delta.col("id").equalTo(dw.col("id")))
            .and(delta.col("country").equalTo(dw.col("country")))
            .and(delta.col("region").equalTo(dw.col("region")))
        )
        .drop(delta.col("loc"))
        .drop(delta.col("id"))
        .drop(delta.col("country"))
        .drop(delta.col("region"))
        .cache()
    }

 val recordsToSend = records.cache()
 val count = recordsToSend.select("loc").distinct().count()

Is there a more efficient way to do this? I am new to Spark. I am pretty sure I am missing something here

Answer 1

I would suggest using SQL to make this more readable.

First, create Temp Views of the dataframes in question. Don't know exactly what data frames you have, so something like

dfToCompare.createOrReplaceTempView("toCompare")
previousDf.createOrReplaceTempView("previous")
anotherDataSet.createOrReplaceTempView("another")

Then you can proceed to do all your opertions in one SQL statement

val records = spark.sql("""select loc, id, country,region
              from toCompare c
              inner join another a
               on a.loc = c.loc  
                and a.id = p.id
                and a.country = c.country
                and a.region = c.region
             where not exists (select null
                                from previous p
                                where p.loc = c.loc  
                                 and p.id = p.id
                                 and p.country = c.country
                                 and p.region = c.region""")

Then you can proceed as before...

val recordsToSend = records.cache()
val count = recordsToSend.select("loc").distinct().count()

Answer 2

I think there's potentially some errors in the code you've pasted as tooCompare and previous are the same, + the third dataset join references deAnon but dw on the table....

For this example answer, assume your current table is called "current", previous is called "previous" and third table is "extra". Then:

val delta = current.join(
              previous, 
              Seq("loc","id","country","region"), 
              "leftanti"
            ).select("loc","id","country","region").distinct

val recordsToSend = delta
                    .join(
                      extra,
                      Seq("loc", "id", "country", "region")
                    )

val count = recordsToSend.select("loc").distinct().count()

This may be more efficient, but I'd appreciate you commenting as to whether it actually was!

Just as an aside: note that I'm using the Seq[String] as a join argument (this requires the column names to be identical on both tables, and won't produce two copies of the columns). However, your original join logic can be written a bit more succinctly, as follows (using my naming conventions):

val recordsToSend = delta
                    .join(
                      extra,
                      delta("loc") === extra("loc")
                        && delta("id") === extra("id")
                        && delta("country") === extra("country")
                        && delta("region") === extra("region")
                    )
                    .drop(delta("loc"))
                    .drop(delta("id"))
                    .drop(delta("country"))
                    .drop(delta("region"))

Even better would be to write a drop function that lets you provide more than one column, but I'm going really off topic now ;-)