除了 Spark Scala 之外的更好的替代品

Question

I have been told that EXCEPT is a very costly operation and one should always try to avoid using EXCEPT . My Use Case -

val myFilter = "rollNo='11' AND class='10'"
val rawDataDf = spark.table(<table_name>)
val myFilteredDataframe = rawDataDf.where(myFilter)
val allOthersDataframe = rawDataDf.except(myFilteredDataframe)

But I am confused, in such use case, what are my alternatives?

Answer 1

Use left anti join as below-

 val df = spark.range(2).withColumn("name", lit("foo"))
    df.show(false)
    df.printSchema()
    /**
      * +---+----+
      * |id |name|
      * +---+----+
      * |0  |foo |
      * |1  |foo |
      * +---+----+
      *
      * root
      * |-- id: long (nullable = false)
      * |-- name: string (nullable = false)
      */
    val df2 = df.filter("id=0")
    df.join(df2, df.columns.toSeq, "leftanti")
      .show(false)

    /**
      * +---+----+
      * |id |name|
      * +---+----+
      * |1  |foo |
      * +---+----+
      */