Spark left_outer连接返回的两个列表中都存在项目

Question

I'm running spark-shell to compare 2 csv files. Each file has the same number of columns and all have 600,000 rows. I'm expecting the 2 files have all the same rows. Here is my script.

val a =
  spark
    .read
    .option("header", "true")
    .option("delimiter", "|")
    .csv("/tmp/1.csv")
    .drop("unwanted_column").
    .cache()

val b = 
   spark
     .read
     .option("header", "true")
     .option("delimiter", "|")
     .csv("/tmp/2.csv")
     .drop("unwanted_column")
     .cache()

val c = a.join(b, Seq("id", "year"), "left_outer").cache()

c.count() // this is returning 600,000

Now I'm trying to find out the difference by randomly picking a line with the same id and year in 2 datasets a and b.

val a1 = a.filter(i => i.get(0).equals("1") && i.get(1).equals("2016")).first()

val b1 = b.filter(i => i.get(0).equals("1") && i.get(1).equals("2016")).first()

Then I try to compare each column in a1 and b1.

(0 to (a1.length -1)).foreach { i =>
  if (a1.getString(i) != null && !a1.getString(i).equals(b1.getString(i))) {
    System.out.println(i + " = " + a1.getString(i) + " = " + b1.getString(i))
  }
}

It didn't print anything. In other words, there is no difference.

I can't tell why c.count() is returning 600,000 like that.

Answer 1

Sorry guys, I guess it was my fault. Actually I was after a.subtract(b). My purpose is to find out the difference between a and b. I was confused about left_outer join.