如何 output 计数来自 Spark dataframe 的两个二进制列的所有成对组合的计数，即使它是零计数？

Question

How to output the count of all pairwise combination of two binary(0/1) columns from a Spark dataframe even when the count is zero?

final_sdf.groupBy('actual', 'prediction').count().show()

Current output is

But my desired output includes the zero groups as below.

Answer 1

Okay, the idea to do this, is first create the missing binary rows, allocate value count to 0, filter, then append the dataset.

Let's assume our main dataset is called df and looks as below:

+------+----------+-----+
|actual|prediction|count|
+------+----------+-----+
|1     |1.0       |944  |
|0     |1.0       |208  |
+------+----------+-----+

First, let's create a column called array for example with value abs(actual - 1) , this way, we get the missing binary value. Then, we explode that back to prediction and we drop our array column.

val df2 = df1
  .withColumn("array", array(col("actual"), abs(col("actual") - 1)))
  .withColumn("prediction", explode(col("array")))
  .drop("array")

+------+----------+-----+
|actual|prediction|count|
+------+----------+-----+
|1     |1         |944  |
|1     |0         |944  |
|0     |0         |208  |
|0     |1         |208  |
+------+----------+-----+

Then we do an anti join ( df1 and df2 ) and overwrite count value with 0.

val df3 = df2.join(df1, Seq("actual", "prediction", "count"), "anti")
  .withColumn("count", lit(0))

+------+----------+-----+
|actual|prediction|count|
+------+----------+-----+
|1     |0         |0    |
|0     |0         |0    |
+------+----------+-----+

Finally, we union these two dataframes:

df1.union(df3).show(10)

+------+----------+-----+
|actual|prediction|count|
+------+----------+-----+
|     1|       1.0|  944|
|     0|       1.0|  208|
|     1|       0.0|    0|
|     0|       0.0|    0|
+------+----------+-----+

which is I hope what you need!