在与Scala Spark中的DataFrame有关的过滤器函数中使用定义的值

Question

I tried to find a similar issue but didn't find anything related.. I'm new to Spark and Scala and I'm having trouble with a specific case.

I have a DataFrame as the following :

+---+---+------------+
|src|dst|relationship|
+---+---+------------+
|  0|238|        41.0|
|  0|159|        46.0|
|238| 12|        36.0|
|  1|235|        44.0|
|  2|139|        50.0|
+---+---+------------+

My problem is : (1) I want get the destination with the lowest value of "relationship" for the src = 0 and (2) reuse this value. I'm able to get something for (1) using both val j = orderedSrc.filter("src == 1").orderBy("relationship").select("dst").take(5) and val h = j(0)(0) .

In my example it would return j: Array[org.apache.spark.sql.Row] = Array([238], [159]) and h: Any = 238 .

My question is concerning (2):

How can use this h value inside the previous query? Something that would look like val j = orderedSrc.filter("src==h").orderBy("relationship").select("dst").take(5) which would return Array[org.apache.spark.sql.Row] = Array([12]) ?

By advance thanks if you can help :-)!

Answer 1

As @Lamanus wrote, the solution was :

orderedSrc.filter($"src" === h).orderBy("relationship").select($"dst").take(5)

Many thanks!

Answer 2

you don't need to use take in the intermediate steps (this wont scale), use a join instead:

val df = Seq(
  (0,238,41.0),
  (0,159, 46.0),
  (238,12,36.0),
  (1, 235, 44.0),
  (2,139,50.0)
).toDF("src","dest","relationship")


val h = df.where($"src"===0)
  .select(min(struct($"relationship",$"dest")).as("min"))

df
  .join(h,df("src")===h("min.dest"),"leftsemi")
  .show()

+---+----+------------+
|src|dest|relationship|
+---+----+------------+
|238|  12|        36.0|
+---+----+------------+

Or the same with Window-functions:

df
  .withColumn("selector",min(when($"src"===0,struct($"relationship",$"dest"))).over(Window.partitionBy()))
  .where($"src"===$"selector.dest")
  .drop($"selector")
  .show()