Scala：将火花 DataFrame 中的值添加到 for 循环中的可变列表

Question

I want to update the elements of a MutableList that was declared outside of a for loop with values from a dataframe. I initialized the list as empty and expect the list to have the n number of elements added when the loop terminates. However, it seems only one element is back to an empty list (never gets updated with new additions) and when the loop terminates, the list is back to empty.

This only happens if I am iterating over a dataFrame, if I iterate over a fied range, say 1-10, the results returned are as expected.

Iterating through dataframe:

val my_list = MutableList[String]()

scala> for (i <-df){
     | my_list += "ok"
     | println(my_list)
     | }
MutableList(ok)
MutableList(ok)

scala> valid_list
res120: scala.collection.mutable.MutableList[String] = MutableList()

Iterating through fixed range

scala> for (i <- 1 to 10) {
     | my_list += "ok"
     | println (my_list)
     | }
MutableList(ok)
MutableList(ok, ok)
MutableList(ok, ok, ok)
MutableList(ok, ok, ok, ok)
MutableList(ok, ok, ok, ok, ok)
MutableList(ok, ok, ok, ok, ok, ok)
MutableList(ok, ok, ok, ok, ok, ok, ok)
MutableList(ok, ok, ok, ok, ok, ok, ok, ok)
MutableList(ok, ok, ok, ok, ok, ok, ok, ok, ok)
MutableList(ok, ok, ok, ok, ok, ok, ok, ok, ok, ok)

scala> my_list
res122: scala.collection.mutable.MutableList[String] = MutableList(ok, ok, ok, ok, ok, ok, ok, ok, ok, ok)

Also open to alternative methods of generating a list from df elements.

Answer 1

I will try to answer here. Is this what you want to achieve

val listValueDataFrame = Seq(("one", 2.0),("two", 1.5),("three", 8.0)).toDF("id", "val")
listValueDataFrame.printSchema
val listOfIds = listValueDataFrame.select("id").collect().map(_(0)).toList
val someOtherDataFrame = Seq(("one", 3.0),("ni", 2.5),("san", 9.0)).toDF("id", "val")
someOtherDataFrame.filter(someOtherDataFrame("id").isin(listOfIds:_*)).show

on my execution this prints the following

root
 |-- id: string (nullable = true)
 |-- val: double (nullable = false)

+---+---+
| id|val|
+---+---+
|one|3.0|
+---+---+

listValueDataFrame: org.apache.spark.sql.DataFrame = [id: string, val: double]
listOfIds: List[Any] = List(one, two, three)
someOtherDataFrame: org.apache.spark.sql.DataFrame = [id: string, val: double]

Does this help at all, was not 100% sure i understood the complete context of the question, but this can be achieved this way. Note that i have used collect and with large number of records this will cause bad performance (data will have to be "collected" and moved to the driver)

Answer 2

Can you try like this,

for (i <-df.collect){
      my_list += "ok"
      println(my_list)
      }

i used scala.collection.mutable.listBuffer and it worked fine.

scala> val a = scala.collection.mutable.ListBuffer[String]()
a: scala.collection.mutable.ListBuffer[String] = ListBuffer()

scala> for ( i <- df.collect) {a+="ok"; println(a)}
ListBuffer(ok)
ListBuffer(ok, ok)
ListBuffer(ok, ok, ok)
ListBuffer(ok, ok, ok, ok)
ListBuffer(ok, ok, ok, ok, ok)
ListBuffer(ok, ok, ok, ok, ok, ok)
ListBuffer(ok, ok, ok, ok, ok, ok, ok)
ListBuffer(ok, ok, ok, ok, ok, ok, ok, ok)

scala> a
res11: scala.collection.mutable.ListBuffer[String] = ListBuffer(ok, ok, ok, ok, ok, ok, ok, ok)

Please try and let me know the result. Cheers....