scala foreach loop return list

Question

I am a newbie to scala (and functional programming, basically). I am trying to loop over a list of rows (can think as strings), where each string will be passed to a different scala method, where I am doing some manipulation on the input string and then return a string to the for-loop.

below is not the working code, but this is what I am expecting to work.

     val input_list = spark.read
          .format("com.crealytics.spark.excel")
          .option("sheetName" , "SchemaInfo")
          .option("useHeader", "true")
          .schema(profilerSchema)
          .load(path)  // this is spark dataframe, which has rows.

 val columnNames : List[String] = new List("Hello" , "world");

         var outputList = new ListBuffer[String]();

        // Here i am iterating the input_list where i pass each ele to getString() method where  
        // it returns the final string which i want to add to outputList. 

         input_list.foreach(i => {
                val res: String = getString(row, columnNames)
                 outputList += res;
               }));


         def getString(row: Row, schemaNames: List[String]) : String  = {

        // some implementation where it returns a string.
        }

Below is the error message I am getting (discard the line number. its getting at the foreach loop.).

Error:(57, 14) overloaded method value foreach with alternatives:
  (func: org.apache.spark.api.java.function.ForeachFunction[org.apache.spark.sql.Row])Unit <and>
  (f: org.apache.spark.sql.Row => Unit)Unit
 cannot be applied to (org.apache.spark.sql.Row => scala.collection.mutable.ListBuffer[String])
    excel_df.foreach{row => (jsonStrList += convertRowToJSON(row, columnNames))};

I am having a hard time writing the logic. any help is really appreciated.

Answer 1

input_list.foreach(i => {
      val res: String = getString(row, columnNames)
       outputList += res;
     });

Your function inside the foreach method returns the outputList value. If you look at the signature, the return value should be Unit - which means the method doesn't return a value. It assumes you do some calculation that doesn't return anything.

You should use map instead of foreach. You don't need foreach and the outputList variable.

input_list.map(row => {
       // Logic to return the item that you want to compute
     })
     .toList()

You have to shift your way of thinking from an imperative style where you declare a collection and then loop through the items of another collection and add the result of your calculations to the first collection to a functional style that uses the map/filter methods.

Example:

List(1,2,3,4,5,6,7,8,9,10)
// filter gives you the list of even numbers between 1 and 10
.filter(i => i % 2 == 0) 
// This gives you the squares of the even numbers between 1 and 10
.map(i => i * i)
// This gives the doubles of the squares of the even numbers
.map(i => i * 2)

You can also do this:

val evenNumbers = List(1,2,3,4,5,6,7,8,9,10)
// filter gives you the list of even numbers between 1 and 10
.filter(i => i % 2 == 0) 

val squares= evenNumbers
// This gives you the squares of the even numbers between 1 and 10
.map(i => i * i)

val doubleSquares = squares
// This gives the doubles of the squares of the even numbers
.map(i => i * 2)

// this will return a tuple with lists when it's the last statement in a function or method.
(squares, doubleSquares)

As you can see there are no ListBuffer objects declared. The example is a bit contrived, you could combine the last two maps into one but I wanted to prove a point. You can do other operations, you can group by items, sort them and so on.

You need to read more about scala and FP in general. This is a good introduction: https://docs.scala-lang.org/overviews/scala-book/introduction.html and this: https://docs.scala-lang.org/overviews/scala-book/passing-functions-around.html .

You can also trying things here inside the browser: https://scastie.scala-lang.org .