如何使用 Spark SQL 在循环时将迭代的行记录保存到新的数据框或列表？

Question

I have one data frame. That data frame is giving me list of records and then I am going to iterate over each row and doing some manipulation.

 for (row <- dataframe.rdd.collect()) {

// var anyval= row.mkString(",").split(",")(take the column);
}

Then I am Making some checks and then if current row will match the requirement then try to create new list or collection to save full row.

Could you please help to with example how to save this row in new data frame using spark sql?

Answer 1

There are different ways to achieve this, the main point is understand the basic behavior main components of spark . For all of them ( dataframe , dataset , rdd ) you can not update the actual value, they are inmutable objects, but you can iterate their items and based on your logic, create a new one, based on an existing one. Examples:

val yourDF = Seq( // Sample
  ("A1", 12, null),       // Record 1
  ("B1", -1, "Mexico"),   // Record 2
  ("C1", 2, "Argentina")  // Record 3
).toDF("id", "some_value", "country") // Column definition

yourDF.show() // Visualize your DF

Above code will output:

+---+----------+---------+
| id|some_value|  country|
+---+----------+---------+
| A1|        12|     null|
| B1|        -1|   Mexico|
| C1|         2|Argentina|
+---+----------+---------+

Based on that is a Dataframe , this is how you can iterate all rows and access their items:

val newDF = yourDF
  .map(item =>{  // Iterate your DF 
    val id = item.getAs[String]("id") // Access their element (from row object - each item in your DF) - You need to specify datatype and 'column_name' on this approach
    val some_value = item.getAs[Integer]("some_value")
    val country = item.getAs[String]("country")
    val outputCountry = if(country != null) country.substring(0,3) else null
    // Output: id, first 3 chars of the country (if it is not null) and `some_value` multiplied by 10
    (id, outputCountry, some_value*10)
  })

newDF.show()

Above code will output:

+---+----+---+
| _1|  _2| _3|
+---+----+---+
| A1|null|120|
| B1| Mex|-10|
| C1| Arg| 20|
+---+----+---+

As you can see, column names are not the same as the first DF , this is because we are creating a new one and we did not specify the column names, we can use either .toDF("column_a", "column_b", "column_c") or use a case class like in the next example.

Let's do same exercise, but using case classes (with Scala ).

case class Country(id: String, some_value: Integer, country: String) // Case class

val newDF = yourDF
  .as[Country] // Cast your DF with a case class to have a Dataset
  .map(country=>{ // iterate dataset
    val id = country.id // Access their element (as object notation, easier!)
    val some_value = country.some_value
    val countryName = country.country
    val outputCountry = if(countryName != null) countryName.substring(0,3) else null
    // Output: id, first 3 chars of the country (if it is not null) and `some_value` multiplied by 10
    Country(id, some_value*10, outputCountry) // Output will use a case class to define the schema of the new object (DAtaset[Country])
  })

newDF.show()

Above code will output:

+---+----------+-------+
| id|some_value|country|
+---+----------+-------+
| A1|       120|   null|
| B1|       -10|    Mex|
| C1|        20|    Arg|
+---+----------+-------+