使用 spark 和 scala 将具有值的 dataframe 列转换为列表

Question

+-----------------------------------------------------------------------------------------------------------------------------------------------+
|Texts                                                                                                                                          |
+----------------------------------------------------------------------------------------------------------------------------------------------+
|RT @xxxxxx: post aqwe qwqq ssdd qaAQ WQWQW CSDWDW!!! 

must RT !                                                                                                                                      |
|RT @xxxxx: aaa in ssss ssss ss sqqq this qqq in "sss" should xxxx xx at xx xaaaa aqw   |
|RT @xxxx: QWW sadad jkhj to hjyhy a eryr rrryryry? ersfsfdsgdgdgg t rtrt ytyyryr.
 
sadwf wwewe ewewe jyiopo;l dwewre etet of the ddgdg-we dfdfdf, @b…                                                                              |
+-----------------------------------------------------------------------------------------------------------------------------------------------+

I want to have these rows of values in Text column in a list using scala and spark.

1. val newList =   myDataframe.select("Texts").rdd.map(_(0)).collect.toList
2. val newList =   myDataframe.select("Texts").collect().map(_(0)).toList
   newList .foreach(println)

both ways aren't giving any output and program doesn't terminate also. No errors are thrown.

Expected output

["RT @xxxxxx: post aqwe qwqq ssdd qaAQ WQWQW CSDWDW!!! must RT !", "RT @xxxxx: aaa in ssss ssss ss sqqq this qqq in "sss" should xxxx xx at xx xaaaa aqw", "RT @xxxx: QWW sadad jkhj to hjyhy a eryr rrryryry? ersfsfdsgdgdgg t rtrt ytyyryr.

sadwf wwewe ewewe jyiopo;l dwewre etet of the ddgdg-we dfdfdf, @b…"]

Please note that sentence in each row in dataframe may contain new line

eg I am going to the the shop.\n Its very expensive

this sentence will be displayed as

 I am going to the shop
 its very expensive

But both will belong to the same row.

Answer 1

Below methods are correct to convert a column of a dataframe into a list

1. val newList =   myDataframe.select("Texts").rdd.map(_(0)).collect.toList
2. val newList =   myDataframe.select("Texts").collect().map(_(0)).toList

But the Dataframe in the question says each row may contain new lines. therefore above mthods won't work directly. First new lines should be removed.

val singleLineDataframe =  myDataframe.withColumn("Texts", regexp_replace(col("Texts"), "[\\r\\n\\n]", "."))
val sentenceList =   singleLineDataframe.select("Texts").rdd.map(r => r(0)).collect.toList
for ( element <- sentenceList)
      println(element)