[英]How to sort each line of a rdd in spark using scala?
我的文本文件具有以下數據:
10,14,16,19,52
08,09,12,20,45
55,56,70,78,53
我想按降序對每一行進行排序。 我已經嘗試了以下代碼
val file = sc.textFile("Maximum values").map(x=>x.split(","))
val sorted = file.sortBy(x=> -x(2).toInt)
sorted.collect()
我得到以下輸出
[[55, 56, 70, 78, 53], [10, 14, 16, 19, 52], [08, 09, 12, 20, 45]]
上面的結果表明整個列表已經按照降序排列了,但是我正在尋找每個值的降序排列
例如
[10,14,16,19,52],[08,09,12,20,45],[55,56,70,78,53]
應該
[52,19,16,14,10],[45,20,12,09,08],[78,70,56,55,53]
請抽出一些時間來回答這個問題。
這是一種方法(未經測試)
val reverseStringOrdering = Ordering[String].reverse
val file = sc.textFile("Maximum values").map(x=>x.split(",").sorted(reverseStringOrdering))
val sorted = file.sortBy(r => r, ascending = false)
sorted.collect()
Spark SQL方式
import org.apache.spark.sql.functions._
val df = Seq(
("10","14","16","19","52"),
("08","09","12","20","45"),
("55","56","70","78","53")).toDF("C1", "C2","C3","C4","C5")
df.withColumn("sortedCol", sort_array(array("C1", "C2","C3","C4","C5"), false))
.select("sortedCol")
.show()
產量
+--------------------+
| sortedCol|
+--------------------+
|[52, 19, 16, 14, 10]|
|[45, 20, 12, 09, 08]|
|[78, 70, 56, 55, 53]|
+--------------------+
檢查一下。
val file = spark.sparkContext.textFile("in/sort.dat").map( x=> { val y = x.split(','); y.sorted.reverse.mkString(",") } )
file.collect.foreach(println)
EDIT1:不同的方法如何應用於上述代碼。
scala> val a = "10,14,16,19,52"
a: String = 10,14,16,19,52
scala> val b = a.split(',')
b: Array[String] = Array(10, 14, 16, 19, 52)
scala> b.sorted
res0: Array[String] = Array(10, 14, 16, 19, 52)
scala> b.sorted.reverse
res1: Array[String] = Array(52, 19, 16, 14, 10)
scala> b.sorted.reverse.mkString(",")
res2: String = 52,19,16,14,10
scala> b.sorted.reverse.mkString("*")
res3: String = 52*19*16*14*10
scala>
EDIT2:
val file = spark.sparkContext.textFile("in/sort.dat").map( x=> { val y = x.split(',').map(_.toInt); y.sorted.reverse.mkString(",") } )
file.collect.foreach(println)
聲明:本站的技術帖子網頁,遵循CC BY-SA 4.0協議,如果您需要轉載,請注明本站網址或者原文地址。任何問題請咨詢:yoyou2525@163.com.