[英]how to truncate values for multiple rows and columns in dataframe in spark scala
我有一個數據幀 df id ABCD 1 1.000234 2.3456 4.6789 7.6934 2 3.7643 4.2323 5.6342 8.567
我想創建另一個數據幀 df1,並將值截斷到小數點后 2 位
id A B C D
1 1.00 2.35 4.68 7.70
2 3.76 4.23 5.63 8.57
有人可以幫助我編寫代碼,因為我的數據框由 70 列和 10000 行組成
這可以使用format_number
函數很容易地完成
val df = Seq(
(1, 1.000234, 2.3456, 4.6789, 7.6934),
(2, 3.7643, 4.2323, 5.6342, 8.567)
).toDF("id", "A", "B", "C", "D")
df.show()
+---+--------+------+------+------+
| id| A| B| C| D|
+---+--------+------+------+------+
| 1|1.000234|2.3456|4.6789|7.6934|
| 2| 3.7643|4.2323|5.6342| 8.567|
+---+--------+------+------+------+
val df1 = df.select(col("id"),
format_number(col("A"), 2).as("A"),
format_number(col("B"), 2).as("B"),
format_number(col("C"), 2).as("C"),
format_number(col("D"), 2).as("D"))
df1.show()
+---+----+----+----+----+
| id| A| B| C| D|
+---+----+----+----+----+
| 1|1.00|2.35|4.68|7.69|
| 2|3.76|4.23|5.63|8.57|
+---+----+----+----+----+
這是動態截斷數據幀中的值的方法之一,而不是硬核方法
import org.apache.spark.sql.functions.round
val df1 = df.columns.foldLeft(df){(df,colName) =>df.withColumn(colName,round(col(colName),3))}
這對我有用
您可以通過導入 org.apache.spark.sql.types._ 來使用 DecimalType(3,2) 進行轉換
scala> val df = Seq(
| (1, 1.000234, 2.3456, 4.6789, 7.6934),
| (2, 3.7643, 4.2323, 5.6342, 8.567)
| ).toDF("id", "A", "B", "C", "D")
df: org.apache.spark.sql.DataFrame = [id: int, A: double ... 3 more fields]
scala> df.show()
+---+--------+------+------+------+
| id| A| B| C| D|
+---+--------+------+------+------+
| 1|1.000234|2.3456|4.6789|7.6934|
| 2| 3.7643|4.2323|5.6342| 8.567|
+---+--------+------+------+------+
scala> import org.apache.spark.sql.types._
import org.apache.spark.sql.types._
scala> val df2=df.columns.filter(_ !="id").foldLeft(df){ (acc,x) => acc.withColumn(x,col(x).cast(DecimalType(3,2))) }
df2: org.apache.spark.sql.DataFrame = [id: int, A: decimal(3,2) ... 3 more fields]
scala> df2.show(false)
+---+----+----+----+----+
|id |A |B |C |D |
+---+----+----+----+----+
|1 |1.00|2.35|4.68|7.69|
|2 |3.76|4.23|5.63|8.57|
+---+----+----+----+----+
scala>
聲明:本站的技術帖子網頁,遵循CC BY-SA 4.0協議,如果您需要轉載,請注明本站網址或者原文地址。任何問題請咨詢:yoyou2525@163.com.