[英]unify the information of a user in a single row as a dataFrame using scala
dfFilter.show()
------------+-----------+-----------+------------+--------+
CONTR |COD | DATE |TYPCOD | Amount |
------------+-----------+-----------+------------+--------+
0004 |4433 |2006-11-04 |RMA | 150.0 |
0004 |4433 |2012-05-14 |FCB | 300.0 |
0004 |1122 |2011-10-17 |RMA | 100.0 |
0004 |1122 |2015-12-05 |FCB | 500.0 |
------------+-----------+-----------+------------+--------+
//
val addColumn = dfFilter.withColumn("RMA_AMOUNT", when(col("TYPCOD")==="RMA", col("Amount")))
.withColumn("DATE_RMA", when(col("TYPCOD")==="RMA", col("DATE")))
.withColumn("FCB_AMOUNT", when(col("TYPCOD")==="FCB", col("Amount")))
.withColumn("DATE_FCB", when(col("TYPCOD")==="FCB", col("DATE")))
addColumn.show()
--------+-----------+-----------+------------+--------+------------+-----------+-----------+-----------+
CONTR |COD | DATE |TYPCOD | Amount | RMA_AMOUNT |DATE_RMA |FCB_AMOUNT |DATE_FCB |
--------+-----------+-----------+------------+--------+------------+-----------+-----------+-----------+
0004 |4433 |2006-11-04 |RMA | 150.0 |150.0 |2006-11-04 |null |null |
0004 |4433 |2012-05-14 |FCB | 300.0 |null |null |300.0 |2012-05-14 |
0004 |1122 |2011-10-17 |RMA | 100.0 |100.0 |2011-10-17 |null |null |
0004 |1122 |2015-12-05 |FCB | 500.0 |null |null |500.0 |2015-12-05 |
--------+-----------+------------+-----------+--------+------------+-----------+-----------+-----------+
I have the same CONTR and COD but that client has different dates and amounts and I want to group them and keep two lines in the dataFrame, I have added columns in relation to the TYPCOD and DATE fields so that later I can only stay with two lines in the dataFrame and thus not lose information.我有相同的CONTR和COD ,但该客户有不同的日期和金额,我想将它们分组并在 dataFrame 中保留两行,我添加了与TYPCOD和DATE字段相关的列,以便以后我只能保留两个dataFrame 中的行,因此不会丢失信息。
it's possible?这是可能的? Expected: ?
预期的: ?
------------+-------------+------------+-----------+-----------+-----------+
CONTR |COD | RMA_AMOUNT |DATE_RMA |FCB_AMOUNT |DATE_FCB |
------------+-------------+------------+-----------+-----------+-----------+
0004 |4433 |150.0 |2006-11-04 |300.0 |2012-05-14 |
0004 |1122 |100.0 |2011-10-17 |500.0 |2015-12-05 |
------------+-------------+------------+-----------+-----------+-----------+
Yes, It's possible.是的,这是可能的。 Please check code below.
请检查下面的代码。
scala> val df = Seq(("0004",4433,"2006-11-04","RMA",150.0),("0004",4433,"2012-05-14","FCB",300.0),("0004",1122,"2011-10-17","RMA",100.0),("0004",1122,"2015-12-05","FCB",500.0)).toDF("contr","cod","date","typcod","amount")
df: org.apache.spark.sql.DataFrame = [contr: string, cod: int ... 3 more fields]
scala> val rma = df.filter($"typcod" === "RMA").select($"contr",$"cod",$"date".as("rma_date"),$"typcod",$"amount".as("rma_amount"))
rma: org.apache.spark.sql.DataFrame = [contr: string, cod: int ... 3 more fields]
scala> rma.show(false)
+-----+----+----------+------+----------+
|contr|cod |rma_date |typcod|rma_amount|
+-----+----+----------+------+----------+
|0004 |4433|2006-11-04|RMA |150.0 |
|0004 |1122|2011-10-17|RMA |100.0 |
+-----+----+----------+------+----------+
scala> val fcb = df.filter($"typcod" === "FCB").select($"contr",$"cod",$"date".as("fcb_date"),$"typcod",$"amount".as("fcb_amount")).drop("contr")
fcb: org.apache.spark.sql.DataFrame = [cod: int, fcb_date: string ... 2 more fields]
scala> fcb.show(false)
+----+----------+------+----------+
|cod |fcb_date |typcod|fcb_amount|
+----+----------+------+----------+
|4433|2012-05-14|FCB |300.0 |
|1122|2015-12-05|FCB |500.0 |
+----+----------+------+----------+
scala> rma.join(fcb,Seq("cod"),"inner").select("contr","cod","rma_amount","rma_date","fcb_amount","fcb_date").show(false)
+-----+----+----------+----------+----------+----------+
|contr|cod |rma_amount|rma_date |fcb_amount|fcb_date |
+-----+----+----------+----------+----------+----------+
|0004 |4433|150.0 |2006-11-04|300.0 |2012-05-14|
|0004 |1122|100.0 |2011-10-17|500.0 |2015-12-05|
+-----+----+----------+----------+----------+----------+
scala>
Use groupBy
then use first(col,ignoreNull=true)
functions for this case.使用
groupBy
然后在这种情况下使用first(col,ignoreNull=true)
函数。
val df=Seq(("0004","4433","2006-11-04","RMA","150.0","150.0","2006-11-04",null.asInstanceOf[String],null.asInstanceOf[String]),("0004","4433","2012-05-14","FCB","300.0",null.asInstanceOf[String],null.asInstanceOf[String],"300.0","2012-05-14"),("0004","1122","2011-10-17","RMA","100.0","100.0","2011-10-17",null.asInstanceOf[String],null.asInstanceOf[String]),("0004","1122","2015-12-05","FCB","500.0",null.asInstanceOf[String],null.asInstanceOf[String],"500.0","2015-12-05")).toDF("CONTR","COD","DATE","TYPCOD","Amount","RMA_AMOUNT","DATE_RMA","FCB_AMOUNT","DATE_FCB")
//+-----+----+----------+------+------+----------+----------+----------+----------+
//|CONTR| COD| DATE|TYPCOD|Amount|RMA_AMOUNT| DATE_RMA|FCB_AMOUNT| DATE_FCB|
//+-----+----+----------+------+------+----------+----------+----------+----------+
//| 0004|4433|2006-11-04| RMA| 150.0| 150.0|2006-11-04| null| null|
//| 0004|4433|2012-05-14| FCB| 300.0| null| null| 300.0|2012-05-14|
//| 0004|1122|2011-10-17| RMA| 100.0| 100.0|2011-10-17| null| null|
//| 0004|1122|2015-12-05| FCB| 500.0| null| null| 500.0|2015-12-05|
//+-----+----+----------+------+------+----------+----------+----------+----------+
df.groupBy("CONTR","COD").agg(first(col("RMA_AMOUNT"),true).alias("RMA_AMOUNT"),first(col("DATE_RMA"),true).alias("DATE_RMA"),first(col("FCB_AMOUNT"),true).alias("FCB_AMOUNT"),first(col("DATE_FCB"),true).alias("DATE_FCB")).show()
//+-----+----+----------+----------+----------+----------+
//|CONTR| COD|RMA_AMOUNT| DATE_RMA|FCB_AMOUNT| DATE_FCB|
//+-----+----+----------+----------+----------+----------+
//| 0004|4433| 150.0|2006-11-04| 300.0|2012-05-14|
//| 0004|1122| 100.0|2011-10-17| 500.0|2015-12-05|
//+-----+----+----------+----------+----------+----------+
//incase if you want to keep TYPCOD and DATE values
df.groupBy("CONTR","COD").agg(concat_ws(",",collect_list(col("TYPCOD"))).alias("TYPECOD"),concat_ws(",",collect_list(col("DATE"))).alias("DATE"),first(col("RMA_AMOUNT"),true).alias("RMA_AMOUNT"),first(col("DATE_RMA"),true).alias("DATE_RMA"),first(col("FCB_AMOUNT"),true).alias("FCB_AMOUNT"),first(col("DATE_FCB"),true).alias("DATE_FCB")).show(false)
//+-----+----+-------+---------------------+----------+----------+----------+----------+
//|CONTR|COD |TYPECOD|DATE |RMA_AMOUNT|DATE_RMA |FCB_AMOUNT|DATE_FCB |
//+-----+----+-------+---------------------+----------+----------+----------+----------+
//|0004 |4433|RMA,FCB|2006-11-04,2012-05-14|150.0 |2006-11-04|300.0 |2012-05-14|
//|0004 |1122|RMA,FCB|2011-10-17,2015-12-05|100.0 |2011-10-17|500.0 |2015-12-05|
//+-----+----+-------+---------------------+----------+----------+----------+----------+
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.