[英]Create an array column from other columns after processing the column values
Let's say I have a spark dataframe that includes the categorical columns (School, Type, Group) 假设我有一个包含类别列(学校,类型,组)的spark数据框
------------------------------------------------------------
StudentID | School | Type | Group
------------------------------------------------------------
1 | ABC | Elementary | Music-Arts
2 | ABC | Elementary | Football
3 | DEF | Secondary | Basketball-Cricket
4 | DEF | Secondary | Cricket
------------------------------------------------------------
I need to add one more column to the dataframe as below: 我需要向数据框添加另一列,如下所示:
--------------------------------------------------------------------------------------
StudentID | School | Type | Group | Combined Array
---------------------------------------------------------------------------------------
1 | ABC | Elementary | Music-Arts | ["School: ABC", "Type: Elementary", "Group: Music", "Group: Arts"]
2 | ABC | Elementary | Football | ["School: ABC", "Type: Elementary", "Group: Football"]
3 | DEF | Secondary | Basketball-Cricket | ["School: DEF", "Type: Secondary", "Group: Basketball", "Group: Cricket"]
4 | DEF | Secondary | Cricket | ["School: DEF", "Type: Secondary", "Group: Cricket"]
----------------------------------------------------------------------------------------
The extra column is combination of all categorical columns but includes a different processing on 'Group' column. 多余的列是所有类别列的组合,但在“组”列上包含不同的处理。 The values of 'Group' column need to be split on '-'.
“组”列的值需要在“-”上分割。
All the categorical columns including 'Group' are contained in a list. 包括“组”的所有分类列都包含在列表中。 The 'Group' column is also input as a String as the column to be split on.
“组”列也作为字符串输入,作为要拆分的列。 The data-frame has other columns which are not used.
数据框还有其他未使用的列。
I am looking for the best performance solution. 我正在寻找最佳性能的解决方案。
If it's a simple array, it can be done with a single 'withColumn' transformation. 如果它是一个简单的数组,则可以通过单个“ withColumn”转换来完成。
val columns = List("School", "Type", "Group")
var df2 = df1.withColumn("CombinedArray", array(columns.map(df1(_)):_*))
However, here because of the additional processing in 'Group' column, the solution doesn't seem straightforward. 但是,由于在“组”列中需要进行其他处理,因此该解决方案似乎并不简单。
Using the spark.sql(), Check this out: 使用spark.sql(),检查以下内容:
Seq(("ABC","Elementary","Music-Arts"),("ABC","Elementary","Football"),("DEF","Secondary","Basketball-Cricket"),("DEF","Secondary","Cricket"))
.toDF("School","Type","Group").createOrReplaceTempView("taba")
spark.sql( """ select school, type, group, array(concat('School:',school),concat('type:',type),concat('group:',group)) as combined_array from taba """).show(false)
Output: 输出:
+------+----------+------------------+------------------------------------------------------+
|school|type |group |combined_array |
+------+----------+------------------+------------------------------------------------------+
|ABC |Elementary|Music-Arts |[School:ABC, type:Elementary, group:Music-Arts] |
|ABC |Elementary|Football |[School:ABC, type:Elementary, group:Football] |
|DEF |Secondary |Basketball-Cricket|[School:DEF, type:Secondary, group:Basketball-Cricket]|
|DEF |Secondary |Cricket |[School:DEF, type:Secondary, group:Cricket] |
+------+----------+------------------+------------------------------------------------------+
If you need it as a dataframe, then 如果需要将其用作数据框,则
val df = spark.sql( """ select school, type, group, array(concat('School:',school),concat('type:',type),concat('group:',group)) as combined_array from taba """)
df.printSchema()
root
|-- school: string (nullable = true)
|-- type: string (nullable = true)
|-- group: string (nullable = true)
|-- combined_array: array (nullable = false)
| |-- element: string (containsNull = true)
Update: 更新:
Dynamically constructing the sql columns. 动态构造sql列。
scala> val df = Seq(("ABC","Elementary","Music-Arts"),("ABC","Elementary","Football"),("DEF","Secondary","Basketball-Cricket"),("DEF","Secondary","Cricket")).toDF("School","Type","Group")
df: org.apache.spark.sql.DataFrame = [School: string, Type: string ... 1 more field]
scala> val columns = df.columns.mkString("select ", ",", "")
columns: String = select School,Type,Group
scala> val arr = df.columns.map( x=> s"concat('"+x+"',"+x+")" ).mkString("array(",",",") as combined_array ")
arr: String = "array(concat('School',School),concat('Type',Type),concat('Group',Group)) as combined_array "
scala> val sql_string = columns + " , " + arr + " from taba "
sql_string: String = "select School,Type,Group , array(concat('School',School),concat('Type',Type),concat('Group',Group)) as combined_array from taba "
scala> df.createOrReplaceTempView("taba")
scala> spark.sql(sql_string).show(false)
+------+----------+------------------+---------------------------------------------------+
|School|Type |Group |combined_array |
+------+----------+------------------+---------------------------------------------------+
|ABC |Elementary|Music-Arts |[SchoolABC, TypeElementary, GroupMusic-Arts] |
|ABC |Elementary|Football |[SchoolABC, TypeElementary, GroupFootball] |
|DEF |Secondary |Basketball-Cricket|[SchoolDEF, TypeSecondary, GroupBasketball-Cricket]|
|DEF |Secondary |Cricket |[SchoolDEF, TypeSecondary, GroupCricket] |
+------+----------+------------------+---------------------------------------------------+
scala>
Update2: UPDATE2:
scala> val df = Seq((1,"ABC","Elementary","Music-Arts"),(2,"ABC","Elementary","Football"),(3,"DEF","Secondary","Basketball-Cricket"),(4,"DEF","Secondary","Cricket")).toDF("StudentID","School","Type","Group")
df: org.apache.spark.sql.DataFrame = [StudentID: int, School: string ... 2 more fields]
scala> df.createOrReplaceTempView("student")
scala> val df2 = spark.sql(""" select studentid, collect_list(concat('Group:', t.sp1)) as sp2 from (select StudentID,School,Type,explode((split(group,'-'))) as sp1 from student where size(split(group,'-')) > 1 ) t group by studentid """)
df2: org.apache.spark.sql.DataFrame = [studentid: int, sp2: array<string>]
scala> val df3 = df.alias("t1").join(df2.alias("t2"),Seq("studentid"),"LeftOuter")
df3: org.apache.spark.sql.DataFrame = [StudentID: int, School: string ... 3 more fields]
scala> df3.createOrReplaceTempView("student2")
scala> spark.sql(""" select studentid, school,group, type, array(concat('School:',school),concat('type:',type),concat_ws(',',temp_arr)) from (select studentid,school,group,type, case when sp2 is null then array(concat("Group:",group)) else sp2 end as temp_arr from student2) t """).show(false)
+---------+------+------------------+----------+---------------------------------------------------------------------------+
|studentid|school|group |type |array(concat(School:, school), concat(type:, type), concat_ws(,, temp_arr))|
+---------+------+------------------+----------+---------------------------------------------------------------------------+
|1 |ABC |Music-Arts |Elementary|[School:ABC, type:Elementary, Group:Music,Group:Arts] |
|2 |ABC |Football |Elementary|[School:ABC, type:Elementary, Group:Football] |
|3 |DEF |Basketball-Cricket|Secondary |[School:DEF, type:Secondary, Group:Basketball,Group:Cricket] |
|4 |DEF |Cricket |Secondary |[School:DEF, type:Secondary, Group:Cricket] |
+---------+------+------------------+----------+---------------------------------------------------------------------------+
scala>
You need to first add an empty column then map it like so (in Java): 您需要首先添加一个空列,然后按如下方式映射它(在Java中):
StructType newSchema = df1.schema().add("Combined Array", DataTypes.StringType);
df1 = df1.withColumn("Combined Array", lit(null))
.map((MapFunction<Row, Row>) row ->
RowFactory.create(...values...) // add existing values and new value here
, newSchema);
It should be fairly similar in Scala. 它在Scala中应该非常相似。
Using regex replacement to start of each field and to "-" in between: 使用正则表达式替换来开始每个字段,并在两者之间使用“-”:
val df1 = spark.read.option("header","true").csv(filePath)
val columns = List("School", "Type", "Group")
var df2 = df1.withColumn("CombinedArray", array(columns.map{
colName => regexp_replace(regexp_replace(df1(colName),"(^)",s"$colName: "),"(-)",s", $colName: ")
}:_*))
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.