I have a List list =["a","b","c"] and i have to add a new column to my dataframe, but first i have to construct it and this column have to be like:
x|y|z| list
strings in my list are columns, i mean that i have to construct my request like :
SELECT x,y,z, list FROM Dataframe
I tried to split strings in the list with
String.join("," , list)
but it is seen like a singl column not multiple columns
Dataset<Row> df= dataframe.withColumn("NewColumn", concat(dataframe.col("x"), lit("|"), dataframe.col("y"),lit("|"), String.join(","list));
Note 1 : the size of my list is editable and the columns too Note 2 : i have to call String.join(","list) in my function withColumn, i don't have the choice
the expected result is a dataframe :
------------------------------------------------------------
x y z a b c **NewColumn**
------------------------------------------------------------
val1 val2 val3 val4 val5 val6 val1|val2|val3|val4|val5|val6
-------------------------------------------------------------
I don't see how to construct my new column, thank you for your help
Function concat
will take params of type org.apache.spark.sql.Column
but you are passing list of type String
.
You have to convert list of String
into list of Column
type and pass those values to concat
def concat(exprs: org.apache.spark.sql.Column*): org.apache.spark.sql.Column
Below code is in scala, You can convert it into java.
val list = List("a","b","c")
dataframe.withColumn(
"NewColumn",
concat(
col("x"),
lit("|"),
col("y"),
lit("|"),
list.map(c => col(c)):_* // I have added this, You may need to convert your list of strings into list of columns, It will work.
)
);
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.