简体   繁体   中英

How to construct a new column based on listwString> in java

I have a List list =["a","b","c"] and i have to add a new column to my dataframe, but first i have to construct it and this column have to be like:

x|y|z| list

strings in my list are columns, i mean that i have to construct my request like :

SELECT x,y,z, list FROM Dataframe

I tried to split strings in the list with

String.join("," , list) 

but it is seen like a singl column not multiple columns

Dataset<Row> df= dataframe.withColumn("NewColumn", concat(dataframe.col("x"), lit("|"), dataframe.col("y"),lit("|"), String.join(","list));

Note 1 : the size of my list is editable and the columns too Note 2 : i have to call String.join(","list) in my function withColumn, i don't have the choice

the expected result is a dataframe :

 ------------------------------------------------------------
  x     y     z     a     b     c     **NewColumn**
 ------------------------------------------------------------
 val1  val2  val3  val4  val5  val6   val1|val2|val3|val4|val5|val6
 -------------------------------------------------------------

I don't see how to construct my new column, thank you for your help

Function concat will take params of type org.apache.spark.sql.Column but you are passing list of type String .

You have to convert list of String into list of Column type and pass those values to concat

def concat(exprs: org.apache.spark.sql.Column*): org.apache.spark.sql.Column

Below code is in scala, You can convert it into java.

val list = List("a","b","c")

dataframe.withColumn(
    "NewColumn", 
    concat(
        col("x"), 
        lit("|"), 
        col("y"),
        lit("|"),
        list.map(c => col(c)):_* // I have added this, You may need to convert your list of strings into list of columns, It will work.
    )
);

The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.

 
粤ICP备18138465号  © 2020-2024 STACKOOM.COM