how to concat all columns in a spark dataframe, using java?

Question

This is how I do do for 2 specific columns:

dataSet.withColumn("colName", concat(dataSet.col("col1"), lit(","),dataSet.col("col2") ));

but dataSet.columns() retruns Sting array, and not Column array. How should I craete a List<Column> ?

Thanks!

Answer 1

Simple Way - Instead of df.columns use concat_ws(",","*") , Check below code.

df.withColumn("colName",expr("concat_ws(',',*)")).show(false)
+---+--------+---+-------------+
|id |name    |age|colName      |
+---+--------+---+-------------+
|1  |Srinivas|29 |1,Srinivas,29|
|2  |Ravi    |30 |2,Ravi,30    |
+---+--------+---+-------------+

Answer 2

This is how I do do for 2 specific columns:

dataSet.withColumn("colName", concat(dataSet.col("col1"), lit(","),dataSet.col("col2") ));

but dataSet.columns() retruns Sting array, and not Column array. How should I craete a List<Column> ?

Thanks!

Answer 3

Java has more verbose syntax. Try this -

 df.withColumn("colName",concat_ws(",", toScalaSeq(Arrays.stream(df.columns()).map(functions::col).collect(Collectors.toList()))));

Use below utility to convert java list to scala seq-

  <T> Buffer<T> toScalaSeq(List<T> list) {
        return JavaConversions.asScalaBuffer(list);
    }

Answer 4

If someone is looking for a way to concat all the columns of a DataFrame in Scala, this is what worked for me:

val df_new = df.withColumn(new_column_name, concat_ws("-", df.columns.map(col): _*))