Concatenate columns to list of columns in Apache Spark DataFrame

Question

Is there a way in spark-sql to add a new column in a dataframe such that:

• the new column contains arrays of strings
• each string is the value of another column

Basically this is the same thing as the concat function except that the concatenation returns a list of strings instead of a big string seperated by an identifier

NB : I'm using the python API

Answer 1

you can use the built-in array function. In scala this would look like:

df
.withColumn("col_arr",array(df.columns.map(c => col(c)):_*))