使用 Colum 将列表传递给 dataframe 中的 udf

Question

I am building dataframe from hive table where i need to transform column based on multiple columns in dataframe, for that i built udf and passing kwargs however i doubt the order of the kwargs gets changed as the order is important. So i decided to use List but i am exploring how can we pass multiple columns as list from dataframe transformation.

function:

val = ''
@udf(returnType = StringType())
def func(list):
   for i in list
      val = val + i
   return val

df = df.withColumn(new_col,func(df["col1"],df["col2"],df["col3"])
df.show()

Answer 1

the below dynamic column approach might solve your problem.

from pyspark.sql.functions import concat
# Creating an example DataFrame
values = [('A1',11,'A3','A4'),('B1',22,'B3','B4'),('C1',33,'C3','C4')]
df = spark.createDataFrame(values,['col1','col2','col3','col4'])
df.show()

'''
+----+----+----+----+
|col1|col2|col3|col4|
+----+----+----+----+
|  A1|  11|  A3|  A4|
|  B1|  22|  B3|  B4|
|  C1|  33|  C3|  C4|
+----+----+----+----+
'''
 
col_list = ['col1','col2']
df = df.withColumn('concatenated_cols2',concat(*col_list))
col_list = ['col1','col2','col3']
df = df.withColumn('concatenated_cols3',concat(*col_list))
col_list = ['col1','col2','col3','col4']
df = df.withColumn('concatenated_cols4',concat(*col_list))
df.show()

'''
+----+----+----+----+------------------+------------------+------------------+
|col1|col2|col3|col4|concatenated_cols2|concatenated_cols3|concatenated_cols4|
+----+----+----+----+------------------+------------------+------------------+
|  A1|  11|  A3|  A4|              A111|            A111A3|          A111A3A4|
|  B1|  22|  B3|  B4|              B122|            B122B3|          B122B3B4|
|  C1|  33|  C3|  C4|              C133|            C133C3|          C133C3C4|
+----+----+----+----+------------------+------------------+------------------+
'''

Answer 2

Thanks Smart_Coder. and Sorry for the delay in getting back to you. Let me give your the full requirement. i will take the dataframe as you mentioned above an example. and i will take 3 columns as input(It should be dynamic but will take these fro now). col1, col2, col3 are the input columns to function.columns values should move from right to left in case of null or empty values. extension of the requirement: then i need to check the count of characters in each value and take only specific no. of characters into that column and rest of the columns should go into next column, if still it is ore than specific number of characters then rest will go into next column. However we need only 3 columns/elements as output.

col1 col2 col2
ASDF QWER NMVB
     QWER NMVB
ASD       NMVB

for suppose i need only 3 characters in each field max.
output will be:
col1 col2 col3
ASD  F    QWE
QWE  R   NMV
ASD  NMV