如何将列表列的数组转换为 pyspark 中的结构

Question

I need to create a json string of non null columns. I created a new column col 3 as list or array with names of non null columns for each row. How do I create struct/json string using the column col3

post code	addr	col3
5678		[post code]
7877	San jose	[post code, addr]

Output expected

col4
{"post code": "5678"}
{"post code": "7877", "addr": "San jose"}

as the column is not iterable I created a string by replacing col3 values with col(names) for every element of aray and applied struct but it gave all array elements as 1 value.

As an alternate I created col4 using

to_json(struct(when (col1 not null,col1),when (col2 not null,col2)))

However wanted to know if we can do by using only col3

Answer 1

df1 = df.withColumn("New", F.to_json(F.struct(F.col('post code'),F.col('addr'))))
df1.show(2,False)

+---------+--------+--------------------------------------+
|post code|addr    |New                                   |
+---------+--------+--------------------------------------+
|5678     |null    |{"post code":"5678"}                  |
|7877     |San jose|{"post code":"7877","addr":"San jose"}|
+---------+--------+--------------------------------------+