如何更新结构字段 spark/scala

Question

I have a struct as part of my json .

store: struct (c1, c2, c3, c4)

I would like to update c2 in place so no new field is created. After update it should be same struct with new value for c2 .

In spark/scala , I have already tried:

df.withColumn("store.c2", newVal)

But this creates a new field store.c2 , Columns not part of struct, I am able to update.

df.withColumn("columnTen", newValue)

does not create new field and updated to newValue .

Answer 1

you can create a new structtype based on an existing schema by using the following:

val newstructtype = StructType(existing_df.schema.map( x => {
if (x.name == "fieldname_to_change") StructField("new_fieldname",TimestampType,true)
else x})

Answer 2

doing this

df.withColumn("store",struct($"store.c1", $"store.c2", $"store.c3", lit(newValue) as "c4"))

will replace the value in the store.c4 field.

Answer 3

Since Spark 3.1+, you can use withField on a struct column:

An expression that adds/replaces field in StructType by name.

df.withColumn("store", $"store".withField("c2", lit(newVal)))