PYSPARK在重复的列名称中设置默认值

Question

In pyspark, I have a dataframe with 10 columns like this :

id, last_name, first_name, manager, shop, location, manager, place, country, status

i would like to set a default value to only the first column manager, i've tried with :

df.withColumn("manager", "x1")

but it gives me an error for ambiguous reference as there is 2 columns with the same name.

Is there a way to do it without renaming the column ?

Answer 1

One work around can be,to recreate the dataframe changing the column names. It's always better to have unique column names.

>>> df = spark.createDataFrame([('value1','value2'),('value3','value4')],['manager','manager'])
>>> df.show()
+-------+-------+
|manager|manager|
+-------+-------+
| value1| value2|
| value3| value4|
+-------+-------+
>>> df1 = df.toDF('manager1','manager2')

>>> from pyspark.sql.functions import lit
>>> df1.withColumn('manager1',lit('x1')).show()
+--------+--------+
|manager1|manager2|
+--------+--------+
|      x1|  value2|
|      x1|  value4|
+--------+--------+