Pyspark指定列的默认值

Question

I'm using Spark 1.6.1 and Python 2.7

I'm trying to figure out how I can specify a default value for a newly added column to a dataframe. Take this scenario where I have a dataframe named df1 that contains

+-------+----+
|user_id| age|
+-------+----+
|  10000|  45|
|  10013|  40|
|  10021|Null|
|  10025|  50|
|  10051|  31|
+-------+----+

Now I want to add a new column called age2 that just has a simple function age + 1

>>> df1 = df1.withColumn("age2", df1["age"]+1)

+-------+----+----+
|user_id| age|age2|
+-------+----+----+
|  10000|  45|  46|
|  10013|  40|  41|
|  10021|Null|Null|
|  10025|  50|  51|
|  10051|  31|  32|
+-------+----+----+

Is there a way I can specify a default value for age2 so that instead of returning a null for age2 when there is a null in age, I can return something like 1 so that I get

+-------+----+----+
|user_id| age|age2|
+-------+----+----+
|  10000|  45|  46|
|  10013|  40|  41|
|  10021|Null|   1|
|  10025|  50|  51|
|  10051|  31|  32|
+-------+----+----+

I know that I can use a UDF to do this, but I want to know if there is a built in way to do it instead.

Answer 1

I would suggest you to use fillna function. Create a new column as you are doing it currently. Next fill the null values with the fillna function

>>> df1 = df1.withColumn("age2", df1["age"]+1)
>>> df1 = df1.na.fill({'age2': 1})