[英]Selecting values from non-null columns in a PySpark DataFrame
There is a pyspark dataframe with missing values: 有一个缺少值的pyspark数据框:
tbl = sc.parallelize([
Row(first_name='Alice', last_name='Cooper'),
Row(first_name='Prince', last_name=None),
Row(first_name=None, last_name='Lenon')
]).toDF()
tbl.show()
Here's the table: 这是表格:
+----------+---------+
|first_name|last_name|
+----------+---------+
| Alice| Cooper|
| Prince| null|
| null| Lenon|
+----------+---------+
I would like to create a new column as follows: 我想创建一个新列,如下所示:
I can construct a simple function: 我可以构造一个简单的函数:
def combine_data(row):
if row.last_name is None:
return row.first_name
elif row.first_name is None:
return row.last_name
else:
return '%s %s' % (row.first_name, row.last_name)
tbl.map(combine_data).collect()
I do get the correct result, but I can't append it to the table as a column: tbl.withColumn('new_col', tbl.map(combine_data))
results in AssertionError: col should be Column
我得到了正确的结果,但我无法将其作为列附加到表中:
tbl.withColumn('new_col', tbl.map(combine_data))
导致AssertionError: col should be Column
What is the best way to convert the result of map
to a Column
? 将
map
结果转换为Column
的最佳方法是什么? Is there a preferred way to deal with null
values? 有没有一种处理
null
值的首选方法?
As always it is best to operate directly on native representation instead of fetching data to Python: 一如既往,最好直接在本机表示上操作,而不是将数据提取到Python:
from pyspark.sql.functions import concat_ws, coalesce, lit, trim
def combine(*cols):
return trim(concat_ws(" ", *[coalesce(c, lit("")) for c in cols]))
tbl.withColumn("foo", combine("first_name", "last_name")).
You just need to use a UDF that receives two columns
as arguments. 您只需要使用接收两
columns
作为参数的UDF 。
from pyspark.sql.functions import *
from pyspark.sql import Row
tbl = sc.parallelize([
Row(first_name='Alice', last_name='Cooper'),
Row(first_name='Prince', last_name=None),
Row(first_name=None, last_name='Lenon')
]).toDF()
tbl.show()
def combine(c1, c2):
if c1 != None and c2 != None:
return c1 + " " + c2
elif c1 == None:
return c2
else:
return c1
combineUDF = udf(combine)
expr = [c for c in ["first_name", "last_name"]] + [combineUDF(col("first_name"), col("last_name")).alias("full_name")]
tbl.select(*expr).show()
#+----------+---------+------------+
#|first_name|last_name| full_name|
#+----------+---------+------------+
#| Alice| Cooper|Alice Cooper|
#| Prince| null| Prince|
#| null| Lenon| Lenon|
#+----------+---------+------------+
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.