[英]how to add rows to a data frame that are in another data frame by a column in pyspark
[英]How to add column to a data frame?
我有以下代碼:
db_fields = ("id", "email", "status", "source")
df = DataFrame(results)
for col in db_fields:
if col not in df.columns:
COLUMN IS MISSING - COMMAND TO ADD COLUMN
例如,如果缺少status
列,則需要將其添加到數據框中,而無需添加任何值。因此,當我將df
導出到csv
我將始終具有相同的字段架構。
我知道要刪除列,我應該這樣做:
df = df.drop(col, 1)
但是我不知道添加具有空值的列的最佳方法是什么。
此方法將在狀態列中添加Null值:
import numpy as np
df['status'] = np.nan
或者:
df['status'] = None
所以:
db_fields = ("id", "email", "status", "source")
for col in db_fields:
if col not in df.columns:
df[col] = None
您可以創建不存在的列的數組,並使用assign
和dictionary創建新的列:
df = pd.DataFrame({'id': ['a1','a2', 'b1'],
'a': ['a1','a2', 'b1'],
'source': ['a1','a2', 'b1']})
print (df)
id a source
0 a1 a1 a1
1 a2 a2 a2
2 b1 b1 b1
db_fields = ("id", "email", "status", "source")
#get missing columns
diff = np.setdiff1d(np.array(db_fields), df.columns)
print (diff)
['email' 'status']
#get original columns not existed in db_fields
diff1 = np.setdiff1d(df.columns, np.array(db_fields)).tolist()
print (diff1)
['a']
#add missing columns with change order
d = dict.fromkeys(diff, np.nan)
df = df.assign(**d)[diff1 + list(db_fields)]
print (df)
a id email status source
0 a1 a1 NaN NaN a1
1 a2 a2 NaN NaN a2
2 b1 b1 NaN NaN b1
#if necessary first db_fields
df = df.assign(**d)[list(db_fields) + diff1]
print (df)
id email status source a
0 a1 NaN NaN a1 a1
1 a2 NaN NaN a2 a2
2 b1 NaN NaN b1 b1
在這里,只需一行就可以簡單明了地看到它:
import numpy as np
db_fields = ("id", "email", "status", "source")
df = DataFrame(results)
for col in db_fields:
if col not in df.columns:
# Add the column
df[col] = np.nan
順便說一句:您也可以使用df.drop(inplace=True)
刪除列。
聲明:本站的技術帖子網頁,遵循CC BY-SA 4.0協議,如果您需要轉載,請注明本站網址或者原文地址。任何問題請咨詢:yoyou2525@163.com.