[英]Fill a list/pandas.dataframe with all the missing data combinations (like complete() in R)
I have data set like the following (this is an example, it actually has 66k rows):我有如下数据集(这是一个例子,它实际上有 66k 行):
Type Food Loc Num
0 Fruit Banana House-1 15
1 Fruit Banana House-2 4
2 Fruit Apple House-2 6
3 Fruit Apple House-3 8
4 Vegetable Broccoli House-3 8
5 Vegetable Lettuce House-4 12
6 Vegetable Peppers House-5 3
7 Vegetable Corn House-4 4
8 Seasoning Olive Oil House-6 2
9 Seasoning Vinegar House-7 2
I'd like to fill all the missing combinations (how many bananas are there in Houses 3-7?, how many peppers are there elsewhere than House-5?) with 0, to get something like:我想用 0 填充所有缺失的组合(3-7 号房子里有多少香蕉?5 号房子以外的其他地方有多少辣椒?),得到类似的结果:
Type Food Loc Num
0 Fruit Banana House-1 15
1 Fruit Banana House-2 4
2 Fruit Banana House-3 0
... fill remaining houses with zeros
6 Fruit Banana House-7 0
7 Fruit Apple House-1 0
8 Fruit Apple House-2 6
9 Fruit Apple House-3 8
... fill remaining houses with zeros
14 Vegetable Broccoli House-1 0
15 Vegetable Broccoli House-2 0
16 Vegetable Broccoli House-3 8
... etc
n Seasoning Vinegar House-7 2
I know that R has the complete
function integrated.我知道R集成了
complete
功能。
Right now I had been working with a list that was digested from the original DataFrame, which I transformed into a dictionary.现在我一直在处理从原始 DataFrame 中提取的列表,我将其转换为字典。
for key,grp in fruit.groupby(level=0):
dir[key] = test.ix[key].values.tolist()
fruit = {'Banana': [[1.0,15.0], [2.0,4.0],
'Apple': [[2.0,6.0], [3.0,8.0]
#Type = {fruit1:[[Loc1,Count1],...,[Locn],[Countn],
#... fruitn:[...]}
I designed this function to apply to the assignation rule of the dictionary:我设计了这个函数来应用于字典的分配规则:
def fill_zeros(list):
final = [0] * 127
for i in list:
final[int(i[0])] = i[1]
return final
That works on individual "fruits":这适用于单个“水果”:
print fill_zeros(test.ix['QLLSEEEKK'].values.tolist())
print fill_zeros(test.ix['GAVPLEMLEIALR'].values.tolist())
print fill_zeros(test.ix['VPVNLLNSPDCDVK'].values.tolist())
But doesn't on the dictionary:但字典上没有:
for key,grp in test.groupby(level=0):
dir[key] = fill_zeros(test.ix[key].values.tolist())
Traceback (most recent call last):
File "peptidecount.py", line 59, in <module>
print fill_zeros(test.ix[str(key)].values.tolist())
File "peptidecount.py", line 43, in fill_zeros
final[int(i[0])] = i[1]
TypeError: 'float' object has no attribute '__getitem__'
Apparently I'm not iterating correctly on the dictionary.显然我没有在字典上正确迭代。 Is there a way to correct that?
有没有办法纠正? Or is there a more suitable function to apply directly on the DataFrame?
或者有更合适的函数可以直接应用于DataFrame?
You could use a reindex
.您可以使用
reindex
。
First you'll need a list of the valid (type, food)
pairs.首先,您需要一个有效
(type, food)
对的列表。 I'll get it from the data itself, rather than writing them out.我将从数据本身中获取它,而不是将它们写出来。
In [88]: kinds = list(df[['Type', 'Food']].drop_duplicates().itertuples(index=False))
In [89]: kinds
Out[89]:
[('Fruit', 'Banana'),
('Fruit', 'Apple'),
('Vegetable', 'Broccoli'),
('Vegetable', 'Lettuce'),
('Vegetable', 'Peppers'),
('Vegetable', 'Corn'),
('Seasoning', 'Olive Oil'),
('Seasoning', 'Vinegar')]
Now we'll generate all the pairs for those kinds
with the houses using itertools.product
.现在我们将使用
itertools.product
为这些kinds
的房子生成所有对。
In [93]: from itertools import product
In [94]: houses = ['House-%s' % x for x in range(1, 8)]
In [95]: idx = [(x.Type, x.Food, house) for x, house in product(kinds, houses)]
In [96]: idx[:2]
Out[96]: [('Fruit', 'Banana', 'House-1'), ('Fruit', 'Banana', 'House-2')]
And now you can use set_index
and reindex
to get the missing observations.现在您可以使用
set_index
和reindex
来获取缺失的观察结果。
In [98]: df.set_index(['Type', 'Food', 'Loc']).reindex(idx, fill_value=0)
Out[98]:
Num
Type Food Loc
Fruit Banana House-1 15
House-2 4
House-3 0
House-4 0
House-5 0
... ...
Seasoning Vinegar House-3 0
House-4 0
House-5 0
House-6 0
House-7 2
[56 rows x 1 columns]
This should work:这应该有效:
cond0 = df.Num.isnull()
cond1 = df.Food == 'Banana'
cond2 = df.Loc.str.match(r'House-[34567]')
cond3 = df.Food == 'Peppers'
cond4 = df.Loc != 'House-5'
missing_bananas = cond0 & cond1 & cond2
missing_peppers = cond0 & cond3 & cond4
missing_food = missing_bananas | missing_peppers
df.loc[missing_food] = df.loc[missing_food].fillna(0)
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.