Pandas 数据帧上的矢量化查找

Question

我有两个 DataFrames 。 . .

df1是一个表，我需要从使用索引、从 df2 中的多个列检索的列对中提取值。

我看到有一个函数get_value在给定索引和列值时可以完美地工作，但是当尝试向量化这个函数以创建一个新列时，我失败了......

df1 = pd.DataFrame(np.arange(20).reshape((4, 5)))

df1.columns = list('abcde')

df1.index = ['cat', 'dog', 'fish', 'bird']

        a   b   c   d   e
cat     0   1   2   3   4
dog     5   6   7   8   9
fish    10  11  12  13  14
bird    15  16  17  18  19

df1.get_value('bird, 'c')

17

现在我需要做的是在df2上创建一个完整的新列——当基于索引对df1进行索引时，来自animal列对， df2指定的letter列有效地矢量化了上面的pd.get_value函数。

df2 = pd.DataFrame(np.arange(20).reshape((4, 5)))

df2['animal'] = ['cat', 'dog', 'fish', 'bird']

df2['letter'] = list('abcd')

    0   1   2   3   4   animal  letter
0   0   1   2   3   4   cat     a
1   5   6   7   8   9   dog     b
2   10  11  12  13  14  fish    c
3   15  16  17  18  19  bird    d

导致 。 . .

    0   1   2   3   4   animal  letter   looked_up
0   0   1   2   3   4   cat     a        0
1   5   6   7   8   9   dog     b        6
2   10  11  12  13  14  fish    c        12
3   15  16  17  18  19  bird    d        18

Answer 1

有一个名为lookup的函数正是这样做的。

df2['looked_up'] = df1.lookup(df2.animal, df2.letter)

df2

    0   1   2   3   4 animal letter  looked_up
0   0   1   2   3   4    cat      a          0
1   5   6   7   8   9    dog      b          6
2  10  11  12  13  14   fish      c         12
3  15  16  17  18  19   bird      d         18

Answer 2

如果寻找更快的方法，那么 zip 将有助于在小数据帧的情况下，即

k = list(zip(df2['animal'].values,df2['letter'].values))
df2['looked_up'] = [df1.get_value(*i) for i in k]

输出：

0   1   2   3   4 animal letter  looked_up
0   0   1   2   3   4    cat      a          0
1   5   6   7   8   9    dog      b          6
2  10  11  12  13  14   fish      c         12
3  15  16  17  18  19   bird      d         18

正如约翰所建议的那样，您可以简化代码，这样会更快。

 df2['looked_up'] = [df1.get_value(r, c) for r, c in zip(df2.animal, df2.letter)]

如果缺少数据，请使用 if else ie

df2['looked_up'] = [df1.get_value(r, c) if not pd.isnull(c) | pd.isnull(r) else pd.np.nan for r, c in zip(df2.animal, df2.letter) ]

对于小数据帧

%%timeit
df2['looked_up'] = df1.lookup(df2.animal, df2.letter)
1000 loops, best of 3: 801 µs per loop

k = list(zip(df2['animal'].values,df2['letter'].values))
df2['looked_up'] = [df1.get_value(*i) for i in k]
1000 loops, best of 3: 399 µs per loop

[df1.get_value(r, c) for r, c in zip(df2.animal, df2.letter)]
10000 loops, best of 3: 87.5 µs per loop

对于大数据框

df3 = pd.concat([df2]*10000)

%%timeit
k = list(zip(df3['animal'].values,df3['letter'].values))
df2['looked_up'] = [df1.get_value(*i) for i in k]
1 loop, best of 3: 185 ms per loop


df2['looked_up'] = [df1.get_value(r, c) for r, c in zip(df3.animal, df3.letter)]
1 loop, best of 3: 165 ms per loop

df2['looked_up'] = df1.lookup(df3.animal, df3.letter)
100 loops, best of 3: 8.82 ms per loop

Answer 3

如果您的值存在于查找数据框中， lookup和get_value是很好的答案。

但是，如果您在查找数据框中不存在（行，列）对，并且希望查找值为NaN - merge和stack是一种方法

In [206]: df2.merge(df1.stack().reset_index().rename(columns={0: 'looked_up'}),
                    left_on=['animal', 'letter'], right_on=['level_0', 'level_1'],
                    how='left').drop(['level_0', 'level_1'], 1)
Out[206]:
    0   1   2   3   4 animal letter  looked_up
0   0   1   2   3   4    cat      a          0
1   5   6   7   8   9    dog      b          6
2  10  11  12  13  14   fish      c         12
3  15  16  17  18  19   bird      d         18

添加不存在的（动物、字母）对进行测试

In [207]: df22
Out[207]:
      0     1     2     3     4 animal letter
0   0.0   1.0   2.0   3.0   4.0    cat      a
1   5.0   6.0   7.0   8.0   9.0    dog      b
2  10.0  11.0  12.0  13.0  14.0   fish      c
3  15.0  16.0  17.0  18.0  19.0   bird      d
4   NaN   NaN   NaN   NaN   NaN  dummy    NaN

In [208]: df22.merge(df1.stack().reset_index().rename(columns={0: 'looked_up'}),
                    left_on=['animal', 'letter'], right_on=['level_0', 'level_1'],
                    how='left').drop(['level_0', 'level_1'], 1)
Out[208]:
      0     1     2     3     4 animal letter  looked_up
0   0.0   1.0   2.0   3.0   4.0    cat      a        0.0
1   5.0   6.0   7.0   8.0   9.0    dog      b        6.0
2  10.0  11.0  12.0  13.0  14.0   fish      c       12.0
3  15.0  16.0  17.0  18.0  19.0   bird      d       18.0
4   NaN   NaN   NaN   NaN   NaN  dummy    NaN        NaN