熊貓應用於數據框分組
[英]Pandas apply to data frame groupby
問題描述
如果我分組（下面的g對象），然后將以下函數應用於df的前1000行，則它可以工作。 但是，如果將其應用於整個df，則會出現以下異常：
def calc_load(x):
...: x.sort('log_timestamp')
...: x['time_stddev'] = x['time'].std()
...: x['time_mean'] = x['time'].mean()
...: return x
...:
c=g.apply(calc_load)
---------------------------------------------------------------------------
........
ValueError Traceback (most recent call last)
<ipython-input-262-f2fe1f013907> in <module>()
----> 1 c=g.apply(calc_load)
2215 tuple(map(int, [tot_items] + list(block_shape))),
-> 2216 tuple(map(int, [len(ax) for ax in axes]))))
2217
2218
ValueError: Shape of passed values is (10, 3943482), indices imply (10, 410450)
這是什么原因，我該如何解決？
更新：
我正在從HDF5商店讀取此表，如下所示：
prob2
Out[374]:
<class 'pandas.io.pytables.HDFStore'>
File path: /tmp/test2.h5
/mytable frame_table (typ->appendable,nrows->410450,ncols->8,indexers->[index])
a=prob2.mytable
a
Out[376]:
<class 'pandas.core.frame.DataFrame'>
Int64Index: 410450 entries, 0 to 9999
Data columns (total 8 columns):
args 410450 non-null values
host 410450 non-null values
kwargs 410450 non-null values
log_timestamp 410450 non-null values
operation 410450 non-null values
slot 410450 non-null values
status 410450 non-null values
time 410450 non-null values
dtypes: float64(1), int64(2), object(5)
如果我像下面這樣往返於CSV，則不會發生異常：
a.to_csv('/tmp/test2.csv')
b=pd.read_csv('/tmp/test2.csv')
b
Out[379]:
<class 'pandas.core.frame.DataFrame'>
Int64Index: 410450 entries, 0 to 410449
Data columns (total 9 columns):
Unnamed: 0 410450 non-null values
args 410450 non-null values
host 410450 non-null values
kwargs 410450 non-null values
log_timestamp 410450 non-null values
operation 410450 non-null values
slot 410450 non-null values
status 410450 non-null values
time 410450 non-null values
dtypes: float64(1), int64(3), object(5)
bg = b.groupby(['host','operation'])
bg.apply(calc_load)
Out[381]:
<class 'pandas.core.frame.DataFrame'>
Int64Index: 410450 entries, 0 to 410449
Data columns (total 11 columns):
Unnamed: 0 410450 non-null values
args 410450 non-null values
host 410450 non-null values
kwargs 410450 non-null values
log_timestamp 410450 non-null values
operation 410450 non-null values
slot 410450 non-null values
status 410450 non-null values
time 410450 non-null values
time_stddev 410371 non-null values
time_mean 410450 non-null values
dtypes: float64(3), int64(3), object(5)
往返（a）之前和往返（b）之后的數據幀看起來相似，但不相同！
a
Out[386]:
<class 'pandas.core.frame.DataFrame'>
Int64Index: 410450 entries, 0 to 9999
Data columns (total 8 columns):
args 410450 non-null values
host 410450 non-null values
kwargs 410450 non-null values
log_timestamp 410450 non-null values
operation 410450 non-null values
slot 410450 non-null values
status 410450 non-null values
time 410450 non-null values
dtypes: float64(1), int64(2), object(5)
b
Out[387]:
<class 'pandas.core.frame.DataFrame'>
Int64Index: 410450 entries, 0 to 410449
Data columns (total 9 columns):
Unnamed: 0 410450 non-null values
args 410450 non-null values
host 410450 non-null values
kwargs 410450 non-null values
log_timestamp 410450 non-null values
operation 410450 non-null values
slot 410450 non-null values
status 410450 non-null values
time 410450 non-null values
dtypes: float64(1), int64(3), object(5)
嗯，這是怎么回事？
1 個解決方案
解決方案1
4 已采納 2013-12-20 13:56:52
按主機/操作分組后，您將有許多重復項。 這就是為什么前1000行概率有效但整個集合無效的原因。
首先重置索引，然后分組並應用。 您可以通過在末尾設置索引來恢復原始索引。 重置索引被轉換為稱為“索引”的列（然后set_index刪除）。
實際上，這是一個相當普遍的模式。 我認為可能是更有用的錯誤消息，請參閱此處 。 由於我不確定groupby應該自動修復（可以）。 因為這可能是用戶錯誤或意圖。
In [26]: df = d.reset_index().groupby(['host','operation']).apply(calc_load).set_index('index')
In [27]: df
Out[27]:
args host kwargs log_timestamp operation slot status time time_stddev time_mean
index
0 [] yy3.segm1.org {} 1385984306000000000 x_gWidgboxParams a12yy3 -101 0.000477 0.061657 0.003226
1 [] yy14.segm1.org {} 1385984306000000000 x_initWidgbox a11yy14 1 0.004177 0.035759 0.005816
10 [] yy32.segm1.org {} 1385984307000000000 gSettings a13yy32 -101 0.009686 0.245170 0.070137
100 [] yy19.segm1.org {} 1385984308000000000 notifyTestsDelivered a16yy19 1 0.000766 0.002825 0.000964
1000 [] yy7.segm1.org {} 1385984320000000000 addWidging2 a12yy7 1 0.002576 0.008525 0.004122
10000 [] yy14.segm1.org {} 1385984461000000000 addWidging2 a13yy14 1 0.001317 0.009431 0.003910
10001 [] yy14.segm1.org {} 1385984461000000000 gxyzinf a13yy14 -101 0.000542 0.001861 0.001074
10002 [] yy20.segm1.org {} 1385984461000000000 x_gbinf I502yy20 -101 0.000522 0.001043 0.000743
10003 [] yy20.segm1.org {} 1385984461000000000 setFlagsOneWidg I502yy20 1 0.001660 0.005404 0.002910
10004 [] yy14.segm1.org {} 1385984461000000000 notifyTestsDelivered a13yy14 1 0.000551 0.002877 0.001156
10005 [] yy20.segm1.org {} 1385984461000000000 gxyzinf I502yy20 -101 0.000521 0.000802 0.000813
10006 [] yy14.segm1.org {} 1385984461000000000 addWidging2 a13yy14 1 0.001256 0.009431 0.003910
10007 [] yy14.segm1.org {} 1385984461000000000 gxyzinf a13yy14 -101 0.000414 0.001861 0.001074
10008 [] yy14.segm1.org {} 1385984461000000000 addWidging2 a13yy14 1 0.001222 0.009431 0.003910
10009 [] yy14.segm1.org {} 1385984461000000000 gxyzinf a13yy14 -101 0.000475 0.001861 0.001074
1001 [] yy7.segm1.org {} 1385984320000000000 gxyzinf a12yy7 -101 0.000783 0.003059 0.001004
10010 [] yy14.segm1.org {} 1385984461000000000 x_initWidgbox a12yy14 1 0.002764 0.035759 0.005816
10011 [] yy32.segm1.org {} 1385984461000000000 x_initWidgbox a15yy32 1 0.057966 0.334923 0.147668
10012 [] yy3.segm1.org {} 1385984461000000000 gSettings a11yy3 -101 0.006519 0.163707 0.017649
10013 [] yy30.segm1.org {} 1385984461000000000 gtfull a13yy30 -101 0.003648 0.116366 0.014088
10014 [] yy6.segm1.org {} 1385984461000000000 x_gbinf a16yy6 -101 0.000621 0.005796 0.001139
10015 [] yy34.segm1.org {} 1385984461000000000 gtfull a14yy34 -101 0.002031 0.015581 0.007747
10016 [] yy34.segm1.org {} 1385984461000000000 x_gbinf a14yy34 -101 0.000546 0.002596 0.001899
10017 [] yy34.segm1.org {} 1385984461000000000 setFlagsOneWidg a14yy34 1 0.001358 0.003515 0.005866
10018 [] yy34.segm1.org {} 1385984461000000000 gxyzinf a14yy34 -101 0.000486 0.004446 0.002018
10019 [] yy25.segm1.org {} 1385984461000000000 gtfull a13yy25 -101 0.002029 0.001793 0.002355
1002 [] yy7.segm1.org {} 1385984320000000000 notifyTestsDelivered a12yy7 1 0.000847 0.003748 0.001081
10020 [] yy32.segm1.org {} 1385984462000000000 gFolderId a15yy32 -101 0.018326 0.187434 0.058200
10021 [] yy25.segm1.org {} 1385984462000000000 x_gbinf a13yy25 -101 0.000589 0.001716 0.000830
10022 [] yy25.segm1.org {} 1385984462000000000 updateWidg a13yy25 1 0.003058 0.004660 0.003973
10023 [] yy25.segm1.org {} 1385984462000000000 clearElems a13yy25 1 0.000661 0.004893 0.001687
10024 [] yy10.segm1.org {} 1385984462000000000 gtfull a18yy10 -101 0.002779 0.069679 0.007495
10025 [] yy13.segm1.org {} 1385984462000000000 gtfull a11yy13 -101 0.001978 0.124069 0.012524
10026 [] yy32.segm1.org {} 1385984462000000000 x_gbinf a14yy32 -101 0.018674 0.190657 0.058083
10027 [] yy10.segm1.org {} 1385984462000000000 x_gbinf a18yy10 -101 0.000874 0.007170 0.001606
10028 [] yy32.segm1.org {} 1385984462000000000 gWidgId a14yy32 1 0.014523 1.518315 0.559983
10029 [] yy13.segm1.org {} 1385984462000000000 x_gbinf a11yy13 -101 0.000577 0.008605 0.001130
1003 [] yy7.segm1.org {} 1385984320000000000 x_gWidgboxParams a12yy7 -101 0.000933 0.001084 0.001442
10030 [] yy13.segm1.org {} 1385984462000000000 setFlagsOneWidg a11yy13 1 0.001611 0.011409 0.004093
10031 [] yy13.segm1.org {} 1385984462000000000 gxyzinf a11yy13 -101 0.000575 0.053991 0.003044
10032 [] yy39.segm1.org {} 1385984462000000000 gtfull a13yy39 -101 0.002005 0.034577 0.003504
10033 [] yy39.segm1.org {} 1385984462000000000 x_gbinf a13yy39 -101 0.000539 0.001371 0.000931
10034 [] yy32.segm1.org {} 1385984462000000000 addWidging2 a15yy32 1 0.122369 1.414068 0.441565
10035 [] yy32.segm1.org {} 1385984462000000000 moveOneWidg a12yy32 1 0.468481 1.303089 0.665778
10036 [] yy32.segm1.org {} 1385984462000000000 gxyzinf a15yy32 -101 0.018006 0.155379 0.040389
10037 [] yy32.segm1.org {} 1385984462000000000 notifyTestsDelivered a15yy32 1 0.006874 0.129650 0.032741
10038 [] yy32.segm1.org {} 1385984462000000000 gxyzinf a12yy32 -101 0.016607 0.155379 0.040389
10039 [] yy39.segm1.org {} 1385984462000000000 updateWidg a13yy39 1 0.003879 0.005466 0.006465
1004 [] yy34.segm1.org {} 1385984320000000000 gtfull a11yy34 -101 0.003681 0.015581 0.007747
10040 [] yy39.segm1.org {} 1385984462000000000 SELECT a13yy39 217831 0.000423 0.000126 0.000551
10041 [] yy39.segm1.org {} 1385984462000000000 clearElems a13yy39 1 0.000705 0.002367 0.001356
10042 [] yy3.segm1.org {} 1385984462000000000 moveOneWidg a15yy3 1 0.002660 0.027428 0.009078
10043 [] yy3.segm1.org {} 1385984462000000000 gxyzinf a15yy3 -101 0.000436 0.041627 0.001913
10044 [] yy39.segm1.org {} 1385984462000000000 gSettings a11yy39 -101 0.002237 0.007467 0.002679
10045 [] yy32.segm1.org {} 1385984462000000000 gSettings a15yy32 -101 0.012113 0.245170 0.070137
10046 [] yy32.segm1.org {} 1385984462000000000 x_gWidgboxParams a15yy32 -101 0.030427 0.143941 0.050055
10047 [] yy13.segm1.org {} 1385984462000000000 moveOneWidg a12yy13 1 0.003796 0.117085 0.017910
10048 [] yy13.segm1.org {} 1385984462000000000 gxyzinf a12yy13 -101 0.000521 0.053991 0.003044
10049 [] yy30.segm1.org {} 1385984462000000000 x_gWidgboxParams a13yy30 -101 0.002451 0.051829 0.003644
1005 [] yy12.segm1.org {} 1385984320000000000 gtfull a15yy12 -101 0.003428 0.005479 0.003063
... ... ... ... ... ... ... ... ... ...
[410450 rows x 10 columns]
python pandas 在 groupby 中應用 function，並將結果添加為數據框中的列
[英]python pandas apply function in groupby, and add results as column in data frame
使用pandas groupby+apply 恢復標准的單索引數據框
[英]Recover a standard, single-index data frame after using pandas groupby+apply
聲明:本站的技術帖子網頁，遵循CC BY-SA 4.0協議，如果您需要轉載，請注明本站網址或者原文地址。任何問題請咨詢:yoyou2525@163.com.
相關問題
使用Pandas將分組聚合應用於原始數據框
如何在 pandas 數據框上應用 groupby 兩次？
python pandas 在 groupby 中應用 function，並將結果添加為數據框中的列
使用pandas groupby+apply 恢復標准的單索引數據框
對Pandas DataFrame中的groupBy數據應用歸一化
使用groupby的結果過濾pandas數據幀
熊貓：將groupby輸出為數據框
在groupby中查找pandas數據幀索引的交集
遍歷 pandas 數據幀或 groupby object
pandas 數據幀的 groupby 給出錯誤的結果
相關標簽