在 Pandas 中循環和應用的替代方案
[英]Alternatives to loop and apply in Pandas
問題描述
我有兩個數據框，一個是 80,000 行長 37 列寬 (dfa)，另一個是 90 行長 (dfb)。 我需要篩選 dfb 中的 90 行值以查找與 dfa 中的行匹配的值，然后 append 將 dfb 中的值篩選到 dfa 中的索引。
但是，需要比較 dfa 中的 10 列，我當前的代碼正在運行，但速度極慢。
我曾嘗試對數據進行矢量化，但我遇到了同樣的速度問題，而且我不知道如何在這種情況下使用 apply function，所以我一直在使用.iterrows()。 我曾想過嘗試並行計算，但我覺得嘗試實現這將是另一場噩夢。
我願意接受有關如何嘗試加快此代碼速度的任何建議
def add_scale():
x = 1
for i in dfa.iterrows():
for f in dfb.iterrows():
if i[1][2] == f[1][0]:
dfa.iloc[x, 27] = f[1][8]
if i[1][4] == f[1][0]:
dfa.iloc[x, 28] = f[1][8]
if i[1][6] == f[1][0]:
dfa.iloc[x, 29] = f[1][8]
if i[1][8] == f[1][0]:
dfa.iloc[x, 30] = f[1][8]
if i[1][10] == f[1][0]:
dfa.iloc[x, 31] = f[1][8]
if i[1][12] == f[1][0]:
dfa.iloc[x, 32] = f[1][8]
if i[1][14] == f[1][0]:
dfa.iloc[x, 33] = f[1][8]
if i[1][16] == f[1][0]:
dfa.iloc[x, 34] = f[1][8]
if i[1][18] == f[1][0]:
dfa.iloc[x, 35] = f[1][8]
if i[1][20] == f[1][0]:
dfa.iloc[x, 36] = f[1][8]
x += 1
print(x)
dfa的負責人。
id
2d11489f-919c-436d-8e7d-e25df44d9dfb
a747fe55-7bb0-4877-b080-9a3f89855c02
9688cb3c-57a0-4e23-b10b-c674e346cce5
a042f8e6-d433-4229-8b6b-304a1c14df98
fe4918d7-6e23-4605-8158-e5a89afc0614
item_1 quantity_1 \
c2f332de-1cdb-43ce-9cb1-f61a06e51d65 1
6ebaafde-8652-4fb8-bea2-a08d661bd56b 1
063b51ab-a8b8-4714-8adc-992d507fd222 2
b6ab20b1-be59-4592-9447-d12fc7a4f405 1
9bdd10b9-2356-494c-958f-04a35514178e 1
item_2 quantity_2 \
1f672f37-50d9-40ff-a063-122bdcd7da2a 1.0
16c36c7a-a6b0-4aca-9f6e-a178074dc15e 1.0
e2341b46-b323-4b41-9865-cbf1625ee810 3.0
c34eab5c-1772-422c-8773-e00c10b10b1c 1.0
4e720d54-fbb0-4c9d-bb99-dc2b17004bf2 1.0
item_3 quantity_3 \
NaN NaN
33671e62-f1d4-4284-b08b-1e4813b9cb4c 3.0
2192e8c2-c66f-4650-9f6e-b5b12a2e8587 1.0
60fddb6f-c6a3-41e1-90ed-febdd13ffbdf 1.0
9493337b-8843-40fe-b97f-4cca3b687ebc 2.0
item_4 quantity_4 \
NaN NaN
e2341b46-b323-4b41-9865-cbf1625ee810 2.0
b6ab20b1-be59-4592-9447-d12fc7a4f405 1.0
NaN NaN
b6ab20b1-be59-4592-9447-d12fc7a4f405 3.0
item_5 quantity_5 \
NaN NaN
257db03b-3711-4e98-9b8f-68890d433a18 1.0
7d2fc54e-c92e-47e4-830e-c434cdd70ffc 1.0
NaN NaN
NaN NaN
item_6 quantity_6 \
NaN NaN
9493337b-8843-40fe-b97f-4cca3b687ebc 3.0
c34eab5c-1772-422c-8773-e00c10b10b1c 2.0
NaN NaN
NaN NaN
item_7 quantity_7 \
NaN NaN
3b10f6f0-6412-4366-bd8d-483c88368511 1.0
d9962506-1685-4502-b3f1-4c885eeb5457 1.0
NaN NaN
NaN NaN
item_8 quantity_8 \
NaN NaN
75929f5e-f3fb-42b1-9e71-0aed7f8e5066 3.0
1f7b73b1-c995-46cf-9781-ed1bcc336345 1.0
NaN NaN
NaN NaN
item_9 quantity_9 \
NaN NaN
819069be-ef26-4670-aa2c-321c53ed6c94 1.0
a708ab2d-b79e-4577-80fb-a30fa155445f 1.0
NaN NaN
NaN NaN
item_10 quantity_10 datetime \
NaN NaN 2019-09-03 10:56
NaN NaN 2019-09-04 21:59
1a747199-994e-4e82-a8ea-1fbc1029256c 1.0 2019-09-04 12:50
NaN NaN 2019-09-05 20:48
NaN NaN 2019-09-05 14:06
food_prep_time_minutes minutes.inday days item_val it1 it2 it3 it4 \
13 23 2 2 NaN NaN NaN NaN
38 45 3 9 NaN NaN NaN NaN
24 27 3 0 NaN NaN NaN NaN
14 43 4 3 NaN NaN NaN NaN
25 29 4 4 NaN NaN NaN NaN
it5 it6 it7 it8 it9 it10
NaN NaN NaN NaN NaN NaN
NaN NaN NaN NaN NaN NaN
NaN NaN NaN NaN NaN NaN
NaN NaN NaN NaN NaN NaN
NaN NaN NaN NaN NaN NaN
dfb 的負責人，未使用的列已刪除
item_id scale
0 5445da62-e213-4f71-b2b0-8b6073647102 3
1 16c36c7a-a6b0-4aca-9f6e-a178074dc15e 1
2 a708ab2d-b79e-4577-80fb-a30fa155445f 1
3 024f545f-a8af-4244-8c9e-da2b92633d59 2
4 8e3e855c-918c-4761-b2d6-0f4aae1c5e0d 3
1 個解決方案
解決方案1
0 2019-11-05 03:16:12
我有兩個數據框，一個是 80,000 行長 37 列寬 (dfa)，另一個是 90 行長 (dfb)。 我需要篩選 dfb 中的 90 行值以查找與 dfa 中的行匹配的值，然后 append 將 dfb 中的值篩選到 dfa 中的索引。
但是，需要比較 dfa 中的 10 列，我當前的代碼正在運行，但速度極慢。
我曾嘗試對數據進行矢量化，但我遇到了同樣的速度問題，而且我不知道如何在這種情況下使用 apply function，所以我一直在使用.iterrows()。 我曾想過嘗試並行計算，但我覺得嘗試實現這將是另一場噩夢。
我願意接受有關如何嘗試加快此代碼速度的任何建議
def add_scale():
x = 1
for i in dfa.iterrows():
for f in dfb.iterrows():
if i[1][2] == f[1][0]:
dfa.iloc[x, 27] = f[1][8]
if i[1][4] == f[1][0]:
dfa.iloc[x, 28] = f[1][8]
if i[1][6] == f[1][0]:
dfa.iloc[x, 29] = f[1][8]
if i[1][8] == f[1][0]:
dfa.iloc[x, 30] = f[1][8]
if i[1][10] == f[1][0]:
dfa.iloc[x, 31] = f[1][8]
if i[1][12] == f[1][0]:
dfa.iloc[x, 32] = f[1][8]
if i[1][14] == f[1][0]:
dfa.iloc[x, 33] = f[1][8]
if i[1][16] == f[1][0]:
dfa.iloc[x, 34] = f[1][8]
if i[1][18] == f[1][0]:
dfa.iloc[x, 35] = f[1][8]
if i[1][20] == f[1][0]:
dfa.iloc[x, 36] = f[1][8]
x += 1
print(x)
dfa的負責人。
id
2d11489f-919c-436d-8e7d-e25df44d9dfb
a747fe55-7bb0-4877-b080-9a3f89855c02
9688cb3c-57a0-4e23-b10b-c674e346cce5
a042f8e6-d433-4229-8b6b-304a1c14df98
fe4918d7-6e23-4605-8158-e5a89afc0614
item_1 quantity_1 \
c2f332de-1cdb-43ce-9cb1-f61a06e51d65 1
6ebaafde-8652-4fb8-bea2-a08d661bd56b 1
063b51ab-a8b8-4714-8adc-992d507fd222 2
b6ab20b1-be59-4592-9447-d12fc7a4f405 1
9bdd10b9-2356-494c-958f-04a35514178e 1
item_2 quantity_2 \
1f672f37-50d9-40ff-a063-122bdcd7da2a 1.0
16c36c7a-a6b0-4aca-9f6e-a178074dc15e 1.0
e2341b46-b323-4b41-9865-cbf1625ee810 3.0
c34eab5c-1772-422c-8773-e00c10b10b1c 1.0
4e720d54-fbb0-4c9d-bb99-dc2b17004bf2 1.0
item_3 quantity_3 \
NaN NaN
33671e62-f1d4-4284-b08b-1e4813b9cb4c 3.0
2192e8c2-c66f-4650-9f6e-b5b12a2e8587 1.0
60fddb6f-c6a3-41e1-90ed-febdd13ffbdf 1.0
9493337b-8843-40fe-b97f-4cca3b687ebc 2.0
item_4 quantity_4 \
NaN NaN
e2341b46-b323-4b41-9865-cbf1625ee810 2.0
b6ab20b1-be59-4592-9447-d12fc7a4f405 1.0
NaN NaN
b6ab20b1-be59-4592-9447-d12fc7a4f405 3.0
item_5 quantity_5 \
NaN NaN
257db03b-3711-4e98-9b8f-68890d433a18 1.0
7d2fc54e-c92e-47e4-830e-c434cdd70ffc 1.0
NaN NaN
NaN NaN
item_6 quantity_6 \
NaN NaN
9493337b-8843-40fe-b97f-4cca3b687ebc 3.0
c34eab5c-1772-422c-8773-e00c10b10b1c 2.0
NaN NaN
NaN NaN
item_7 quantity_7 \
NaN NaN
3b10f6f0-6412-4366-bd8d-483c88368511 1.0
d9962506-1685-4502-b3f1-4c885eeb5457 1.0
NaN NaN
NaN NaN
item_8 quantity_8 \
NaN NaN
75929f5e-f3fb-42b1-9e71-0aed7f8e5066 3.0
1f7b73b1-c995-46cf-9781-ed1bcc336345 1.0
NaN NaN
NaN NaN
item_9 quantity_9 \
NaN NaN
819069be-ef26-4670-aa2c-321c53ed6c94 1.0
a708ab2d-b79e-4577-80fb-a30fa155445f 1.0
NaN NaN
NaN NaN
item_10 quantity_10 datetime \
NaN NaN 2019-09-03 10:56
NaN NaN 2019-09-04 21:59
1a747199-994e-4e82-a8ea-1fbc1029256c 1.0 2019-09-04 12:50
NaN NaN 2019-09-05 20:48
NaN NaN 2019-09-05 14:06
food_prep_time_minutes minutes.inday days item_val it1 it2 it3 it4 \
13 23 2 2 NaN NaN NaN NaN
38 45 3 9 NaN NaN NaN NaN
24 27 3 0 NaN NaN NaN NaN
14 43 4 3 NaN NaN NaN NaN
25 29 4 4 NaN NaN NaN NaN
it5 it6 it7 it8 it9 it10
NaN NaN NaN NaN NaN NaN
NaN NaN NaN NaN NaN NaN
NaN NaN NaN NaN NaN NaN
NaN NaN NaN NaN NaN NaN
NaN NaN NaN NaN NaN NaN
dfb 的負責人，未使用的列已刪除
item_id scale
0 5445da62-e213-4f71-b2b0-8b6073647102 3
1 16c36c7a-a6b0-4aca-9f6e-a178074dc15e 1
2 a708ab2d-b79e-4577-80fb-a30fa155445f 1
3 024f545f-a8af-4244-8c9e-da2b92633d59 2
4 8e3e855c-918c-4761-b2d6-0f4aae1c5e0d 3
解決方案2
-1 2019-11-05 03:24:57
好吧，我覺得很愚蠢，@furas 是正確的。 我刪除了額外的循環，還有沒有循環的方法呢？
探索將條件應用於除 Pandas 之外的每一行數據框的替代方法
[英]Exploring alternatives to apply conditions to every row of dataframe other than pandas apply
聲明:本站的技術帖子網頁，遵循CC BY-SA 4.0協議，如果您需要轉載，請注明本站網址或者原文地址。任何問題請咨詢:yoyou2525@163.com.