append rows using pd.concat() with python from a 'for loop'
Question
After an update, I am getting the following message:
'The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead.'
I am conducting an artificial star experiment, where I have three files with columns:
b1: ['Id', 'x', 'y', 'bmag'] - artificial stars with b-filter
i1: ['Id', 'x', 'y', 'imag'] - artificial stars with I-filter
biart: ['Id', 'x', 'y', 'bmag', 'imag'] - measured stars
**I have calculated the minimum radial distance between stars in b1 (artificial) and biart (measured) and writing a dataframe with the 'Id', 'bmag' and 'imag' if the minimum distance is satisfied. **
The code goes through each row of b1 and determines which row of biart has minimum distance, then saves it in a dataframe
I am having trouble re-writing my code with pd.concat() instead of df.append() . Please help!
extract = pd.DataFrame(columns=['Id_art', 'x_art', 'y_art',
'bmag_art', 'imag_art',
'dist_d',
'Id_meas', 'x_meas', 'y_meas',
'bmag_meas', 'imag_meas'])
for i in range(len(b1.index)):
x = b1['x'].iloc[i]
y = b1['y'].iloc[i]
dist = np.sqrt((x - biart['x'])**2 + (y - biart['y'])**2)
if (min(dist))<=1/2:
extract = extract.append({'Id_art': b1['Id'].iloc[i],
'x_art':b1['x'].iloc[i],
'y_art': b1['y'].iloc[i],
'bmag_art':b1['bmag'].iloc[i],
'imag_art':i1['imag'].iloc[i],
'dist_d':min(dist),
'Id_meas':biart['Id'].iloc[dist.idxmin()],
'x_meas':biart['x'].iloc[dist.idxmin()],
'y_meas':biart['y'].iloc[dist.idxmin()],
'bmag_meas':biart['bmag'].iloc[dist.idxmin()],
'imag_meas':biart['imag'].iloc[dist.idxmin()]},
ignore_index=True)
I am new to python. Can you please help explain how to rewrite the code below to pd.concat()?
extract.append({'Id_art': b1['Id'].iloc[i],
'x_art':b1['x'].iloc[i],
'y_art': b1['y'].iloc[i],
'bmag_art':b1['bmag'].iloc[i],
'imag_art':i1['imag'].iloc[i],
'dist_d':min(dist),
'Id_meas':biart['Id'].iloc[dist.idxmin()],
'x_meas':biart['x'].iloc[dist.idxmin()],
'y_meas':biart['y'].iloc[dist.idxmin()],
'bmag_meas':biart['bmag'].iloc[dist.idxmin()],
'imag_meas':biart['imag'].iloc[dist.idxmin()]},
ignore_index=True)
please see below the first 30 values of the three data files:
b1:
Id x y bmag z1 z2 z3 z4 z5
1 1163.630 1327.069 16.4552 0 0 0 0 0
2 1747.045 1477.734 16.1293 0 0 0 0 0
3 3236.084 1252.050 16.5628 0 0 0 0 0
4 2516.600 888.832 17.7005 0 0 0 0 0
5 1868.000 1376.218 16.3139 0 0 0 0 0
6 2484.642 941.331 16.7632 0 0 0 0 0
7 2984.336 640.185 16.1351 0 0 0 0 0
8 2612.480 1664.341 16.3573 0 0 0 0 0
9 1973.405 731.071 18.7704 0 0 0 0 0
10 3391.978 130.435 16.1769 0 0 0 0 0
11 165.021 942.634 17.4829 0 0 0 0 0
12 2491.430 1397.675 16.5589 0 0 0 0 0
13 2886.291 1367.745 16.3339 0 0 0 0 0
14 939.302 726.039 16.8744 0 0 0 0 0
15 2296.265 686.110 16.1132 0 0 0 0 0
16 682.263 932.721 16.5750 0 0 0 0 0
17 2393.509 1776.220 16.8953 0 0 0 0 0
18 277.072 1879.963 16.1769 0 0 0 0 0
19 1824.305 1616.309 16.2134 0 0 0 0 0
20 2986.818 833.414 16.2364 0 0 0 0 0
21 2547.293 1067.399 16.4345 0 0 0 0 0
22 707.099 591.125 16.1352 0 0 0 0 0
23 1987.175 899.605 17.0269 0 0 0 0 0
24 2853.484 761.114 16.4994 0 0 0 0 0
25 2664.596 897.447 16.7100 0 0 0 0 0
26 730.565 905.192 18.1313 0 0 0 0 0
27 2760.565 1894.611 17.9226 0 0 0 0 0
28 3427.851 1883.343 18.2891 0 0 0 0 0
29 2165.442 769.363 17.9948 0 0 0 0 0
30 2982.916 1094.855 17.1777 0 0 0 0 0
i1:
Id x y imag z1 z2 z3 z4 z5
1 1168.121 1332.597 15.1104 0 0 0 0 0
2 1751.536 1483.263 14.3248 0 0 0 0 0
3 3240.575 1257.578 15.3140 0 0 0 0 0
4 2521.091 894.360 17.1246 0 0 0 0 0
5 1872.491 1381.746 14.8184 0 0 0 0 0
6 2489.133 946.859 15.6679 0 0 0 0 0
7 2988.827 645.714 14.3458 0 0 0 0 0
8 2616.971 1669.869 14.9122 0 0 0 0 0
9 1977.896 736.599 18.6215 0 0 0 0 0
10 3396.469 135.964 14.4787 0 0 0 0 0
11 169.512 948.163 16.8041 0 0 0 0 0
12 2495.921 1403.203 15.3067 0 0 0 0 0
13 2890.782 1373.273 14.8621 0 0 0 0 0
14 943.793 731.567 15.8544 0 0 0 0 0
15 2300.756 691.638 14.2601 0 0 0 0 0
16 686.754 938.249 15.3363 0 0 0 0 0
17 2398.000 1781.748 15.8890 0 0 0 0 0
18 281.563 1885.491 14.4787 0 0 0 0 0
19 1828.796 1621.837 14.5791 0 0 0 0 0
20 2991.309 838.942 14.6377 0 0 0 0 0
21 2551.784 1072.927 15.0697 0 0 0 0 0
22 711.590 596.653 14.3461 0 0 0 0 0
23 1991.666 905.133 16.1022 0 0 0 0 0
24 2857.975 766.643 15.1955 0 0 0 0 0
25 2669.087 902.975 15.5765 0 0 0 0 0
26 735.056 910.720 17.7406 0 0 0 0 0
27 2765.056 1900.140 17.4449 0 0 0 0 0
28 3432.342 1888.871 17.9612 0 0 0 0 0
29 2169.933 774.891 17.5477 0 0 0 0 0
30 2987.407 1100.383 16.3397 0 0 0 0 0
biart:
Id x y bmag imag
1 3.463 964.699 16.0450 16.9640
2 5.084 1567.960 16.9260 18.4700
3 5.105 236.018 15.0720 16.1090
4 5.236 364.229 17.4830 15.8440
5 5.478 1434.603 17.3220 16.7400
6 5.564 453.920 16.9000 17.2100
7 6.015 1186.374 17.9310 15.5080
8 6.017 945.186 17.5790 17.2270
9 6.031 1001.720 15.4920 16.7710
10 6.337 747.211 16.0480 14.9890
11 6.376 1243.525 15.8780 14.8180
12 6.426 635.425 17.3890 17.2010
13 6.432 72.437 16.3430 16.7180
14 6.584 982.933 17.7750 17.9360
15 6.630 688.179 16.8100 17.2390
16 6.827 1559.448 17.6680 17.1000
17 6.855 502.682 18.0290 17.4950
18 7.260 1807.838 17.8060 17.3680
19 7.289 981.327 18.3170 17.6140
20 7.383 1485.190 16.4730 14.4650
21 7.461 1141.420 16.3860 15.6110
22 7.696 1213.424 15.9370 14.7050
23 7.779 1211.424 16.9000 15.2840
24 7.853 821.390 15.1270 16.2590
25 7.868 1325.808 18.3270 17.9500
26 7.954 1789.587 16.2340 14.7330
27 8.132 194.533 17.1350 16.5100
28 8.194 1390.519 16.3770 15.4670
29 8.555 299.428 17.7980 16.5920
30 8.724 982.512 17.9880 17.1510
1 answers
solution1
1 2022-06-24 17:26:27
Instead of adding rows to a dataframe, you can generate each dataframe from a single row and then concat all the rows.
Although it may surprise, it is more efficient.
list_a = []
for i in range(len(b1.index)):
x = b1['x'].iloc[i]
y = b1['y'].iloc[i]
dist = np.sqrt((x - biart['x'])**2 + (y - biart['y'])**2)
if (min(dist))<=1/2:
list_a.append(pd.DataFrame({'Id_art': b1['Id'].iloc[i],
'x_art':b1['x'].iloc[i],
'y_art': b1['y'].iloc[i],
'bmag_art':b1['bmag'].iloc[i],
'imag_art':i1['imag'].iloc[i],
'dist_d':min(dist),
'Id_meas':biart['Id'].iloc[dist.idxmin()],
'x_meas':biart['x'].iloc[dist.idxmin()],
'y_meas':biart['y'].iloc[dist.idxmin()],
'bmag_meas':biart['bmag'].iloc[dist.idxmin()],
'imag_meas':biart['imag'].iloc[dist.idxmin()]},
index=[0])``
now concat.
df_all = pd.concat(list_a, ignore_index=True)
and delete intermediate list
del list_a
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.