append rows from a 'for loop' to a dataframe using pd.concat with python

Question

After an update, I am getting the following message:

'The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead.'

I am having trouble re-writing my code with pd.concat() instead of df.append() . Please help!

I am conducting an artificial star experiment, where I have three files with columns:

b1: ['Id', 'x', 'y', 'bmag'] - artificial stars with b-filter

i1: ['Id', 'x', 'y', 'imag'] - artificial stars with I-filter

biart: ['Id', 'x', 'y', 'bmag', 'imag'] - measured stars

I have calculated the minimum radial distance between stars in b1 (artificial) and biart (measured) by keeping the 'Id', 'bmag' and 'imag' for both files if the minimum distance is satisfied.

extract = pd.DataFrame(columns=['Id_art', 'x_art', 'y_art', 
                                  'bmag_art', 'imag_art', 
                                  'dist_d',
                                  'Id_meas', 'x_meas', 'y_meas',                                 
                                  'bmag_meas', 'imag_meas'])

for i in range(len(b1.index)):
    
    x = b1['x'].iloc[i]
    y = b1['y'].iloc[i]
    
    dist = np.sqrt((x - biart['x'])**2 + (y - biart['y'])**2)
    
    if (min(dist))<=1/2:
        
        print(b1['Id'].iloc[i], 
              b1['x'].iloc[i], 
              b1['y'].iloc[i],        
              b1['bmag'].iloc[i], 
              i1['imag'].iloc[i], 
              min(dist),
              biart['Id'].iloc[dist.idxmin()],
              biart['x'].iloc[dist.idxmin()],
              biart['y'].iloc[dist.idxmin()], 
              biart['bmag'].iloc[dist.idxmin()],
              biart['imag'].iloc[dist.idxmin()])
            
        extract = extract.append({'Id_art': b1['Id'].iloc[i],
                                    'x_art':b1['x'].iloc[i], 
                                   'y_art': b1['y'].iloc[i], 
                                 'bmag_art':b1['bmag'].iloc[i],
                                 'imag_art':i1['imag'].iloc[i],
                                   'dist_d':min(dist), 
                         'Id_meas':biart['Id'].iloc[dist.idxmin()], 
                          'x_meas':biart['x'].iloc[dist.idxmin()],
                          'y_meas':biart['y'].iloc[dist.idxmin()], 
                     'bmag_meas':biart['bmag'].iloc[dist.idxmin()],
                    'imag_meas':biart['imag'].iloc[dist.idxmin()]},
                                     ignore_index=True)

how do I rewrite the code below to pd.concat()?

extract.append({'Id_art': b1['Id'].iloc[i],
                  'x_art':b1['x'].iloc[i], 
                 'y_art': b1['y'].iloc[i],
               'bmag_art':b1['bmag'].iloc[i],
               'imag_art':i1['imag'].iloc[i],
               'dist_d':min(dist),
               'Id_meas':biart['Id'].iloc[dist.idxmin()], 
                'x_meas':biart['x'].iloc[dist.idxmin()],
                'y_meas':biart['y'].iloc[dist.idxmin()], 
             'bmag_meas':biart['bmag'].iloc[dist.idxmin()],
             'imag_meas':biart['imag'].iloc[dist.idxmin()]},
                                     ignore_index=True)

``

Answer 1

Change the dataframe to a list, and have the pd.DataFrame constructor consume the list at the end:

# Make a list first
extract = []

for i in range(len(b1.index)):
    
    x = b1['x'].iloc[i]
    y = b1['y'].iloc[i]
    
    dist = np.sqrt((x - biart['x'])**2 + (y - biart['y'])**2)
    
    if (min(dist))<=1/2:
        
        print(b1['Id'].iloc[i], b1['x'].iloc[i], b1['y'].iloc[i],        
              b1['bmag'].iloc[i], i1['imag'].iloc[i], 
              min(dist),
              biart['Id'].iloc[dist.idxmin()],
              biart['x'].iloc[dist.idxmin()],
              biart['y'].iloc[dist.idxmin()], 
              biart['bmag'].iloc[dist.idxmin()],
              biart['imag'].iloc[dist.idxmin()])
            
        # Append to the list here
        extract.append({'Id_art': b1['Id'].iloc[i],
                                      'x_art':b1['x'].iloc[i], 
                                      'y_art': b1['y'].iloc[i], 
                                      'bmag_art':b1['bmag'].iloc[i],
                                      'imag_art':i1['imag'].iloc[i],
                                      'dist_d':min(dist), 
                            'Id_meas':biart['Id'].iloc[dist.idxmin()], 
                              'x_meas':biart['x'].iloc[dist.idxmin()],
                              'y_meas':biart['y'].iloc[dist.idxmin()], 
                        'bmag_meas':biart['bmag'].iloc[dist.idxmin()],
                       'imag_meas':biart['imag'].iloc[dist.idxmin()]})

Then, after the loop is finished filling the list:

# Consume the data here
extract = pd.DataFrame(extract)