Fastest way to update table nulls in postgresql from dataframe

Question

I have a pandas dataframe and matching postgresql table, where every cell in both is either null or a timestamp. For each cell in the table where the cell value equals null, and the corresponding dataframe cell value is a timestamp, I want to update the table cell value. What's the fastest way to do this?

Currently I'm pulling in the whole table into a dataframe, comparing the two dataframes in python (cell by cell), entering those values into a 3rd dataframe (call it DFC), and then destroying the old table and building a new table from DFC. This seems inefficient.

Example:

**Data Frame**   **Postgres Table**
     A    B           A    B
1   NaN   5      1   NaN  NaN
2    8   NaN     2    7   NaN



**Goal State Postgres Table**
     A    B
1   NaN   5
2    7   NaN

Current Code:

import pandas as pd
from pandas import DataFrame

d = {'A': ['None', 8], 'B': [5, 'None']}
df = pd.DataFrame(data=d)
out = {'A': ['None', 'None'], 'B': ['None', 'None']}
outdf = pd.DataFrame(data=out)
tbl = pd.read_sql_query('select * from "exampletable"',con=engine)
for i, row in df.iterrows():
    for j in ['A', 'B']:
        if df.at[i, j] != 'None' and tbl.at[i, j] == 'None':
            outdf.at[i, j] = df.at[i, j]
        else:
            outdf.at[i, j] = tbl.at[i, j]
df.to_sql('exampletable', engine, if_exists='replace')
print(outdf.to_string())

Answer 1

IIUC, you can merge the two databases but maintain a record of what records come from each. Then you can check if your A column is empty and fill in the B column with the B from df2.

outdf = df1.join(df2, on=columns, how="outer", rsuffix='_df2', lsuffix='_df1')
outdf['B'] = outdf.apply(lambda x: x['B_df2'] if pd.isnull(x['A']), axis=1)

Edit : you'd want to filter back down to different rows.

outdf = outdf.loc[:, [columns with _df1 suffix]]
outdf.columns = [i.replace('_df1', '') for i in columns]
outdf = outdf.sort_values(by='B')
outdf = outdf.drop_duplicates([columns you're not filling in], keep='first')