如何从 Pandas 表中的先前加载中选择存储在文件中的大于最大时间戳的记录

Question

I have SQL script from which I am extracting data storing into a data frame.

Current Code

cursor.execute("""
    SELECT * FROM ofs.ord_add oa
            WHERE
            oa.is_active = 'Y' AND   oa.or_ad_id = 5820  AND   oa.flag_value = 'Y'
    """)
    data3=cursor.fetchall()
    columns = [column[0] for column in cursor.description]
    order_addition = pd.DataFrame(data3,columns=columns)

** Current Output- Run1**

ID     IS_ACTIVE   OR_AD_ID   FLAG_VALUE   INSERT TIMESTAMP
12300     Y          5820        Y         2020-01-06 08:12:53
14340     Y          5820        Y         2020-01-19 06:11:53

** Current Output-Run2**

ID     IS_ACTIVE   OR_AD_ID   FLAG_VALUE   INSERT TIMESTAMP
12300     Y          5820        Y         2020-01-06 08:12:53
14340     Y          5820        Y         2020-01-19 06:11:53
22368     Y          5820        Y         2020-01-22 08:12:53
34567     Y          5820        Y         2020-01-24 06:11:53

I am want use a condition in my current code such that :

1) max insert timestamp from the last run is stored in a file

2) In the next run of the SQL query, Only records greater than max timestamp stored in the file is loaded into the dataframe.

Expected Output after Run2

  ID     IS_ACTIVE   OR_AD_ID   FLAG_VALUE   INSERT TIMESTAMP
  22368     Y          5820        Y         2020-01-22 08:12:53
  34567     Y          5820        Y         2020-01-24 06:11:53

How can this be done in python ?

Answer 1

First, create a variable from the very first run:

unique_ids = tuple(dataframe.IDs.unique.tolist())

Then you execute the sql query the next time

cursor.execute("""
     SELECT * FROM ofs.ord_add oa
              WHERE
             oa.is_active = 'Y' AND   oa.or_ad_id = 5820  AND   oa.flag_value = 'Y'
             AND oa.ID NOT IN %(unique_ids)s; 
"""
({'unique_ids':unique_ids})
)

do test the variable place holding ... i may have bungled it in this code. But the whole idea is that before u run the query again, get the variable then pass it on to the query. https://www.psycopg.org/docs/usage.html has a guide on how to pass variables in. love to get your feedback if it works or not