python and mysql downloading data into a csv

Question

im using python pandas and storing mysql queries into dataframes and then downloading the results into excel files

   query = """  ...  """
   DF= pd.read_sql(query, connection)
   writer = pd.ExcelWriter('excel.xlsx',engine='xlsxwriter')
   DF.to_excel(writer,'sheet1') 
   writer.save()

but when I need to update my file I have to run the SQL query to retrieve all the existing data + the new rows now this can take a while each time to execute and puts the server into an unneeded load

so, for example I have an excel file with 100 row, and the DB has 110 row i would run a query to retrieve data from row 90 to row 110 from the database and would update the excel with the additional 10 rows

thanks

Answer 1

Check if the file exists or not and then write to the file

import os
exists = os.path.exists(file) # check if the file already exists
df=pd.read_sql(query, connection) #read the data using limit or offset here
open_mode = None
if exists:
    open_mode = 'a'
    header = False
else:
    open_mode = 'w'
    header = True

with open(file, open_mode) as f:
    df.to_excel(f, header=header, index=False)

Answer 2

There are several ways to do this. First of all, you need to read the data in excel, that is, load the existing data in a dataframe. The other alternative is to store a pointer somewhere, for example the amount of rows in the dataframe or the last id(assuming an auto_increment key is at your disposal) from the database, then modify the query accordingly. For example storing the last id, say 1000 in a file. Then reading it and moving on from there.

with open('last_id') as f:
    last_id = f.read()

sql = "SELECT * FROM `mytable` WHERE `id` > %d" % last_id

Alternatively store the size of the extracted data and use the offset syntax .

Then it's just a question of concatenating the two dataframes(existing+new).