Error trying to save new table to MySQL with sqlalchemy

Question

Hi does somebody has any troubleshooting ideas to solve this problem?

I have a standard python-sql connection at my local machine:

from sqlalchemy import create_engine

engine = create_engine("mysql+pymysql://root:*******@localhost/my_DB")

con = engine.connect()

this DB consists of 200+ tables where I store stock/market information and I need to update it daily, in order to do that I usually construct a loop through all the tables to fetch up to date information from yahoo_finance using pandas datareader.

Once loaded into a new DF I use

df_new.to_sql(name = stock_ticker, con = con, if_exists = 'replace', index = False)

to save the new table into my DB.

The code above works just fine when I execute one by one, but when I try to implement the same idea on a loop it just breaks, sometimes on the very first instance of the loop:

for stock in Stocks:

df_new = yahoo_quote(stock)

df_new.to_sql(name = stock_ticker, con = con, if_exists = 'replace', index = False)

My first thought was that somehow I was exhuasting my machine/sql with so many calls, so I tried to add a time.sleep(5) and make sure I erased all the information from memory on each instance, but none of that seems to work. And, as I said, sometimes the computer just breaks on the very first loop.

By "break" I mean that it just keeps running forever without saving the table, usually it takes little less than 1 second to save a table, but when this happens I can leave it running for 10+ minutes and it still won't save it.

Answer 1

if_exists= 'replace' option is drop the table before inserting new values. API reference
your code repeats drop and create same Table in the loop.
If you want to replace all data, first time call df_new.to_sql set if_exists= 'replace' , and second time call set if_exists= 'append' .