[英]Error trying to save new table to MySQL with sqlalchemy
Hi does somebody has any troubleshooting ideas to solve this problem?嗨,有人有解决此问题的任何故障排除想法吗?
I have a standard python-sql connection at my local machine:我在本地机器上有一个标准的 python-sql 连接:
from sqlalchemy import create_engine
engine = create_engine("mysql+pymysql://root:*******@localhost/my_DB")
con = engine.connect()
this DB consists of 200+ tables where I store stock/market information and I need to update it daily, in order to do that I usually construct a loop through all the tables to fetch up to date information from yahoo_finance using pandas datareader.这个数据库由 200 多个表组成,我在其中存储股票/市场信息,我需要每天更新它,为了做到这一点,我通常构建一个遍历所有表的循环,以使用 Pandas 数据读取器从 yahoo_finance 获取最新信息。
Once loaded into a new DF I use一旦加载到我使用的新 DF 中
df_new.to_sql(name = stock_ticker, con = con, if_exists = 'replace', index = False)
to save the new table into my DB.将新表保存到我的数据库中。
The code above works just fine when I execute one by one, but when I try to implement the same idea on a loop it just breaks, sometimes on the very first instance of the loop:当我一个接一个执行时,上面的代码工作得很好,但是当我尝试在循环中实现相同的想法时,它就会中断,有时在循环的第一个实例上:
for stock in Stocks:
df_new = yahoo_quote(stock)
df_new.to_sql(name = stock_ticker, con = con, if_exists = 'replace', index = False)
My first thought was that somehow I was exhuasting my machine/sql with so many calls, so I tried to add a time.sleep(5) and make sure I erased all the information from memory on each instance, but none of that seems to work.我的第一个想法是不知何故我用这么多调用来耗尽我的机器/sql,所以我尝试添加一个 time.sleep(5) 并确保我从每个实例的内存中删除了所有信息,但似乎没有一个工作。 And, as I said, sometimes the computer just breaks on the very first loop.
而且,正如我所说,有时计算机会在第一个循环中中断。
By "break" I mean that it just keeps running forever without saving the table, usually it takes little less than 1 second to save a table, but when this happens I can leave it running for 10+ minutes and it still won't save it.通过“中断”我的意思是它只是在不保存表格的情况下永远运行,通常保存表格只需不到 1 秒,但是当发生这种情况时,我可以让它运行 10 分钟以上,但仍然无法保存它。
if_exists= 'replace'
option is drop the table before inserting new values. if_exists= 'replace'
选项是在插入新值之前删除表。 API reference API参考
your code repeats drop and create same Table in the loop.您的代码重复删除并在循环中创建相同的表。
If you want to replace all data, first time call df_new.to_sql
set if_exists= 'replace'
, and second time call set if_exists= 'append'
.如果要替换所有数据,第一次调用
df_new.to_sql
set if_exists= 'replace'
,第二次调用 set if_exists= 'append'
。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.