[英]how to close pandas read_sql connection
The following code yields an error that says the database is locked:以下代码产生一个错误,表明数据库已锁定:
df_chunks = pd.read_sql('cdr', engine, chunksize=100000)
engine.dispose()
with engine.connect() as conn:
trans = conn.begin()
query = """
CREATE TABLE test (row_id BIGINT,ego_id BIGINT,alter_id BIGINT)
"""
print(query)
conn.execute(query)
trans.commit()
conn.close()
but if I do it the other way around:但如果我反过来做:
with engine.connect() as conn:
trans = conn.begin()
query = """
CREATE TABLE test (row_id BIGINT,ego_id BIGINT,alter_id BIGINT)
"""
print(query)
conn.execute(query)
trans.commit()
conn.close()
df_chunks = pd.read_sql('cdr', engine, chunksize=100000)
it works and I am able to generate the new table.它有效,我能够生成新表。 So from here, it seems that the problem is that
pd.read_sql
(see docs ) locks the database.因此,从这里开始,问题似乎在于
pd.read_sql
(请参阅文档)锁定了数据库。 I found this question but engine.dispose()
didn't work for me.我发现了这个问题,但
engine.dispose()
对我不起作用。 What's the way to go around this problem? go 解决这个问题的方法是什么?
Because chunksize
builds a generator of data frames and you never do anything with df_chunks
, this object may still hold a pointer to source object, engine
, and therefore "locks" the database.因为
chunksize
构建了一个数据帧生成器,而您从不使用df_chunks
做任何事情,这个 object 可能仍然持有指向源 object, engine
的指针,因此“锁定”数据库。 To resolve your first attempt, consider running an iteration on the df_chunks
:要解决您的第一次尝试,请考虑在
df_chunks
上运行迭代:
with engine.connect() as conn:
trans = conn.begin()
query = """CREATE TABLE test (row_id BIGINT,ego_id BIGINT,alter_id BIGINT)
"""
print(query)
conn.execute(query)
trans.commit()
df_chunks = pd.read_sql('cdr', engine, chunksize=100000)
for df in df_chunks:
# DO SOMETHING WITH EACH df
engine.dispose() # ALLOWED SINCE GENERATOR IS EXHAUSTED AFTER for LOOP
Alternatively, to resolve your second attempt, integrate your read_sql
call inside with
block and use the conn
object.或者,要解决您的第二次尝试,请将您的
read_sql
调用with
块集成并使用conn
object。
with engine.connect() as conn:
trans = conn.begin()
query = """CREATE TABLE test (row_id BIGINT,ego_id BIGINT,alter_id BIGINT)
"""
print(query)
conn.execute(query)
trans.commit()
# INDENT LINE AND USE conn OBJECT
df_chunks = pd.read_sql('cdr', conn, chunksize=100000)
for df in df_chunks:
# DO SOMETHING WITH EACH df
engine.dispose() # CLOSE engine OBJECT NOT conn
Also, when using with
as a context manager, it is unnecessary to call close: conn.close()
.此外,当使用
with
作为上下文管理器时,无需调用 close: conn.close()
。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.