pandas to_sql() 错误 - 对于准备好的语句，超出了 memory 20 MB 的限制（每个 session）

Question

I'm already using a chunksize of 50 and im running into memory errors. I need to upload about 250M records in total so if I use any lower of a chunksize I dont think it will finish within the next decade. Any tips on how to get around this?

Database = Azure SQL Server

final_df.to_sql('table', con=engine, schema='example', index=False, chunksize=50, if_exists='append', dtype={'ID':sqlalchemy.types.VARCHAR(length=255),'MSA_NAME':sqlalchemy.types.VARCHAR(length=255)})

Answer 1

I'd like to suggest an alternative.

You could export the data as CSV and have it uploaded to one of Azure Storage Services ( Blob for instance - You could probably use AZCopy to achieve.

Then use Azure Data Factory in order to copy/import the files to the Azure SQL.

Copy and transform data in Azure SQL Database by using Azure Data Factory

You could refer to this thread which has a similar discussion.

Answer 2

I solved this by using fast_executemany=True and method='multi' -

engine = sqlalchemy.create_engine('mssql+pyodbc:///?odbc_connect=%s' % params, echo=False, connect_args={'autocommit': True}, fast_executemany=True)

...

final_df.to_sql('table', con=engine, schema='example', index=False, method='multi',  if_exists='append', dtype={'ID':sqlalchemy.types.VARCHAR(length=255),'MSA_NAME':sqlalchemy.types.VARCHAR(length=255)})