有没有更快的方法使用 python 将 dataframe 插入到 SQL？

Question

We have two parts to get final data frame into SQL.

1. downlaoding from datasets from Azure and transforming using python.
2. Uploading transformed data into Azure and then inserting the final dataframe into SQL

Downloading, transforming and uploading takes 5 mins but insertion to SQL is taking quite long time. I used below code for faster insertion.

server = 'XXXX.database.windows.net' 
database = 'XXX' 
username = 'XXX' 
password = 'XXXX' 
driver= '{ODBC Driver 17 for SQL Server}' 
cnxn = pyodbc.connect('DRIVER={ODBC Driver 17 for SQL Server};SERVER='+server+';DATABASE='+database+';UID='+username+';PWD='+ password)



params = urllib.parse.quote_plus('DRIVER='+driver+
                      ';SERVER='+server+
                      ';PORT=1433;DATABASE='+database+
                      ';UID='+username+
                      ';PWD='+ password)
    
engine = 
sa.create_engine("mssql+pyodbc:///?odbc_connect={}".format(params),fast_executemany=True)
conn = engine.connect()



with engine.connect() as connection:
    
    try:  
      
         df_copy.to_sql('XXXX',connection,if_exists = 'append',index=False,chunksize=500)

    except SQLAlchemyError as e:
     error = str(e.__dict__['orig'])
     print(error)
    

conn.close()

Final data frame contains 97000 rows with 127 columns.

SQL Server configuration: Purchased Azure SQL 10 DTUS 250GB of storage.

The error is

Exception has occurred: OperationalError (pyodbc.OperationalError) ('08S01', '[08S01] [Microsoft][ODBC Driver 17 for SQL Server]TCP Provider: An existing connection was forcibly closed by the remote host.\r\n (10054) (SQLExecute); [08S01] [Microsoft][ODBC Driver 17 for SQL Server]Communication link failure (10054)')

I have also used connect_args={'connect_timeout': 2400} inside create engine but after 40-50 mins we are receiving the same error mgs. I think 50 mins for 97k records is quite long time. Any way I could improve the process? Also, I'm currently running on my local machine which has 16GB ram and 12th Gen Intel(R) Core(TM) i7-1265U 1.80 GHz processor. Also, we use Jenkins for deployment. Will there be any faster performance if we test it on Jenkins?

Answer 1

hello there you should try to specify your chunksize in your call df.to_sql(engine, connect(), index=False, if_exists='append', method=None, chunksize = 50000)