将大型 dask 数据帧写入文件

Question

I have a large BCP file (12GB) that I have imported into dask and did some data wrangling that I wish to import to SQL server. The file has been reduced from 40+ columns to 8 columns and I wish to find the best method to import to SQL server. I have tried using the following:

import sqlalchemy as sa
import dask.dataframe as dd
from dask.diagnostics import ProgressBar
from urllib.parse import quote_plus

pbar = ProgressBar()
pbar.register()
#windows authentication 
#to_sql_uri = quote_plus(engine)
ddf.to_sql('test', 
           uri='mssql+pyodbc://TEST_SERVER/TEST_DB?driver=SQL Server?Trusted_Connection=yes', if_exists='replace', index=False)

This method is taking too long (3 days and counting). I had suspected this may be the case, so I also tried to write to a BCP file with the intention of using SQL BCP, but again this is taking a number of days:

df_train_grouped.compute().to_csv("F:\TEST_FILE.bcp", sep='\t')

I am relatively new to dask and can't seem to find an easy to follow example on the most efficient method to do this.

Answer 1

There is no need for you to use compute , this materialises the dataframe into memory and is likely the bottleneck for you. You can instead do

df_train_grouped.to_csv("F:\TEST_FILE*.bcp", sep='\t')

which will create a number of output files in parallel - which is probably exactly what you want.

Note that profiling will determine whether your process is IO bound (eg, by the disc itself), in which case there is nothing you can do, or whether one of the process-based schedulers (ideally the distributed scheduler) can help with GIL-holding tasks.

Changing to a multiprocessing scheduler as follows improved performance in this particular case:

dask.config.set(scheduler='processes')  # overwrite default with multiprocessing scheduler
df_train_grouped.to_csv("F:\TEST_FILE*.bcp", sep='\t', chunksize=1000000)