将文本数据的大熊猫df保存到磁盘会由于耗尽所有RAM而使Colab崩溃。 有解决方法吗？

Question

I have a very large Pandas dataframe that I would like to save to disk to use later. The dataframe only contains string data.

However, no matter what format I use, the saving process crashes my Google Colab enviroment due to using up all available RAM, except CSV, which doesn't complete even after 5 hours.

but that also crashes the enviroment.

Is there a workaround to saving a large text pandas dataframe to disk?

I have tried to_json , to_feather , to_parquet , to_pickle , and they all crash the enviroment.

I also tried to_sql by using

from sqlalchemy import create_engine
engine = sqlalchemy.create_engine("sqlite:///database.db")
df.to_sql("table", engine)

I would like to save my dataframe to disk within a reasonable time without crashing the enviroment.

Answer 1

Use the chunksize argument with an appropriate number, eg

df.to_csv('filename.csv', chunksize=100000)

This tells Python to convert the data into .csv 100000 lines at a time, rather than essentially store an entire second copy of your dataframe in RAM before dumping it to the disk.

Similar for .to_sql : Pandas would write in batches, rather than everything at once.

Answer 2

Instead of using Pandas method "to_csv()", use Dask Dataframe to write the csv file; it will be quicker than pandas method. Dask write function will break your file into mulitple chuncks and store it. Code:

#Reading file
import dask.dataframe as dd

df = dd.from_pandas(pd.DataFrame(load_boston().data),npartitions=10)

def operation(df):
   df['new'] = df[0]
   return df[['new']]

#Writing the file
df.pipe(operation).to_csv('boston*.csv')

NOTE: Install Dask package before use:

Using Conda:

conda install -c conda-forge dask

Using pip:

pip install "dask[complete]"    # Install everything

References:

[1] https://docs.dask.org/en/latest/install.html

[2] https://gist.github.com/hussainsultan/f7c2fb9f11008123bda405c5b024a79f