如何处理格式为 SAS7DBAT 文件的大于 30 GiB 的大型数据集？

Question

I have these 30 GiB SAS7BDAT files which correspond to a year's worth of data. When I try importing the file using pd.read_sas() I get a memory-related error. Upon research, I hear mentions of using Dask, segmenting the files into smaller chunks, or SQL. These answers sound pretty broad, and since I'm new, I don't really know where to begin. Would appreciate if someone could share some details with me. Thanks.

Answer 1

I am not aware of a partitioned loader of this sort of data for dask. However, the pandas API apparently allows you to stream the data by chunks, so you could write these chunks to other files in any convenient format, and then process those either serially or with dask. The best value of chunksize will depend on your data and available memory.

The following should work, but I don't have any of this sort of data to try it on.

with pd.read_sas(..., chunksize=100000) as file_reader:
    for i, df in enumerate(file_reader):
        df.to_parquet(f"{i}.parq")

then you can load the parts (in parallel) with

import dask.dataframe as dd
ddf = dd.read_parquet("*.parq")