pd.read_sav 和 pyreadstat 太慢了。 如果必须使用 SAV/SPSS 文件格式，如何为大数据加速 Pandas？

Question

I've been transitioning away from SPSS for syntax writing/data management where I work to python and pandas for higher levels of functionality and programming. The issue is, reading SPSS files into pandas is SO slow. i work with bigger datasets (1 million or more rows often with 100+ columns). it seems that there are some pretty cool plugins out there to speed up processing CSV files such as Dask and Modin, but i don't think these work with SPSS files. i'd like to continue using pandas, but i have to stick with the SPSS file format (it's what everyone else where i work uses).

Are there any tips on how to accomplish faster data processing outside of computer upgrades and or file chunking?

Answer 1

You can try to parallelize reading your file:

As an example I have a file "big.sav" which is 294000 rows x 666 columns. Reading the file with pyreadstat.read_sav (which is what pd.read_spss uses in the background) takes 115 seconds. By parallelizing it I get 29 seconds:

first I create a file worker.py:

def worker(inpt):
    import pyreadstat
    offset, chunksize, path = inpt
    df, meta = pyreadstat.read_sav(path, row_offset=offset, row_limit=chunksize)
    return df

and then in the main script I have this:

import multiprocessing as mp
from time import time

import pandas as pd
import pyreadstat

from worker import worker

# calculate the number of rows in the file
_, meta = pyreadstat.read_sav("big.sav", metadataonly=True)
numrows = meta.number_rows
# calculate number of cores in the machine, this could also be set manually to some number, i.e. 8
numcores = mp.cpu_count()
# calculate the chunksize and offsets
divs = [numrows // numcores + (1 if x < numrows % numcores else 0)  for x in range (numcores) ]
chunksize = divs[0]
offsets = [indx*chunksize for indx in range(numcores)] 
# pack the data for the jobs
jobs = [(x, chunksize, "big.sav") for x in offsets]

pool = mp.Pool(processes=numcores)
# let's go!
t0=time()
chunks = pool.map(worker, jobs)
t1=time()
print(t1-t0) # this prints 29 seconds
# chunks is a list of dataframes in the right order
# you can concatenate all the chunks into a single big dataframe if you like
final = pd.concat(chunks, axis=0, ignore_index=True)

EDIT:

1. pyreadstat version 1.0.3 has had a big improvement in performance of about 5x.
2. In addition a new function "read_file_multiprocessing" has been added that is a wrapper around the previous code shared in this answer. It can give up to another 3x improvement, making (up to) a 15 times improvement compared to the previous version!

You can use the function like this:

import pyreadstat

fpath = "path/to/file.sav" 
df, meta = pyreadstat.read_file_multiprocessing(pyreadstat.read_sav, fpath)