多进程 pandas pd.read_sas

Question

I have an extremely large dataset from SAS and want to load it into Python using multiprocessing (if possible). My current code is:

import pandas as pd
from multiprocessing import Pool

sas_file = pd.read_sas('path',
                       encoding='ISO-8859-1',
                       chunksize=100000,
                       iterator=True)


def process_sas(chunk):
    dfs.append(chunk)


if __name__ == '__name__':
    pool=Pool()
    pool.map(process_sas, sas_file)

However, dfs is not defined using this method. Is there any way to multiprocess the SAS data set? Separating the data into chunks is not a requirement.

Thanks,

Answer 1

Depending on where you defined dfs
Quick fix but not ideal: put dfs within the global scope
and then declare global dfs inside the respective function and/or method.

def process_sas(chunk):
    global dfs
    dfs.append(chunk)