如何使用多处理将多个Excel工作表导入到pandas中？

Question

I am trying to use multiprocessing on a 12-core machine to read an Excel file – a 60MB file with 15 sheets and 10,000 rows each. Importing all the sheets with pandas.read_csv and no parallelisation still takes about 33 seconds.

If I use pool.map() it works, but it takes longer than the non-parallel version: 150 seconds vs 33!

If I use pool.map_async() it takes 36 seconds, but I can`t seem to access (and cannot therefore check) the output!

My questions are:

• what am I doing wrong? both pool.map and pool.map_async take roughly the same time even if I set nrows=10 in the read_single_sheet function; same time whether it reads 10 rows or 10,000 – how is that possible?
• How do I get the results of pool.map_async()? I have tried output = [p.get() for p in dataframes] but it doesn`t work:

MapResult object is not iterable

• Is this more of a IO-bound than a CPU-bound problem? Still, why does pool.map take so long?

Reading the same data from CSV (each Excel sheet saved to a separate CSV) takes 2 seconds on my machine. However, CSV is not really a good option for what I need to do. I often have 10 to 20 mid-sized tabs; converting them manually may often take longer than waiting for pandas to read them, plus if I receive updated versions I have to do the manual conversion again.

I know I could use a VBA script in Excel to automatically save each sheet to CSV, but data types are most often inferred correctly when reading from Excel – not so with CSV, especially for dates (my dates are never ISO yyyy-mm-dd): I'd have to identify the date fields, specify the format, etc – just reading from Excel would often be faster. Especially because these tasks tend to be one-offs: I import the data once, maybe twice or 3 times if I receive an update, store it in SQL and then all my Python scripts read from SQL.

The code I am using to read the file is:

import numpy as np
import pandas as pd
import time
import multiprocessing
from multiprocessing import Pool
def parallel_read():
    pool = Pool(num_cores)
    # reads 1 row only, to retrieve column names and sheet names
    mydic = pd.read_excel('excel_write_example.xlsx', nrows=1, sheet_name=None)
    sheets =[]
    for d in mydic:
        sheets.extend([d])
    dataframes  = pool.map( read_single_sheet , sheets  )
    return dataframes

def parallel_read_async():
    pool = Pool(num_cores)
    # reads 1 row only, to retrieve column names and sheet names
    mydic = pd.read_excel('excel_write_example.xlsx', nrows=1, sheet_name=None)
    sheets =[]
    for d in mydic:
        sheets.extend([d])
    dataframes  = pool.map_async( read_single_sheet , sheets  ) 
    output = None
    # this below doesn`t work - can`t understand why
    output = [p.get() for p in dataframes]
    return output

def read_single_sheet(sheet):
    out = pd.read_excel('excel_write_example.xlsx', sheet_name=sheet )
    return out

num_cores = multiprocessing.cpu_count() 

if __name__=='__main__':
    start=time.time()
    out_p = parallel_read()
    time_par = time.time() -start

    out_as = parallel_read_async()
    time_as = time.time() - start - time_par

The code I used to create the Excel is:

import numpy as np
import pandas as pd

sheets = 15
rows= int(10e3)

writer = pd.ExcelWriter('excel_write_example.xlsx')

def create_data(sheets, rows):
    df = {} # dictionary of dataframes
    for i in range(sheets):
        df[i] = pd.DataFrame(data= np.random.rand(rows,30) )
        df[i]['a'] = 'some long random text'
        df[i]['b'] = 'some more random text'
        df[i]['c'] = 'yet more text'
    return df

def data_to_excel(df, writer):
    for d in df:
        df[d].to_excel(writer, sheet_name = str(d), index=False)
    writer.close()

df = create_data(sheets, rows)
data_to_excel(df, writer)

Answer 1

I am posting this as an answer because, while it doesn't answer the question of how to do it in Python, it still provides a feasible alternative to speed up the reading time materially, so it can be of interest to any Python user ; additionally, it relies only on open-source software, and requires the user to learn only a couple of commands in R.

My solution is: do it in R!

I posted about it here , which also shows my (very minimal) code ; basically, on the same file, R's readxl took 5.6 seconds. To recap:

• Python from xlsx: 33 seconds
• Python from CSV: ca. 2 seconds
• R from xlsx: 5.6 seconds

The link also has an answer which shows parallelising can speed up the process even more.

I believe the key difference is that the pandas.read_cs v relies on C code, while pandas.read_excel relies on more Python code. R's readxl is probably based on C. It might be possible to use a C parser to import xlsx files into Python, but AFAIK no such parser is available as of now.

It is a feasible solution because, after importing into R, you can easily export to a format which retains all the information on data types, and which Python can read from (SQL, parquet, etc). Not everyone will have a SQL server available, but formats like parquet or sqlite don't require any additional software.

So the changes to the workflow are minimal: the initial data loading, which, at least in my case, tends to be a one-off, is in R, and everything else continues to be in Python.

I also noticed that exporting the same sheets to SQL is much faster with R and DBI::dbWriteTable than with pandas (4.25 sec vs 18.4 sec).

Answer 2

A couple of things are going on here:

• The 36 seconds that parallel_read_async seems to be taking is in fact entirely taken up by the call to pd.read_excel('excel_write_example.xlsx', nrows=1, sheet_name=None) . map_async returns immediately, giving you the MapResult object, and you're immediately causing an exception by trying to iterate over it, so in this version essentially no work is being done by the read_single_sheet function.
• Furthermore, pd.read_excel with sheet_name=None is taking exactly as long as with sheet_name='1' etc. - so in your parallel_read function, each process is doing the work of parsing every row of every sheet. This is why it takes so much longer.

And now that I've written out, I remember that my company ran into this same problem, and we ended up implementing our own xlsx parser because of it. There's simply no way with xlrd - which pandas uses - to open an xlsx file without parsing it completely.

If you have the option to produce (or receive?) xls files instead, those should be much quicker to work with. Besides that, the export-to-csv option may be your best bet, if the speed of the non-parallel processing is unacceptable.

Answer 3

Here is an outline of how you can bypass file lock and achieve concurrency with little changes to your code:

import io
import xlrd
from functools import partial

def read_sheet(buff, sheetname):
    # reads 1 row only, to retrieve column names and sheet names
    df = pd.read_excel(buff, sheetname=sheetname)
    return df

if __name__=='__main__':
    start=time.time()
    time_par = time.time() -start
    xl = xlrd.open_workbook("myfile.xls")  # you fill in this
    sheets = xl.book.sheet_names()
    buff = io.BytesIO()
    xl.dump(buff)
    buff.seek(0)
    target = partial(read_sheet, buff)
    with Pool(num_processes) as p:
        dfs = p.map(target, sheetnames)
    time_as = time.time() - start - time_par