如何提高使用 python 读取多个 excel 文件的运行时间？

Question

I created function that iterates over a folder containing excel files and creates a list of all the headers across all sheets. I t works fine but is VERY slow . Do you have any ideas on how to improve it? THANKS!

import glob

# file directory
path = r'C:\Users\John\Excel_folder' 
all_files = glob.glob(path + "/*.xlsx")

def get_columns(file):    
    sheets = pd.ExcelFile(file).sheet_names
    for sheet in sheets:
        for i in (list(pd.read_excel(file, sheet, nrows=0).columns)):
                  col.append(i)
col=[]
for i in all_files:
    get_columns(i)

col

Answer 1

you can pass None to sheet_name in read_excel to read all sheets at once. It creates a dictionary of dataframe, so at the end you can do with list comprehension.

def get_columns(file):
    return [c 
            for df in pd.read_excel(file, 
                                    sheet_name=None, 
                                    nrows=0).values() 
            for c in df.columns]

col = [c for file in all_files for c in get_columns(file)]

it should be faster because you open once the file instead of many times.