Python：循环通过 excel 表并写入 csv

Question

I have a very large dataset (>100gb). It has many excel files (.xlsx). Each xlsx file has many sheets. The data in each sheet is shown in the below picture.

I would like to combine these sheets into a csv file, and change this wide format to a long format so that:

1. The first column contains the excel file name,
2. The second column contains the sheet's name,
3. The third, fourth, and fifth are the "ticker", "Name", "Detail Holding Type" column from the picture above,
4. The sixth column would be the "date" (which are the first row), and
5. The final column contains the number.

What would be the most effective way to do this? I have the code to loop through files and sheets, but cannot transpose the (wide) data to the long format that I am after. Below is my attempt to loop:

import csv
from os import listdir
from os.path import isfile, join

mypath = "E:/data_download/Python_test_files/"
file_lists = [f for f in listdir(mypath) if isfile(join(mypath, f))]
import xlrd


for file in file_lists:
    book = xlrd.open_workbook(f'{mypath}{file}')
    sheet_names = book.sheet_names()
    print(sheet_names)
    for sheet in book.sheets():
        for row in sheet.get_rows():

Answer 1

taking things step by step (and take in mind that in order for the process to be as fast as possible, you have to use native python as much as you can and only use the other libraries when you absolutely HAVE TO.): so you want one csv file out of all those sheets. what you should do is first make a 2D list of all the rows from all the sheets, and however you want them constructed like you've mentioned, that you want to include in the csv file, and then finally import them into the csv file with the Dataframe class using the pandas library:

import pandas as pd
my_list = [...] # your 2D list containing the rows
dataset= pd.DataFrame(my_list, columns=['column1','column2', '...') # the name of your columns
dataset.to_csv('/PATH/file.csv')