迭代xlsx文件并删除unicode python openpyxl

Question

I'm trying to convert all of the excel files on my computer to CSV files (sheet by sheet). Some of the .xlsx files are massive (over 100MB). I'm still having a couple issues:
1. My function to remove non-unicode characters is very slow
2. I'm not sure that I'm using openpyxl's iteration properly, as I'm still using a lot of memory and am afraid that if I really let this thing run, it'll hit a memory error
Also, looking for any coding help in general, as I'm still very new to code in general.

import csv
from formic import FileSet
from openpyxl import load_workbook
import re
from os.path import basename
import os
import string


def uclean(s): # Clean out non-unicode chars for csv.writer - SLOW
    try:
        return ''.join(char for char in s if char in string.printable).strip()
    except:
        return ''

def fclean(s): # Clean out non-filename-safe chars
    return ''.join([c for c in s if re.match(r'\w', c)])

xlsx_files = FileSet(directory='C:\\', include='**\\*.xlsx') # the whole computer's excel files
for filename in xlsx_files:
    wb = load_workbook(filename, use_iterators=True, read_only=True)  # This is still using > 600 MBs
    for sheet in wb.worksheets:
        i = wb.worksheets.index(sheet)
        bf = os.path.splitext(
            basename(filename))[0]
        sn = fclean(str(wb.get_sheet_names()[i]))
        f = bf + '_' + sn + '.csv'
        if not os.path.exists(f):
            with open(f, 'wb') as outf:
                out_writer = csv.writer(outf)
                for row in sheet.iter_rows():
                    out_writer.writerow([uclean(cell.value) for cell in row])

Answer 1

Using encode will be a lot faster:

#lines is some French text
In [80]: %timeit [s.encode('ascii', errors='ignore').strip() for s in lines]
10000 loops, best of 3: 15.3 µs per loop

In [81]: %timeit [uclean(s) for s in lines]                          
1000 loops, best of 3: 522 µs per loop

As for your openpyxl question, I'll have to get back to you -- the only thing I can think of right now is that it might be possible to load just one worksheet at a time rather than the whole workbook. Keep in mind that since wb is local to the loop, it's going to be replaced with a new object each iteration, so it's not like you're going to use an additional 600mb of memory for each file.

Answer 2

Read-only mode really does read cells one at a time so memory use is minimal. However, based on you wanting to convert all the text to ascii I wonder if the reason is that there is a lot of text in the Excel files. Excel employs an optimisation where it stores all strings in a big list which cells reference. If you have a lot of unique strings then it is possible that these are the root of any memory issues as we have to keep them in memory in order to be able to read them.

Regarding conversion: you can probably use a wrapper to save to UTF-8 and so remove any inline encoding whatsoever.