使用 openpyxl 从大文件中删除行

Question

i'm working with openpyxl on a.xlsx file which has around 10K products, of which some are "regular items" and some are products that need to be ordered when required. For the project I'm doing I would like to delete all of the rows containing the items that need to be ordered.

I tested this with a small sample size of the actual workbook and did have the code working the way I wanted to. However when I tried this in the actual workbook with 10K rows it seems to be taking forever to delete those rows (it has been running for nearly and hour now).

Here's the code that I used:

wb = openpyxl.load_workbook('prod.xlsx')
sheet = wb.get_sheet_by_name('Sheet1')
def clean_workbook():
    for row in sheet:
        for cell in row:
            if cell.value == 'ordered':
                sheet.delete_rows(cell.row)

I would like to know is there a faster way of doing this with some tweaks in my code? Or is there a better way to just read just the regular stock from the workbook without deleting the unwanted items?

Answer 1

You can open with read-only mode, and import all content into a list, then modify in list is always a lot more faster than working in excel. After you modify the list, made a new worksheet and upload your list back to excel. I did this way with my 100k items excel.

Answer 2

Deleting rows in loops can be slow because openpyxl has to update all the cells below the row being deleted. Therefore, you should do this as little as possible. One way is to collect a list of row numbers, check for contiguous groups and then delete using this list from the bottom.

A better approach might be to loop through ws.values and write to a new worksheet filtering out the relevant rows. Copy any other relevant data such as formatting, etc. Then you can delete the original worksheet and rename the new one.

ws1 = wb['My Sheet']
ws2 = wb.create_sheet('My Sheet New')

for row in ws1.values:
    if row[x] == "ordered": # we can assume this is always the same column
       continue
    ws2.append(row)

del wb["My Sheet"]
ws2.title = "My Sheet"

For more sophisticated filtering you will probably want to load the values into a Pandas dataframe, make the changes and then write to a new sheet.