使用 Python 读取 .xlsx 文件的最快方法

Question

I'm trying to read data from a.xlsx file into a MySQL database using Python.

Here's my code:

wb = openpyxl.load_workbook(filename="file", read_only=True)
ws = wb['My Worksheet']

conn = MySQLdb.connect()
cursor = conn.cursor()

cursor.execute("SET autocommit = 0")

for row in ws.iter_rows(row_offset=1):
     sql_row = # data i need
     cursor.execute("INSERT sql_row")

conn.commit() 

Unfortunately, openpyxl 's ws.iter_rows() is painfully slow. I've tried similar methods using the xlrd and pandas modules. Still slow. Any thoughts?

Answer 1

You really need to benchmark your code and provide information about the size of the worksheet and the time taken to process it.

openpyxl's read-only mode is essentially a memory optimisation which avoids loading the whole worksheet into memory. When it comes to parsing Excel worksheets most of the work involved is converting XML to Python and there are limits to this.

However, two optimisations do spring to mind:

• keep your SQL statement outside the loop
• use executemany to pass lots of rows at once to the driver

These can be combined in something like like

INSERT_SQL = "INSERT INTO mytable (name, age…) VALUES (%s, %s, …)"
c.executemany(INSERT_SQL, ws.values)

If you only want a subset of the rows then look at using itertools.islice

This should be faster than your current code but you shouldn't expect miracles.

When it comes to pure performance, xlrd is a little faster than openpyxl when reading worksheets because it has a smaller memory footprint, largely related to being a read-only library. But it always loads a whole workbook into memory, which might not be want you want.

Answer 2

For reading, try http://github.com/AndyStricker/FastXLSX It claimed to use expat for event based parsing and stream based zip reader. Only shared string table has to kept in memory.

If still to slow, you can try compile it with Nuitka. I used to get 25% speed boost with Nuitka compiled lib.

For writing, try http://github.com/kz26/PyExecelerate

For MySQL, try the cython enabled CyMySQL http://github.com/nakagami/CyMySQL for bulk insertion, in my experience, this noticeably boosted the insertion speed compared to pymysql. The difference I saw only applies to massive bulk insertion in tight loop. Try different larger batch size to get best speed possible.

Answer 3

There is a rust lib calamine and python`s bindings for it. It gives 10x-20x speedup on read.

from python_calamine import get_sheet_data

recs: list[list] = get_sheet_data("myfile.xlsx", sheet=0)

If you wanna turn it into a pd.DataFrame :

df = pd.DataFrame.from_records(recs)