Excel file in Python with XLRD

Question

I'm looking to load an excel file (xlsx) which is about 35MB and has close to 100k rows of data into a sqlite database for some research. The file has about 40 columns and I might want to selectively load columns into a sqlite DB.

I'm approaching this as a straightforward XLRD and load to Sqlite problem. Is there a better way of doing this - such as using a different module?

And given the volume of data, would the SQLite manager plugin for Firefox be the right utility to view some of the data?

Answer 1

I would definitely use pandas for such operations. It has implemented wrappers for many data inputs (including excel). It is based on numpy and features a bunch of statistical methods to apply on your data. You can easily select the columns of your choice, and then directly store them in any database.

Its main data structure is called dataframe .

An example of the code you could use to load, and store data.

import pandas as pd
dataframe = pd.read_excel(YOUR_FILE)
dataframe.to_sql(*args)

Answer 2

You could use a python command line tool like this one:

https://pypi.org/project/xlsx2sqlite/

Then once you have your data imported you can view it using sqlitemanager or dbeaver database tools.