I know beforehand what columns I don't need from an excel file and I'd like to avoid them when reading the file to improve the performance. Something like this:
import pandas as pd
df = pd.read_excel('large_excel_file.xlsx', skip_cols=['col_a', 'col_b',...,'col_zz'])
There is nothing related to this in the documentation. is there any workaround for this?
If your version of pandas allows (check first if you can pass a function to usecols), I would try something like:
import pandas as pd
df = pd.read_excel('large_excel_file.xlsx', usecols=lambda x: 'Unnamed' not in x,)
This should skip all columns without header names. You could substitute 'Unnamed' with a list of column names you do not want.
You can use the following technique. Let the columns we don't want(want to skip) are 2 5 8 , then find all reamining columns we DO WANT TO KEEP as cols
such that:
In [7]: cols2skip = [2,5,8]
In [8]: cols = [i for i in range(10) if i not in cols2skip]
In [9]: cols
Out[9]: [0, 1, 3, 4, 6, 7, 9]
and then we can use those remaining columns(which we DO WANT TO KEEP ) using usecols
:
df = pd.read_excel(filename, usecols=cols)
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.