Ignore columns that only contain missing values using pd.read_csv

Question

I have created an application that reads in data using pd.read_csv. There are some datasets that we get that have columns that only contain missing values (empty cells). Is there any way using pandas to not load those columns into the dataframe? As the dataset can be quite large it would be more convenient to ignore them in the pre-loading stage.

Of course I could delete them from the excel sheet, but I am aiming to make the data loading as automated as possible.

Answer 1

You can use pd.read_csv(file, keep_default_na=False). All NA values will not be loaded, but instead of them will be empty rows, so this will save memory. For more details you can read: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_csv.html .