如何从多个 CSV 文件中读取特定列，并使用 Python 跳过某些文件中不存在的列 Pandas

Question

I have data about online transactions that are stored in CSV files, one file per day. These files contain over 100 columns, but I only want to extract a few of them (eg user_id, event_type, event_time, event_store, sale_amount). The columns included in the files have changed over time, so that more recent files have different column names that I also would like to extract (eg discount_amount). I want to only extract the columns that I need in order to avoid loading a lot of unnecessary data.

So far, I have tried to use the pandas.read_csv("file_name.csv", usecols=col_list) argument in order to only load the columns I want. However, as not all CSV files contain these desired columns, when one of those files passes through the loop, it fails with the error message that the specific column was not found. Is there any way to make python skip a column that does not exist in a csv file, rather than produce an error and terminate?

Here is what I have so far:

data = []

col_list = ["user_id", "event_type", "event_time", "event_store", "sale_amount", "discount_amount"]

for obj in files:
    csv_obj = client.get_object(Bucket=bucket_name, Key=obj)
    body = csv_obj['Body']
    csv_string = body.read().decode('utf-8')
    temp = pd.read_csv(StringIO(csv_string), usecols=col_list)
    data.append(temp)

# combining all dataframes into one
event_data = pd.concat(data, ignore_index=True)

Thanks for any help given!

Answer 1

You could try to read only the columns names from the csv file and check them with your desired columns as follows:

import csv 

desired_col = ["user_id", "event_type"]  # I selected only two values

for file_name in csv_files:

    csv_cols = next(csv.reader(open(file_name))) # read only the csv columns names

    cols = [col for col in desired_col if col in csv_cols]

    df = pd.read_csv(file_name, usecols=cols)

Then, each time you read a new csv file, you need first to read the names of columns and then check desired_columns against csv_columns.