python - how to create a dictionary with a lot of dataframes using a glob.glob

Question

I have created a dictionary that contains a bunch of dataframes using the following code

files = ('auction_aggregated_curves_germany_austria_20100101.csv', 'auction_aggregated_curves_germany_austria_20100102.csv', 'auction_aggregated_curves_germany_austria_20100103.csv', 'auction_aggregated_curves_germany_austria_20100104.csv', 'auction_aggregated_curves_germany_austria_20100105.csv')

dfs = ('df1', 'df2', 'df3', 'df4', 'df5')

list_of_dfs = {}
for df, file in zip(dfs, files):
       list_of_dfs[df] = pd.read_csv(file, skiprows=1)

However I wonder if there is an easier way to automatize the process using glob.iglob to call a bunch of cvs files which are only different in the last number -which indicates the date in year, month and day-. I have more than 365 files and it would be really helpful if someone can help me to avoid writing all the file names.

Thanks in advance.

Answer 1

You can use the pathlib module for this. It includes a glob method.

from pathlib import Path

dataframes = {}

csv_root = Path(".")

for csv_path in csv_root.glob("*.csv"):
    key = csv_path.stem  # the filename without the ".csv" extension
    dataframes[key] = pd.read_csv(csv_path)

Using this code with your example data, the dataframes dict would look like this:

dataframes == {
    "auction_aggregated_curves_germany_austria_20100101": <DataFrame(...)>,
    "auction_aggregated_curves_germany_austria_20100102": <DataFrame(...)>,
    # etc...
}