大熊猫递归read_csv，同时将列添加到每个

Question

I'm recursively reading many csv's in multiple directories, and each time a read one in I want to add a column called num which is just the index of which csv it was in the list.

path = r'data/'
all_files = glob.glob(os.path.join(path, "**/*.csv"), recursive=True)

After I have the filenames I want to read each in and then add the column, but leave it as a generator to simply concat afterwards. Is it possible to enumerate a generator?

df_from_each_file = (pd.read_csv(f) for f in all_files)
df_from_each_file = (df.insert(0,'num',i,allow_duplicates=True) for i, df in enumerate(df_from_each_file))
concatenated_df   = pd.concat(df_from_each_file, ignore_index=True)

This just returns a bunch of None df's

Answer 1

Use enumerate and DataFrame.assign within the generator like:

path = r'data/'
all_files = glob.glob(os.path.join(path, "**/*.csv"), recursive=True)

df_from_each_file = (pd.read_csv(f).assign(num=i) for i, f in enumerate(all_files))    
concatenated_df = pd.concat(df_from_each_file, ignore_index=True)