多个 CSV 合二为一，文件名作为 Pandas 中的列名

Question

I have a directory with a hundred the CSV files inside. One of the CSV looks like this;

Time    ID
09:00   A
..      ..

I want to join all of the csv into one dataframe with including name of file (append by axis=1) I used this code:

files = glob.glob(data/*.csv')
df = pd.concat([pd.read_csv(fp).assign(File=os.path.basename(fp).split('.')[0]) for fp in files], axis=1)
df.to_csv('new.csv')
df

I got a result looks like this

Time    ID  File  Time  ID  File    ..
09:00   A   01    09:00 B   02      ..
..      ..  ..    ..    ..  ..      ..

I want to join the ID column name with the file name as a column name. my expected result looks like this:

Time    01_ID   Time    02_ID   ..
09:00   A       09:00   B       ..
..      ..      ..      ..      ..

Answer 1

You can use dictionary comprehension first:

comp = {os.path.basename(fp).split('.')[0]: pd.read_csv(fp) for fp in files}
df = pd.concat(comp, axis=1)

And then filter in list comprehension for convert MultiIndex in columns :

df.columns = [f"{a}_{b}" if b == 'ID' else b for a, b in df.columns]
print (df)
    Time 01_ID   Time 02_ID
0  09:00     A  09:00     B

df.to_csv('new.csv')

EDIT: Better solution is create unique columns names:

df.columns = df.columns.map('_'.join)
print (df)
  01_Time 01_ID 02_Time 02_ID
0   09:00     A   09:00     B