Append 文件名到 pandas 中的 csv 文件

Question

I'm trying to append the file name of my CSV files as a column name in those CSV files and I have the basic idea and code of how to do it, just can't integrate it in my current code. Its probably very easy.

This is how I'm reading my CSV files and appending them in a dataframe

big_frame = pd.concat([pd.read_csv(f, skiprows=0 , header=None , index_col= False ,names=col_Names) for f in glob.glob('filepath' + "/*.csv")],
                      ignore_index=True)

and I know I just need to add these two lines somewhere in the code

frame['filename'] = os.path.basename(f)
f.append(frame)

any help?

For example, I have 3 CSV files, each with the same column names as shown below.

Column A Column B Column C 

I want to concatenate them all in a big data frame with a new column that has there original CSV file name like

Column A Column B Column C filename
                            file 1
                            file 2
                            file 3

Answer 1

You can use df.assign and you can open files using Path.glob from pathlib module.

from pathlib import Path

big_frame = pd.concat(
    [pd.read_csv(file.name, skiprows=0, header=None, index_col=False, names=col_Names).assign(filname=file.name)
     for file in Path('filepath').glob('*.csv')],
    ignore_index=True)

Answer 2

Use DataFrame.assign() after read_csv to add a column as soon as it's read:

big_frame = pd.concat([pd.read_csv(f, ...).assign(filename=os.path.basename(f))
                       for f in glob.glob('filepath' + "/*.csv")],
                      ignore_index=True)

(The ... refers to all the other paramters to read_csv .)

Other changes:

1. pd.concat() accepts a generator so you don't need to create a list of df's with the list comprehension. It just uses more memory than needed and since you're reading off disk, provides no performance improvement. And when you use a generator expression, it will need extra parentheses. Note the extra indent for readability:

    big_frame = pd.concat((pd.read_csv(f, ...).assign(filename=os.path.basename(f)) for f in glob.glob('filepath' + "/*.csv")), ignore_index=True)

2. For the globbing, use os.path.join (since filepath is a variable name and not the actual path:

    glob.glob(os.path.join(filepath, '*.csv'))

   Or use pathlib.Path and Path.glob as in deadshots's answer.

With all the params:

big_frame = pd.concat((pd.read_csv(f,
                                   skiprows=0,
                                   header=None,
                                   index_col=False,
                                   names=col_Names,
                        ).assign(filename=os.path.basename(f))
                            for f in glob.glob(os.path.join(filepath, '*.csv'))
                       ),
                       ignore_index=True)

---

Btw, I do this when mass reading CSV's, except I don't use only the basename because I want the full path to the file included. Especially useful when reading same-format CSV's from different sources/directories.