Append multiple CSVs with the name of the name of each CSV in all rows - Python

Question

I have this code in Python which appends multiple CSVs that are located in the same file. It works OK.

import os
import pandas as pd

targetdir = r'E:/Base 2015-1/Carreras'

filelist = os.listdir(targetdir) 

big_df=pd.DataFrame()

for filename in filelist:
    big_df = big_df.append(pd.read_csv(os.path.join(targetdir, filename), header= None),ignore_index=True)

However, I would like to create a column in the output with the name of the CSV each row is from before appending. For example:

CSV File: A1
1 2 3
a b 3

CSV File: A2
2 4 1
a e r

Append:
1 2 3 A1
a b 3 A1
2 4 1 A2
a e r A2

So, How Can I change my code to get that?

Answer 1

I tested that the following does what you have specified:

# from for loop onward... 
for filename in filelist:
    tmpdf = pd.read_csv(os.path.join(targetdir, filename),sep=' ',header=None)
    tmpdf[len(tmpdf.columns)] = filename # add column with filename 
    bigdf = pd.concat([bigdf,tmpdf],ignore_index=True)

Answer 2

You can use the converters parameter to change or format values on the fly. I am not sure if would append a column if one didn't exist but you can give this a try:

def file_converter(file_number):
    # This can be made more complicated if you want A..Z
    return 'A{0}'.format(file_number)

for file_number, filename in enumerate(filelist):
    file_path = os.path.join(targetdir, filename)
    # Modify the 4th column contents
    big_df = big_df.append(pd.read_csv(file_path, converters={3:file_converter(file_number)}))