Pandas Concatenate dataframes

Question

This is driving me nuts! I have several Dataframe that I am trying to concatenate with pandas. The index is the filename. When I use df.to_csv for individual data frames I can see the index column (filename) along with the column of interest. When I concatenate along the filename axis I only get the column of interest and numbers. No filename.

Here is the code I am using as is. It works as I expect up until the "all_filename" line.

for filename in os.listdir(directory):
    if filename.endswith("log.csv"):
        df = pd.read_fwf(filename, skiprows=186, nrows=1, names=["Attribute"])
        df['System_Library_Name'] = [x.split('/')[6] for x in df['Attribute']]
        df2= pd.concat([df for filename in os.listdir(directory)], keys=[filename])
        df2.to_csv(filename+"log_info.csv", index=filename)
        
        all_filenames = glob.glob(os.path.join(directory,'*log_info.csv'))
        cat_log = pd.concat([pd.read_csv(f) for f in all_filenames ])
        cat_log2= cat_log[['System_Library_Name']]
        cat_log2.to_excel("log.xlsx", index=filename)

I have tried adding keys=filename to the 3rd to last line and giving the index a name with df.index.name=

I have used similar code before and had it work fine, however this is only one column that I am using from a larger original input file if that makes a difference.

Any advice is greatly appreciated!

Answer 1

df = pd.concat(
          # this is just reading one value from each file, yes?
         [pd.read_fwf(filename, skiprows=186, nrows=1, names=["Attribute"])
            .set_index(pd.Index([filename]))
            .applymap(lambda x: x.split('/')[6])
            .rename(columns={'Attribute':'System_Library_Name'})
          for filename in glob.glob(os.path.join(directory,'*log.csv'))
         ]
     )
df.to_xlsx("log_info.xlsx")