Python loop only printing final loop of the dataframe as output to Excel

Question

Been stuck on this (most likely very simple) issue but my output to Excel only prints the values in the final loop. I believe this is something to do with indentation, and have tried putting the values out of the for loop which has been unsuccessul. Here is some dummy code to help understand the issue:

#dummy code

#reads each file in folder, runs through a bunch of functions, and prints output in console
import os,glob
import csv
import pandas as pd
filename ='path to folder'
for filename in glob.glob(os.path.join(folder_path, '*.txt')):
    with open(filename, 'r') as f:
        text = f.read()
        data =function1(file= filename)
        data= function2(file= filename)
        data = function3(file= filename)
        final =pd.DataFrame(data)
        print(final)
        final.to_excel('output.xlsx')

Output in console looks like this, which is correct and what I would like to export to csv:

0  some text here ...
1  more text...
2  clear text...
3  final data...
                                                   0
0  yes no...
1  does lots...
2  happy sunflower ...
3  ate food...

                                                  0
0  final data ...
1  apple strawberry...
2  different dataset...
3  dinne meals ...

Any advice would be appreciated. Thank you.

Answer 1

You are overwritting output.csv on each iteration of the for loop. Try doing this instead

import pandas as pd
#use enumerate to have a counter variable at each index
for filename, n in enumerate(glob.glob(os.path.join(folder_path, '*.txt'))):
    with open(filename, 'r') as f:
        #your logic
        #this will be overwritten on next pass of for loop
        final = pd.DataFrame(data)
        #write to file with appended index- avoids file being overwritten on next pass
        final.to_csv("output_%d.csv" % n)

Now your dataframes will all be saved in the form output_x.csv. Ie if three files are found you will see the following files: output_1.csv, output_2.csv, output_3.csv