merging files from separate folders

Question

I have two folders. Files in folder 1 look like this:

Year   Pressure
1995   1.2
1996   2.7
1997   3.1 
1998   5.6

Files in folder 2 look like this:

Year   NDVI
1995   1.0
1995   2.8
1995   0.2
1996   1.2
1996   0.9
1997   6.7
1997   5.7
1998   3.4
1998   1.2

there are 53 files in each of the folders. I want to merge them, based on the file order (they do all have corresponding names but they are in same order anyways)

I am using this so far:

import pandas as pd
import os

#path to folder 1
pth1=(r'D:\Sheyenne\Grazing_Regressions\NDVI\grazing')
#path to folder 2
pth2=(r'D:\Sheyenne\Grazing_Regressions\NDVI\NDVI')
#output pathway
outfile=(r'D:\Sheyenne\Grazing_Regressions\NDVI\final_merge')

for f in os.listdir(pth1):
    df = pd.read_csv(os.path.join(pth1, f))
    for f2 in os.listdir(pth2):
        df2=pd.read_csv(os.path.join(pth2, f2))
        outpath=os.path.join(outfile, f2)
        finalmerge=pd.merge(df,df2, left_on='Year', right_on='Year', how='right')
        finalmerge.to_csv(outpath)

but it is only merging the last file from pth1 to all of the files in pth2

Answer 1

You can use a single loop to keep things simple like this:

for f, f2 in zip(os.listdir(pth1),os.listdir(pth2)):
    df = pd.read_csv(os.path.join(pth1, f))
    df2 = pd.read_csv(os.path.join(pth2, f2))

    outpath=os.path.join(outfile, f2)

    finalmerge=pd.merge(df, df2, left_on='Year', right_on='Year', how='right')
    finalmerge.to_csv(outpath)

Answer 2

I'm unfamiliar with pandas but you could probably do this just by writing to a new file with the csv built in package if the files are structured in the same order. Something like

import os
import csv

path_one = 'your/path/here'
path_two = 'your/other_path/here'

one = open(path_one, 'r')
two = open(path_two, 'r')

headers = ['Year', 'NDVI', 'Pressure']
things_to_add = []

for i, line in enumerate(one):
    if i > 0:
        things_to_add.append(line.split(',')[1])


one.close()
ending_file = open('path/to/end/file.csv', 'w')
writer = csv.writer(ending_file)
writer.writerow(headers)

for i, line in enumerate(two):
    if i > 0:
        writer.writerow([line.split(',')[0], line.split(',')[1], things_to_add[i - 1])

two.close()
ending_file.close()