append content of one csv file to another using python

Question

I have 2 csv files:

• output.csv
• output1.csv

output.csv has a 5 columns of titles.
output1.csv has about 40 columns of different types of data.

I need to append all the content of output1.csv to output.csv. How can I do this?
could somebody please give me a hint on how to go about it ???
i have the following code :

reader=csv.DictReader(open("test.csv","r"))
allrows = list(reader)

keepcols = [c for c in allrows[0] if all(r[c] != '0' for r in allrows)]

print keepcols
writer=csv.DictWriter(open("output.csv","w"),fieldnames='keepcols',extrasaction='ignore')
writer.writerows(allrows)

    with open("test1.csv","r") as f:
        fields=next(f).split()
        # print(fields)
        allrows=[]
        for line in f:
            line=line.split()
            row=dict(zip(fields,line))
            allrows.append(row)
            # print(row)
        keepcols = [c for c in fields if any(row[c] != '0' for row in allrows)]
        print keepcols
        writer=csv.DictWriter(open("output1.csv","w"),fieldnames=keepcols,extrasaction='ignore')
        writer.writerows(allrows)

test.csv generates output.csv
test1.csv generates output1.csv

im trying to see if i can make both files generate my output in the same file..

Answer 1

If I understand your question correctly, you want to create a csv with 41 columns - the 1 from output.csv followed by the 40 from output1.csv. I assume they have the same number of rows (if not - what is the necessary behavior?)

Try using the csv module:

import csv
reader = csv.reader(open('output.csv', 'rb'))
reader1 = csv.reader(open('output1.csv', 'rb'))
writer = csv.writer(open('appended_output.csv', 'wb'))
for row in reader:
    row1 = reader1.next()
    writer.writerow(row + row1)

If your csv files are formatted with special delimiters or quoting characters, you can use the optional keyword arguments for the csv.reader and csv.writer objects. See Python's csv module documentation for details...

EDIT: Added 'b' flag, as suggested.

Answer 2

This recent discussion looks very similar to what you are looking for except that the OP there wanted to concatenate mp3 files.

EDIT:

import os, sys
target = '/path/to/target'
src1 = '/path/to/source1.csv'
src2 = '/path/to/source2.csv'
tf = open(target, 'a')
tf.write(open(src1).read())
tf.write(open(src2).read())
tf.close()

try this, this should work since you simply want to do the equivalent of cat src1 src2 > target of shell command

Answer 3

"I need to append all the content of output1.csv to output.csv." ... taken literally that would mean write each row in the first file followed by each row in the second file. Is that what you want??

titles of what? the 40 columns in the other file?? If this is so, then assuming that you want the titles written as a row of column headings:

import csv
titles = [x[0] for x in csv.reader(open('titles.csv', 'rb'))]
writer = csv.writer(open('merged.csv', 'wb'))
writer.writerow(titles)
for row in csv.reader(open('data.csv', 'rb')):
    writer.writerow(row)

Answer 4

You could also use a generator from the reader if you want to pass a condition:

import csv
def read_generator(filepath:str):
   with open(filepath, 'rb'):
     reader = csv.reader(f)
     for row in reader:
       if row[0] == condition:
         yield row

and then write from that with:

writer = csv.writer(open("process.csv", "rb"))
write.writerow(read_generator(file_to_read.csv))