CSV Writer only writing first line in file

Question

So I have patent data I wish to store from an XML to a CSV file. I've been able to run my code through each iteration of the invention name, date, country, and patent number, but when I try to write the results into a CSV file something goes wrong.

The XML data looks like this (for one section of many):

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE us-patent-grant SYSTEM "us-patent-grant-v42-2006-08-23.dtd" [ ]>
<us-patent-grant lang="EN" dtd-version="v4.2 2006-08-23" file="USD0584026-20090106.XML" status="PRODUCTION" id="us-patent-grant" country="US" date-produced="20081222" date-publ="20090106">
<us-bibliographic-data-grant>
<publication-reference>
<document-id>
<country>US</country>
<doc-number>D0584026</doc-number>
<kind>S1</kind>
<date>20090106</date>
</document-id>
</publication-reference>

My code for running through and writing these lines one-by-one is:

for xml_string in separated_xml(infile): # Calls the output of the separated and read file to parse the data
    soup = BeautifulSoup(xml_string, "lxml")     # BeautifulSoup parses the data strings where the XML is converted to Unicode
    pub_ref = soup.findAll("publication-reference") # Beginning parsing at every instance of a publication
    lst = []  # Creating empty list to append into

    for info in pub_ref:  # Looping over all instances of publication

# The final loop finds every instance of invention name, patent number, date, and country to print and append into

        with open('./output.csv', 'wb') as f:
            writer = csv.writer(f, dialect = 'excel')

            for inv_name, pat_num, date_num, country in zip(soup.findAll("invention-title"), soup.findAll("doc-number"), soup.findAll("date"), soup.findAll("country")):
            #print(inv_name.text, pat_num.text, date_num.text, country.text)
            #lst.append((inv_name.text, pat_num.text, date_num.text, country.text))
                writer.writerow([inv_name.text, pat_num.text, date_num.text, country.text])

And lastly, the output in my .csv file is this:

"Content addressable information encapsulation, representation, and transfer",07475432,20090106,US

I'm unsure where the issue lies and I know I'm still quite a newbie at Python but can anyone find the problem?

Answer 1

The problem lies in this line with open('./output.csv', 'wb') as f:

If you want to write all rows into a single file, use mode a . Using wb will overwrite the file and thus you are only getting the last line.

Read more about the file mode here: https://docs.python.org/2/tutorial/inputoutput.html#reading-and-writing-files

Answer 2

You open the file in overwrite mode ( 'wb' ) inside a loop. On each iteration you erase what could have been previously written. The correct way is to open the file outside the loop:

...
with open('./output.csv', 'wb') as f:
    writer = csv.writer(f, dialect = 'excel')

    for info in pub_ref:  # Looping over all instances of publication

# The final loop finds every instance of invention name, patent number, date, and country to print and append into



        for inv_name, pat_num, date_num, country in zip(soup.findAll("invention-title"), soup.findAll("doc-number"), soup.findAll("date"), soup.findAll("country")):
            ...