Problems Importing CSV File as a List into Python

Question

I generated a list of urls into a Python list named MasterList using Beautiful Soup and I would like to save this list as a text or .csv file to then import into other Python scraping scripts as a list that I can iterate over and feed into Beautiful Soup. I don't want to use pickle or import the functions that generate MaterList into my other script files because generating MasterList takes a very long time so I only want to generate it once and use it repeatedly. So I saved MasterList using the following

import csv

ListCsv=csv.writer(open("MasterList.csv","w"))
ListCsv.writerow(MasterList)

and the file shows up in my directory as a .csv file populated with the correct information. However, when I try running the code

test = csv.reader(open("MasterList.csv","r"))
test = list(test)
print(test)

I don't get any errors and the list test is empty. That is, the print(test) command displays [] . I've looked around many forums and tried different variations of importing the csv file to save as a list and nothing has worked. Depending on the syntax and method I use, I either get and empty list or print(test) displays <_csv.reader object at 0x1014c78a0> .

I should add that I am using Python Ver 2.7.3 on Mac OSX 10.7.5 and I have installed the latest version of the csv module. Any help would be greatly appreciated.

Answer 1

You should close the file object explicitly or better use a with statement for handling file as it'll automatically close the file for you, otherwise the data you actually wrote to the file might not be flushed to it yet.

Demo:

>>> f = open('abc.csv', 'w')
>>> writer = csv.writer(f)
>>> writer.writerow(range(10))
>>> list(csv.reader(open("abc.csv"))) #empty result, nothing flushed to file yet.
[]
>>> f.close()             #close the file object, now the data is flushed to file
>>> list(csv.reader(open("abc.csv")))
[['0', '1', '2', '3', '4', '5', '6', '7', '8', '9']]

Better use the with statement:

with open('abc.csv', 'w') as f:
    writer = csv.writer(f)
    writer.writerow(range(10))
with open('abc.csv') as f:
    reader = csv.reader(f)
    print list(reader)