How to ignore blank rows in a csv file

Question

I'm using dictreader to open some csv files, adding them to one big list of dictionaries, and then using dictwriter to write the list of dictionaries out to one csv file.

The problem I'm having is that the resultant csv file has a bunch of blank rows between rows with data. I guess when the csv files are being read, it's not ignoring blank rows.

Could someone please send me in the right direction to find how I say to ignore the blank rows?

I've tried finding this in the csv module but no joy.

Any help would be much appreciated please.

Hi! Thanks for replying! I more want dictreader to read rows if there is anything of interest in them, but will ignore a row only if it's totally blank. Eg if I had

{'1': '', '2': 'two', '3': ''},
{'1': '', '2': '', '3': ''}

I would just want to keep

{'1': '', '2': 'two', '3': ''} 

I've found that the following works for me

for dictionary in csv.DictReader(open(filename)):
    if any(x != '' for x in dictionary.itervalues()):

Answer 1

You can read a fake file object that skips the blank lines in the real file. I'm not familiar with exactly what you're doing, but this will work better than mac's answer if the blank lines are making your reading process crash, or you really don't want the blank lines ever in there.

class BlankLineSkipper(object):
    def __init__(self, file):
        self.file = file
    def __iter__(self):
        return (line for line in self.file if line.strip())
    def read(self):
        return ''.join(self)

>>> print open('lol.csv').read()
5,7,8

1,2,3

abc,lol,haha


>>> list(csv.reader(open('lol.csv')))
[['5', '7', '8'], [], ['1', '2', '3'], [], ['abc', 'lol', 'haha'], []]

>>> list(csv.reader(BlankLineSkipper(open('lol.csv'))))
[['5', '7', '8'], ['1', '2', '3'], ['abc', 'lol', 'haha']]

(You might need to implement readline() or something else to make your code work, depending on how it uses the file object.)

Answer 2

If I understand you correctly, you simply have to filter your dictionary for blank lines before dumping it to a file. A trivial example for getting you started:

>>> d = {'l1': 'data', 'l2': '   '}
>>> dict([(k, v) for k, v in d.iteritems() if v.strip()])
{'l1': 'data'}

Does this help?

Answer 3

Your iterating through lines form the dictreader and then only "using" lines that have values. mac's list comprehension to check for values should be the correct method to clear out unwanted lines by returning an empty dictionary. You can write out the resulting dictionary, {'2': 'two'}, using the dictwriter object even if all keys aren't present.

The sample below performs a check of the line dict to see if it has any values assigned.

column_headers = ["1", "2", "3"]
dictwriter = csv.DictWriter(open("output.csv", "wb"), fieldnames=column_headers)
for line in dictreader:
    # check if the line contains "interesting" values.
    # --> Result will be empty list if not and evaluate to False
    if [True for v in line.values() if v.strip()]: 
        # line is not EMPTY, process as desired
        dictwriter.writerow(line)