Extracting elements from a list (inc sub-lists) with a for loop, creating a sublist and exporting it to CSV file, Python

Question

I have a list of elements (some elements contain sub lists) called data_all ie

data_all=[{u'heatindexm': u'-9999', u'windchillm': u'-999', u'wdire': u'SW', u'wdird': u'230', u'windchilli': u'-999', u'hail': u'0', u'heatindexi': u'-9999', u'precipi': u'-9999.00', u'thunder': u'0', u'pressurei': u'29.44', u'snow': u'0', u'pressurem': u'997', u'fog': u'0', u'icon': u'mostlycloudy', u'precipm': u'-9999.00', u'conds': u'Mostly Cloudy', u'tornado': u'0', u'hum': u'82', u'tempi': u'50.0', u'tempm': u'10.0', u'dewptm': u'7.0', u'rain': u'0', u'dewpti': u'44.6', u'date': {u'mday': u'11', u'hour': u'00', u'min': u'20', u'mon': u'05', u'pretty': u'12:20 AM BST on May 11, 2014', u'year': u'2014', u'tzname': u'Europe/London'}, u'visi': u'6.2', u'vism': u'10.0', u'utcdate': {u'mday': u'10', u'hour': u'23', u'min': u'20', u'mon': u'05', u'pretty': u'11:20 PM GMT on May 10, 2014', u'year': u'2014', u'tzname': u'UTC'}, u'wgusti': u'-9999.0', u'metar': u'METAR EGBB 102320Z 23009KT 200V260 9999 SCT021 BKN027 10/07 Q0997', u'wgustm': u'-9999.0', u'wspdi': u'10.4', u'wspdm': u'16.7'}, {u'heatindexm': u'-9999', u'windchillm': u'-999', u'wdire': u'SW', u'wdird': u'230', u'windchilli': u'-999', u'hail': u'0', u'heatindexi': u'-9999', u'precipi': u'-9999.00', u'thunder': u'0', u'pressurei': u'29.44', u'snow': u'0', u'pressurem': u'997', u'fog': u'0', u'icon': u'mostlycloudy', u'precipm': u'-9999.00', u'conds': u'Mostly Cloudy', u'tornado': u'0', u'hum': u'82', u'tempi': u'50.0', u'tempm': u'10.0', u'dewptm': u'7.0', u'rain': u'0', u'dewpti': u'44.6', u'date': {u'mday': u'11', u'hour': u'00', u'min': u'50', u'mon': u'05', u'pretty': u'12:50 AM BST on May 11, 2014', u'year': u'2014', u'tzname': u'Europe/London'}, u'visi': u'6.2', u'vism': u'10.0', u'utcdate': {u'mday': u'10', u'hour': u'23', u'min': u'50', u'mon': u'05', u'pretty': u'11:50 PM GMT on May 10, 2014', u'year': u'2014', u'tzname': u'UTC'}, u'wgusti': u'-9999.0', u'metar': u'METAR EGBB 102350Z 23010KT 200V270 9999 BKN024 BKN033 10/07 Q0997', u'wgustm': u'-9999.0', u'wspdi': u'11.5', u'wspdm': u'18.5'}, {u'heatindexm': u'-9999', u'windchillm': u'-999', u'wdire': u'WSW', u'wdird': u'240', u'windchilli': u'-999', u'hail': u'0', u'heatindexi': u'-9999', u'precipi': u'-9999.00', u'thunder': u'0', u'pressurei': u'29.44', u'snow': u'0', u'pressurem': u'997', u'fog': u'0', u'icon': u'cloudy', u'precipm': u'-9999.00', u'conds': u'Overcast', u'tornado': u'0', u'hum': u'82', u'tempi': u'50.0', u'tempm': u'10.0', u'dewptm': u'7.0', u'rain': u'0', u'dewpti': u'44.6', u'date': {u'mday': u'11', u'hour': u'01', u'min': u'20', u'mon': u'05', u'pretty': u'1:20 AM BST on May 11, 2014', u'year': u'2014', u'tzname': u'Europe/London'}, u'visi': u'5.0', u'vism': u'8.0', u'utcdate': {u'mday': u'11', u'hour': u'00', u'min': u'20', u'mon': u'05', u'pretty': u'12:20 AM GMT on May 11, 2014', u'year': u'2014', u'tzname': u'UTC'}, u'wgusti': u'-9999.0', u'metar': u'METAR EGBB 110020Z 24010KT 210V270 8000 BKN018 OVC025 10/07 Q0997', u'wgustm': u'-9999.0', u'wspdi': u'11.5', u'wspdm': u'18.5'}]

NB: I've only shown three elements above, the actual list is >1,000 elements

I am using a for loop to extract the values of interest from the elements contained within the list, pass these elements of interest to a separate list which is then exported as a CSV file using the following code:

#define output file location and create empty list
import csv
Output_file="outputfile.csv"
interesting_data = []
#I then implement a for loop so that the values of interest from each element in the data_all list
#can be obtained
for values in data_all:
    data_string_sample=((values['utcdate']['mday']),(values['utcdate']['mon']),(values['utcdate']['year']),(values['utcdate']['hour']),(values['utcdate']['min']),(values['tempm']),(values['hum']),(values['pressurem']))
    interesting_data.append(data_string_sample)
    #append elements of interest onto the new list, interesting_data
    #pushing the list to the csv file by way of csv.writer
    #NB specfiting ab instead of wb so the data is appended to the CSV file instead of overwriting it
    #on subsequent passes
    with open(Output_file, "ab") as resultFile:
        #Override the default white space generator ('\r\n')
        #and specfiying a new lineterminator of '\n' so that each
        #list element starts on the next sequential row instead
        #of skipping a row
        writer = csv.writer(resultFile, lineterminator='\n')
        writer.writerows(interesting_data)
#end code

My problem is that instead of the csv file showing 3 rows of data (ie the elements within interesting_data) it seems to add a the first element as row one to the data file, repeate the loop again but add element one as row2 and element 2 as row 3, repeate the loop again and add element one as row4, element 2 as row5, element 3 as row6 and so on to produce 6 rows ie the csv file output looks like:

10  5   2014    23  20  10  82  997
10  5   2014    23  20  10  82  997
10  5   2014    23  50  10  82  997
10  5   2014    23  20  10  82  997
10  5   2014    23  50  10  82  997
11  5   2014    0   20  10  82  997

While I want it to look like:

10  5   2014    23  20  10  82  997
10  5   2014    23  50  10  82  997
11  5   2014    0   20  10  82  997

I know I'm on the right track because interesting_data has the elements I want

>>>interesting_data
[(u'10', u'05', u'2014', u'23', u'20', u'10.0', u'82', u'997'), (u'10', u'05', u'2014', u'23', u'50', u'10.0', u'82', u'997'), (u'11', u'05', u'2014', u'00', u'20', u'10.0', u'82', u'997')]

Any help / insights would be greatly appreciated!

Answer 1

Move file writing part of code out of for loop.

for values in data_all:
    ........
    ........
with open(Output_file, "ab") as resultFile:
    ........
    ........