I'm using a for loop to add a dictionary to a list of dictionaries (which I'm calling data_file), using data_file.append()
It works fine :) But then later I try to add some more dictionaries to data_file in another for loop and I use data_file.append() again and it doesn't work. It doesn't add those dictionaries to data_file
Does anyone know what I'm doing wrong?
I don't get an error. It just produces a file that only has the dictionaries from massage_generators. It doesn't take on anything from travel_generators.
I've even tried commenting out the first for loop, the one for massage_generators, and in that case it does add in the travel_generators dictionaries. It's like I can't use .append() twice?
Any help would be much appreciated please!
I'm sorry it's not very elegant, I'm only just learning this coding stuff! :)
Thanks
import csv
import copy
import os
import sys
generators = list()
for filename in os.listdir(os.getcwd()):
root, ext = os.path.splitext(filename)
if root.startswith('Travel Allowance Auto Look up') and ext == '.csv':
travel = filename
open_file = csv.DictReader(open(travel))
generators.append(open_file)
travel_generators = generators[:]
massage_generators = generators[:]
data_file = [] # result will be stored here (List of dicts)
travel_remap = {'FINAL_TRAVEL_ALL':'AMOUNT'}
massage_remap = {'MASSAGE_BIK':'AMOUNT'}
for generator in massage_generators:
for dictionary in generator:
dictionary['PAYMENT_TYPE_CODE'] = 'MASSAGE_BIK'
dictionary['COMMENT'] = 'Massage BIK'
dictionary['IS_GROSS'] = 'FALSE'
dictionary['PAYMENT_TO_DATE'] = '01/01/2099'
dictionary['PAID MANUALLY'] = 'FALSE'
for old_key, new_key in massage_remap.iteritems():
if old_key not in dictionary:
continue
dictionary['AMOUNT'] = dictionary['MASSAGE_BIK']
del dictionary['MASSAGE_BIK']
if (dictionary['AMOUNT'] != '0' and dictionary['AMOUNT'] != ''):
data_file.append(dictionary)
for generator in travel_generators:
for dictionary in generator:
dictionary['PAYMENT_TYPE_CODE'] = 'TRANSPORTATION_ALLOWANCE'
dictionary['COMMENT'] = 'Annual travel allowance paid in monthly installments'
dictionary['IS_GROSS'] = 'TRUE'
dictionary['PAYMENT_TO_DATE'] = '01/01/2099'
dictionary['PAID MANUALLY'] = 'FALSE'
for old_key, new_key in travel_remap.iteritems():
if old_key not in dictionary:
continue
dictionary['AMOUNT'] = dictionary['FINAL_TRAVEL_ALL']
del dictionary['FINAL_TRAVEL_ALL']
if (dictionary['AMOUNT'] != 'Not Applicable' and dictionary['AMOUNT'] != '0' and dictionary['AMOUNT'] != '' and dictionary['AMOUNT'] != '#N/A'):
data_file.append(dictionary)
keys = ['EMPID', 'Common Name', 'PAYMENT_TYPE_CODE', 'CURRENCY', 'AMOUNT', 'EFFECTIVE_DATE',
'COMMENT', 'PAYMENT_FROM_DATE', 'PAYMENT_TO_DATE', 'IS_GROSS', 'HIDDEN_UNTIL', 'PAID MANUALLY', 'PAYMENT_DATE']
bulk_upload = open('EMEA Bulk Upload.csv', 'wb')
dict_writer = csv.DictWriter(bulk_upload, keys, restval='', extrasaction='ignore')
dict_writer.writer.writerow(keys)
dict_writer.writerows(data_file)
print "Everything saved! Look up EMEA Bulk Upload.csv"
Here's your problem:
open_file = csv.DictReader(open(travel))
generators.append(open_file)
travel_generators = generators[:]
massage_generators = generators[:]
Your travel_generators
and massage_generators
lists are using the same open file to iterate over and get data; the problem with that is once the file has been iterated over once by the csv.DictReader
in massage_generators
it has been exhausted and you get no more data when you try to read it again in travel_generators
.
Try this instead:
generators.append(travel) # just the filename
travel_generators = generators[:]
massage_generators = generators[:]
and then in your loops:
for filename in massage_generators:
for dictionary in csv.DictReader(open(filename)):
Also, in your first loop where you are gathering filenames: be aware that the loop is only saving the last filename that matches. If that is your intention you can make it clear by adding break
on the line after travel = filename
.
In the example you provided, generators
, travel_generators
, and massage_generators
are all identical, and each contain the same instance of open_file
. When you iterate massage_generators
, you reach the end of the file. Then when you iterate travel_generators
, there's nothing left to iterate in open_file
.
At the very least, you could change the following line:
open_file = csv.DictReader(open(travel))
To:
open_file = map(None, csv.DictReader(open(travel)))
open_file
is now a list and not a generator and can be iterated and re-iterated.
However, I would recommend you iterate through the file once, and build your two dictionaries, and write to a DictWriter
instance in each iteration.
open_file = csv.DictReader(open(travel))
...
dict_writer = csv.DictWriter(bulk_upload, keys, restval='', extrasaction='ignore')
dict_writer.writer.writerow(keys)
for line in open_file:
massage_dictionary={}
massage_dictionary.update(line)
... # manipulate massage_dictionary
dict_writer.writerow(massage_dictionary)
travel_dictionary={}
travel_dictionary.update(line)
... # manipulate travel_dictionary
dict_writer.writerow(travel_dictionary)
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.