Python Dictionary to CSV Issue

Question

I put together a python script to clean CSV files. The reformatting works, but the data rows the writer writes to the new CSV file are wrong. I am constructing a dictionary of all rows of data before writing using writer.writerows(). When I check the dictionary using print statements, the correct data is appending to the list. However, after appending, the incorrect values are in the dictionary.

import csv

data = []
with open(r'C:\\Data\\input.csv', 'r') as csv_file:
    csv_reader = csv.reader(csv_file, delimiter=',')
    line_count = 0
    street_fields = []                                  # Store new field names in list
    street_fields.append("startdate")
    street_fields.append("starttime")
    street_fields.append("sitecode")
    street_fields.append("recordtime")
    street_fields.append("direction")
    street_fields.append("turnright")
    street_fields.append("wentthrough")
    street_fields.append("turnleft")
    street_fields.append("pedestrians")
    for row in csv_reader:                              # Read input rows
        if line_count == 0:
            startdate = row[1]                          # Get Start Date from B1
            line_count += 1
        elif line_count == 1:
            starttime = row[1]                          # Get Start Time from B2
            line_count += 1
        elif line_count == 2:
            sitecode = str(row[1])                      # Get Site code from B3
            line_count += 1
        elif line_count == 3:
            street_count = len(row) - 3                 # Determine number of streets in report
            streetnames = []
            i = 1
            while i < street_count:
                streetnames.append(row[i])              # Add streets to list
                i += 4
            line_count += 1
        elif line_count > 4:
            street_values = {}                          # Create dictionary to store new row values
            n = 1
            for street in streetnames:
                turnright = 0 + n                                       
                wentthrough = 1 + n
                turnleft = 2 + n
                pedestrians = 3 + n
                street_values["startdate"] = startdate
                street_values["starttime"] = starttime
                street_values["sitecode"] = sitecode
                street_values["recordtime"] = row[0]
                street_values["direction"] =  street
                street_values["turnright"] = int(row[turnright])
                street_values["wentthrough"] = int(row[wentthrough])
                street_values["turnleft"] = int(row[turnleft])
                street_values["pedestrians"] = int(row[pedestrians])
                data.append(street_values)                                  # Append row dictionary to list
                #print(street_values)                                       ### UNCOMMENT TO SEE CORRECT ROW DATA ###
                #print(data)                                                ### UNCOMMENT TO SEE INCORRECT ROW DATA ###
                n += 4
            line_count += 1
        else:
            line_count += 1
with open(r'C:\\Data\\output.csv', 'w', newline='', encoding="utf-8") as w_scv_file:
    writer = csv.DictWriter(w_scv_file,fieldnames=street_fields)
    writer.writerow(dict((fn,fn) for fn in street_fields))                  # Write headers to new CSV
    writer.writerows(data)                                                  # Write data from list of dictionaries

An example of the list of dictionaries created (JSON):

[  
   {  
      "startdate":"11/9/2017",
      "starttime":"7:00",
      "sitecode":"012345",
      "recordtime":"7:00",
      "direction":"Cloud Dr. From North",
      "turnright":0,
      "wentthrough":2,
      "turnleft":11,
      "pedestrians":0
   },
   {  
      "startdate":"11/9/2017",
      "starttime":"7:00",
      "sitecode":"012345",
      "recordtime":"7:00",
      "direction":"Florida Blvd. From East",
      "turnright":4,
      "wentthrough":433,
      "turnleft":15,
      "pedestrians":0
   },
   {  
      "startdate":"11/9/2017",
      "starttime":"7:00",
      "sitecode":"012345",
      "recordtime":"7:00",
      "direction":"Cloud Dr. From South",
      "turnright":15,
      "wentthrough":4,
      "turnleft":6,
      "pedestrians":0
   },
   {  
      "startdate":"11/9/2017",
      "starttime":"7:00",
      "sitecode":"012345",
      "recordtime":"7:00",
      "direction":"Florida Blvd. From West",
      "turnright":2,
      "wentthrough":219,
      "turnleft":2,
      "pedestrians":0
   },
   {  
      "startdate":"11/9/2017",
      "starttime":"7:00",
      "sitecode":"012345",
      "recordtime":"7:15",
      "direction":"Cloud Dr. From North",
      "turnright":1,
      "wentthrough":3,
      "turnleft":8,
      "pedestrians":0
   }
]

What actually writes to the CSV:

Note the Direction field and data rows are incorrect. For some reason when it loops through the streetnames list, the last street name and the corresponding row values persist for the individual record time.

Do I need to delete my variables before re-assigning them values?

Answer 1

It looks like you are appending the same dictionary to the list over and over.

In general, when appending a nuber of separate dictionaries to a list, I would use mylist.append(mydict.copy()) , otherwise later on when you assign new values within a dictionary of the same name you are really just updating your old dictionary, including entries in your list that point to a dictionary of the same name (see mutable vs immutable objects in python ).

In short: If you want the dictionary in the list to be a separate entity from the new one, create a deep copy using dict.copy() when appending it to the list.