在python中以CSV比较数据

Question

I am using python 2.7 to clean some data from a CSV file before chucking it into a MySQL database.

Each row is a user entry with a timestamp.

Before I send the data to the db, I want to check the CSV for duplicate rows (two rows with the same username), and then use the timers (on the same rows) to check which one to keep.

# set up data container
data = []

# read csv file
with open(file, 'rU') as f:
    # create file reader
    reader = csv.reader(f)

    # skip first row (headers)
    next(reader)

    # gather data in a table 
    for row in reader:
        data.append(row)

I think I am getting confused with comparing items in a 2d array... I know that the usernames are in data[][1] and the timer (int) is in data[][52] .

I tried to create a new list like this:

usernames = []
cleaner_list = data
for row in data:
    if row[1] is in usernames:
         # dupe
    else:
        usernames.append(row[2])

But I keep going out of range when trying to compare the data such as like this:

if row[1] is in usernames:
      if row[52] > usernames[row[2]][52]:
            # delete row[52] from cleaner_data
      else:
            # delete the equivalent row in usernames from cleaner_data

I feel that I am overthinking this but I can't use a set as I need the data to stay in line. I thought about creating some sort of enum list of the unique usernames and filter the CSV column with that, but I wouldn't know how to keep the correct references to the row when I find a duplicate and need to check its timer before deleting it. Any help would be really appreciated!

Answer 1

I'd do the following: Keep a dictionary of users with associated last timestamps. If you find something newer while scanning the CSV, replace the old value.

cleaner_data = {}
for row in data:
  if row[1] not in cleaner_data:    # user name not yet seen: add
    cleaner_data[row[1]] = row
  else:
    if row[52] > cleaner_data[row[1]][52]:    # already seen, but newer timestamp: replace
      cleaner_data[row[1]] = row