解析CSV文件并修改列

Question

I'd like to change a CSV file in a specific way. This is my sample CSV file:

name,time,Operations
Cassandra,2015-10-06T15:07:22.333662984Z,INSERT
Cassandra,2015-10-06T15:07:24.334536781Z,INSERT
Cassandra,2015-10-06T15:07:27.339662984Z,READ
Cassandra,2015-10-06T15:07:28.344493608Z,READ
Cassandra,2015-10-06T15:07:28.345221189Z,READ
Cassandra,2015-10-06T15:07:29.345623750Z,READ
Cassandra,2015-10-06T15:07:31.352725607Z,UPDATE
Cassandra,2015-10-06T15:07:33.360272493Z,UPDATE
Cassandra,2015-10-06T15:07:38.366408708Z,UPDATE

I know how to read from CSV file using python parser but I'm totally a beginner with that. I need to get such an output:

start_time,end_time,operation
2015-10-06T15:07:22.333662984Z,2015-10-06T15:07:24.334536781Z,INSERT    
2015-10-06T15:07:27.339662984Z,2015-10-06T15:07:29.345623750Z,READ
2015-10-06T15:07:31.352725607Z,2015-10-06T15:07:38.366408708Z,UPDATE

Comment: The start time is a timestamp given at beginning of specific query (insert/read,update) and accordingly the end time is completion of the query.

Thanks.

Answer 1

It appears from your sample that you can (presumably) guarantee that the first entry of a certain kind in the Operations column, and the last entry of that same kind are the start and stop times. If you can't guarantee this, then it's slightly more complicated, but let's assume you can't – to be more robust.

One thing we can assume is that the data represented in the CSV is the entirety. If you're missing entries for a particular operation, there's little we can do. We also want to read the timestamps, which we can do using the dateutil.parser module.

So we can start by setting up a short dictionary for keeping track of our values, and a function for populating the dictionary, which accepts one line at a time.

import dateutil.parser

ops = dict()

def update_ops(opsdict, row):

    # first get the timestamp and op name in a useable format
    timestamp = dateutil.parser.parse(row[1])
    op_name = row[2]

    ## now populate, or update the dictionary
    if op_name not in opsdict:
        # sets a new dict entry with the operation's timestamp.
        # since we don't know what the start time and end time 
        # is yet, for the moment set them both.
        opsdict[op_name] = { 'start_time': timestamp,
                            'end_time': timetstamp }
    else:
        # now evaluate the current timestamp against each start_time
        # and end_time value. Update as needed.
        if opsdict[op_name]['start_time'] > timestamp:
            opsdict[op_name]['start_time'] = timestamp
        if opsdict[op_name]['end_time'] < timestamp:
            opsdict[op_name]['end_time'] = timestamp

Now that we have a function to do the sorting, run through the CSV file reader and populate ops . When we're done, we can generate a new CSV file with the contents from our dictionary.

import csv

cr = csv.reader(open('/path/to/your/file.csv'))
cr_head = cr.next()    # throw away the first row

for row in cr:
    update_ops(ops, row)

# Now write a new csv file – csv.writer is your friend :)
with open('new_operation_times.csv', 'w') as newcsv:
    cw = csv.writer(newcsv)

    # first write your header. csv.writer accepts lists for each row.
    header = 'start_time,end_time,operation'.split(',')
    cw.writerow(header)

    # now write out your dict values. You may want them sorted, 
    # but how to do that has been answered elsewhere on SE.
    for opname, timesdict in ops.items():
        row = [ opname, timesdict['start_time'], timesdict['end_time'] ]
        cw.writerow(row)

And you're done! I've tried to make this as elaborate as possible so it's clear what's going on. You can probably collapse a lot of this into fewer, more clever steps (such as reading from one csv and writing it out directly). But if you follow the KISS principle, you'll have an easier time reading this later on, and learning from it again.