PYTHON-比较两个不同的文件； 如果包含ID的行已经存在，请更新，否则追加该行

Question

I have a master file which is the one which will constantly be updated and a file which is created every minute. I want to be able to compare the new file which is created every minute to the already existing master file. So far I've got:

with open("jobs") as a:
   new = a.readlines()

count=0
for item in new:
   new[count]=new[count].split(",")
   count+=1

This will allow me to compare the first index([0] of each line in my master file. Now at this point I start to confuse myself. I'm guessing it would be something along the lines of:

counter=0
for item in new:
    if new[counter][0] not in master:
        end = open("end","a")
        end.write(str(new[counter]) + "\n")
        counter+=1
        end.close()
    else:
         REPLACE LINES THAT ALREADY EXIST IN MASTER FILE WITH NEW LINE

The IDs won't necessarily be in the same order every time the new file comes in and the new file may contains more entries than the master file at some point.

If I haven't made sense or missed some information out then please let me know and I'll try and clarify. Thanks.

Answer 1

Sounds like a csv problem to me.

unfortunately, it is not clear from your question, if you want to modify the masterfile itself, an out-file, or both. this does the second (it takes a masterfile and an updatefile, both in csv format, and prints the merged thing unsorted to an out-file). If this is not what you want, or if you got data comma-seperated, but without fieldnames on top, change as you need, should be easy enough.

import csv
with open("master.csv") as m, open("update.csv") as u, open("out.csv", "w") as o:
    master = { line['ID']: line for line in csv.DictReader(m) }
    update = { line['ID']: line for line in csv.DictReader(u) }
    master.update(update)
    fields = csv.DictReader(open("master.csv")).fieldnames
    out = csv.DictWriter(o, fields)
    out.writeheader()
    out.writerows(master.values())

with master.csv as such:

ID,Name,Foo,Bar,Baz,Description
1000001,Name here:1,1001,1,description here
1000002,Name here:2,1002,2,description here
1000003,Name here:3,1003,3,description here
1000004,Name here:4,1004,4,description here
1000005,Name here:5,1005,5,description here
1000006,Name here:6,1006,6,description here
1000007,Name here:7,1007,7,description here
1000008,Name here:8,1008,8,description here
1000009,Name here:9,1009,9,description here

and update.csv as such:

ID,Name,Foo,Bar,Baz,Description
1000003,UPDATED Name here:3,1003,3, UPDATED description here
1000010,NEW ITEM Name here:9,1009,9,NEW ITEM description here 

it outputs to out.csv:

ID,Name,Foo,Bar,Baz,Description
1000010,NEW ITEM Name here:9,1009,9,NEW ITEM description here ,
1000008,Name here:8,1008,8,description here,
1000009,Name here:9,1009,9,description here,
1000006,Name here:6,1006,6,description here,
1000007,Name here:7,1007,7,description here,
1000004,Name here:4,1004,4,description here,
1000005,Name here:5,1005,5,description here,
1000002,Name here:2,1002,2,description here,
1000003,UPDATED Name here:3,1003,3, UPDATED description here,
1000001,Name here:1,1001,1,description here,

Note that the order is not preserved (not clear from question if neccessary). But it is fast and clean.

Answer 2

Maybe something like this will work:

#First create a set of all the ids contained in a masterfile
master_set = set()
with open('masterfile.txt') as mf:

    for ele in mf:
        master_set.add(ele.split(',')[0])

#if id is not in masterfile (or set) append the line to masterfile
with open('tempfile.txt') as temp, open('masterfile.txt', 'a') as mf:
    for line in temp:
        index = line.split(',')[0]
        if not index in master_set:
            master_set.add(index)
            mf.write(line)

I have not tested it.