[英]Python/CSV unique rows with unique values per row in a column
Have this dataset, dataset is fictional:有这个数据集,数据集是虚构的:
cat sample.csv
id,fname,lname,education,gradyear,attributes
"6F9619FF-8B86-D011-B42D-00C04FC964FF",john,smith,mit,2003,qa
"6F9619FF-8B86-D011-B42D-00C04FC964FF",john,smith,harvard,2007,"test|admin,test"
"6F9619FF-8B86-D011-B42D-00C04FC964FF",john,smith,harvard,2007,"test|admin,test"
"6F9619FF-8B86-D011-B42D-00C04FC964FF",john,smith,ft,2012,NULL
"6F9619FF-8B86-D011-B42D-00C04FC964F1",john,doe,htw,2000,dev
When I run this script, which parses the csv and finds unique rows, concating rows in on column when more are found:当我运行这个脚本时,它解析 csv 并找到唯一的行,当找到更多行时,在列中连接行:
parse-csv.py解析-csv.py
import itertools
from itertools import groupby
import csv
import pprint
import argparse
if __name__ == '__main__':
parser = argparse.ArgumentParser(description='sql dump parser')
parser.add_argument('-i','--input', help='input file', required=True)
parser.add_argument('-o','--output', help='output file', required=True)
args = parser.parse_args()
inputf = args.input
outputf = args.output
t = csv.reader(open(inputf, 'rb'))
t = list(t)
def join_rows(rows):
return [(e[0] if i < 1 else '|'.join(e)) for (i, e) in enumerate(zip(*rows))]
myfile = open(outputf, 'wb')
wr = csv.writer(myfile, quoting=csv.QUOTE_ALL, lineterminator='\n')
for name, rows in groupby(t, lambda x:x[0]):
wr.writerow(join_rows(rows))
#print join_rows(rows)
And than another script, which makes sure each colum has only unique values separated by "|"而不是另一个脚本,它确保每个列只有由“|”分隔的唯一值
unique.py独特的.py
import csv
import sys
from collections import OrderedDict
import argparse
csv.field_size_limit(sys.maxsize)
import argparse
if __name__ == '__main__':
parser = argparse.ArgumentParser(description='sql dump parser - unique')
parser.add_argument('-i','--input', help='input file', required=True)
parser.add_argument('-o','--output', help='output file', required=True)
args = parser.parse_args()
inputf = args.input
outputf = args.output
with open(inputf) as fin, open(outputf, 'wb') as fout:
csvin = csv.DictReader(fin)
csvout = csv.DictWriter(fout, fieldnames=csvin.fieldnames, quoting=csv.QUOTE_ALL,lineterminator='\n')
csvout.writeheader()
for row in csvin:
for k, v in row.items():
row[k] = '|'.join(OrderedDict.fromkeys(v.split('|')))
csvout.writerow(row)
It works for the sample.csv它适用于 sample.csv
Output:输出:
$ python parse-csv.py -i sample.csv -o sample-out.csv
$ python unique.py -i sample-out.csv -o sample-final.csv
$ cat sample-final.csv
"id","fname","lname","education","gradyear","attributes"
"6F9619FF-8B86-D011-B42D-00C04FC964FF","john","smith","mit|harvard|ft","2003|2007|2012","qa|test|admin,test|NULL"
"6F9619FF-8B86-D011-B42D-00C04FC964F1","john","doe","htw","2000","dev"
But when I do the same for this:但是当我为此做同样的事情时:
(dataset is fictional) (数据集是虚构的)
sample2.csv样本2.csv
id,lastname,firstname,middlename,address1,address2,city,zipcode,city2,zipcode2,emailaddress,website
"E387F3C1-F6E9-40DD-86AB-A7149C67F61C","Technical Support",NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL
"648EEB5D-0586-444A-B86F-4EB2446BBC93","Palm","Samuel","J",NULL,NULL,NULL,NULL,NULL,NULL,"",NULL
"A94FAD4E-27DB-48FE-B89E-C37B408C5DD5","Mait","A.V.",NULL,NULL,NULL,NULL,NULL,NULL,NULL,"mait@yahoo.com",NULL
"E387F3C1-F6E9-40DD-86AB-A7149C67F61C","Technical Support",NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL
"648EEB5D-0586-444A-B86F-4EB2446BBC93","Palm","Samuel","J",NULL,NULL,NULL,NULL,NULL,NULL,"",NULL
"A94FAD4E-27DB-48FE-B89E-C37B408C5DD5","Mait","A.V.",NULL,NULL,NULL,NULL,NULL,NULL,NULL,"mait@yahoo.com",NULL
"FDFCA22A-EE19-4997-B892-90B2006FE328","Drago","Paul",NULL,"","","","",NULL,NULL,"psd@gmail.com",NULL
"FDFCA22A-EE19-4997-B892-90B2006FE328","Drago","Paul",NULL,"","","","",NULL,NULL,"psd@gmail.com",NULL
"FDFCA22A-EE19-4997-B892-90B2006FE328","Drago","Paul",NULL,"","","","",NULL,NULL,"psd@gmail.com",NULL
"FDFCA22A-EE19-4997-B892-90B2006FE328","Drago","Paul",NULL,"","","","",NULL,NULL,"psd@gmail.com",NULL
"FDFCA22A-EE19-4997-B892-90B2006FE328","Drago","Paul",NULL,"","","","",NULL,NULL,"psd@gmail.com",NULL
The output is:输出是:
$ python parse-csv.py -i sample2.csv -o sample2-out.csv
$ python unique.py -i sample2-out.csv -o sample2-final.csv
$ cat sample2-final.csv
"id","lastname","firstname","middlename","address1","address2","city","zipcode","city2","zipcode2","emailaddress","website"
"E387F3C1-F6E9-40DD-86AB-A7149C67F61C","Technical Support","NULL","NULL","NULL","NULL","NULL","NULL","NULL","NULL","NULL","NULL"
"648EEB5D-0586-444A-B86F-4EB2446BBC93","Palm","Samuel","J","NULL","NULL","NULL","NULL","NULL","NULL","","NULL"
"A94FAD4E-27DB-48FE-B89E-C37B408C5DD5","Mait","A.V.","NULL","NULL","NULL","NULL","NULL","NULL","NULL","mait@yahoo.com","NULL"
"E387F3C1-F6E9-40DD-86AB-A7149C67F61C","Technical Support","NULL","NULL","NULL","NULL","NULL","NULL","NULL","NULL","NULL","NULL"
"648EEB5D-0586-444A-B86F-4EB2446BBC93","Palm","Samuel","J","NULL","NULL","NULL","NULL","NULL","NULL","","NULL"
"A94FAD4E-27DB-48FE-B89E-C37B408C5DD5","Mait","A.V.","NULL","NULL","NULL","NULL","NULL","NULL","NULL","mait@yahoo.com","NULL"
"FDFCA22A-EE19-4997-B892-90B2006FE328","Drago","Paul","NULL","","","","","NULL","NULL","psd@gmail.com","NULL"
Why it doesn't properly get unique rows and columns like it did for sample.csv????为什么它不能像 sample.csv 那样正确地获得唯一的行和列????
Anybody has any ideas?有人有任何想法吗?
Thanks in advance!提前致谢! Chewing on this for a long time now ....咀嚼这个很长时间了......
Your first file is sorted, while the second is not.您的第一个文件已排序,而第二个文件未排序。 Please see this discussion请看 这个讨论
All you need is this:你只需要这个:
t = list(t)
t[1:] = sorted(t[1:])
Here's my simple minded solution to your problem (as I understand it), using a dictionary:这是我使用字典对您的问题(据我所知)的简单解决方案:
import csv
t = csv.reader(open("sample2.csv", 'rb'))
t = list(t)
def parsecsv(data):
# Assumes that the first column is the unique id and that the first
# row contains the column titles and that all rows have same # of columns
L = len(data[0])
csvDict = {}
for entry in data: # build a dict csvDict to represent data
if entry[0] in csvDict: # already have entry so add to it...
for i in range(L - 1): # loop through columns
if csvDict[entry[0]][i] != 'NULL': #check if data exists in column
if (entry[i] not in csvDict[entry[0]][i]) and (entry[i] != 'NULL'):
csvDict[entry[0]][i] += '|' + entry[i]
else:
csvDict[entry[0]][i] = entry[i]
else:
csvDict[entry[0]] = [None]*(L - 1)
for i in range(L - 1): # loop through columns
csvDict[entry[0]][i] = entry[i]
return csvDict
out = parsecsv(t)
for entry in out:
print entry + ' = ' + str(out[entry])
This should be independent of sorted data sets, etc...这应该独立于排序的数据集等......
Let me know if it helps!如果有帮助,请告诉我!
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.