![](/img/trans.png)
[英]Is there a faster way to check the similar data value in a column from csv file using python?
[英]Faster way to check for value in csv?
我有一些从csv查找的代码,然后如果csv中不存在它,则从google地图中检索它。 我有100,000多条记录,大约需要2个小时。 关于如何加快速度的任何想法? 谢谢!
from csv import DictReader
import codecs
def find_school(high_school, city, state):
types_of_encoding = ["utf8"]
for encoding_type in types_of_encoding:
with codecs.open('C:/high_schools.csv', encoding=encoding_type, errors='replace') as csvfile:
reader = DictReader(csvfile)
for row in reader:
#checks the csv file and sees if the high school already exists
if (row['high_school'] == high_school.upper() and
row['city'] == city.upper() and
row['state'] == state.upper()):
return dict(row)['zipcode'],dict(row)['latitude'],dict(row)['longitude'],dict(row)['place_id']
else:
#hits Google Maps api
#executes
df['zip'],df['latitude'], df['longitude'], df['place_id'] = zip(*df.apply(lambda row: find_school(row['high_school'].strip(), row['City'].strip(), row['State'].strip()), axis=1))
CSV文件片段
high_school,city,state,address,zipcode,latitude,longitude,place_id,country,location_type
GEORGIA MILITARY COLLEGE,MILLEDGEVILLE,GA,"201 E GREENE ST, MILLEDGEVILLE, GA 31061, USA",31061,33.0789184,-83.2235169,ChIJv0wUz97H9ogRwuKm_HC-lu8,USA,UNIVERSITY
BOWIE,BOWIE,MD,"15200 ANNAPOLIS RD, BOWIE, MD 20715, USA",20715,38.9780387,-76.7435378,ChIJRWh2C1fpt4kR6XFWnAm5yAE,USA,SCHOOL
EVERGLADES,MIRAMAR,FL,"17100 SW 48TH CT, MIRAMAR, FL 33027, USA",33027,25.9696495,-80.3737813,ChIJQfmM_I6j2YgR1Hdq0CC4apo,USA,SCHOOL
您每次都要进行检查都没有必要读取文件。 只需将文件加载到内存中一次,然后使用元组键将您感兴趣的字段创建一个新字典即可。
import csv
lookup_dict = {}
with open('C:/Users/Josh/Desktop/test.csv') as infile:
reader = csv.DictReader(infile)
for row in reader:
lookup_dict[(row['high_school'].lower(), row['city'].lower(),
row['state'].lower())] = row
现在,您只需检查要测试的值是否已经是lookup_dict
的键。 如果不是,那么您查询Google地图。
由于您的编辑显示您正在使用它来apply
数据框,因此应在函数外部计算lookup_dict
并将其作为参数传递。 这样,该文件仍然只能读取一次。
lookup_dict = {}
with open('C:/Users/Josh/Desktop/test.csv') as infile:
reader = csv.DictReader(infile)
for row in reader:
lookup_dict[(row['high_school'].lower(), row['city'].lower(),
row['state'].lower())] = row
def find_school(high_school, city, state, lookup_dict):
result = lookup_dict.get((high_school.lower(), city.lower(), state.lower()))
if result:
return result
else:
# Google query
pass
a = find_school('georgia military college', 'milledgeville', 'ga', lookup_dict)
#df['zip'],df['latitude'], df['longitude'], df['place_id'] = zip(*df.apply(lambda row: find_school(row['high_school'].strip(), row['City'].strip(), row['State'].strip()), axis=1))
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.