关于找到两个字典交集的最佳方法的建议

Question

I have posted a similar question before, but after reworking the project, I've gotten here:

With two csv files ( new.csv , scrapers.csv ) -

new.csv contains a single column:
'urls' = whole URLs

scrapers.csv contains two columns:
'scraper_dom' = A simplification of specific URL domains
'scraper_id' = An associated scraper_id that is used to import URLs to a separately managed database

Question

My goal here is to iterate through new.csv (parsing out fnetloc using urlparse ) and perform a lookup on scrapers.csv to return a set of matching 'scraper_id' given a set of 'urls' (the way a VLOOKUP would work, or a JOIN in SQL), once urlparse does it's thing to isolate the netloc within the URL (the result of fnetloc ).

My next big issue is that urlparse does not parse the URLs (from new.csv ) to the exact simplification found in the scrapers.csv file, so I'd be reliant on a sort of partial match until I can figure out the regular expressions to use for that part of it.

I've imported pandas because previous attempts found me creating DataFrames and performing a pd.merge but I couldn't get that to work either...

Current code, commented out bits at the bottom are failed attempts, just thought I'd include what I've tried thus far.
( ## are just intermediate print lines I put in to check output of the program)

import pandas as pd, re
from urllib.parse import urlparse
import csv
sd = {}
sid = {}
#INT = []
def fnetloc(any):
    try:
        p = urlparse(any)
        return p.netloc
    except IndexError:
        return 'Error'
def dom(any):
    try:
        r = any.split(',')
        return r[0]
    except IndexError:
        return 'Error'
def ids(any):
    try:
        e = any.split(',')
        return e[0]
    except IndexError:
        return 'Error'

with open('scrapers.csv',encoding='utf-8',newline='') as s:
    reader = enumerate(csv.reader(s))
    s.readline()
    for j, row in reader:
        dict1 = dict({'scraper_dom':dom(row[0]), 'scraper_id':ids(row[1])})
        sid[j + 1] = dict1
for di in sid.keys():
    id = di
    ##print(sid[di]['scraper_dom'],sid[di]['scraper_id'])

with open('new.csv',encoding='UTF-8',newline='') as f:
    reader = enumerate(csv.reader(f))
    f.readline()
    for i, row in reader:
        dict2 = dict({'scraper_domain': fnetloc(row[0])})
        sd[i + 1] = dict2
for d in sd.keys():
    id = d
    ##print(sd[d]['scraper_domain'])

    #def tryme(  ):
        #return filter(sd.has_key, sid)
    #print(list(filter(sid, sd.keys())))

Sample of desired output.

Answer 1

You just need a procedure that can take a fnetloc and a list of scrapers and check to see if there is a scraper that matches that fnetloc:

def fnetloc_to_scraperid(fnetloc: str, scrapers: List[Scraper]) -> str:
    try:
        return next(x.scraper_id for x in scrapers if x.matches(fnetloc))
    except:
        return "[no scraper id found]"

I also recommend that you use some classes instead of keeping everything in csv row objects--it reduces errors in your code, in the long run, and greatly advances your sanity.

This script worked on the sample data I fed it:

import csv
from urllib.parse import urlparse
from typing import List

def fnetloc(any) -> str:
    try:
        p = urlparse(any)
        return p.netloc
    except IndexError:
        return 'Error'

class Scraper:
    def __init__(self, scraper_dom: str, scraper_id: str):
        self.scraper_dom = scraper_dom
        self.scraper_id = scraper_id
    def matches(self, fnetloc: str) -> bool:
        return fnetloc.endswith(self.scraper_dom)


class Site:
    def __init__(self, url: str):
        self.url = url
        self.fnetloc = fnetloc(url)
    def get_scraperid(self, scrapers: List[Scraper]) -> str:
        try:
            return next(x.scraper_id for x in scrapers if x.matches(self.fnetloc))
        except:
            return "[no scraper id found]"

sites = [Site(row[0]) for row in csv.reader(open("new.csv"))]
scrapers = [Scraper(row[0], row[1]) for row in csv.reader(open("scrapers.csv"))]

for site in sites:
    print(site.url, site.get_scraperid(scrapers), sep="\t")