在开发Web刮板模块上做出pythonic决策

Question

This is a fairly high level question. I have developed many different web scrapers that work in different websites.

I have many different versions of the functions named getName() and getAddress().

Is it pythonic/not terrible coding practice to do this in the function of a module? If this is bad to do, could someone give me a high level tip on how to manage this kind of library-of-scrapers?

def universalNameAdressGrab(url):
   page = pullPage(url)
   if 'Tucson.com' in url:
       import tucsonScraper
       name = getName(page)     #this is the getName for Tucson
       address = getAddress(page)
   elif 'NewYork.com' in url:
       import newyorkScraper
       name = getName(page)   #this is the getName for NewYork
       address = getAddress(page)
   return {'name':name, 'address':address}

Answer 1

It is probably more pythonic to import everything at the top of the file. After that you can reference the functions by module and remove a lot of duplicated code. You may run into issues with URL capitalization, so I would standardize that as well. You could use urlparse for that. I would consider something like the following more pythonic:

import tucsonScraper
import newyorkScraper

def universalNameAdressGrab(url):
    page = pullPage(url)
    scraper = None

    if 'Tucson.com' in url:
        scraper = tucsonScraper
    elif 'NewYork.com' in url:
        scraper = newyorkScraper
    else:
        raise Exception("No scraper found for url")

    return {'name': scraper.getName(page), 'address': scraper.getAddress(page)}