如何在Scrapy Spider中访问管道数据库池

Question

First, here is what I'm trying to do:

I have an XMLFeedSpider that goes through a list of products (the nodes) in an XML file and creates items that are saved to my database in a pipeline. The first time I see a product I need to create requests to do some scraping on the url field of the product to get images, etc. On subsequent reads of the feed if I see the same product I don't want to waste time/resources doing this and just want to skip making these extra requests. To see which products to skip I need to access my database to see if the product exists.

Here are various ways I could think of to do this:

1. Just create a db request for each product within the spider. This seems like a bad idea.
2. In my item store pipeline I'm already creating a database pool as follows: dbpool = adbapi.ConnectionPool('psycopg2', cp_max=2, cp_min=1, **dbargs) and it would seem more efficient to just use that so I'm not constantly creating new database connects. I don't know how to access the instantiated pipeline class though in my spider (that is probably more of a general python question).
   Note: this guy is basically asking this same question but didn't really get the answer he was looking for. How to get the pipeline object in Scrapy spider
3. Maybe before starting the crawl load all of the product urls into memory so I can compare them when processing the products? Where would be a good place to do this?
4. Other suggestion?

Update: this is my pipeline with db pool

class PostgresStorePipeline(object):
    """A pipeline to store the item in a MySQL database.
    This implementation uses Twisted's asynchronous database API.
    """

    def __init__(self, dbpool):
        print "Opening connection pool..."
        dispatcher.connect(self.spider_closed, signals.spider_closed)
        self.dbpool = dbpool

    @classmethod
    def from_settings(cls, settings):
        dbargs = dict(
            host=settings['MYSQL_HOST'],
            database=settings['MYSQL_DBNAME'],
            user=settings['MYSQL_USER'],
            password=settings['MYSQL_PASSWD'],
            #charset='utf8',
            #use_unicode=True,
        )
        dbpool = adbapi.ConnectionPool('psycopg2', cp_max=2, cp_min=1, **dbargs)
        return cls(dbpool)

Answer 1

I think you mean URL by item , remember that for scrapy an item is a data output, and a pipeline is a mechanism to deal with those output items.

Of course you don't need to open many connections to do your db queries, but you will have to do the necessary queries. It depends on how many records you have on your database to only do one query or one per URL , you should test which one is better on your case.

I would recommend setting your own DUPEFILTER_CLASS with something like:

from scrapy.dupefilters import RFPDupeFilter

class DBDupeFilter(RFPDupeFilter):

    def __init__(self, *args, **kwargs):
        # self.cursor = .....                       # instantiate your cursor
        super(DBDupeFilter, self).__init__(*args, **kwargs)

    def request_seen(self, request):
        if self.cursor.execute("myquery"):          # if exists
            return True
        else:
            return super(DBDupeFilter, self).request_seen(request)

    def close(self, reason):
        self.cursor.close()                         # close  your cursor
        super(DBDupeFilter, self).close(reason)

UPDATE

The problem here is that the DUPEFILTER_CLASS doesn't offer the spider on its request_seen object or even to get in on the constructor, so I think your best shot is with a Downloader Middleware , where you can raise a IgnoreRequest exception.

1. Instantiate the db connection on the spider, you could do this on the spider itself (the constructor), or you can also add it through a signal on the Middleware or Pipeline, we'll add it on the Middleware:

    from scrapy.exceptions import IgnoreRequest class DBMiddleware(object): def __init__(self): pass @classmethod def from_crawler(cls, crawler): o = cls() crawler.signals.connect(o.spider_opened, signal=signals.spider_opened) return o def spider_opened(self, spider): spider.dbpool = adbapi.ConnectionPool('psycopg2', cp_max=2, cp_min=1, **dbargs) def process_request(self, request, spider): if spider.dbpool... # check if request.url inside the database raise IgnoreRequest() 

2. Now on your Pipeline, remove the instantiation of dbpool and get it from the spider argument when necessary, remember than process_item receives the item and the spider as argument, so you should be able to use spider.dbpool to check your db connection.

3. Remember to activate your middleware .

That way you should only be doing one instance of the db connection inside the spider object.