Scrapy-如何跟踪起始URL

Question

Given a pool of start urls I would like to identify in the parse_item() function the origin url.

As far as I'm concerned the scrapy spiders start crawling from the initial pool of start urls, but when parsing there is no trace of which of those urls was the initial one. How it would be possible to keep track of the starting point?

Answer 1

If you need a parsing url inside the spider, just use response.url:

def parse_item(self, response):
    print response.url 

but in case you need it outside spider I can think of following ways:

1. Use scrapy core api
2. You can also call scrapy from an external python module with OS command (which apparently is not recommended):

in scrapycaller.py

from subprocess import call
urls = 'url1,url2'
cmd = 'scrapy crawl myspider -a myurls={}'.format(urls)
call(cmd, shell=True)

Inside myspider:

class mySpider(scrapy.Spider):
    def __init__(self, myurls=''):              
        self.start_urls = myurls.split(",")