I have 3 URL files all of them have same structure so same spider can be used for all lists. A special need is that all three need to be crawled simultaneously.
is it possible to crawl them simultaneously without creating multiple spiders?
I believe this answer
start_urls = ["http://example.com/category/top/page-%d/" % i for i in xrange(4)] + \
["http://example.com/superurl/top/page-%d/" % i for i in xrange(55)]
in Scrap multiple urls with scrapy only joins two list, but not to run them at the same time.
Thanks very much
use start_requests instead of start_urls ... this will work for u
class MySpider(scrapy.Spider):
name = 'myspider'
def start_requests(self):
for page in range(1,20):
yield self.make_requests_from_url('https://www.example.com/page-%s' %page)
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.