Web 使用 Scrapy 在链接内抓取

Question

I am web-scraping information from a site that has several pages of data. Within each scrape, I am extracting a handful of information. However, I want to also go inside the link of what I am scraping and scrape information from there as well and then return back to the site and continue scraping. How would I do this using Scrapy?

Answer 1

CrawlSpider library can be used along with scrapy to recursively crawl through a huge graph of web pages.

More information:

Crawling a site recursively using scrapy

https://realpython.com/web-scraping-and-crawling-with-scrapy-and-mongodb/

https://mherman.org/blog/recursively-scraping-web-pages-with-scrapy/

Answer 2

You could use recursion to achieve the desired objective. Start by scraping a start link, and then recursively iterate the links inside it and it goes on. Beware that this recursion might take very long, and might eventually ban your scraper in some cases. Try limiting the recursion depth to 2 or 3.

Code Snippet:

from scrapy.linkextractors import LinkExtractor
from scrapy.contrib.spiders import CrawlSpider, Rule
from scrapy.item import Item, Field
from scrapy.http import Request

class MyItem(Item):
    url= Field()

class MySpider(CrawlSpider):
    name = 'twitter.com'
    allowed_domains = ['twitter.com']
    start_urls = ['http://www.twitter.com']

    rules = (Rule(LinkExtractor(), callback='parse_url', follow=False), )

    def parse_url(self, response):
        item = MyItem()

        ## Do your processing here

        item['url'] = response.url
        request = Request(response.url)
        yield request