抓取抓取时具有相同内容（例如/ product）的抓取网址

Question

How can I crawl multiple pages with same condition with scrapy?Example: I want to identify all the product pages in an eCommerce site that doesn´t contains product photo (or something)

class SomewebsiteProductSpider(scrapy.Spider):
name = "test"
allowed_domains = ["test.com"]

start_urls = [test.com/product]

In many eamples I´ve seen the start url always correspond to a single page.

It´s possible? Thanks!

Answer 1

If you want to identify all items of a webpage it's good practice to start with one page -- typically the main-page of the site -- and start the crawling from there. You want to use the page of the site where all categories are listed that you are interested in.

With scrapy you can define which links the spider should follow and which pages it should parse and return information to you.

So it's possible and scrapy is a great tool for that.