python scrapy如何知道访问的链接

Question

lets say i am scraping over thousands of pages.

Then, when i scrap on a page, i want to know if this page has been scraped before. Then, i decided if i scrap on it or not.

I want to know if scrapy by default save the scraped pages or not.

what i have tried

i save the scraped links in a file, then i read it to know if a specific link has been scraped before. However, i think that scrapy should have a build-in feature to do that.

right?

Answer 1

scrapy has that functionality built in and will filter those requests for you, see scrapy request in the docs

dont_filter (boolean) – indicates that this request should not be filtered by the scheduler. This is used when you want to perform an identical request multiple times, to ignore the duplicates filter. Use it with care, or you will get into crawling loops. Default to False.

so when creating requests you can decide if you want to recrawl the same url or not.

for more implementation info see the default RFPDupeFilter in the code

there is a settings entry called DUPEFILTER_CLASS in case you wish to replace the default one with some other dedup logic