error when run scrapy crawl my_scraper -o ehadith.csv

Question

I really need help to solve my problem.

I have an error:

"[scrapy.core.engine] DEBUG: Crawled (200) <GET http://www.islam.gov.my/robots.txt> (referer: None)" 

When I try running scrapy crawl my_scraper -o ehadith.csv

• this is my spiders, ehadith_scrapy.py : https://hastebin.com/ererirawit.py
• this is my output: https://hastebin.com/zinibiqado.coffeescript

Answer 1

That's not an error. That's a debug level log telling you that you're spider successfully downloaded the domain's robots.txt file.

The other issue you are having is 403 responses. Try using the AutoThrottle extension to reduce requests concurrency.