Scrapy 为 reddit.com 返回 301

Question

I'm using reddit as the basis for learning scrappy. It was working fine for a while, but now it always returns a 301 redirect, even when simply calling the shell with "scrapy shell www.reddit.com". Any ideas how to fix this?

Answer 1

在 url 中使用 https/http 方案

scrapy shell https://www.reddit.com

Answer 2

I think it's something related to reddit itself as it seems blocking your IP or user agent, you need to try tweaking the following : 1- Raise the DOWNLOAD_DELAY in scrapy settings 2- Try to change your user agent 3- Use proxy with scrapy

For more info on the settings: http://doc.scrapy.org/en/latest/topics/settings.html

For scrapy proxy : http://doc.scrapy.org/en/latest/topics/downloader-middleware.html