I am trying to print the 301 status for scrapy requests. But, the page redirects and shows 200 everytime

Question

I am trying to print the 301 status for scrapy requests. But, the page redirects and I get 200 status code every time. I do not want the redirection to happen and just print the 301 code. I am new to Scrapy and can not figure it out.

I have the following code: enter image description here

Answer 1

You can use the meta attribute in your request, and you pass the key:value

• 'dont_redirect':'True
• 'handle_httpstatus_list': [301]

---

class RedirectSpider(scrapy.Spider):
name = 'redirect'
url = 'http://www.wayfair.com/outdoor/sb0/fire-pits-c215964.html/'

def start_requests(self):
    yield scrapy.Request(
        url=self.url,
        meta={'dont_redirect': True, 'handle_httpstatus_list': [301]},
        callback=self.parse,
    )

def parse(self, response):
    print(response.status)