如何停止scrapy 301重定向并停止解析重定向的页面

Question

I am trying to crawl a page that redirects scrapy for whatever reason via 301 to the english version and then the site gets parsed which it should not as the rules clearly exclude the URL.

While searching for a solution on how to stop any redirect I came upon the following code:

meta ={'dont_redirect': True}

Unfortunatelly this does not have any effect. My spider class looks like this:

class GetbidSpider(CrawlSpider):
    name = 'test'
    meta ={'dont_redirect': True}
    allowed_domains = ['www.example.de']
    start_urls = ['https://www.example.url/bla.html']

    rules = (

        Rule(
            LinkExtractor(allow=['.*Mein-String.*[a-z]::[0-9].*']), 
            callback='parse_item'
        ),

        Rule(
            LinkExtractor(allow=['^.*de\/((?!My-String|:_:|productListingStyle|\.php).)*$']), 
            follow=True
        ),
    )

Is this the right place to configure the redirect and why is scrapy parsing the input while the first URL rule will not execute?

Answer 1

Why what you tried doesn't work:

• The rules only determine what requests will be created by your spider, they don't control redirect logic.
• meta is an attribute of a Request , and only works on a per-request basis.

How to disable redirects:

The easiest way to disable redirects globally is to set the REDIRECT_ENABLED setting to False .