堆栈内存溢出
  繁体   English   中英

Scrapy 403 错误（但使用 Insomnia 的 http.client.HTTPSConnection 没有错误

[英]Scrapy 403 error (but no error with http.client.HTTPSConnection using Insomnia

 原文  2022-01-29 17:00:11  8   1    python/ https/ scrapy/ http-headers/ web-crawler

问题描述

我正在尝试将我的 api 爬虫转换为 Scrapy api 爬虫，但在 Z8A5DA5DA52ED126447D359E70C05721A8AAZ 爬虫中出现 403 错误 2022-01-29 11:48:14 [scrapy.core.engine] 调试：已爬网 (403) <POST https://api2.realtor.ca/Listing.svc/PropertySearch_Post/> （参考： Z5E056C500A1C4B6A7110BADE50D8 .realtor.ca/ )

我在 scrapy 方面做错了什么？ 我使用 Insomnia 来生成有效的 POST 请求。 但是scrapy好像处理不了，不知道为什么。

工作代码：

import http.client
import json
conn = http.client.HTTPSConnection("api2.realtor.ca")

payload = "ZoomLevel=11&LatitudeMax=43.98268&LongitudeMax=-78.96028&LatitudeMin=43.43223&LongitudeMin=-79.79249&Sort=6-D&PropertySearchTypeId=1&TransactionTypeId=2&PropertyTypeGroupID=1&Currency=CAD&CurrentPage=1&ApplicationId=1&CultureId=1&Version=7.0&RecordsPerPage=200"

headers = {
    'cookie': "gig_bootstrap_3_mrQiIl6ov44s2X3j6NGWVZ9SDDtplqV7WgdcyEpGYnYxl7ygDWPQHqQqtpSiUfko=gigya-pr_ver4; visid_incap_2271082=JLYUNoQeQ8aK7XCukUKVyLTR4GEAAAAAQUIPAAAAAAD5E9h/kC3MKXYFk8j751oC; reese84=3:kSScWfkGFsGrIfH0oClFcA==:7I6p2X8egFPYsN4vhOQSzcG+bY31pkxS/fXEEvP9k7YrM9pKJIRxaMywKm1+U1DhuZilDxW9dT97UvcEqz9H8d6b81yJvbuEOsNCc1hlGdVd3ZjT2uSjne23lT3z3KZ2lj0sxMQRKgafW63RLwfBQWtCgxWHMp9k338xzLwnTwqzOMI+qkBLPMwUolKoJP3IL6gzbljA6PY7Pyaa/R7VMRlGzcnk+M/ILnqZV8kJ+xV8sukeGZAcs4nJqJbvRr6vSE+fsAe/Jct5iAOnYTGoC+IyUdOAIWn2jEXTjwdjndvGdsKGvvkvcnkIjX1/JNTFH922HZPL3q9LzFCEgMhuYgygFRFjCqaz/HJ0zr0DXL9uTjiripD29T9lA0k0fGs+vyMz0ApV2Ni9UJsosr747CQMLROcZCQSIik2p+Taflc3cIAZgituGSiJss6o94xm:40hT11A7LacDXW+pa+D9sBQF5meDdTDqnBCeCZs/jls=; visid_incap_2269415=SnrfkV/xRM2/zMAwUwhS+8xj4GEAAAAAQkIPAAAAAACALOWhAWPmyCDb18oF6pHY12yEiWlUbIgr; nlbi_2269415=Vq3lX/59Tg3xXSVkkG5lugAAAAB+8mCpAlS1F5Timw4irGua; _gid=GA1.2.1583316853.1643415717; ASP.NET_SessionId=ov1iizcd3npobsilupzii5ep; nlbi_2271082=uYrYPyycJwjJZMEUcbDG1QAAAABYkXBfZhGO2s/HVfvOZarc; incap_ses_303_2269415=tkGkDvUztDFYLuufDHk0BPFK9WEAAAAAaFTGABendGhY7YQ6DfHGqg==; incap_ses_303_2271082=R007Dwk7akm7gfCfDHk0BPhS9WEAAAAAEzSDycWG9SwrHnLXWGiOuQ==; reese84=3:xy6Z/sx1YkpUszEJSoRL3A==:lovsunYlSfQpJkP59Bs11wY2+LlgzOMeiL3ObotvVpDwGDJVy4RKfSLLKuvVMLLcTstWzewelf4RKJ3e6v5hAVv9wkqa01hiSd1TIDBTDdyPzcKMI0xlq6r0G2P+8dMx91eZ0jKEx40QdURnU3XwLghg1BALZ+aWt4US7pC1FIftQksLhz7QlnyBw8pl2ucIJ9JIyuM3gBjNaP4hvYyc17UOnBvf37wtLhWeFb+fomUnnLqyTag5dM/vASoIg+Uo+lH9yQI2K9xGm0KveqgF2nUre9Z+UG1gwHWRBEIjygnhZZjnJOR20wxQU7gOZ8YqW+DJdczgSiqbn93I1um+VwOlf8bD6zCq99miEtaVOdVlGZesCvoe9d9JciEryAFMlYcn3RuLvycVNPQVrdCP9REneI+J1AmfXeSveEGpLhnSZs64rniGIf7iT0lRY9c1:f519Rour27xdzG5PjxP0BlHw/5uwjBdnwdY9Zd3AWpU=; nlbi_2269415_2147483646=5DLdcUXrjg0v1GhykG5lugAAAAA0AOEmZsShmR8VQ3d3LJzx; _dc_gtm_UA-12908513-11=1; _ga=GA1.2.2104488426.1642095581; _gac_UA-12908513-11=1.1643468700.Cj0KCQiA6NOPBhCPARIsAHAy2zAmFT3_yol1CanQDHoHW_z8aJ6HgaY2f7iilRt6yGvssuzmDbbh8FoaAkpxEALw_wcB; _ga_Y07J3B53QP=GS1.1.1643467388.20.1.1643468729.19; _4c_=%7B%22_4c_s_%22%3A%22bZNbj5swEIX%2FSuTnhfgKJm%2BrVKpWatWqUp8jgwdiLcHIOKHbKP%2B9Y3JZ7ap%2BAZ85%2BoxnDmcy72EgG1ZIIQtdCiGUfiKv8DaRzZkEZ9PjRDakrUBWitqsVIxmEjjNdNGYzIAxFFdbcUmeyJ%2BFVemSKc2oKi5PxA53hoXWHPt4tymqpZaSCS3Q5sZ486WPYVJrWWquH17kX5XkvRMN%2BW89zA%2FUtaBoKcUH66KgtRlv1jNpvAVksipnMsdLthPS41%2BUMkEpvo%2FB22MTd%2FFtTMYZ6tVkX7Fg4eQa2M3Oxn0icMbf1T24bh8XWVVJHgNueE6vS1UFlYWUGkuzG6yfP3Nu6mdOHfw8QWJt98EfYPXd166HVVVi0eP8yPNgg8cRYkOghRAW8z7Gcdqs1%2FM85533XQ954w9rNE0upmsFMH30IW%2FMTcM0vMvZIn8zQ3c03dKutPVdB3b1MqSgmH4C1H4Gf3JDkyw%2FhmiC8yhu%2FXGIIeG2ZjA2oX7B5CwM0Zneh60%2FHCC4xvTLiY8K%2BmBIAx9Dus7X593vly%2Bpi4zKlBRe5JgaTiuFsSOXew6FUooJLjTDQUZk6kIuTUfH6REiK2lZK6gzJoBnsq55hjNpMuxzyy2tVQv6EW38TSoqmCjlDSmuwMvlHw%3D%3D%22%7D; _gali=lnkNextResultsPage",
    'authority': "api2.realtor.ca",
    'sec-ch-ua': '" Not;A Brand";v="99", "Google Chrome";v="97", "Chromium";v="97"',
    'accept': "*/*",
    'content-type': "application/x-www-form-urlencoded; charset=UTF-8",
    'sec-ch-ua-mobile': "?1",
    'user-agent': "Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/97.0.4692.99 Mobile Safari/537.36",
    'sec-ch-ua-platform': '"Android"',
    'origin': "https://www.realtor.ca",
    'sec-fetch-site': "same-site",
    'sec-fetch-mode': "cors",
    'sec-fetch-dest': "empty",
    'referer': "https://www.realtor.ca/",
    'accept-language': "en-US,en;q=0.9"
    }

conn.request("POST", "/Listing.svc/PropertySearch_Post", payload, headers)

res = conn.getresponse()
data = res.read()

out = json.loads(data.decode("utf-8"))

results = out.get('Results')
for r in results:
    print(r.get('Id'), r.get('Property').get('Price'))

print(len(results))

带有 403 错误的 scrapy 代码损坏：

    # -*- coding: utf-8 -*-
import scrapy
import json
from scrapy.exceptions import CloseSpider

class RealtorSpider(scrapy.Spider):
    name = 'realtor'
    allowed_domains = ['api2.realtor.ca']

    api = "https://api2.realtor.ca/Listing.svc/PropertySearch_Post/"

    payload = "ZoomLevel=11&LatitudeMax=43.98268&LongitudeMax=-78.96028&LatitudeMin=43.43223&LongitudeMin=-79.79249&Sort=6-D&PropertySearchTypeId=1&TransactionTypeId=2&PropertyTypeGroupID=1&Currency=CAD&CurrentPage=1&ApplicationId=1&CultureId=1&Version=7.0&RecordsPerPage=200"

    headers = {
        'cookie': "gig_bootstrap_3_mrQiIl6ov44s2X3j6NGWVZ9SDDtplqV7WgdcyEpGYnYxl7ygDWPQHqQqtpSiUfko=gigya-pr_ver4; visid_incap_2271082=JLYUNoQeQ8aK7XCukUKVyLTR4GEAAAAAQUIPAAAAAAD5E9h/kC3MKXYFk8j751oC; reese84=3:kSScWfkGFsGrIfH0oClFcA==:7I6p2X8egFPYsN4vhOQSzcG+bY31pkxS/fXEEvP9k7YrM9pKJIRxaMywKm1+U1DhuZilDxW9dT97UvcEqz9H8d6b81yJvbuEOsNCc1hlGdVd3ZjT2uSjne23lT3z3KZ2lj0sxMQRKgafW63RLwfBQWtCgxWHMp9k338xzLwnTwqzOMI+qkBLPMwUolKoJP3IL6gzbljA6PY7Pyaa/R7VMRlGzcnk+M/ILnqZV8kJ+xV8sukeGZAcs4nJqJbvRr6vSE+fsAe/Jct5iAOnYTGoC+IyUdOAIWn2jEXTjwdjndvGdsKGvvkvcnkIjX1/JNTFH922HZPL3q9LzFCEgMhuYgygFRFjCqaz/HJ0zr0DXL9uTjiripD29T9lA0k0fGs+vyMz0ApV2Ni9UJsosr747CQMLROcZCQSIik2p+Taflc3cIAZgituGSiJss6o94xm:40hT11A7LacDXW+pa+D9sBQF5meDdTDqnBCeCZs/jls=; visid_incap_2269415=SnrfkV/xRM2/zMAwUwhS+8xj4GEAAAAAQkIPAAAAAACALOWhAWPmyCDb18oF6pHY12yEiWlUbIgr; nlbi_2269415=Vq3lX/59Tg3xXSVkkG5lugAAAAB+8mCpAlS1F5Timw4irGua; _gid=GA1.2.1583316853.1643415717; ASP.NET_SessionId=ov1iizcd3npobsilupzii5ep; nlbi_2271082=uYrYPyycJwjJZMEUcbDG1QAAAABYkXBfZhGO2s/HVfvOZarc; incap_ses_303_2269415=tkGkDvUztDFYLuufDHk0BPFK9WEAAAAAaFTGABendGhY7YQ6DfHGqg==; incap_ses_303_2271082=R007Dwk7akm7gfCfDHk0BPhS9WEAAAAAEzSDycWG9SwrHnLXWGiOuQ==; reese84=3:xy6Z/sx1YkpUszEJSoRL3A==:lovsunYlSfQpJkP59Bs11wY2+LlgzOMeiL3ObotvVpDwGDJVy4RKfSLLKuvVMLLcTstWzewelf4RKJ3e6v5hAVv9wkqa01hiSd1TIDBTDdyPzcKMI0xlq6r0G2P+8dMx91eZ0jKEx40QdURnU3XwLghg1BALZ+aWt4US7pC1FIftQksLhz7QlnyBw8pl2ucIJ9JIyuM3gBjNaP4hvYyc17UOnBvf37wtLhWeFb+fomUnnLqyTag5dM/vASoIg+Uo+lH9yQI2K9xGm0KveqgF2nUre9Z+UG1gwHWRBEIjygnhZZjnJOR20wxQU7gOZ8YqW+DJdczgSiqbn93I1um+VwOlf8bD6zCq99miEtaVOdVlGZesCvoe9d9JciEryAFMlYcn3RuLvycVNPQVrdCP9REneI+J1AmfXeSveEGpLhnSZs64rniGIf7iT0lRY9c1:f519Rour27xdzG5PjxP0BlHw/5uwjBdnwdY9Zd3AWpU=; nlbi_2269415_2147483646=5DLdcUXrjg0v1GhykG5lugAAAAA0AOEmZsShmR8VQ3d3LJzx; _dc_gtm_UA-12908513-11=1; _ga=GA1.2.2104488426.1642095581; _gac_UA-12908513-11=1.1643468700.Cj0KCQiA6NOPBhCPARIsAHAy2zAmFT3_yol1CanQDHoHW_z8aJ6HgaY2f7iilRt6yGvssuzmDbbh8FoaAkpxEALw_wcB; _ga_Y07J3B53QP=GS1.1.1643467388.20.1.1643468729.19; _4c_=%7B%22_4c_s_%22%3A%22bZNbj5swEIX%2FSuTnhfgKJm%2BrVKpWatWqUp8jgwdiLcHIOKHbKP%2B9Y3JZ7ap%2BAZ85%2BoxnDmcy72EgG1ZIIQtdCiGUfiKv8DaRzZkEZ9PjRDakrUBWitqsVIxmEjjNdNGYzIAxFFdbcUmeyJ%2BFVemSKc2oKi5PxA53hoXWHPt4tymqpZaSCS3Q5sZ486WPYVJrWWquH17kX5XkvRMN%2BW89zA%2FUtaBoKcUH66KgtRlv1jNpvAVksipnMsdLthPS41%2BUMkEpvo%2FB22MTd%2FFtTMYZ6tVkX7Fg4eQa2M3Oxn0icMbf1T24bh8XWVVJHgNueE6vS1UFlYWUGkuzG6yfP3Nu6mdOHfw8QWJt98EfYPXd166HVVVi0eP8yPNgg8cRYkOghRAW8z7Gcdqs1%2FM85533XQ954w9rNE0upmsFMH30IW%2FMTcM0vMvZIn8zQ3c03dKutPVdB3b1MqSgmH4C1H4Gf3JDkyw%2FhmiC8yhu%2FXGIIeG2ZjA2oX7B5CwM0Zneh60%2FHCC4xvTLiY8K%2BmBIAx9Dus7X593vly%2Bpi4zKlBRe5JgaTiuFsSOXew6FUooJLjTDQUZk6kIuTUfH6REiK2lZK6gzJoBnsq55hjNpMuxzyy2tVQv6EW38TSoqmCjlDSmuwMvlHw%3D%3D%22%7D; _gali=lnkNextResultsPage",
        'authority': "api2.realtor.ca",
        'sec-ch-ua': '" Not;A Brand";v="99", "Google Chrome";v="97", "Chromium";v="97"',
        'accept': "*/*",
        'content-type': "application/x-www-form-urlencoded; charset=UTF-8",
        'sec-ch-ua-mobile': "?1",
        'user-agent': "Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/97.0.4692.99 Mobile Safari/537.36",
        'sec-ch-ua-platform': '"Android"',
        'origin': "https://www.realtor.ca",
        'sec-fetch-site': "same-site",
        'sec-fetch-mode': "cors",
        'sec-fetch-dest': "empty",
        'referer': "https://www.realtor.ca/",
        'accept-language': "en-US,en;q=0.9"
        }

    def start_requests(self):
        
        yield scrapy.Request(url=self.api,
                        callback=self.parse,
                        method='POST', 
                        headers=self.headers,
                        body=json.dumps(self.payload) )  

    def parse(self, response):
        print('PARSE')
        results = json.loads(response.body).get('Results')
        print('results')
        for r in results:
            yield {
                'Id': r.get('Id'),
                'Price': r.get('Property').get('Price')
            }

if __name__ == '__main__':
    import os
    from scrapy.cmdline import execute

    os.chdir(os.path.dirname(os.path.realpath(__file__)))

    SPIDER_NAME = RealtorSpider.name
    try:
        execute(
            [
                'scrapy',
                'crawl',
                SPIDER_NAME,
                '-s',
                'FEED_EXPORT_ENCODING=utf-8',
            ]
        )
    except SystemExit:
        pass

1 个解决方案

解决方案1
 0  2022-01-30 09:56:48

  1. url 中有多余的/ 。 删除它以避免 404 错误。
  2. 您不需要json.dumps因为payload已经是一个字符串。
  3. 让 scrapy 自己处理 cookies。
  4. 我猜您的标题是错误的，因为更改它们后我不再获得 403 状态。
  5. 了解如何从脚本运行 scrapy 。

蜘蛛.py：

import scrapy


class RealtorSpider(scrapy.Spider):
    name = 'realtor'
    allowed_domains = ['api2.realtor.ca']

    custom_settings = {
        'DOWNLOAD_DELAY': 0.5,
    }
    api = "https://api2.realtor.ca/Listing.svc/PropertySearch_Post"

    payload = "ZoomLevel=11&LatitudeMax=43.98268&LongitudeMax=-78.96028&LatitudeMin=43.43223&LongitudeMin=-79.79249&Sort=6-D&PropertySearchTypeId=1&TransactionTypeId=2&PropertyTypeGroupID=1&Currency=CAD&CurrentPage=1&ApplicationId=1&CultureId=1&Version=7.0&RecordsPerPage=200"

    headers = {
        "Accept": "*/*",
        "Accept-Encoding": "gzip, deflate, br",
        "Accept-Language": "en-US,en;q=0.5",
        "Cache-Control": "no-cache",
        "Connection": "keep-alive",
        "Content-Type": "application/x-www-form-urlencoded; charset=UTF-8",
        "DNT": "1",
        "Host": "api2.realtor.ca",
        "Origin": "https://www.realtor.ca",
        "Pragma": "no-cache",
        "Referer": "https://www.realtor.ca/",
        "Sec-Fetch-Dest": "empty",
        "Sec-Fetch-Mode": "cors",
        "Sec-Fetch-Site": "same-site",
        "Sec-GPC": "1",
        "TE": "trailers",
        'User-Agent': "Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/97.0.4692.99 Mobile Safari/537.36",
    }
    
    def start_requests(self):
        yield scrapy.Request(url=self.api,
                             callback=self.parse,
                             method='POST',
                             headers=self.headers,
                             body=self.payload)

    def parse(self, response):
        print('PARSE')
        results = response.json().get('Results')
        print('results')
        for r in results:
            yield {
                'Id': r.get('Id'),
                'Price': r.get('Property').get('Price')
            }

主要.py：

from scrapy.crawler import CrawlerProcess
from scrapy.utils.project import get_project_settings

if __name__ == "__main__":
    spider = 'realtor'
    settings = get_project_settings()
    process = CrawlerProcess(settings)
    process.crawl(spider)
    process.start()

output：

{'Id': '23983739', 'Price': '$799,000'}
{'Id': '23983759', 'Price': '$579,000'}
{'Id': '23983752', 'Price': '$750,000'}
...
...
...

暂无
暂无

声明:本站的技术帖子网页，遵循CC BY-SA 4.0协议，如果您需要转载，请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.

相关问题 在Python中正确格式化http.client.HTTPSConnection 如何通过 http.client.HTTPSConnection 设置 SNI 如何在python中禁用http.client.HTTPSConnection类的ssl验证？ http.client.HTTPSConnection请求错误无法捕获 错误 403：scrapy 中未处理或不允许 HTTP 状态代码 scrapy spider错误：403 HTTP状态代码未处理或不允许 Scrapy 下载图片 403 错误 Scrapy：所有请求均出现403错误 Scrapy返回403错误（禁止） 使用httlib的HTTPSConnection和PKCS＃12证书时出错
相关标签
 
粤ICP备18138465号  © 2020-2024 STACKOOM.COM