有没有办法绕过这个 403 python 请求响应

Question

I am trying to scrape some data from this website https://www.sportsmansguide.com/productlist/military-surplus/all-military-surplus?d=122&c=199&sb=na&ipp=ALL and when I send requests to this website it return 403 response

Here is my code:

import requests

URL = "https://www.sportsmansguide.com/productlist/military-surplus/all-military-surplus?d=122&c=199&sb=na&pg=1"


headers = {
    "authority": "www.sportsmansguide.com",
    "method": "GET",
    "path": "/productlist/military-surplus/all-military-surplus?d=122&c=199&sb=na",
    "scheme": "https",
    "accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9",
    "accept-language": "en-US,en;q=0.9",
    "cache-control": "max-age=0",
    "referer": "https://www.sportsmansguide.com/productlist/military-surplus/all-military-surplus?d=122&c=199&sb=na&ipp=ALL",
    "sec-ch-ua": '"Chromium";v="94", "Google Chrome";v="94", ";Not A Brand";v="99"',
    "sec-ch-ua-mobile": "?0",
    "sec-ch-ua-platform": '"Windows"',
    "sec-fetch-dest": "empty",
    "sec-fetch-mode": "cors",
    "sec-fetch-site": "same-origin",
    "sec-fetch-user": "?1",
    "upgrade-insecure-requests": "1",
    "user-agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4606.81 Safari/537.36",
    "x-requested-with": "XMLHttpRequest"
}

r = requests.get(URL, headers=headers)

print(r.status_code)

403

Answer 1

Try to use "User-Agent" instead of "user-agent", but you may be banned by ip.

Debug you browser and Python connections with Charles HTTPS proxy, see the difference and eliminate it:

https://www.charlesproxy.com/

If you are blocked by ip, use Tor:

https://tor.eff.org/

More in this book: Mitchell, Ryan E - Web scraping with Python:

http://libgen.rs/book/index.php?md5=0560CF60528E817CD8C87555BCD57C9C