Powershell Invoke-WebRequest 有效，但 Python Requests 无效

Question

This is about a weird situation where the Powershell Invoke-WebRequest works as intended and the Python Requests does not.

I am trying to scrape a ecommerce site using python. Part of the scraping is to test if an item can be added to cart. Using the Chrome Developer tools F12, I was able to extract the following Powershell scripts.

Step 1 - Request a customer session

$session = New-Object Microsoft.PowerShell.Commands.WebRequestSession
$session.UserAgent = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4844.51 Safari/537.36"
$secPasswd=ConvertTo-SecureString "password" -AsPlainText -Force
$myCreds=New-Object System.Management.Automation.PSCredential -ArgumentList "username",$secPasswd
Invoke-WebRequest -UseBasicParsing -Uri "https://bck.hermes.com/customer-session?locale=de_de" `
-Proxy 'http://proxyaddress' `
-ProxyCredential $mycreds `
-WebSession $session `
-Headers @{
"sec-ch-ua"="`" Not A;Brand`";v=`"99`", `"Chromium`";v=`"99`", `"Google Chrome`";v=`"99`""
  "Accept"="application/json, text/plain, */*"
  "Cache-Control"="no-cache"
  "DNT"="1"
  "sec-ch-ua-mobile"="?0"
  "sec-ch-ua-platform"="`"Windows`""
  "Origin"="https://www.hermes.com"
  "Sec-Fetch-Site"="same-site"
  "Sec-Fetch-Mode"="cors"
  "Sec-Fetch-Dest"="empty"
  "Referer"="https://www.hermes.com/"
  "Accept-Encoding"="gzip, deflate, br"
  "Accept-Language"="en-US,en;q=0.9,ja;q=0.8,zh-CN;q=0.7,zh-TW;q=0.6,zh;q=0.5"
} | Select-Object -Expand RawContent

The response would give me a "ECOM_SESS" cookie along with a bunch others.

I would then pass the ECOM_SESS cookie to the next step.

Step 2 - add to cart

$session = New-Object Microsoft.PowerShell.Commands.WebRequestSession
$session.UserAgent = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4844.51 Safari/537.36"
$session.Cookies.Add((New-Object System.Net.Cookie("ECOM_SESS", "XXXXXXXXXXXXXXXX", "/", ".hermes.com")))
$secPasswd=ConvertTo-SecureString "password" -AsPlainText -Force
$myCreds=New-Object System.Management.Automation.PSCredential -ArgumentList "username",$secPasswd
Invoke-WebRequest -UseBasicParsing -Uri "https://bck.hermes.com/add-to-cart" `
-Proxy 'http://proxyaddress' `
-ProxyCredential $mycreds `
-Method "POST" `
-WebSession $session `
-Headers @{
"sec-ch-ua"="`" Not A;Brand`";v=`"99`", `"Chromium`";v=`"99`", `"Google Chrome`";v=`"99`""
  "Accept"="application/json, text/plain, */*"
  "DNT"="1"
  "sec-ch-ua-mobile"="?0"
  "sec-ch-ua-platform"="`"Windows`""
  "Origin"="https://www.hermes.com"
  "Sec-Fetch-Site"="same-site"
  "Sec-Fetch-Mode"="cors"
  "Sec-Fetch-Dest"="empty"
  "Referer"="https://www.hermes.com/"
  "Accept-Encoding"="gzip, deflate, br"
  "Accept-Language"="en-US,en;q=0.9,ja;q=0.8,zh-CN;q=0.7,zh-TW;q=0.6,zh;q=0.5"
} `
-ContentType "application/json" `
-Body "{`"locale`":`"de_de`",`"items`":[{`"category`":`"direct`",`"sku`":`"H079082CCAC`"}]}"

With the Powershell script above, the process works perfectly and I would get responses from each of the two steps. Note this is with a rotating IP proxy which refreshes the IP on each request to prevent bot detection.

However, when I tried to integrate this into my Python code, I would encounter the requirement of captcha upon Step 2, irrespective of the proxy server used.

Here is the relevant python code:

from __future__ import print_function
import bs4
import requests
from requests.cookies import RequestsCookieJar
import jsons

def main():
    url1= "https://bck.hermes.com/customer-session?locale=de_de"
    url2 = "https://bck.hermes.com/add-to-cart"
    proxies1 = {
        "http": "xxxxxxxxxxxxxxxxxx"
    }
    headers1 = {
            'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4844.51 Safari/537.36',         
            'sec-ch-ua': '" Not A;Brand";v="99", "Chromium";v="99", "Google Chrome";v="99"',
            'Accept': 'application/json, text/plain, */*',
            'Cache-Control': 'no-cache',
            'DNT': '1',
            'sec-ch-ua-mobile': '?0',
            'sec-ch-ua-platform': '"Windows"',
            'Origin': 'https://www.hermes.com',
            'Sec-Fetch-Site': 'same-site',
            'Sec-Fetch-Mode': 'cors',
            'Sec-Fetch-Dest': 'document',
            'Referer': 'https://www.hermes.com/',
            'Accept-Encoding': 'gzip, deflate, br',
            'Accept-Language': 'en-US,en;q=0.9,ja;q=0.8,zh-CN;q=0.7,zh-TW;q=0.6,zh;q=0.5'
    }
    headers2 = {
            'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4844.51 Safari/537.36',
            'sec-ch-ua': '" Not A;Brand";v="99", "Chromium";v="99", "Google Chrome";v="99"',
            'Accept': 'application/json, text/plain, */*',
            'DNT': '1',
            'sec-ch-ua-mobile': '?0',
            'sec-ch-ua-platform': '"Windows"',
            'Origin': 'https://www.hermes.com',
            'Sec-Fetch-Site': 'same-site',
            'Sec-Fetch-Mode': 'cors',
            'Sec-Fetch-Dest': 'empty',
            'Referer': 'https://www.hermes.com/',
            'Accept-Encoding': 'gzip, deflate, br',
            'Accept-Language': 'en-US,en;q=0.9,ja;q=0.8,zh-CN;q=0.7,zh-TW;q=0.6,zh;q=0.5'
    }
    
    body2 = {"locale":"de_de","items":[{"category":"direct","sku":"H079082CCAC"}]}


    #Step 1

    f = requests.get(url1, headers=headers1,proxies=proxies1)
    print(f"1Response Body: {f.text}\n")
    ECOM_SESS = f.cookies['ECOM_SESS']
    cookieJar = RequestsCookieJar()
    cookieJar.set('ECOM_SESS', ECOM_SESS, domain='.hermes.com', path='/')

    #Step 2
    g = requests.post(url2, headers=headers2,cookies=cookieJar,proxies=proxies1,json=body2)
    print(f"2Response Body: {g.text}\n")

   

if __name__ == '__main__':
    main()

Running the Python code here, Step 1 would nicely give the intended response with the cookies needed to pass onto Step 2. However, Step 2 would always result in a captcha response.

I am just curious as to the difference between the Powershell Invoke-WebRequest method and the Python Requests method, as there has to be something fundamentally different for the former to avoid captcha completely and the latter to always get hit with captcha.

Would appreciate any thoughts and insights from you guys! Thanks!

Answer 1

I'm not sure specifically what it is about requests that's triggering the bot protection on the site, but based on this you might have luck using:

requests.request("POST", url2, headers=headers2, cookies=cookieJar, proxies=proxies1, json=body2)

Alternatively you could try urllib3 instead of Requests.

Here's your powershell script simplified too just as an excercise.

$secPasswd=ConvertTo-SecureString "password" -AsPlainText -Force
$myCreds=New-Object System.Management.Automation.PSCredential -ArgumentList "username",$secPasswd
$headers = @{
"sec-ch-ua"='" Not A;Brand";v="99", "Chromium";v="99", "Google Chrome";v="99"'
  "DNT"="1"
  "sec-ch-ua-mobile"="?0"
  "sec-ch-ua-platform"="`"Windows`""
  "Origin"="https://www.hermes.com"
  "Sec-Fetch-Site"="same-site"
  "Sec-Fetch-Mode"="cors"
  "Sec-Fetch-Dest"="empty"
  "Referer"="https://www.hermes.com/"
}
Invoke-WebRequest -UseBasicParsing -Uri "https://bck.hermes.com/customer-session?locale=de_de" `
-Proxy 'http://proxyaddress' `
-ProxyCredential $mycreds `
-SessionVariable session `
-Headers $headers
Invoke-WebRequest -UseBasicParsing -Uri "https://bck.hermes.com/add-to-cart" `
-Proxy 'http://proxyaddress' `
-ProxyCredential $mycreds `
-Method POST `
-WebSession $session `
-Headers $headers `
-ContentType "application/json" `
-Body "{`"locale`":`"de_de`",`"items`":[{`"category`":`"direct`",`"sku`":`"H079082CCAC`"}]}"