curl 请求在 python3 中没有很好地转换

Question

I am using python3.7.7 and I am trying to make a POST request. This POST seems to have some anomalies because it always generates a 403 error.
In Chrome the problem does not appear, the error code is 200(OK) and the POST responds with expeceted data
To narrow the problem down, I extracted the curl request from the browser and tried to hardcode it into python.
So, I have the working curl extracted from browser:

curl 'http://www.zjnsf.gov.cn/h/01/news_list.aspx?t=%u57fa%u91d1%u7ed3%u9898%u9879%u76ee%u6e05%u5355' --data '__VIEWSTATE=%2FwEPDwUJOTIwODkyNjA5D2QWAmYPZBYCZg9kFgICAw9kFgICAQ9kFgICBQ9kFgQCAQ9kFgICAw8PFgIeBFRleHQFDOS%2FoeaBr%2BaQnOe0omRkAgMPZBYCAgcPZBYCZg8PFgQeBFJPV1MC5BIeBUlOREVYZmQWCGYPDxYCHgdWaXNpYmxlaGRkAgEPDxYCHwNoZGQCAg8PFgIfA2dkZAIDDw8WAh8DZ2RkZN0f2oaGWjQWIew4DBiZrFuBSFq0&__VIEWSTATEGENERATOR=E98323FB&ctl00%24ctl00%24ContentPlaceHolder1%24ContentPlaceHolder1%24newData=&ctl00%24ctl00%24ContentPlaceHolder1%24ContentPlaceHolder1%24ctl16=&ctl00%24ctl00%24ContentPlaceHolder1%24ContentPlaceHolder1%24ctl05=&ctl00%24ctl00%24ContentPlaceHolder1%24ContentPlaceHolder1%24ctl06=' --compressed --insecure

And the python code for it:

 post_data = {} url = 'http://www.zjnsf.gov.cn/h/01/news_list.aspx?t=%u57fa%u91d1%u7ed3%u9898%u9879%u76ee%u6e05%u5355' post_data['__VIEWSTATE'] = '%2FwEPDwUJOTIwODkyNjA5D2QWAmYPZBYCZg9kFgICAw9kFgICAQ9kFgICBQ9kFgQCAQ9kFgICAw8PFgIeBFRleHQFDOS%2FoeaBr%2BaQnOe0omRkAgMPZBYCAgcPZBYCZg8PFgQeBFJPV1MC5BIeBUlOREVYZmQWCGYPDxYCHgdWaXNpYmxlaGRkAgEPDxYCHwNoZGQCAg8PFgIfA2dkZAIDDw8WAh8DZ2RkZN0f2oaGWjQWIew4DBiZrFuBSFq0' post_data['__VIEWSTATEGENERATOR'] = 'E98323FB' post_data['ctl00%24ctl00%24ContentPlaceHolder1%24ContentPlaceHolder1%24newData'] = '' post_data['ctl00%24ctl00%24ContentPlaceHolder1%24ContentPlaceHolder1%24ctl16'] = '' post_data['ctl00%24ctl00%24ContentPlaceHolder1%24ContentPlaceHolder1%24ctl05'] = '' post_data['ctl00%24ctl00%24ContentPlaceHolder1%24ContentPlaceHolder1%24ctl06'] = '' p_data = json.dumps(post_data) news_search_page = req_session.post(url, data=p_data)

This always responds with 403 error.
Any idea what might be wrong OR in what direction to investigate?

Answer 1

Maybe you need to add a header with some content. Some API's require you to show which browser (User-Agent) you are using, you can create a header like this:

HEADERS = {'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.102 Safari/537.36'}

And then the API will think that you are using an MacBook with Chrome.

and btw: the headers are added to the code like this:

news_search_page = req_session.post(url, data=p_data, headers=HEADERS)

You can also try to write json instead of data:

news_search_page = req_session.post(url, json=p_data, headers=HEADERS)

Answer 2

Ok, so now I tried myself, and I don't know if I did it right because I don't understand Chineese, But i found out that the header with a user agent was essential for creating a get request.

With this code i got a 200 response. I hope you can use it.

import requests
HEADERS = {'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.102 Safari/537.36'}
link = 'http://www.zjnsf.gov.cn/h/01/news_list.aspx'
post_data = {
        '__VIEWSTATE': '%2FwEPDwUJOTIwODkyNjA5D2QWAmYPZBYCZg9kFgICAw9kFgICAQ9kFgICBQ9kFgQCAQ9kFgICAw8PFgIeBFRleHQFDOS%2FoeaBr%2BaQnOe0omRkAgMPZBYCAgcPZBYCZg8PFgQeBFJPV1MC5BIeBUlOREVYZmQWCGYPDxYCHgdWaXNpYmxlaGRkAgEPDxYCHwNoZGQCAg8PFgIfA2dkZAIDDw8WAh8DZ2RkZN0f2oaGWjQWIew4DBiZrFuBSFq0',
        '__VIEWSTATEGENERATOR': 'E98323FB',
        'ctl00%24ctl00%24ContentPlaceHolder1%24ContentPlaceHolder1%24newData': '',
        'ctl00%24ctl00%24ContentPlaceHolder1%24ContentPlaceHolder1%24ctl16': '',
        'ctl00%24ctl00%24ContentPlaceHolder1%24ContentPlaceHolder1%24ctl05': '',
        'ctl00%24ctl00%24ContentPlaceHolder1%24ContentPlaceHolder1%24ctl06': '',
        }
r = requests.post(link, json=post_data, headers=HEADERS)
print(r)

Answer 3

I used Wireshark to detect the small differences between the 2 POST requests; it seems that the link was the actual problem, unicode characters are written in a text editor like this: '%u57fa' something happens in the belly of Python that the characters are miss-interpreted.
For eg char '%u57fa' will be transformed in '%25u57fa'; so python extra encodes the characters like '%' one more time before making the request.
So long story short, I changed the link from: /h/01/news_list.aspx?t=%u57fa%u91d1%u7ed3%u9898%u9879%u76ee%u6e05%u5355
TO:
http://www.zjnsf.gov.cn/h/01/news_list.aspx?t=基金结题项目清单

this solves the problem, but I don't really understand who creates the problem, since unicode values for the characters are OK:

57FA -> 基