Python请求与curl

Question

我正在尝试使用具有特定标头和正文有效负载的POST从URL中提取数据。 使用curl或Advanced Rest Client之类的工具，我可以获取包含所需数据和漂亮代码200的JSON。但是，如果我尝试使用python请求执行相同的请求，它将无法正常工作。 我收到代码405和错误的json。

这些是我正在使用的配置：

curl -H "Content-Type: application/json" --data @payload.json -X POST https://www.walmart.com/terra-firma/fetch?rgs=REVIEWS_MAP

json文件payload.json如下所示：

{"productId":"10VJLBWPW82T","paginationContext":{"sort":"relevancy","page":3}}

这是python3代码：

import requests

url = "http://www.walmart.com/terra-firma/fetch?rgs=REVIEWS_MAP"

head = {'Content-type':'application/json'}
payload = {"productId":"10VJLBWPW82T","paginationContext":{"sort":"relevancy","page":3}}

rep = requests.post(url, headers=head, data=payload)

有人知道如何解释这种差异吗？

Answer 1

有两件事，首先在请求中使用'json'参数，而不是用于发布json的'data'参数。 其次，对该特定网页使用https而非http协议。

import requests

url = "https://www.walmart.com/terra-firma/fetch?rgs=REVIEWS_MAP"
head = {'Content-type':'application/json'}
payload = {"productId":"10VJLBWPW82T","paginationContext":{"sort":"relevancy","page":3}}
rep = requests.post(url, headers=head, json=payload)
print (rep.status_code)
print (rep.json())

输出：

200
{'status': 'OK', 'header': {'headerAttributes': {'timing': {'total': {'threads': 3, 'orchestrate': 1, 'delay': 0, 'elapsed': 30, 'futures': 37}}, 'tenant': {'WM_BU_ID': '0', 'WM_VERTICAL_ID': '0', 'WM_LOCALE_ID': 'en_US', 'WM_MART_ID': '0', 'WM_CHANNEL_ID': '0'}, 'server': {'ver': '84.0.2', 'env': 'prod-a', 'dc': 'prod-dfw8', 'name': 'app-273848643-9-280204056', 'sha': 'bd15b234fd7d120a2aa489becec41385c5e34719'}, 'context': {'browser': 'CRAWLER', 'device': 'UNSPECIFIED', 'topTxId': 'b0886fbe-1e8-161e8486db3000', 'access': True}}}, 'errors': [], 'payload': {'products': {}, 'reviews': {'10VJLBWPW82T': {'percentageFourCount': 2, 'percentageFiveCount': 91, 'ratingValueTwoCount': 1, 'customerReviews': [{'showRecommended': True, 'photos': [], 'userNickname': 'murrie56', 'positiveFeedback': 5, 'negativeFeedback': 0, 'userAttributes': {}, 'rating': 5.0, 'authorId': '6b61e60fc8b3f358b84858dbe0fd50f7', 'reviewTitle': 'LOVE IT! LOVE IT! LOVE IT!', 'reviewText': "I've been using this ever since it was introduced back in the late 70's, I was in High School. I still get compliments on it. I'll never wear anything else as long as it's available. I buy in bulk.", 'videos': [], 'reviewSubmissionTime': '11/30/2014', 'reviewId': '40087713', 'recommended': True}, {'showRecommended': False, 'photos': [], 'userNickname': 'Tony', 'positiveFeedback': 0, 'negativeFeedback': 0, 'userAttributes': {}, 'rating': 5.0, 'authorId': 'ce2dbcdd5602bd2fde14f2d40bfc3217', 'reviewText': 'my girl friend loved it', 'videos': [], 'reviewSubmissionTime': '12/18/2017', 'reviewId': '186608534'}, {'showRecommended': True, 'photos': [], 'userNickname': 'SandyC', 'positiveFeedback': 5, 'badges': [{'id': 'VerifiedPurchaser', 'badgeType': 'Custom', 'contentType': 'REVIEW'}], 'userAge': '45 - 54', 'negativeFeedback': 0, 'userAttributes': {'Usage': 'Every day', 'Age': '45 - 54', 'Ownership': 'Longer than one year', 'Gender': 'Female'}, 'rating': 5.0, 'authorId': '6a0c52dc6fc5c63ccff7bde8ad1f3e9a', 'reviewTitle': 'Have been using this perfume for years', 'reviewText': 'Love it!', 'userLocation': 'Barnwell, SC', 'userGender': 'Female', 'reviewSubmissionTime': '12/21/2013', 'reviewId': '19488031', 'videos': [], 'recommended': True}, {'showRecommended': True, 'photos': [], 'userNickname': 'corkmiester', 'positiveFeedback': 6, 'negativeFeedback': 1, 'userAttributes': {}, 'rating': 5.0, 'authorId': '95edbfc402035dd0237b48ef92a5401f', 'reviewTitle': 'Lifetime Favorite', 'reviewText': "I've used this product since my teenage years, and I'm happy to say that the scent has never varied, never changed. It smells the same as it did then and is my all-time favorite perfume. I love the staying power of this product. An application in the morning lasts all day. I've had many a compliment over the years when wearing this and the affordable price makes it that much better. I hope they never stop making this stuff. I'll be a happy camper as long as I can find it. :)", 'videos': [], 'reviewSubmissionTime': '12/10/2014', 'reviewId': '41086799', 'recommended': True}, {'showRecommended': True, 'photos': [], 'userNickname': 'Clydesdales', 'positiveFeedback': 4, 'negativeFeedback': 0, 'userAttributes': {}, 'rating': 5.0, 'authorId': '0b1608e40c638ce49215bfba6624e27d', 'reviewTitle': 'GL McC', 'reviewText': 'Have used this cologne for years and I can always get it at Walmart. The other places I have gotten it before no longer carry the product. Thanks Walmart for carrying this item!!', 'videos': [], 'reviewSubmissionTime': '3/30/2016', 'reviewId': '84475295', 'recommended': True}], 'overallRatingRange': 5.0, 'totalReviewCount': 45, 'activeFilters': {}, 'percentageTwoCount': 2, 'roundedAverageOverallRating': 4.8, 'recommendedPercentage': 95, 'ratingValueFourCount': 1, 'pagination': {'total': 43, 'currentSpan': '11-15', 'next': {'num': 0, 'url': 'sort=relevancy&page=4', 'gap': False, 'active': False}, 'pages': [{'num': 1, 'url': 'sort=relevancy&page=1', 'gap': False, 'active': False}, {'num': 2, 'url': 'sort=relevancy&page=2', 'gap': False, 'active': False}, {'num': 3, 'url': 'sort=relevancy&page=3', 'gap': False, 'active': True}, {'num': 4, 'url': 'sort=relevancy&page=4', 'gap': False, 'active': False}, {'num': 5, 'url': 'sort=relevancy&page=5', 'gap': False, 'active': False}, {'num': 6, 'url': 'sort=relevancy&page=6', 'gap': False, 'active': False}, {'num': 0, 'gap': True, 'active': False}, {'num': 9, 'url': 'sort=relevancy&page=9', 'gap': False, 'active': False}], 'previous': {'num': 0, 'url': 'sort=relevancy&page=2', 'gap': False, 'active': False}}, 'percentageOneCount': 2, 'ratingValueFiveCount': 41, 'ratingValueThreeCount': 1, 'ratingValueOneCount': 1, 'activeSort': 'relevancy', 'percentageThreeCount': 2, 'averageOverallRating': 4.7778}}}}