Python请求与curl

Question

I'm trying to extract data from URL using POST with specific headers and body payload. Using curl or a tool like Advanced Rest Client, I'm able to get the JSON containing the data I want and the pretty code 200. However, if I try doing the same request using python requests, it won't work. I get a code 405 and bad json.

These are the configs I'm using:

curl -H "Content-Type: application/json" --data @payload.json -X POST https://www.walmart.com/terra-firma/fetch?rgs=REVIEWS_MAP

the json file payload.json, looks like this:

{"productId":"10VJLBWPW82T","paginationContext":{"sort":"relevancy","page":3}}

And here is the python3 code:

import requests

url = "http://www.walmart.com/terra-firma/fetch?rgs=REVIEWS_MAP"

head = {'Content-type':'application/json'}
payload = {"productId":"10VJLBWPW82T","paginationContext":{"sort":"relevancy","page":3}}

rep = requests.post(url, headers=head, data=payload)

Does anybody know how to explain this difference ?

Answer 1

Two things, first use the 'json' parameter in the request not 'data' for posting json. Secondly use https not http protocol for this particular web-page.

import requests

url = "https://www.walmart.com/terra-firma/fetch?rgs=REVIEWS_MAP"
head = {'Content-type':'application/json'}
payload = {"productId":"10VJLBWPW82T","paginationContext":{"sort":"relevancy","page":3}}
rep = requests.post(url, headers=head, json=payload)
print (rep.status_code)
print (rep.json())

Outputs:

200
{'status': 'OK', 'header': {'headerAttributes': {'timing': {'total': {'threads': 3, 'orchestrate': 1, 'delay': 0, 'elapsed': 30, 'futures': 37}}, 'tenant': {'WM_BU_ID': '0', 'WM_VERTICAL_ID': '0', 'WM_LOCALE_ID': 'en_US', 'WM_MART_ID': '0', 'WM_CHANNEL_ID': '0'}, 'server': {'ver': '84.0.2', 'env': 'prod-a', 'dc': 'prod-dfw8', 'name': 'app-273848643-9-280204056', 'sha': 'bd15b234fd7d120a2aa489becec41385c5e34719'}, 'context': {'browser': 'CRAWLER', 'device': 'UNSPECIFIED', 'topTxId': 'b0886fbe-1e8-161e8486db3000', 'access': True}}}, 'errors': [], 'payload': {'products': {}, 'reviews': {'10VJLBWPW82T': {'percentageFourCount': 2, 'percentageFiveCount': 91, 'ratingValueTwoCount': 1, 'customerReviews': [{'showRecommended': True, 'photos': [], 'userNickname': 'murrie56', 'positiveFeedback': 5, 'negativeFeedback': 0, 'userAttributes': {}, 'rating': 5.0, 'authorId': '6b61e60fc8b3f358b84858dbe0fd50f7', 'reviewTitle': 'LOVE IT! LOVE IT! LOVE IT!', 'reviewText': "I've been using this ever since it was introduced back in the late 70's, I was in High School. I still get compliments on it. I'll never wear anything else as long as it's available. I buy in bulk.", 'videos': [], 'reviewSubmissionTime': '11/30/2014', 'reviewId': '40087713', 'recommended': True}, {'showRecommended': False, 'photos': [], 'userNickname': 'Tony', 'positiveFeedback': 0, 'negativeFeedback': 0, 'userAttributes': {}, 'rating': 5.0, 'authorId': 'ce2dbcdd5602bd2fde14f2d40bfc3217', 'reviewText': 'my girl friend loved it', 'videos': [], 'reviewSubmissionTime': '12/18/2017', 'reviewId': '186608534'}, {'showRecommended': True, 'photos': [], 'userNickname': 'SandyC', 'positiveFeedback': 5, 'badges': [{'id': 'VerifiedPurchaser', 'badgeType': 'Custom', 'contentType': 'REVIEW'}], 'userAge': '45 - 54', 'negativeFeedback': 0, 'userAttributes': {'Usage': 'Every day', 'Age': '45 - 54', 'Ownership': 'Longer than one year', 'Gender': 'Female'}, 'rating': 5.0, 'authorId': '6a0c52dc6fc5c63ccff7bde8ad1f3e9a', 'reviewTitle': 'Have been using this perfume for years', 'reviewText': 'Love it!', 'userLocation': 'Barnwell, SC', 'userGender': 'Female', 'reviewSubmissionTime': '12/21/2013', 'reviewId': '19488031', 'videos': [], 'recommended': True}, {'showRecommended': True, 'photos': [], 'userNickname': 'corkmiester', 'positiveFeedback': 6, 'negativeFeedback': 1, 'userAttributes': {}, 'rating': 5.0, 'authorId': '95edbfc402035dd0237b48ef92a5401f', 'reviewTitle': 'Lifetime Favorite', 'reviewText': "I've used this product since my teenage years, and I'm happy to say that the scent has never varied, never changed. It smells the same as it did then and is my all-time favorite perfume. I love the staying power of this product. An application in the morning lasts all day. I've had many a compliment over the years when wearing this and the affordable price makes it that much better. I hope they never stop making this stuff. I'll be a happy camper as long as I can find it. :)", 'videos': [], 'reviewSubmissionTime': '12/10/2014', 'reviewId': '41086799', 'recommended': True}, {'showRecommended': True, 'photos': [], 'userNickname': 'Clydesdales', 'positiveFeedback': 4, 'negativeFeedback': 0, 'userAttributes': {}, 'rating': 5.0, 'authorId': '0b1608e40c638ce49215bfba6624e27d', 'reviewTitle': 'GL McC', 'reviewText': 'Have used this cologne for years and I can always get it at Walmart. The other places I have gotten it before no longer carry the product. Thanks Walmart for carrying this item!!', 'videos': [], 'reviewSubmissionTime': '3/30/2016', 'reviewId': '84475295', 'recommended': True}], 'overallRatingRange': 5.0, 'totalReviewCount': 45, 'activeFilters': {}, 'percentageTwoCount': 2, 'roundedAverageOverallRating': 4.8, 'recommendedPercentage': 95, 'ratingValueFourCount': 1, 'pagination': {'total': 43, 'currentSpan': '11-15', 'next': {'num': 0, 'url': 'sort=relevancy&page=4', 'gap': False, 'active': False}, 'pages': [{'num': 1, 'url': 'sort=relevancy&page=1', 'gap': False, 'active': False}, {'num': 2, 'url': 'sort=relevancy&page=2', 'gap': False, 'active': False}, {'num': 3, 'url': 'sort=relevancy&page=3', 'gap': False, 'active': True}, {'num': 4, 'url': 'sort=relevancy&page=4', 'gap': False, 'active': False}, {'num': 5, 'url': 'sort=relevancy&page=5', 'gap': False, 'active': False}, {'num': 6, 'url': 'sort=relevancy&page=6', 'gap': False, 'active': False}, {'num': 0, 'gap': True, 'active': False}, {'num': 9, 'url': 'sort=relevancy&page=9', 'gap': False, 'active': False}], 'previous': {'num': 0, 'url': 'sort=relevancy&page=2', 'gap': False, 'active': False}}, 'percentageOneCount': 2, 'ratingValueFiveCount': 41, 'ratingValueThreeCount': 1, 'ratingValueOneCount': 1, 'activeSort': 'relevancy', 'percentageThreeCount': 2, 'averageOverallRating': 4.7778}}}}