[英]How to include body to a POST request in Python with requests
我需要從頁面中提取所有巡航。 我發現我需要提出2個要求。 一個用於獲取總結果+參考單個巡航,另一個用於巡航本身。 到目前為止,ii已成功發出請求並獲得了JSON
。 問題是該請求僅返回首頁中包含的引用。 最初我用這個:
https://www.pocruises.com.au/sc_ignore/b2c/cruiseresults/searchresults
作為POST
請求網址。 我是請求中的一些標頭:
Host: www.pocruises.com.au
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Firefox/52.0
Accept: application/json, text/plain, */*
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate, br
Referer: https://www.pocruises.com.au/cruises/search
Content-Type: application/json;charset=utf-8
Content-Length: 613
Cookie: ASP.NET_SessionId=zm50ahxa4uoiwcuowgkeizcn; SC_ANALYTICS_GLOBAL_COOKIE=f459da6904ee4cea8a809455f37b09c5|False; optimizelyEndUserId=oeu1480435687102r0.9194470051495017; optimizelySegments=%7B%223906442756%22%3A%22none%22%2C%223911484226%22%3A%22ff%22%2C%223917223299%22%3A%22direct%22%2C%223920524266%22%3A%22false%22%2C%224924982297%22%3A%22true%22%7D; optimizelyBuckets=%7B%227883701652%22%3A%227867701359%22%7D; _ga=GA1.3.1128287966.1480435688; AdBlockDetected=false; _msuuid_29439mm27589=D0670AB5-C96A-4706-A563-9F29FCA3D9D2; gwcc=%7B%22fallback%22%3A%221300159454%22%2C%22clabel%22%3A%22ykseCIGywFkQhoDuxQM%22%2C%22backoff%22%3A86400%2C%22backoff_expires%22%3A1480522087%7D; gaFindACruise=; _gat=1; optimizelyPendingLogEvents=%5B%22n%3Dengagement%26u%3Doeu1480435687102r0.9194470051495017%26wxhr%3Dtrue%26time%3D1480437791.708%26f%3D7274530066%2C7883701652%26g%3D3909534788%22%5D
Connection: keep-alive
到現在為止,如果不算我返回的僅有6個結果,它就可以工作。 在撰寫本文時,他們必須為106。 響應包含具有總數(正確)和頁數,總頁數等的元數據。 然后我看到了: Firefox屏幕快照此Request body
包含所有頁面中導航所需的全部內容。 到目前為止,這是我的全部代碼:
import requests
url = "https://www.pocruises.com.au/sc_ignore/b2c/cruiseresults/searchresults"
session = requests.session()
data = {"searchParameters": {"p": [], "c": [], "d": [], "s": [], "ms": [], "adv": [], "sort": "dpa", "page": 5},
"renderingParameters": {"DefaultSortOption": "dpa", "LargeScreenFlag": "true", "NewModelIsLoading": "false",
"PagingAnchor": "", "ViewStyleSelectorVisible": "true", "FilterBarVisible": "true",
"NumberOfResultsAndSortByPanelVisible": "true", "DefaultResultsView": "Grid",
"MaxNumberOfResults": 0, "PaginationEnabled": "true", "KeepPageState": "true",
"PageSize": 9, "DefaultResultsGrouping": "Itinerary", "Duration": [],
"CruiseItinerary": [], "Voyage": [], "ExcludeVoyage": [], "PromoCode": [],
"AdditionalPromoCodes": []}}
headers = {"Host": "www.pocruises.com.au",
"User-Agent": "Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Firefox/52.0",
"Accept": "application/json, text/plain, */*", "Accept-Language": "en-US,en;q=0.5",
"Content-Type": "application/json;charset=utf-8",
"Cookie": "ASP.NET_SessionId=zm50ahxa4uoiwcuowgkeizcn; "
"SC_ANALYTICS_GLOBAL_COOKIE=f459da6904ee4cea8a809455f37b09c5|False; "
"optimizelyEndUserId=oeu1480435687102r0.9194470051495017; "
"optimizelySegments=%7B%223906442756%22%3A%22none%22%2C%223911484226%22%3A%22ff%22%2C"
"%223917223299%22%3A%22direct%22%2C%223920524266%22%3A%22false%22%2C%224924982297%22%3A%22true"
"%22%7D; optimizelyBuckets=%7B%227883701652%22%3A%227867701359%22%7D; "
"_ga=GA1.3.1128287966.1480435688; AdBlockDetected=false; "
"_msuuid_29439mm27589=D0670AB5-C96A-4706-A563-9F29FCA3D9D2; "
"gwcc=%7B%22fallback%22%3A%221300159454%22%2C%22clabel%22%3A%22ykseCIGywFkQhoDuxQM%22%2C"
"%22backoff%22%3A86400%2C%22backoff_expires%22%3A1480522087%7D; gaFindACruise=; _gat=1; "
"optimizelyPendingLogEvents=%5B%5D",
"Connection": "keep-alive"}
session.headers.update(headers)
page = session.post(url, headers=headers, data=data)
cruise_data = page.json()
print(cruise_data)
session.headers.update(headers)
page = session.post(url, headers=headers, data=data)
cruise_data = page.json()
但是現在我收到此錯誤:
Traceback (most recent call last):
File "/home/fixxxer/PycharmProjects/POCruses/main.py", line 31, in <module>
cruise_data = page.json()
File "/usr/lib/python3.5/site-packages/requests/models.py", line 841, in json
return complexjson.loads(self.text, **kwargs)
File "/usr/lib/python3.5/json/__init__.py", line 319, in loads
return _default_decoder.decode(s)
File "/usr/lib/python3.5/json/decoder.py", line 339, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/lib/python3.5/json/decoder.py", line 357, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
我想我做錯了body
部位。 如何在請求中添加此正文? 編輯:如果我刪除data
參數,請求工作正常,但信息不是我所需要的。如果我用瀏覽器打開頁面並檢查發送到瀏覽器的響應,我會得到: ScreenShot即使沒有,我也會得到不同body
。 這就是我知道該body
至關重要的信息。
它接縫該頁面期望數據為JSON
因此您需要
page = session.post(url, json=data)
順便說一句:
服務器總是向新客戶端分配新cookie,尤其是cookie行ASP.NET_SessionId
這樣更好的GET
主頁可以在執行其他請求之前獲取新cookie。
在session.headers.update(headers)
您不必在get()
/ post()
使用headers=headers
。 如果某些請求需要一些額外的標頭,則可能只需要更改一些標頭即可。
如果您使用json=
那么它將自動添加標頭"Content-Type": "application/json;charset=utf-8"
完整的代碼如下所示:
import requests
# -- create session ---
session = requests.session()
headers = {
"User-Agent": "Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Firefox/52.0",
"Accept": "application/json, text/plain, */*",
"Accept-Language": "en-US,en;q=0.5",
"Connection": "keep-alive"
}
# set headers for all requests
session.headers.update(headers)
# --- get cookies ---
url = "https://www.pocruises.com.au/"
page = session.get(url)
# --- search ---
data = {
"searchParameters": {
"p": [],
"c": [],
"d": [],
"s": [],
"ms": [],
"adv": [],
"sort": "dpa",
"page": 5
},
"renderingParameters": {
"DefaultSortOption": "dpa",
"LargeScreenFlag": "true",
"NewModelIsLoading": "false",
"PagingAnchor": "",
"ViewStyleSelectorVisible": "true",
"FilterBarVisible": "true",
"NumberOfResultsAndSortByPanelVisible": "true",
"DefaultResultsView": "Grid",
"MaxNumberOfResults": 0,
"PaginationEnabled": "true",
"KeepPageState": "true",
"PageSize": 9,
"DefaultResultsGrouping": "Itinerary",
"Duration": [],
"CruiseItinerary": [],
"Voyage": [],
"ExcludeVoyage": [],
"PromoCode": [],
"AdditionalPromoCodes": []
}
}
url = "https://www.pocruises.com.au/sc_ignore/b2c/cruiseresults/searchresults"
# `json=` add `"Content-Type": "application/json;charset=utf-8"`
page = session.post(url, json=data)
print(page.text)
cruise_data = page.json()
print(cruise_data)
聲明:本站的技術帖子網頁,遵循CC BY-SA 4.0協議,如果您需要轉載,請注明本站網址或者原文地址。任何問題請咨詢:yoyou2525@163.com.