![](/img/trans.png)
[英]Can't scrape a certain field from a webpage using requests even when that very field is available in page source
[英]Unable to scrape csrf token (available in page source) from a webpage using requests
我正在嘗試從網站上抓取 csrf 令牌。 但是,即使頁面源中的令牌可用,我創建的腳本也會慘遭失敗。 這是網站 url 。
我試過:
import requests
from bs4 import BeautifulSoup
url = 'https://fanniemae.mbs-securities.com/fannie/search?issrSpclSecuType=Super&status=Active'
with requests.Session() as s:
s.headers['User-Agent'] = 'Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.104 Safari/537.36'
r = s.get(url)
soup = BeautifulSoup(r.text,"lxml")
csrf = soup.select_one("[name='_csrf']").get("content")
print(csrf)
如何使用請求從該站點刮取 csrf 令牌?
這里的技巧是在標頭中包含Accept
鍵和值以獲得所需的響應。 這就是我使用請求從該站點獲取表格內容的方式:
import requests
from bs4 import BeautifulSoup
url = 'https://fanniemae.mbs-securities.com/fannie/search?issrSpclSecuType=Super&status=Active'
link = 'https://fanniemae.mbs-securities.com/api/search/fannie'
params = {
'issrSpclSecuType': 'Super',
'status': 'Active',
'page': 1,
'max_results': 100,
'sortField': 'cusip',
'sortAsc': 'true'
}
with requests.Session() as s:
s.headers['User-Agent'] = 'Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.104 Safari/537.36'
s.headers['Accept'] = 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9'
r = s.get(url)
soup = BeautifulSoup(r.text,"lxml")
s.headers['x-csrf-token'] = soup.select_one("[name='_csrf']")["content"]
s.headers['referer'] = 'https://fanniemae.mbs-securities.com/fannie/search?issrSpclSecuType=Super&status=Active'
res = s.get(link,params=params)
for item in res.json():
print(item['cusip'])
聲明:本站的技術帖子網頁,遵循CC BY-SA 4.0協議,如果您需要轉載,請注明本站網址或者原文地址。任何問題請咨詢:yoyou2525@163.com.