[英]Unable to scrape csrf token (available in page source) from a webpage using requests
I'm trying to scrape csrf token from a website.我正在尝试从网站上抓取 csrf 令牌。 However, the script that I created fails miserably even when the very token is available in page source.
但是,即使页面源中的令牌可用,我创建的脚本也会惨遭失败。 This is the site url .
这是网站 url 。
I've tried with:我试过:
import requests
from bs4 import BeautifulSoup
url = 'https://fanniemae.mbs-securities.com/fannie/search?issrSpclSecuType=Super&status=Active'
with requests.Session() as s:
s.headers['User-Agent'] = 'Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.104 Safari/537.36'
r = s.get(url)
soup = BeautifulSoup(r.text,"lxml")
csrf = soup.select_one("[name='_csrf']").get("content")
print(csrf)
How can I scrape csrf token from that site using requests?
如何使用请求从该站点刮取 csrf 令牌?
The trick here is to include Accept
key and value within headers to get the required response.这里的技巧是在标头中包含
Accept
键和值以获得所需的响应。 This is how I fetch tabular content from that site using requests:这就是我使用请求从该站点获取表格内容的方式:
import requests
from bs4 import BeautifulSoup
url = 'https://fanniemae.mbs-securities.com/fannie/search?issrSpclSecuType=Super&status=Active'
link = 'https://fanniemae.mbs-securities.com/api/search/fannie'
params = {
'issrSpclSecuType': 'Super',
'status': 'Active',
'page': 1,
'max_results': 100,
'sortField': 'cusip',
'sortAsc': 'true'
}
with requests.Session() as s:
s.headers['User-Agent'] = 'Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.104 Safari/537.36'
s.headers['Accept'] = 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9'
r = s.get(url)
soup = BeautifulSoup(r.text,"lxml")
s.headers['x-csrf-token'] = soup.select_one("[name='_csrf']")["content"]
s.headers['referer'] = 'https://fanniemae.mbs-securities.com/fannie/search?issrSpclSecuType=Super&status=Active'
res = s.get(link,params=params)
for item in res.json():
print(item['cusip'])
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.