我需要获取新闻文章数据。 我正在使用来自python的请求/获取，但出现此错误：403禁止
[英]I need to get news article data. I'm using request/get from python but I got this error: 403 forbidden
问题描述
Here is the code: 
这是代码：
from requests import get
from bs4 import BeautifulSoup
headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36'}
url = 'https://business.inquirer.net/category/latest-stories/page/10'
response = get(url)
print(response.text[:500])
html_soup = BeautifulSoup(response.text, 'html.parser')
type(html_soup)
And this is the result i got: 这就是我得到的结果：
<html>
<head><title>403 Forbidden</title></head>
<body bgcolor="white">
<center><h1>403 Forbidden</h1></center>
<hr><center>nginx</center>
</body>
</html>
I have read that putting a header will solve the error but I tried putting the header which i copied from the devtool when i inspected the site but it doesn't solve my problem please help me 我已经读到放置标头可以解决该错误，但是我尝试放置在检查站点时从devtool复制的标头，但不能解决我的问题，请帮助我
4 个解决方案
解决方案1
1 2019-02-26 08:02:13
You don't use headers variable anywhere so, you don't pass it with a request. 您不会在任何地方使用标头变量，因此，您不会在请求中传递它。 You can do that with code like this: 您可以使用以下代码执行此操作：
from bs4 import BeautifulSoup
from urllib.request import Request, urlopen
siteurl = "https://business.inquirer.net/category/latest-stories/page/10"
hdr = {'User-Agent': 'Mozilla/5.0'}
req = Request(siteurl,headers=hdr)
page = urlopen(req)
soup = BeautifulSoup(page)
print(soup)
解决方案2
0 2019-02-26 08:05:03
When try to scrap data from this site with using BeautifulSoap site doesn't display their data. 当尝试使用BeautifulSoap网站从该网站刮取数据时，不会显示其数据。
When you try : 当您尝试：
from bs4 import BeautifulSoup
from urllib import urlopen
url = "https://business.inquirer.net/category/latest-stories/page/10"
open_page = urlopen(url)
source = BeautifulSoup(open_page,"html.parser")
print source
You will see a line like : 您将看到如下一行：
<p>The owner of this website (business.inquirer.net) has banned your access based on your browser's signature (4af0dedd3eebcb40-ua48).</p>
So dont attent to do it with using BeautifulSoap . 因此，不要试图使用BeautifulSoap来做到这一点。 Using Selenium is more easier. 使用Selenium更容易。
from selenium import webdriver
options = webdriver.ChromeOptions()
driver=webdriver.Chrome(chrome_options=options, executable_path=r'your driver path')
driver.get('https://business.inquirer.net/category/latest-stories/page/10')
x = driver.find_elements_by_css_selector("div[id='ch-ls-head']")
for a in x:
print a.text
driver.close()
OUTPUT : 输出：
TAXATION
DOF clarifies: Rice tariffication law takes effect on March 5
FEBRUARY 19, 2019 BY: BEN O. DE VERA
BANKS
HSBC reports net profit at $12.6B in 2018
FEBRUARY 19, 2019
CURRENCIES
Asian shares gain on hopes for progress on China-US trade
FEBRUARY 19, 2019
ECONOMY
Amro sees higher PH growth in 2019 on easing inflation, infra boost
FEBRUARY 19, 2019 BY: BEN O. DE VERA
TELECOMMUNICATIONS
Poe to DICT: Stop ‘dilly-dallying’ over 3rd telco project
FEBRUARY 19, 2019 BY: CHRISTIA MARIE RAMOS
SOCIAL SECURITY
SSS contribution collections grow by P22.19B in 2018
FEBRUARY 18, 2019 BY: CHRISTIA MARIE RAMOS
STOCKS
World stocks mixed ahead of further China-US trade talks
FEBRUARY 18, 2019
TRADE
Rice tariffication starts on March 3
FEBRUARY 18, 2019 BY: BEN O. DE VERA
AGRICULTURE/AGRIBUSINESS
NFA-Bohol workers wear black to mourn ‘death of the rice industry’
FEBRUARY 18, 2019 BY: LEO UDTOHAN
BONDS
Treasury: RTBs to be sold to individual investors online in Q1
FEBRUARY 18, 2019 BY: BEN O. DE VERA
解决方案3
0 2019-02-26 08:07:09
Simply worked for me 只是为我工作
from bs4 import BeautifulSoup
import urllib.request
response = urllib.request.urlopen('https://business.inquirer.net/category/latest-stories/page/10')
html = response.read()
soup = BeautifulSoup(html,"html5lib")
text = soup.get_text(strip=True)
print (text)
解决方案4
0 已采纳 2019-02-26 08:27:51
Try including a header, many sites block requests without headers: 尝试包含标头，许多站点会阻止没有标头的请求：
r = requests.get(url, headers=...)
Check the requests docs for more info: http://docs.python-requests.org/en/master/user/quickstart/ 检查请求文档以获取更多信息： http : //docs.python-requests.org/en/master/user/quickstart/
声明:本站的技术帖子网页，遵循CC BY-SA 4.0协议，如果您需要转载，请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.