I am trying to get some data from a wine website.
But I can not assess the data and there have a usage violation message.
The url : https://www.wine-searcher.com/find/drc/2013
The prettify() result something like that:
<div id="bodycontainer">
<div class="colmask contentparent">
<div id="colheader">
<div class="colmask articlecontainer">
<div class="colmidtemp3">
<div class="collefttemp3">
<div class="col1wraptemp3">
<div class="col1temp3">
<div>
<h1 style="margin:50px 0 0">
Usage Violation
</h1>
<div style="margin-bottom:50px;padding:50px 10px;background-color:#FFFACD">
<h2 style="font-size:1.4em">
Blocked
</h2>
<p style="font-size:1.2em">
The IP Address [xx.xxx.xxx.xx] you are using has been used in violation of Wine-Searcher's usage guidelines.
<b>
If you think you have received this message in error restart your web browser and retry accessing wine-searcher.com.
</b>
</p>
<p style="font-size:1.2em">
To re-gain access to Wine-Searcher please
<a href="mailto:wsexcessiveuse@wine-searcher.com?subject=Blocked IP=1 ID=PVBXC7PJCM80025">
Contact Us
</a>
.
</p>
</div>
</div>
</div>
</div>
Is there any possible ways to get the data from the url? Thank you so much.
My coding here :
# -*- coding: utf-8 -*-
import bs4
import re
import requests
import sys
from bs4 import BeautifulSoup
name = "Wine.txt"
k = open(name, "w", encoding='utf-8')
Stat_url = "https://www.wine-searcher.com/find/drc/2012"
page = requests.get(Stat_url)
soup = bs4.BeautifulSoup(page.text,'lxml')
k.write(soup.prettify())
It looks like they added some protection to their page to prevent that, what you try to do ;) They sell access via API https://www.wine-searcher.com/api.lml and allow in a trial 100 calls in 5 days. Maybe this is enough for you?
I would guess that BeautifulSoup is trying to many requests in to few time? (Maybe limit it to one per 10 seconds and let it run overnight?) Can the agent-id in BS be changed to something more common like a regular browser agent-id?
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.