[英]Scraping Multiple Page using Python and BeautifulSoup - Website url does not work
[英]Scraping the second page of a website in Python does not work
该页面具有非常异步的性质,有XHR请求构成搜索结果,并使用requests
在代码中模拟它们。 示例代码是您的起点:
from bs4 import BeautifulSoup
import requests
url = 'http://www.amazon.com/Best-Sellers-Books-Architecture/zgbs/books/173508/#2'
ajax_url = "http://www.amazon.com/Best-Sellers-Books-Architecture/zgbs/books/173508/ref=zg_bs_173508_pg_2"
def get_books(data):
soup = BeautifulSoup(data)
for title in soup.select("div.zg_itemImmersion div.zg_title a"):
print title.get_text(strip=True)
with requests.Session() as session:
session.get(url)
session.headers = {
'User-Agent': 'Mozilla/5.0 (Linux; U; Android 4.0.3; ko-kr; LG-L160L Build/IML74K) AppleWebkit/534.30 (KHTML, like Gecko) Version/4.0 Mobile Safari/534.30',
'X-Requested-With': 'XMLHttpRequest'
}
for page in range(1, 10):
print "Page #%d" % page
params = {
"_encoding": "UTF8",
"pg": str(page),
"ajax": "1"
}
response = session.get(ajax_url, params=params)
get_books(response.content)
params["isAboveTheFold"] = "0"
response = session.get(ajax_url, params=params)
get_books(response.content)
并且不要忘记成为一名良好的网络爬虫公民并遵守使用条款。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.