[英]Element inside <span> tag not found back with Beautiful Soup
I have tried using Beautiful Soup with Python to crawl data from a website.我曾尝试使用 Beautiful Soup 和 Python 从网站抓取数据。
When I inspect the website itself I see the following:当我检查网站本身时,我看到以下内容:
"<span id="test"> 567 </span>"
but when I use Beautiful soup, all I see is:但是当我使用美丽的汤时,我看到的是:
"<span id="test"></span>"
It is as if the number has been hidden for security and protection purposes but how can I get this info from parsing?就好像出于安全和保护目的,该号码已被隐藏,但我如何从解析中获取此信息? I think it is a dynamic JS element but I don't know for sure and how to access it.
我认为它是一个动态的 JS 元素,但我不确定以及如何访问它。
Here is an example code to implement it:这是实现它的示例代码:
import scrapy
from selenium import webdriver
from bs4 import BeautifulSoup as bs
class SomeSpider(scrapy.Spider):
name = "some"
allowed_domains = ["yourdomain.com"]
def __init__(self, *a, **kw):
super(SomeSpider, self).__init__(*a, **kw)
self.start_urls = ['http://www.yoururl.com',]
firefox_profile = webdriver.FirefoxProfile()
self.driver = webdriver.Firefox(firefox_profile=firefox_profile)
def parse(self, response):
self.driver.get(response.url)
page = TextResponse(response.url, body=self.driver.page_source, encoding='utf-8')
#HERE whatever you want do do eg. text = page.xpath('//span[@id="test"]').extract_first()
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.