[英]How to get dynamic page content with capybara
嘗試從元素中獲取文本:
<div class="points-text" data-reactid=".2765swfgy68.1.2.2.1:$sony-xperia-z1-compact.2.0.4.0.0.1">29,367 points</div>
我猜網站使用Reactjs,即使使用Poltergeist驅動程序,水豚也無法獲得內容。 有什么解決方法嗎?
這是我的代碼:
require 'rubygems'
require 'capybara'
require 'capybara/poltergeist'
Capybara.default_driver = :poltergeist
Capybara.register_driver :poltergeist do |app|
Capybara::Poltergeist::Driver.new(app, {js_errors: false})
end
class WebScraper
include Capybara::DSL
def get_page_data(url)
visit(url)
doc = Nokogiri::HTML(page.html)
p doc.css('.points-text')
end
end
scraper = WebScraper.new
puts scraper.get_page_data('http://versus.com/en/sony-xperia-z1-compact')
如果您已經使用Capybara進行了訪問,則無需使用Nokogiri解析html。
def get_page_data(url)
visit(url)
p find(:css, '.points-text').text
end
將使用class points-text類在元素中顯示可見文本
此代碼適用於selenium和webkit,但不適用於poltergeist驅動程序。 從屏幕截圖中,我可以看到該網站上的javascript甚至沒有執行。
require 'capybara'
require 'capybara-webkit'
require 'capybara/poltergeist'
Capybara.default_driver = :poltergeist
# Capybara.default_driver = :selenium
# Capybara.default_driver = :webkit
Capybara.register_driver :poltergeist do |app|
Capybara::Poltergeist::Driver.new(app, {js_errors: false})
end
Capybara::Webkit.configure do |config|
config.allow_unknown_urls
end
class WebScraper
include Capybara::DSL
def get_page_data(url)
visit(url)
max_same_times = 3
same_times = 0
old_points = nil
20.times do |i|
doc = Nokogiri::HTML(page.html)
# p doc.css('.points-text')
points = doc.css('.points-text')
# p [same_times, max_same_times]
if same_times == max_same_times
break
end
if points.length > 0
points = points[0].text
# p [old_points, points]
if old_points == points
same_times += 1
else
same_times = 0
end
old_points = points
end
page.save_screenshot("#{i}.png")
end
old_points
end
end
scraper = WebScraper.new
puts scraper.get_page_data('http://versus.com/en/sony-xperia-z1-compact')
聲明:本站的技術帖子網頁,遵循CC BY-SA 4.0協議,如果您需要轉載,請注明本站網址或者原文地址。任何問題請咨詢:yoyou2525@163.com.