When using Nokogiri with RoR, I'm not getting all the information from the webpage, not sure why?

Question

This is my Scraper Controller

class ScraperController < ApplicationController

    def getinformation
        require 'open-uri'
        require 'nokogiri'
        @information = Nokogiri::HTML(open('https://ibotta.com/rebates'))
    end

end

And this is a webpage of the information that I'm getting from nokogiri https://rails-tutorial2-chriscma.c9users.io/scraper/getinformation

I'm not getting any of the product names, and I'm not sure why?

Answer 1

The page you're trying to scrape is dynamically generated by executing javascript, so you won't be able to use nokogiri to download the content. It looks like the offers on the page are loaded from https://ibotta.com/web_v1/offers.json , but this isn't accessible directly. Therefore, I think you'll need to use something which can execute javascript like selenium / phantomjs / chrome headless / watir etc in order to load the page.