How do I save the scraped data from Nokogiri to a Rails database?

Question

I want to save the scraped data to the database so that I can implement search and sorting functionality on it.

I tried creating a new Rake task and updating attributes but for that I need to run rake fetch-data every time the data is scraped.

app/controller

def show
  url = @scrapper.url 
  data = Nokogiri::HTML(open(url))
  @doc= data.css(".s-item-container")
end

app/views/show

 <% @doc.each do |item| %>
   <tr>
     <td><%= item.css(".s-access-title").text %></td>
     <td><%= item.css(".s-price").text %></td>
     <td><%= item.css("span+ .a-text-normal").text %></td>
   </tr>
 <% end %>

The data I'm getting

It didnt work. Is there any other way to achieve this?

Answer 1

1. Create a database rake db:create
2. Create 'Product' model rails g model Product title:string price:decimal rating:float
3. Create a rake task. Parse data and save it in the database.

 doc = Nokogiri::HTML(open(@scrapper.url )) items = doc.css(".s-item-container") items.each do |item| Product.create!( title: item.css(".s-access-title").text.strip, price: item.css(".s-price").text.to_d, rating: item.css("span+ .a-text-normal").text.to_f) end 

to prevent duplicates

items.each do |item|
    title = item.css(".s-access-title").text.strip
    product = Product.find_or_initialize(title: title)
    product.price = item.css(".s-price").text.to_d
    product.rating = item.css("span+ .a-text-normal").text.to_f
    product.save!
  end

4. Get data from Product model in your controller and show it in the view