[英]Ruby on Rails store code to execute in database
I have a model Shop
in my database which sum up everything I want to know about a shop ( name
, url
, price
) 我的数据库中有一个模型Shop
,该模型汇总了我想了解的有关商店的所有信息( name
, url
, price
)
I would like some advise on what is the best way to deal with my situation. 我想就什么是解决我的情况的最佳方法提供一些建议。 Basically, what I want is to scrap website (which don't have API) to get the price displayed on the page. 基本上,我想要的是取消网站(没有API)以获取页面上显示的价格。
For example, let's say that I want to get the price from this page , every time a user visit the page X and the price from this page every time he comes to the page Y and so on with 1000+ pages. 例如,让我们说,我想从这个得到的价格页面 ,在用户每次访问页面X价格从此时页面每次他来到了Y形等与1000+的网页的时间。
The Shops
in my database would be : 我数据库中的Shops
为:
Shop #1 : {:name => "Tshirt", :url => "XXXXX", :price => "PRICE_FROM_THE_URL"}
Shop #2 : {:name => "Veste", :url => "XXXXX", :price => "PRICE_FROM_THE_URL"}
I see two options to update the price every time a user ask it : 每当用户提出要求时,我都会看到两个选项来更新价格:
code
and do price = eval(Shop.code)
像这样,添加列code
并执行price = eval(Shop.code)
self.id
类似于根据self.id
的开关盒 I tried both options. 我尝试了两种选择。 Both works as expected, but my concerns is that option #1 looks like the "ugly" one but easier to maintain, while option #2 is not well fitted if you have 1000+ shops to record and every one has a different scrapping method. 两者都能按预期工作,但我担心的是,选项#1看起来像“丑陋”的选项,但更易于维护,而如果您要记录1000多家商店,而每个选项都有不同的报废方法,则选项#2不太适合。 I will end up with thousands of code line and it will become impossible to understand. 我最终将获得成千上万的代码行,并且它将变得无法理解。
Nokogiri lets you scrape content by css selectors . Nokogiri允许您通过CSS选择器抓取内容。 Knowing that, considen following design guidelines: 知道这一点,请考虑以下设计准则:
Create a model with selectors for specified shop, name it: ShopSelectorGroup
(it can also be created as ActiveRecord model, to store selectors in database). 使用指定商店的选择器创建一个模型,命名为: ShopSelectorGroup
(也可以将其创建为ActiveRecord模型,以将选择器存储在数据库中)。
class ShopSelectorGroup attr_accessor: :price_selector, :other_selector, :shop_name end
Then create a class Scraper
which will be configured by injecting an instance of ShopSelectorGroup
class. 然后创建一个Scraper
类,该类将通过注入ShopSelectorGroup
类的实例进行配置。
require 'nokogiri' require 'open-uri' class Scraper def initialize(selector_group) @selector_group = selector_group end def scrape(url) open(url) do |content| Nokogiri::HTML(content).css(@selector_group.price_selector).each do |data| yield data end end end end
Use it as follows: 如下使用它:
selector_group = ShopSelectorGroup.new selector_group.price_selector = 'span.price' # or when used as ActiveRecord model # selector_group = ShopSelectorGroup.findByShopName('MyShop') scraper = Scraper.new(selector_group) scraper.scrape(url) do |data| p data # or persist data in database end
Hope this helps! 希望这可以帮助!
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.