如何在Rails中加速抓取

Question

This code works fine locally, but on Heroku it takes more than 30 seconds due to a request timeout:

if @url
  @arr = Array.new
  begin
    doc = Nokogiri::HTML(open(@url))
    doc.css(".new-cars-results-box").each do |item|
      hash = Hash.new

      type = item.at_css(".new-car-name").text
      link = "http://uae.yallamotor.com"+item.at_css(".new-car-name")[:href]
      @arr << [link,type]
    end
  rescue
  end

end

How can I speed this up?

Answer 1

you query the DOM 2 times any result box when you can just query once for all the '.new-car-name', and you creating for each one useless hash

Try this:

if @url
    @arr = Array.new
  begin
    doc = Nokogiri::HTML(open(@url))
    url_prefix = 'http://uae.yallamotor.com'
    doc.css(".new-cars-results-box > .new-car-name").each do |item|
      type = item.text
      link = url_prefix + item[:href]
      @arr << [link,type]
    end
  rescue
  end
end

also try to replace this line

doc.css(".new-cars-results-box > .new-car-name").each do |item|

with this:

doc.css(".new-car-name").each do |item|