[英]Should I be using EM::Synchrony::Multi or EM::Synchrony::FiberIterator with Goliath?
也許這是錯誤的方法,但是我正在嘗試使用EM::Synchrony::Multi
或EM::Synchrony::FiberIterator
並行 化Goliath中的em-hiredis放置和查找。 但是,我似乎無法訪問config中初始化的基本值。 我不斷收到method_missing
錯誤。
這是我要嘗試的基本簡化版本:
/lib/config/try.rb
config['redisUri'] = 'redis://localhost:6379/0'
config['redis_db'] ||= EM::Hiredis.connect
config['user_agent'] = "MyCrawler Mozilla/5.0 Compat etc."
這是基本的Goliath設置
/try.rb
require "goliath"
require "em-hiredis"
require "em-synchrony/fiber_iterator"
require "em-synchrony/em-hiredis"
require "em-synchrony/em-multi"
class Try < Goliath::API
use Goliath::Rack::Params
use Goliath::Rack::DefaultMimeType
def response(env)
case env['REQUEST_PATH']
when "/start" then
start_crawl()
body = "STARTING"
[200, {}, body]
end
end
def start_crawl
urls = ["http://www.example.com/",
"http://www.example.com/photos/",
"http://www.example.com/video/",
]
EM::Synchrony::FiberIterator.new(urls, 3).each do |url|
p "#{user_agent}"
redis_db.sadd 'test_queue', url
end
# multi = EM::Synchrony::Multi.new
# urls.each_with_index do |url, index|
# p "#{user_agent}"
# multi.add index, redis_db.sadd('test_queue', url)
# end
end
end
但是,在Goliath不知道是在配置中初始化的user_agent
或redis_db
地方,我總是收到錯誤redis_db
。
[936:INFO] 2012-09-21 23:47:10 :: Starting server on 0.0.0.0:9000 in development mode. Watch out for stones.
/Users/ewu/.rvm/gems/ruby-1.9.3-p194@crawler/gems/goliath-1.0.0/lib/goliath/api.rb:143:in `method_missing': undefined local variable or method `user_agent' for #<Try:0x007ff5a431c4e0 @opts={}> (NameError)
from ./lib/try.rb:27:in `block in start_crawl'
from /Users/ewu/.rvm/gems/ruby-1.9.3-p194@crawler/gems/em-synchrony-1.0.2/lib/em-synchrony/fiber_iterator.rb:10:in `call'
from /Users/ewu/.rvm/gems/ruby-1.9.3-p194@crawler/gems/em-synchrony-1.0.2/lib/em-synchrony/fiber_iterator.rb:10:in `block (2 levels) in each'
...
...
...
理想情況下,我可以使FiberIterator
工作,因為我還有其他條件要檢查:
EM::Synchrony::FiberIterator.new(urls, 3).each do |new_url}
is_member = redis_db.sismember('crawled_urls', new_url)
is_member += redis_db.sismember('queued_urls', new_url)
if is_member == 0
redis_db.lpush 'crawl_queue', new_url
redis_db.sadd 'queued_urls', new_url
end
end
我不認為您的配置文件正在加載。 try.rb的名稱必須與config目錄中的robojin.rb文件的名稱匹配。
聲明:本站的技術帖子網頁,遵循CC BY-SA 4.0協議,如果您需要轉載,請注明本站網址或者原文地址。任何問題請咨詢:yoyou2525@163.com.