[英]Sent different tuples from 1 spout to different bolt in Apache Storm
[英]storm redis spout tuples lost with no exception
我已經建立了風暴拓撲(1個工作程序),其中spout(在Java中)使redis的事件從Redis出隊(使用blpop)並轉移到螺栓。 但是一個觀察結果是,當隊列超過200萬且在風暴靈氣/主管/動物園管理員/工人日志中未發現任何警告/異常時,螺栓上沒有發生某些事件(clojure,6噴口螺紋,50螺栓螺紋)。
在本地,此方案不復制虛擬數據。 在群集中沒有看到網絡延遲/數據包丟失。 平均處理延遲為100毫秒。 如何找到將其修復的原因。
(ns event-processor
(:import [backtype.storm StormSubmitter LocalCluster]
java.util.UUID
storm_jedis.RedisQueueSpout
)
(:use [backtype.storm clojure config])
(:require [clojure.tools.logging :as log])
(:require [clj-redis.client :as redis])
(:import (redis.clients.jedis Jedis JedisPool JedisPoolConfig))
(:gen-class))
(defmacro process-event [tuple]
(log/info "processing event")
)
(defbolt execute-ls-closure ["word"] {:prepare true}
[conf context collector]
(let [counts (atom {})]
(bolt
(execute [tuple]
(let [
timestart (. System currentTimeMillis)
tuple-message (.get (get tuple "message") 0)
string-to-emit (process-event tuple)
]
(emit-bolt! collector [string-to-emit] :anchor tuple)
(ack! collector tuple)
)))))
(defn mk-topology []
(topology
;{"1" (spout-spec sentence-spout)
{"1" (spout-spec redis-spout :p 6)
}
{"3" (bolt-spec {"1" :shuffle }
execute-ls-closure
:p 50)
}))
(defn run-local! []
(let [cluster (LocalCluster.)]
(.submitTopology cluster "word-count" {TOPOLOGY-DEBUG true} (mk-topology))
(Thread/sleep 10000)
(.shutdown cluster)
))
(defn submit-topology! [name]
(StormSubmitter/submitTopology
name
{TOPOLOGY-DEBUG true
TOPOLOGY-WORKERS 1}
(mk-topology)))
(defn -main
([]
(run-local!))
([name]
(submit-topology! name)))
如果它不會使您的拓撲速度減慢太多,則可以使用Config.setDebug(true)
https://github.com/apache/storm/blob/f2ced23fa4e3f699558663baef4ee582ee148fa2/storm-client/src/jvm/org/ apache / storm / Config.java#L1763 。
否則,我會嘗試向您的螺栓添加一些調試日志記錄,並為Redis spout啟用日志記錄,以判斷元組是否由於Storm或Redis集成而丟失。
我還注意到您使用的是舊版Storm。 您可以嘗試升級。
聲明:本站的技術帖子網頁,遵循CC BY-SA 4.0協議,如果您需要轉載,請注明本站網址或者原文地址。任何問題請咨詢:yoyou2525@163.com.