Un-anchoring tuples in Storm

Question

I have a mostly-linear Storm topology. There is a bolt in the middle that's doing work that may take time. Some tuples might be able to get processed immediately, some others the bolt will stash somewhere for later. (It's trying not to hammer external resources too much.)

I'm using BaseBasicBolt -derived bolts up until this one, which is a BaseRichBolt subclass. So the tuples it receives are correctly anchored to the whole tree up to the spout.

Can I ack these tuples and emit brand new, unanchored tuples from my bolt? If the bolts after this one (all BaseBasicBolts again) emit anchored tuples, will replaying work correctly and how?

Answer 1

Yes, you can emit unachored tuples. However, there is no way to recover those tuples.

If you special bolt acks a tuple, the spout will drop the source tuple that are required to recompute this input tuple. Thus, you cannot recover this input tuple any more and therefore you can't recompute the unanchored result tuple.

IMHO, the only way to get fault-tolerance in you case, is to increase message timeout such that the long processing tuples in your special bolt do not get timed-out. Your emitted tuples need to get anchored regularly to the stashed input tuples and those input tuplse can be acked when youe removed them from stashing space.