How to find out if a transaction is successfully committed in Apache Storm Trident

Question

I'm trying to get started with Storm Trident and have the topology setup and running with IOpaquePartitionedTridentSpout and backed by OpaqueMap .

However, I'm struggled to find out the way to let my spout/function know if a transaction is successfully committed or not. I don't see any ack or fail method as in regular Storm spout/bolt interface.

My use case is to only emit the tuple of a category when the previous one of the same category is processed and persisted (or failed). Because I will then use the processed data to update my next tuple of the category. Tuples from different categories can be process in parallel.

The stream is partitioned by category with partitionBy method.

Setting max_spout_pending to 1 eliminates the problem because Trident only commits 1 batch at a time. But that's not scalable. Setting to any value greater than 1 makes the tuples of the same category, if they are emitted in two consecutive batches, being processed before the previous transaction committed.

Or should I have one spout for each category and set the max_spout_pending to 1?

Thanks

Answer 1

I suggest you look into implementing an ITridentSpout . It has a success() method in the coordinator that's called when the batch is processed successfully. The same metadata from the coordinator is sent to all emitters so they'll need to divide the work up.