apache storm wordcount: how to estimate time taken to process a file

Question

I'm developing WordCount in Java with Apache Storm; sentences are taken from a file and I want to estimate time taken by Storm to process the file.

My Question is how to know that Storm LocalCluster has finished processing all sentences of the file and how to estimate the time taken.

Answer 1

There is no clean solution for your problem and it is not really possible to estimate the time it takes.

If you enable fault-tolerance mechanism (ie, assign message IDs at spout, and anchor and ack in bolts), you can track the incoming acks in your spout. If no tuples are pending any more, ie, all acks are received, you know that everything got processed completely.