Apache Storm - Spout/Bolt crashing with high latency

Question

While using Storm 1.1.0 with my topology I faced the problem, that Storm reschedules or let the Bolts and Spouts crash, when the topology has a Bolt with high latency.

Now i created a LatencyTest-Topology for testing and playing around with this problem.

I have a Spout, which emit random values:

public void nextTuple() {
    outputCollector.emit(new Values(Math.random()));
}

And I have a Bolt, that recieves these values and sleeps for a specific time.

public void execute(Tuple tuple) {
    double input = tuple.getDouble(0);
    try {
        Thread.sleep(this.latencyMS);
    } catch (InterruptedException e) {
        e.printStackTrace();
    }
    outputCollector.ack(tuple);
}

So if i set latencyMS to 10, i can see, that Storm works "fine" (just 2000 acked tuples in the Bolt)for 3 Minutes. Then the Bolts latency goes up to 60-100ms instead of 10ms and Storm starts to "reassign" (Log in the Nimbus) the executors. Then all statistics in the UI are going to 0.

Image 1: No Crash and 2000 Tuples acked with high latency

Image 2: Crashed an no information in the UI

Since i am working with files in the Spout of my real topology, it is not acceptable to reopen these.

I played around with some timeout values in my storm.yaml and with the "config.setMaxSpoutPending(200);" option, but nothing seems to have any effect. I am using a 3 Node Zookeeper Cluster and a 5 Node Storm Cluster for this test.

Do you have any ideas how to solve or understand this? I need Storm to go on, even if the latency is very high.

Answer 1

MaxSpoutPending不是超时配置，您应该配置的是topology.message.timeout 。