程序停留在URL上，无法前进并且不会超时

Question

My program is a web crawler and its been stuck on a URL that apparently corresponds to a random Chinese site. For some reason its not throwing an exception and the connection is not timing out. I would have thought that these lines would prevent that.

static URLConnection in;
in = curURL.openConnection();
in.setConnectTimeout(2000);
pageSource = new StreamedSource(in);

I'm nearly positive this is the issue, any checks on the heap dump for memory leaks turned up nothing.

Answer 1

setConnectTimeout() only controls the timeout for establishing the connection. Once it has been started, it can last for a long time (basically until the server closes it). For instance you might be downloading a very large file over a slow link.

One solution would be to add a watchdog thread monitoring the connections and closing those which exceed some time limit.