Jsoup1.7.2无法使用Google App Engine 1.7.5

Question

I'm working on a project where my application is hosted on Google App Engine and uses Jsoup html parsing library. In my application I'm using TaskQueues with the default queueu, the only task in that queue is to connect to a URL and start parsing the page. No errors or warnings appear in the log files, it just exits as it doesn't see the line of Jsoup parsing the document. Here is a snippet of my code:

log.warning("Before connection");
Document doc = Jsoup.connect("http://en.wikipedia.org/").get();
log.warning("After connection");

The TaskQueue works fine, I've tested and I'm 100% sure that there is no problem with it. I've tried to manually connect to the webpage and download it then pass it to Jsoup and start parsing it there, the connection worked fine and the webpage was downloaded successfully, but yet Jsoup wasn't able to do anything. My biggest problem is that there no errors and no warnings in the log file; so I don't know what it is going on.

Answer 1

App engine restricts a number of of classes, I would assume that either the Jsoup.connect(url) or the .parse method depends on one of those restricted classes and throws an exception.

To eliminate the chance of the Jsoup.connect causing issues then I'd suggest you use App Engine URL Fetch to get a String of the page at the URL and then use:

Document doc = Jsoup.parse(htmlString);

However if there are issues with the parse then you really need to get errors/logging working and there's not a lot of info here yet to be able to suggest something. Try to put the problematic code in a try-catch block and see if you can catch an exception.

Additionally try a later version of GAE SDK (1.8.1 is the current one). I've previously had a conflict with GAE SDK's checkRestricted method that interfered with Jsoup, so that may be the case with 1.7.5.

Answer 2

The problem was that I was using Jsoup 1.7.2 which is clearly not very compatible with Google App Engine. I switched back to Jsoup 1.7.1 and the problem was fixed.