获取网页并保存在数据库中？

Question

如何获取HTML页面并将其保存到JAVA中的数据库？有没有简单的方法可以做到这一点？

Answer 1

Receiving a file over http is pretty easy using the URL class:

String rawHtml = IOUtils.toString(new URL("http://yahoo.com").openStream());

IOUtils is taken from org.apache.commons.io, the toString method reads the whole input stream into one String. Unfortunately by using java.net.URL you cannot control anything (cookies, header information, ..) besides the website's address :-/ Personally, I use this approach wherever I can since the HttpClient's API is too complex (too many LOC) to simply retrieve the source code of a website.

Answer 2

Not sure about your exact requirements.

For something simple you can use HttpClient

For something more complex, you can use Nutch It does crawling, indexing and searching as well.