Java - 下载网页源代码的最佳方式 html

Question

I'm writing a little crawler. What is the best way to download a web page's source html? I'm currently using little piece of code below but some times the result is just half of the page's source... I don't know what's the problem. Some people suggested that I should use Jsoup but using,get.html() function from Jsoup also returns half of the page's source if it's too long. Since I'm writing a crawler. it's very important that the method support unicode (UTF-8) and the efficiency is also very important. I wanted to know the best modern way to do it so I asked you guys since I'm new to Java. Thanks.

Code:

public static String downloadPage(String url)
    {
        try
        {
            URL pageURL = new URL(url);
            StringBuilder text = new StringBuilder();
            Scanner scanner = new Scanner(pageURL.openStream(), "utf-8");
            try {
                while (scanner.hasNextLine()){
                    text.append(scanner.nextLine() + NL);
                }
            }
            finally{
                scanner.close();
            }
            return text.toString();
        }
        catch(Exception ex)
        {
            return null;
        }
    }

Answer 1

I use commons-io String html = IOUtils.toString(url.openStream(), "utf-8");

Answer 2

Personally, I'm very pleased with the Apache HTTP library http://hc.apache.org/httpcomponents-client-ga/ . If you're writing a web crawler, which I am also, you may greatly appreciate the control it gives over things like cookies and client sharing and the like.