没有第三方库，如何完整解析HTML？

Question

I am puzzled with this question.

I can parse a HTML like below way.

package org.owls.parser.html;

import java.io.BufferedReader;
import java.io.InputStreamReader;
import java.net.HttpURLConnection;
import java.net.URL;

public class HTMLParser {
    public static String getHTTPStringsFromWeb(String urlStr) throws Exception {
        StringBuffer sb = new StringBuffer();
        URL url = new URL(urlStr);
        HttpURLConnection con = (HttpURLConnection) url.openConnection();

        BufferedReader br = null;
        if(con.getResponseCode() == HttpURLConnection.HTTP_OK)
        {
            br = new BufferedReader(new InputStreamReader(con.getInputStream()));
            String line = "";
            while((line = br.readLine()) != null){
                sb.append(line);
            }
            br.close();
        }
        return sb.toString();
    }
}

This code works well, but there is a problem. This code can not get dynamic data which made of ajax result.

So I want to get full page. Is it possible?

People talk about jsoup, but I want to know is there anyway to get this with native.

Thanks :D

Answer 1

There is an inherent problem in what you are trying to do, you need a web browser/environment to execute the ajax requests. reading them into a string and looking for url's is not enough, the functions may be doing something special with the data that you won't be able to support.

You will have to use something like phantomjs which can load and parse pages in a headless environment