Jsoup 无法获取带有嵌套标签的外部 HTML

Question

after I connect to an Instagram page using Jsoup, I want to extract the whole outer html from a Tag. Somehow when I inspect the page and copy the outer html from the tag I get loads of lines, while I get only few using Jsoup (somehow the html of the nested tags gets ignored) Any help would be appreciated how to get the whole html!

Code:

Document doc = Jsoup.connect("https://www.instagram.com/myUsername").get();

Element link = doc.selectFirst("span");
String linkOuter = link.outerHtml();
System.out.println(linkOuter);

Output :

<span id="react-root">
  <svg width="50" height="50" viewbox="0 0 50 50" 
  style="position:absolute;top:50%;left:50%;margin:-25px 0 0 
  -25px;fill:#c7c7c7">
    <path d="M25 1c-6.52 0-7.34.03-9.9.14-2.55.12-4.3.53-5.82..." />
</svg></span>

Image of the structure:

EDIT: I wwant that the whole HTML of the span tag gets saved (I want the same result with HtmlUnit/Jsoup as when I right click on the tag click on edit html and then right click-> copy outer html!

Answer 1

Unfortunately, Instagram is a Web app built with javascript framework react. That means, that final HTML is not returned from the server, but rather it's generated by javascript on client side on a browser after the initial loading of the page.

To see HTML generated by react you'd need to evaluate javascript code, which is returned from the server. JSoup is simple HTML parser and can't evaluate js, so you'd have to use another library, like for example HtmlUnit.

For example:

WebClient webClient = new WebClient(BrowserVersion.CHROME);
webClient.getOptions().setJavaScriptEnabled(true); // enable javascript
webClient.getOptions().setThrowExceptionOnScriptError(false); //even if there is error in js continue
HtmlPage page = webClient.getPage(new URL("https://www.instagram.com/myUsername"));
webClient.waitForBackgroundJavaScript(5000); // important! wait when javascript finishes rendering

page.getElementById("react-root");