Removing HTML entities while preserving line breaks with JSoup

Question

I have been using JSoup to parse lyrics and it has been great until now, but have run into a problem.

I can use Node.html() to return the full HTML of the desired node, which retains line breaks as such:

Glóandi augu, silfurnátt
<br />Bl&oacute;&eth; alv&ouml;ru, starir &aacute;
<br />&Oacute;&eth;ur hundur er &iacute; v&iacute;gam&oacute;&eth;, &iacute; maga... m&eacute;r
<br />
<br />Kolni&eth;ur gref, kvik sem dreg h&eacute;r
<br />Kolni&eth;ur svart, hvergi bjart n&eacute;

But has the unfortunate side-effect, as you can see, of retaining HTML entities and tags.

However, if I use Node.text() , I can get a better looking result, free of tags and entities:

Glóandi augu, silfurnátt Blóð alvöru, starir á Óður hundur er í vígamóð, í maga... mér Kolniður gref, kvik sem dreg hér Kolniður svart,

Which has another unfortunate side-effect of removing the line breaks and compressing into a single line.

Simply replacing <br /> from the node before calling Node.text() yields the same result, and it seems that that method is compressing the text onto a single line in the method itself, ignoring newlines.

Is it possible to have the best of both worlds, and have tags and entities replaced correctly which preserving the line breaks, or is there another method or way of decoding entities and removing tags without having to replace them manually?

Answer 1

(disclaimer) I haven't used this API ... but a quick look at the docs suggests that you could visit each descendent node and dump out its text contents. Breaks could be inserted when special tags like <br> are encountered.

The TextNode.getWholeText() call also looks useful.

Answer 2

based on another answer from stackoverflow I added a few fixes and came with

    String text = Jsoup.parse(html.replaceAll("(?i)<br[^>]*>", "br2nl").replaceAll("\n", "br2nl")).text();
    text = text.replaceAll("br2nl ", "\n").replaceAll("br2nl", "\n").trim();

Hope this helps