Jsoup-将html文本转换为字符串列表

Question

Using Jsoup I want to be able add text existing in each html tag to a List<String> in order.

This is fairly easy using BeautifulSoup4 in python but I'm having a hard time in Java.

BeautifulSoup Code:

from bs4 import BeautifulSoup
from bs4.element import Comment
import urllib.request


def tag_visible(element):
    if element.parent.name in ['style', 'script', 'head', 'title', 'meta', '[document]']:
    return False
    if isinstance(element, Comment):
        return False
    return True


def text_from_html(body):
    soup = BeautifulSoup(body, 'html.parser')
    texts = soup.findAll(text=True)
    visible_texts = filter(tag_visible, texts)

    text_list =[]

    for t in visible_texts:
        text_list.append(t.strip())

    return list(filter(None, text_list))

html = urllib.request.urlopen('https://someURL.com/something').read()
print(text_from_html(html))

This code will print ["text1", "text2", "text3",...]

---

My initial attempt was to follow the Jsoup documentation for text conversion.

Jsoup Code Attempt-1:

Document doc = Jsoup.connect('https://someURL.com/something')
                        .userAgent("Bot")
                        .get();
Elements divElements = doc.select("*")
List<String> texts = divElements.eachText();
System.out.println(texts);

What ends up happening is a duplication of texts ["text1 text2 text3","text2 text3", "text3",...]

My assumption is that Jsoup goes through each Element and prints out every text within that Element including the text existing in each child node. Then it goes to the child node and prints out the remaining text, so on and so forth.

I have seen many people specify Tag/Attributes via cssQuery to bypass this problem but my project requires to do this for any scrape-able website.

Any suggestion is appreciated.

Answer 1

Your assumption is right - but BeautifulSoup would probably do the same. Only the text=True in findAll(text=True) limits the result to pure text-nodes. To have the equivalent in JSoup use the following selector:

Elements divElements = doc.select(":matchText");