如何计算HTML源中的单词（文本）数

Question

I have some html documents for which I need to return the number of words in the document. This count should only include actual text (so no html tags eg html, br, etc).

Any ideas how to do this? Naturally, I would prefer to re-use some code.

Thanks,

Assaf

Answer 1

• Strip out the HTML tags , get the text content , reuse Jsoup

• Read file line by line , hold a Map<String, Integer> wordToCountMap and read through and operate on the Map

Answer 2

Solution with jsoup

private int countWords(String html) throws Exception {
    org.jsoup.nodes.Document dom = Jsoup.parse(html);
    String text = dom.text();

    return text.split(" ").length;
}

Answer 3

I would add an extra step to Jigar's answer:

• Parse out the document text using JSoup or Jericho or Dom4j

• Tokenise the resulting text. This depends on your definition of a "word". It is unlikely to be as simple as splitting on white-space. And you'll need to deal with punctuation etc. So take a look at the various Tokeniser's available eg from the Lucene or Stanford NLP projects. Here are some simple examples you will encounter:

  "Today I'm going to New York!" - Is "I'm" one word or two? What about "New York"?

  "We applied two meta-filters in the analysis" - Is "meta-filter" one word or two?

And what about badly formatted text, eg missing of a space at the end of a sentence:

"So we went there.And on arrival..."

Tokenising is tricky...

• Iterate through your tokens and count them up, eg using a HashMap.