Find important text in arbitrary HTML using PHP?

Question

I have some random HTML layouts that contain important text I would like to extract. I cannot just strip_tags() as that will leave a bunch of extra junk from the sidebar/footer/header/etc.

I found a method built in Python and I was wondering if there is anything like this in PHP.

The concept is rather simple: use information about the density of text vs. HTML code to work out if a line of text is worth outputting. (This isn't a novel idea, but it works!) The basic process works as follows:

1. Parse the HTML code and keep track of the number of bytes processed.
2. Store the text output on a per-line, or per-paragraph basis.
3. Associate with each text line the number of bytes of HTML required to describe it.
4. Compute the text density of each line by calculating the ratio of text t> o bytes.
5. Then decide if the line is part of the content by using a neural network.

You can get pretty good results just by checking if the line's density is above a fixed threshold (or the average), but the system makes fewer mistakes if you use machine learning - not to mention that it's easier to implement!

Update: I started a bounty for an answer that could pull main content from a random HTML template. Since I can't share the documents I will be using - just pick any random blog sites and try to extract the body text from the layout. Remember that the header, sidebar(s), and footer may contain text also. See the link above for ideas.

Answer 1

• phpQuery is a server-side, chainable, CSS3 selector driven Document Object Model (DOM) API based on jQuery JavaScript Library.

---

UPDATE 2

• DEMO: http://so.lucafilosofi.com/find-important-text-in-arbitrary-html-using-php/
• tested on a casual blogs list taken from Technorati Top 100 and Best Blogs of 2010

1. many blogs make use of CMS ;
2. blogs html structure is the same almost the time.
3. avoid common selectors like #sidebar, #header, #footer, #comments, etc..
4. avoid any widget by tag name script, iframe
5. clear well know content like:
   1. /\\d+\\scomment(?:[s])/im
   2. /(read the rest|read more).*/im
   3. /(?:.*(?:by|post|submitt?)(?:ed)?.*\\s(at|am|pm))/im
   4. /[^a-z0-9]+/im

---

search for well know classes and ids:

• typepad.com .entry-content
• wordpress.org .post-entry .entry .post
• movabletype.com .post
• blogger.com .post-body .entry-content
• drupal.com .content
• tumblr.com .post
• squarespace.com .journal-entry-text
• expressionengine.com .entry

• gawker.com .post-body

• Ref: The blog platforms of choice among the top 100 blogs

---

$selectors = array('.post-body','.post','.journal-entry-text','.entry-content','.content');
$doc = phpQuery::newDocumentFile('http://blog.com')->find($selectors)->children('p,div');

---

search based on common html structure that look like this:

<div>
<h1|h2|h3|h4|a />
<p|div />
</div>

---

$doc = phpQuery::newDocumentFile('http://blog.com')->find('h1,h2,h3,h4')->parent()->children('p,div');

Answer 2

Domdocument can be used to parse html documents, which can then be queried through PHP.

Edit: wikied

Answer 3

I worked on a similar project a while back. It's not as complex as the Python script but it will do a good job. Check out the Simple HTML PHP Parser

http://simplehtmldom.sourceforge.net/

Answer 4

Depending on your HTML structure and if you have id's or classes in place you can get a little complicated and use preg_match() to specifically get any information between a certain start and end tag. This means that you should know how to write regular expressions.

You can also look into a browser emulation PHP class. I've done this for page scraping and it works well enough depending on how well formatted the DOM is. I personally like SimpleBrowser
http://www.simpletest.org/api/SimpleTest/WebTester/SimpleBrowser.html

Answer 5

I have developed a HTML parser and filter PHP package that can be used for that purpose.

It consists of a set of classes that can be chained together to perform a series of parsing, filtering and transformation operations in HTML/XML code.

It was meant to deal with real world pages, so it can deal with malformed tag and data structures, so it can preserve as much as the original document as possible.

One of the filter classes it comes with can do DTD validation. Another can discard insecure HTML tags and CSS to prevent XSS attacks. Another can simply extract all document links.

All those filter classes are optional. You can chain them together the way you want, if you need any at all.

So, to solve your problem, I do not think there is already a specific solution for that in PHP anywhere, but a special filter class could be developed for it. Take a look at the package. It is thoroughly documented.

If you need help, just check my profile and mail me and I may even develop the filter that does exactly what you need, eventually inspired in any solutions that exist for other languages.