algorithm to find 'article' in webpage?

Question

some browser plugin, like readability can extract the 'article' from a webpage. Does anyone has idea about how to do it? What's the difference between the real articles and ads or comments?

Answer 1

Well, it depends how you want to define "real articles"...

Taking HTML5 into consideration, a webpage is constructed of semantic tags. Pages no longer have to be built with elements like <div> that have exactly no semantic meaning. In HTML5 you may use <section> , <article> , <header> and so on . Those elements can give an application pretty good sense of what is the main content of a webpage (eg print <article> s and skip <nav> s...)

Of course, not many pages use those tags yet. Furthermore, the tags might get abused and lose their meaning. In that case I'd stick to some statistics, eg selecting the largest elements in a HTML document. Moreover, if you have to scrape a webpage, you could use a modification of some pattern-matching algorithm, DIPRE for instance.