在Python中从网页提取文本时如何避免注释

Question

I am trying to extract only text from web page but i am facing some problem like texts which are not in written in the page but they are written in code which are comments such as: "include footer", "sidebar.php end" etc. Also the unwanted things are also coming which i really do not want. Here are the links which i am using for test case ie:

1) http://ai-depot.com/articles/the-easy-way-to-extract-useful-text-from-arbitrary-html

2) http://www.tutorialspoint.com/cplusplus/index.htm

3) http://www.cplusplus.com/doc/tutorial/program_structure/

(so that i can ensure my code is extracting text from any page)

here is the code which i am facing trouble:

import urllib
from bs4 import BeautifulSoup
url = "http://ai-depot.com/articles/the-easy-way-to-extract-useful-text-from-arbitrary-html/" 
html = urllib.urlopen(url).read()
soup = BeautifulSoup(html)

for script in soup(["script", "style","a","p","li","<!-->","small","<div id=\"footer\">","<div id=\"footer\">","<div id=\"bottom\">"]):
    script.extract()    

text = soup.findAll(text=True)
for p in text:
    print unicode(p)
fo = open('file.txt', 'w')
fo.seek(0, 2)
fo.writelines( unicode(p) )
fo.close()

in this code i have used number 1 link and when i did "inspect element" on that page i found so many comments in that code and this code is extracting them as well. So help please.....

Answer 1

One way would be to use a regex to strip/skip comments when your code encounters a line that the regex matches as a comment.

Alternatively, you might be able to use an HTML parser as well. Python has one built into its standard library.

https://docs.python.org/2/library/htmlparser.html