你如何将Python的urllib2.urlopen（）转换为文本？

Question

I'm doing a program on python that does the following:

• Gets info from a web.
• Puts it on a .txt file.

I've used urllib2.urlopen() for giving me the HTML code, but I want the info of the page. I say:

urllib2.urlopen() gets HTML. But I want that HTML written on text, I don't want HTML code!!

My program at the moment:

import urllib2
import time
url = urllib2.urlopen('http://www.dev-explorer.com/articles/using-python-httplib')
html = url.readlines()
for line in html:
    print line

time.sleep(5)

Answer 1

You have to use some method to read what you are opening:

url = urllib2.urlopen('someURL')
html = url.readlines()
for line in html:
    #At this level you already have a str in 'line'
    #do something

Also you have other methods: read, readline

Edit:

As I said in one of my comments in this thread, maybe you need to use BeautifulSoup to scrap what you want. So, I think this was already solved here .

You have to install BeautifulSoup:

pip install BeautifulSoup

Then you have to do what is in the example:

from bs4 import BeautifulSoup
import urllib2    
import re

html = urllib.urlopen('someURL').read()
soup = BeautifulSoup(html)
texts = soup.findAll(text=True)

def visible(element):
    if element.parent.name in ['style', 'script', '[document]', 'head', 'title']:
        return False
    elif re.match('<!--.*-->', str(element)):
        return False
    return True

visible_texts = filter(visible, texts)

And if you have some problem with ascii characters, you have to change str(element) to unicode(element) in the visible function.

Answer 2

You could use the requests package which is my preference over urllib. This returns all the html from the web page.

import requests

response  = requests.get('http://stackoverflow.com/questions/34157599/how-do-you-convert-pythons-urllib2-urlopen-to-text')

with open('test.txt' 'w' ) as f:
   f.writelines(response.text)
f.close()