使用 urllib 和 BeautifulSoup 通过 Python 从网络检索信息

Question

I can get the html page using urllib, and use BeautifulSoup to parse the html page, and it looks like that I have to generate file to be read from BeautifulSoup.

import urllib                                       
sock = urllib.urlopen("http://SOMEWHERE") 
htmlSource = sock.read()                            
sock.close()                                        
--> write to file

Is there a way to call BeautifulSoup without generating file from urllib?

Answer 1

from BeautifulSoup import BeautifulSoup

soup = BeautifulSoup(htmlSource)

No file writing needed: Just pass in the HTML string. You can also pass the object returned from urlopen directly:

f = urllib.urlopen("http://SOMEWHERE") 
soup = BeautifulSoup(f)

Answer 2

You could open the url, download the html, and make it parse-able in one shot with gazpacho :

from gazpacho import Soup
soup = Soup.get("https://www.example.com/")