使用正则表达式抓取网页

Question

I'm running into a wall why this code does not work, even thought it's the same code as on an online tutorial Python Web Scraping Tutorial 5 (Network Requests) . I tried running the code also via online Python interpreter.

import urllib
import re

htmltext = urllib.urlopen("https://www.google.com/finance?q=AAPL")

regex = '<span id="ref_[^.]*_l">(.+?)</span>'
pattern = re.compile(regex)
results = re.findall(pattern,htmltext)
results

I get:

re.pyc in findall(pattern, string, flags)
175 
176     Empty matches are included in the result."""
--> 177     return _compile(pattern, flags).findall(string)
178 
179 if sys.hexversion >= 0x02020000:

TypeError: expected string or buffer 

Expected result(s):

112.71

Help appreciated. I tried using "read()" on the url but that didn't work. According to documentation even empty results should be included. Thanks

Answer 1

If you follow the tutorial until the end :) :

% python2                                                                                                     
>>> import urllib
>>> data = urllib.urlopen('https://www.google.com/finance/getprices?q=AAPL&x=NASD&i=10&p=25m&f=c&auto=1').read()
>>> print data.split()[-1]
112.71

Never use regex to web scrape

I make improvement to fetch last array element simpler

Answer 2

问题是您实际上并没有从请求中读取 HTML。

htmltext = urllib.urlopen("https://www.google.com/finance?q=AAPL").read()