Python：urlopen无法下载整个网站

Question

Greetings,

I have done:

import urllib

site = urllib.urlopen('http://www.weather.com/weather/today/Temple+TX+76504')
site_data = site.read()
site.close()

but it doesn't compare to viewing the source when loaded in firefox.

I suspected the user agent and did this:

class AppURLopener(urllib.FancyURLopener):
    version = "Mozilla/5.0 (X11; U; Linux i686; zh-CN; rv:1.9.2.8) Gecko/20100722 Ubuntu/10.04 (lucid) Firefox/3.6.8"

urllib._urlopener = AppURLopener()

and downloaded it, but it still doesn't download the whole website.

Can someone please help me do user agent switching, if that is the likely culprit?

Thanks, Narnie

Answer 1

It's more likely that there is an iframe in the code or that javascript is modifying the DOM. If theres an iframe, you'll have to parse the page to get the url for the iframe or just do it manually if it's a one-off. If it's javascript, I hear that selenium-rc is good but have no first hand experience with it.

Answer 2

本地显示的下载页面可能看起来由于多种原因而有所不同，例如存在相对链接（可以在页面标题元素中添加例如<base href="http://www.weather.com/today/">进行修复），或者非功能性ajax请求（请参阅规避同源策略的方法 ）。