How to get HTML with expanded containers using Python?

Question

When I get HTML of the page, eg

response = urllib2.urlopen('http://www.wunderground.com/us/fl/miami/precipitation')
html = response.read()

I get HTML with collapsed containers, eg

<h2>6-Hour Precipitation Forecast</h2>
<div id="precip-statement"></div>
<div id="precip-graph">

while the real HTML looks like that:

Clearly, I need to extract 6-hour forecast, which I cannot do having it collapsed into <div id="precip-statement"></div>

I will be very thankful if you can help me with this issue. Thank you

Answer 1

The content is loaded dynamically using ajax. You can sniff this request with Chrome. Press F12 -> Network -> XHR and look at requests, one of them (wwir.json) returns a nice json that you can parse using:

 import json
 weather = json.loads(response)

It looks like they use API key from api.weather.com, which probably means you should get your own.