Code runs in shell but not from the module

Question

This code for downloading a json file from a url runs alright when I type commands one by one in the python shell. However, when I try to run the module containing this code I get: ValueError: No JSON object could be decoded . Any ideas why is that? I run python 2.7.

import urllib2
from urllib2 import Request
import json
import re

url1 = "http://www.skyscanner.net/flights/lond/nyca/130514/130525/airfares-from-london-to-new-york-in-may-2013.html"

req = Request(url1)
res = urllib2.urlopen(req)
the_page = res.read()
theText = str(the_page)

myre = re.compile(r'"SessionKey":"((([a-z0-9]+-)+)[a-z0-9]{12})"')
match = re.search(myre, theText)

print match.group(1)

url2 = "http://www.skyscanner.net/dataservices/routedate/v2.0/"+str(match.group(1))
htmltext = urllib2.urlopen(url2)
data = json.load(htmltext)

the whole code now:

import urllib2
from urllib2 import Request, urlopen, URLError, HTTPError
import json
import re


url1 = "http://www.skyscanner.net/flights/lond/nyca/130514/130525/airfares-from-london-to-new-york-in-may-2013.html"

req = Request(url1)
res = urllib2.urlopen(req)
the_page = res.read()
theText = str(the_page)

myre = re.compile(r'"SessionKey":"((([a-z0-9]+-)+)[a-z0-9]{12})"')
match = re.search(myre, theText)

url2 = "http://www.skyscanner.net/dataservices/routedate/v2.0/%s" % str(match.group(1))

req2 = urllib2.Request(url2)

try:
    response = urlopen(req2)
except HTTPError as e:
    print 'The server couldn\'t fulfill the request.'
    print 'Error code: ', e.code
except URLError as e:
    print 'We failed to reach a server.'
    print 'Reason: ', e.reason
else:
    data = json.loads(response.read())

print data["SessionKey"]

Answer 1

Ha! You were correct, it is a difference between shell use and scripted use. Whatever that url1 is doing, it hasn't completed (remotely) by the time you open url2 . Putting a short wait time in makes it all work!

import time

url2 = "http://www.skyscanner.net/dataservices/routedate/v2.0/%s" % str(match.group(1))

time.sleep(5)

req2 = urllib2.Request(url2)

so it wasn't your code at all... ho hum!