使用urllib.request.urlopen返回空白，但是从Chrome浏览器中我可以看到响应中有数据

Question

It is my first question and appreciate that you could provide some hints to me.

I am developing a spider using python to crawl the odds entry from a website. In that website, there is a onclick event to pop up a window to show the change of odds. I checked from Chrome that it links to a url, " http://odds.500.com/fenxi1/inc/yazhiajax.php?fid=554629&id=3&t= " + str(t) + "&r=1" Here t is a Javascript (new Date).getTime(). I can see the odds change from Chrome's preview and response. However, when I run below code to fetch the data. It shows blank. And when I navigate to the url through Chrome, it shows blank either.

enter image description here

import datetime
import re
import urllib.request
import lottery
import gzip
import time

# getTime simulates Javascript getTime function
def getTime():
    return int(time.time() * 1000)

user_agent = "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.80 Safari/537.36'"
referer = "http://odds.500.com/fenxi/yazhi-554629.shtml"
headers = {'User-Agent': user_agent, 'Referer': referer}

t = getTime()
url_str = "http://odds.500.com/fenxi1/inc/yazhiajax.php?fid=554629&id=3&t=" + str(t) + "&r=1"
print(url_str)
req = urllib.request.Request(url_str, headers = headers)
response = urllib.request.urlopen(req).read()
print(response)

Answer 1

The HTTP request does not consist only of URL. You can see in Chrome's Developer tools (ctrl+shift+I) all requests and responses in Network tab.

I opened your referer URL ( http://odds.500.com/fenxi/yazhi-554629.shtml ) in chrome and clicked one of the items in "盘" column. I believe that is what you are trying to mimic. It sent a request with many cookies. That is probably you problem.

You should probably make your crawler open the first URL, gather all the cookies and then make the second requset with the cookies.

That might be a challenge though, depending on what you have to do to gather the cookies.

Note also that when you the response is not a complete HTML document - it is a JSON list of HTML segments.

EDIT: Found the Answer

I had to check the headers again. If you just add this header, you will get a response: X-Requested-With: XMLHttpRequest . So:

url = 'http://odds.500.com/fenxi1/inc/yazhiajax.php?fid=554629&id=3&t=1449930953112&r=1'
headers = {'X-Requested-With': 'XMLHttpRequest'}
urllib.request.urlopen(urllib.request.Request(url, headers=headers)).read()

It returns some binary data... I'll leave that to you to decode. Hint: respnse headers say Content-Encoding:gzip , so it's zipped...

Answer 2

Using proper headers should resolve the issue. Try checking the headers below:

headers = {
                        'Host': "twitter.com",
                        'User-Agent': "Mozilla/5.0 (Windows NT 6.1; Win64; x64)",
                        'Accept': "application/json, text/javascript, */*; q=0.01",
                        'Accept-Language': "de,en-US;q=0.7,en;q=0.3",
                        'X-Requested-With': "XMLHttpRequest",
                        'Referer': url,
                        'Connection': "keep-alive"
                }

Answer 3

I would rather use urllib3 .

To get a page using urllib3:

   import urllib3
   url = http://odds.500.com/fenxi1/inc/yazhiajax.php?fid=554629&id=3&t=" + str(t) + "&r=1"
   http = urllib3.PoolManager()
   page = http.request('GET',url)

There you have your page.