从urllib2切换到请求，具有相同参数的结果却奇怪地不同

Question

I'm trying to grab a cookie from a POST request. Previously, I used urllib2, which still works fine but I wanted to switch to the clearer library python-requests. Unfortunately I get an error on the page.

Since the request is HTTPS I can't sniff them to locate the difference.

urllib2 code:

NINTENDO_LOGIN_PAGE = "https://id.nintendo.net/oauth/authorize/"
MIIVERSE_CALLBACK_URL = "https://miiverse.nintendo.net/auth/callback"
parameters = {'client_id': 'ead88d8d450f40ada5682060a8885ec0',
              'response_type': 'code',
              'redirect_uri': MIIVERSE_CALLBACK_URL,
              'username': MIIVERSE_USERNAME,
              'password': miiverse_password}

data = urlencode(parameters)
self.logger.debug(data)
req = urllib2.Request(NINTENDO_LOGIN_PAGE, data)
page = urllib2.urlopen(req).read()
self.logger.debug(page)

Result (good):

[...]
<div id="main-body">
    <div id="try-miiverse">
        <p class="try-miiverse-catch">A glimpse at some of the posts that are currently popular on Miiverse.</p>
        <h2 class="headline">Miiverse Sampler</h2>
        <div id="slide-post-container" class="list post-list">
        [...]

Requests code:

req = requests.post(NINTENDO_LOGIN_PAGE, data=parameters)
self.logger.debug(req.text)

Result (bad):

[...]
<div id="main-body">
    <h2 class="headline">Activity Feed</h2>

    <div class="activity-feed content-loading-window">
        <div>
            <img src="https://d13ph7xrk1ee39.cloudfront.net/img/loading-image-green.gif" alt=""></img>
            <p class="tleft"><span>Loading activity feed...</span></p>
        </div>
    </div>
    <div class="activity-feed content-load-error-window none"><div>
    <p>The activity feed could not be loaded. Check your Internet connection, wait a moment and then try reloading.</p>
    <div class="buttons-content"><a href="/" class="button">Reload</a></div>
    </div>
</div>
[...]

Thanks in advance for any hints towards solving this.

Update 1: Thank you all for your responses!

As suggested by @abarnert, I checked the redirects.

resp = urllib2.urlopen(req)
print(resp.geturl()) # https://miiverse.nintendo.net/

req = requests.post(NINTENDO_LOGIN_PAGE, data=parameters)
print(req.url) # https://miiverse.nintendo.net/
print(req.history) # (<Response [303]>, <Response [302]>)

It seems they did both follow a redirect, but ended up in the same place.

@sigmavirus24, very useful website, thank you for making me discover it. Here are the results (I edited the order of parameters so they are easily comparable):

urllib2:

{
  "args": {},
  "data": "",
  "files": {},
  "form": {
    "client_id": "ead88d8d450f40ada5682060a8885ec0",
    "response_type": "code",
    "redirect_uri": "https://miiverse.nintendo.net/auth/callback",
    "username": "Wiwiweb",
    "password": "password"
  },
  "headers": {
    "Accept-Encoding": "identity",
    "Connection": "close",
    "Content-Length": "170",
    "Content-Type": "application/x-www-form-urlencoded",
    "Host": "httpbin.org",
    "User-Agent": "Python-urllib/2.7"
  },
  "json": null,      
  "origin": "24.85.129.188",
  "url": "http://httpbin.org/post"
}

requests:

{
  "args": {},
  "data": "",
  "files": {},
  "form": {
    "client_id": "ead88d8d450f40ada5682060a8885ec0",
    "response_type": "code",
    "redirect_uri": "https://miiverse.nintendo.net/auth/callback",
    "username": "Wiwiweb",
    "password": "password"
  },
  "headers": {
    "Accept": "*/*",
    "Accept-Encoding": "gzip, deflate, compress",
    "Connection": "close",
    "Content-Length": "170",
    "Content-Type": "application/x-www-form-urlencoded"
    "Host": "httpbin.org",
    "User-Agent": "python-requests/1.2.3 CPython/2.7.5 Windows/7"
  },
  "json": null,
  "origin": "24.85.129.188"
  "url": "http://httpbin.org/post",
}

Looks like some headers are slightly different. I don't have any other idea so I might as well try to completely copy the urllib2 headers. Spoofing the user agent might be it.

Update 2: I have added these headers to the "requests" request:

headers = {'User-Agent': 'Python-urllib/2.7',
           'Accept-Encoding': 'identity'}

I am still getting the same results... The only difference between the requests now is the "requests" one has an extra header: "Accept": "*/*" . I'm not sure this is the problem.

Could it be coming from the redirect?

Answer 1

Well, I didn't quite solve "why" the redirects are different, but I found out where to get my cookie using requests.

I figured the difference between the two libraries had something to do with the way they handle redirects. So I checked the history of both requests. For 'requests' that's as easy as doing req.history , but for urllib2, I used this bit of code:

class MyHTTPRedirectHandler(urllib2.HTTPRedirectHandler):
    def http_error_302(self, req, fp, code, msg, headers):
        print("New request:")
        print(headers)
        return urllib2.HTTPRedirectHandler.http_error_302(self, req, fp, code, msg, headers)

opener = urllib2.build_opener(MyHTTPRedirectHandler, urllib2.HTTPCookieProcessor())
urllib2.install_opener(opener)

Checking the history allowed me to see that the 'requests' request had the 'set-cookie' header during it's first redirect (so the second request out of three), but not at the end. That's good enough for me because I know where to get it now: req.history[1].cookies['ms']

As a curious note, because of that bit that I added to the urllib2 request, it started returning the same thing as the 'requests' request! Even changing it to that:

class MyHTTPRedirectHandler(urllib2.HTTPRedirectHandler):
    pass

opener = urllib2.build_opener(MyHTTPRedirectHandler, urllib2.HTTPCookieProcessor())
urllib2.install_opener(opener)

is enough to make it completely change its response to the same thing 'requests' returned all along (That bit I marked as 'bad' in the question).

I'm stumped, but knowing where to find the cookie is good enough for me. Maybe someone curious and bored will be interested in trying to find the cause.

Thank you all for your help :)