使用Python请求重定向错误太多

Question

HTTP requests are working fine on my localhost, but running the same HTTP requests using the python requests library on my server returns a "Too Many Redirects" error

When I enter

localhost/terminal/jfk

in a browser, I get a json dictionary as expected.

However, when I run the following in python using the python requests library on my server

requests.get('http://splitmyri.de/terminal/jfk')

I receive a "Too Many Redirects" error from the requests module.

Any thoughts as to what's causing the error? Or recommendations for narrowing potential causes?

Answer 1

Asha,

After you responded that this really is your site, I checked out why I was getting an empty GoDaddy.com page. The problem is that your DNS entry for splitmyri.de is returning two different A records...

Analyzing results, using 107.10.141.119 :
Hard coding 107.10.141.119 splitmyri.de in my /etc/hosts file allowed me to make queries to 107.10.141.119, which returns "this is my site" in index.html (then I checked http://splitmyri.de/terminal/ and got an empty json hash, with a mime type=[application/json]). Checking the reverse entry for this address returns an Amazon AWS host (the kind of entry I expect for a real web page). Now your code above works as expected...

[mpenning@Bucksnort ~]$ python
Python 2.5.2 (r252:60911, Jan 24 2010, 14:53:14) 
[GCC 4.3.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import requests
>>> requests.get('http://splitmyri.de/terminal/')
<Response [200]>
>>> requests.get('http://splitmyri.de/terminal/').content
u'{}'
>>> # requests.get('http://splitmyri.de/terminal/jfk').content returns a huge json hash 

Analyzing results using 68.178.232.100 :
Performing the same drill with queries to 68.178.232.100 gets an empty GoDaddy.com page. Reverse mapping this address turns up a typical entry for a parked domain. Sending a request here returns TooManyRedirects as you saw in your attempts...

[mpenning@Bucksnort ~]$ python
Python 2.5.2 (r252:60911, Jan 24 2010, 14:53:14) 
[GCC 4.3.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import requests
>>> foo = requests.get('http://splitmyri.de/terminal/jfk')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "build/bdist.linux-i686/egg/requests/api.py", line 50, in get
  File "build/bdist.linux-i686/egg/requests/api.py", line 37, in request
  File "build/bdist.linux-i686/egg/requests/sessions.py", line 170, in request
  File "build/bdist.linux-i686/egg/requests/models.py", line 383, in send
  File "build/bdist.linux-i686/egg/requests/models.py", line 210, in _build_response
requests.exceptions.TooManyRedirects
>>> 

SOLUTION :
Fix your DNS for splitmyri.de (removing the A record for 68.178.232.100) and all will be good.

Including the DNS info below as an FYI...

[mpenning@Bucksnort ~]$ dig splitmyri.de

; <<>> DiG 9.6-ESV-R4 <<>> splitmyri.de
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 54051
;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 2, ADDITIONAL: 4

;; QUESTION SECTION:
;splitmyri.de.                  IN      A

;; ANSWER SECTION:
splitmyri.de.           3193    IN      A       68.178.232.100
splitmyri.de.           3193    IN      A       107.20.141.119

;; AUTHORITY SECTION:
splitmyri.de.           3193    IN      NS      ns49.domaincontrol.com.
splitmyri.de.           3193    IN      NS      ns50.domaincontrol.com.

;; ADDITIONAL SECTION:
ns49.domaincontrol.com. 3193    IN      A       216.69.185.25
ns49.domaincontrol.com. 3193    IN      AAAA    2607:f208:206::19
ns50.domaincontrol.com. 3193    IN      A       208.109.255.25
ns50.domaincontrol.com. 3193    IN      AAAA    2607:f208:302::19

;; Query time: 1 msec
;; SERVER: 127.0.0.1#53(127.0.0.1)
;; WHEN: Wed Oct 26 05:14:51 2011
;; MSG SIZE  rcvd: 205

[mpenning@Bucksnort ~]$ 
[mpenning@Bucksnort ~]$ dig -x 107.20.141.119

; <<>> DiG 9.6-ESV-R4 <<>> -x 107.20.141.119
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 41049
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 6, ADDITIONAL: 6

;; QUESTION SECTION:
;119.141.20.107.in-addr.arpa.   IN      PTR

;; ANSWER SECTION:
119.141.20.107.in-addr.arpa. 300 IN     PTR     ec2-107-20-141-119.compute-1.amazonaws.com.

;; AUTHORITY SECTION:
141.20.107.in-addr.arpa. 900    IN      NS      pdns1.ultradns.net.
141.20.107.in-addr.arpa. 900    IN      NS      pdns2.ultradns.net.
141.20.107.in-addr.arpa. 900    IN      NS      pdns3.ultradns.org.
141.20.107.in-addr.arpa. 900    IN      NS      pdns5.ultradns.info.
141.20.107.in-addr.arpa. 900    IN      NS      pdns4.ultradns.org.
141.20.107.in-addr.arpa. 900    IN      NS      pdns6.ultradns.co.uk.

;; ADDITIONAL SECTION:
pdns1.ultradns.net.     86400   IN      A       204.74.108.1
pdns1.ultradns.net.     86400   IN      AAAA    2001:502:f3ff::1
pdns2.ultradns.net.     86400   IN      A       204.74.109.1
pdns3.ultradns.org.     86400   IN      A       199.7.68.1
pdns4.ultradns.org.     86400   IN      A       199.7.69.1
pdns4.ultradns.org.     86400   IN      AAAA    2001:502:4612::1

;; Query time: 306 msec
;; SERVER: 127.0.0.1#53(127.0.0.1)
;; WHEN: Wed Oct 26 05:09:47 2011
;; MSG SIZE  rcvd: 392

[mpenning@Bucksnort ~]$ dig -x 68.178.232.100

; <<>> DiG 9.6-ESV-R4 <<>> -x 68.178.232.100
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 38578
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 2, ADDITIONAL: 2

;; QUESTION SECTION:
;100.232.178.68.in-addr.arpa.   IN      PTR

;; ANSWER SECTION:
100.232.178.68.in-addr.arpa. 3600 IN    PTR     parkwebwin-v01.prod.mesa1.secureserver.net.

;; AUTHORITY SECTION:
232.178.68.in-addr.arpa. 3600   IN      NS      cns1.secureserver.net.
232.178.68.in-addr.arpa. 3600   IN      NS      cns2.secureserver.net.

;; ADDITIONAL SECTION:
cns1.secureserver.net.  3600    IN      A       208.109.255.100
cns2.secureserver.net.  3600    IN      A       216.69.185.100

;; Query time: 173 msec
;; SERVER: 127.0.0.1#53(127.0.0.1)
;; WHEN: Wed Oct 26 05:12:06 2011
;; MSG SIZE  rcvd: 171

[mpenning@Bucksnort ~]$ 

Answer 2

If you need to narrow down the problem, a good idea is to use wireshark and analyse the different connections and their content. Like this you will see what is transferred over the wire. Good for debugging TCP related problems. Additionally, you may use the pdb Python module and debug your program. Issue pdb.set_trace() right before the call, then single step into the function and see what it does. More info about pdb can be found in the Python documentation or by pressing ? .