python request.get中的特定字符串给出ValueError

Question

Trying to get this specific page...

request = requests.get('http://market.yandex.ru/catalog/90555/list')

...gives me a strange error:

ValueError                                Traceback (most recent call last)
    C:\Python34\lib\site-packages\requests\packages\urllib3\response.py in read_chunked(self, amt)
    406                 try:
--> 407                     self.chunk_left = int(line, 16)
    408                 except ValueError:

ValueError: invalid literal for int() with base 16: ''

I figured out that some part of a string is to blame. I was experimenting with it and the results are even more weird:

# No error
http://market.ru/catalog/90555/list
http://market.yandex.ru/catalo

# Error
http://market.yandex.ru/catalog

PS By the way, the problem has occured today. Just recently I had no problems with getting this very page (using the same method).

Answer 1

You are being rate limited, but the server does so in a way that violates the HTTP specification. Their response headers promise a Chunked transfer encoding, then do not send such a response.

If you look at the URL with curl in verbose mode, you get the following output:

$ curl -v https://market.yandex.ru/catalog/90555/list
* Hostname was NOT found in DNS cache
*   Trying 213.180.204.22...
* Connected to market.yandex.ru (213.180.204.22) port 443 (#0)
* TLS 1.2 connection using TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256
* Server certificate: market.yandex.ru
* Server certificate: Certum Level IV CA
* Server certificate: Certum CA
> GET /catalog/90555/list HTTP/1.1
> User-Agent: curl/7.37.1
> Host: market.yandex.ru
> Accept: */*
> 
< HTTP/1.1 302 Found
* Server nginx is not blacklisted
< Server: nginx
< Date: Mon, 18 May 2015 18:53:15 GMT
< Content-Type: text/html; charset=UTF-8
< Transfer-Encoding: chunked
< Connection: keep-alive
< Keep-Alive: timeout=120
< X-Forwardtouser-Y: 1
< Set-Cookie: spravka=dD0xNDAwNDM5MTk1O2k9ODQuOTIuOTguMTcwO3U9MTQwMDQzOTE5NTUxNjUwOTExMjtoPWNkMzVlMzBlMjgxMTg4YWM0YjYyZDg3OTg4ZjUyNWFj; domain=.yandex.ru; path=/; expires=Wed, 17-Jun-2015 18:53:15 GMT
< Location: http://market.yandex.ru/showcaptcha?cc=1&retpath=http%3A//market.yandex.ru/catalog/90555/list%3F_bfd13d35fbf1551a835f050d3775fc4b&t=0/1431975195/029660aeb063916c78e30ebd9444fd4b&s=4dd645e7048b399008278208fa776ba9
< Set-Cookie: uid=CniLolVaNRthdR2JDtV0Ag==; path=/
< 
* transfer closed with outstanding read data remaining
* Closing connection 0
curl: (18) transfer closed with outstanding read data remaining

The are sending you a redirect , but the Transfer-Encoding: chunked header in the response means that the client side has to load chunks, which are not there.

The redirect leads to a captcha:

http://market.yandex.ru/showcaptcha?cc=1&retpath=http%3A//market.yandex.ru/catalog/90555/list%3F_bfd13d35fbf1551a835f050d3775fc4b&t=0/1431975195/029660aeb063916c78e30ebd9444fd4b&s=4dd645e7048b399008278208fa776ba9
#                       ^^^^^^^^^^^