BadStatusLine Error in using Python, Requests

Question

I use Requests in Python but always encounter BadStatusLine Error.

My code is like followings:

import requests

ip = 'xx.xx.xx.xx'
port = 80
proxies={
        'http': 'http://%s:%s'%(ip, port),
        'https': 'https://%s:%s'%(ip, port)
    }

url = 'https://url'
d = {'active_id': 117}

r=requests.post(url, data=d, proxies=proxies)

The exeptions shows:

File "/root/untitled-5.py", line 13, in <module>
  r=requests.post(url, data=d,proxies=proxies, headers={'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.75 Safari/537.36'},verify=False)
File "/usr/lib/python2.7/dist-packages/requests/api.py", line 110, in post
  return request('post', url, data=data, json=json, **kwargs)
File "/usr/lib/python2.7/dist-packages/requests/api.py", line 56, in request
  return session.request(method=method, url=url, **kwargs)
File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 488, in request
  resp = self.send(prep, **send_kwargs)
File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 609, in send
  r = adapter.send(request, **kwargs)
File "/usr/lib/python2.7/dist-packages/requests/adapters.py", line 473, in send
  raise ConnectionError(err, request=request)

requests.exceptions.ConnectionError: ('Connection aborted.', BadStatusLine("''",))

I find some solutions from google. And I try to change the last line into:

1 r=requests.post(url, data=d, proxies=proxies, varify=False)
2 r=requests.post(url, data=d, proxies=proxies, headers={'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.75 Safari/537.36'})

But no one really works. I wanna know why and how to handle this. Thx

Answer 1

You get this error when your Python client receives an empty response (header /body).

BTW newer Python version will throw a different Exception, It can happen when the server disconnects the connections, or a network issue.

In my case, we spent some weeks trying to reproduce it until we found the main cause, We have a Python application sending requests to a service behind Nginx loadbalancer,

We found that Nginx disconnects the connection when the client exceeded the default Nginx configurations client_header_timeout / client_body_timeout (60 sec), it's the time that Nginx will wait for additional data packet from the client.

You can follow this reference http://nginx.org/en/docs/http/ngx_http_core_module.html#client_body_timeout

When our application is loaded with CPU processing, connections are established but header and request body have a long delay until transmitted, over 82 seconds.

So Nginx closes the connection and returns reset packet (recorded tcpdump with empty body and headers), officially it should return status code 408, which is not happening.

We solved it by increasing the client_header_timeout / client_body_timeout to 180s for both params:

server {
..
client_body_timeout 180s;
client_header_timeout 180s;
}