python-requests - user-agent is being overriden

Question

I have

    logindata = {
        'username': 'me',
        'password': 'blbla'
    }
    payload = {'from':'me', 'lang':'en', 'url':csv_url}
    headers = {
        'User-Agent': 'Mozilla/5.0'
    }
    api_url = 'http://dev.mypage.com/admin/app/import/'

    with requests.Session() as s:
        s.post(api_url, data=json.dumps(logindata), headers=headers)

        print s.headers

        # An authorised request.
        r = s.get(api_url, params=payload, headers=headers)

I am getting rejected but it is because of 403 forbidden. And I printed the headers, I get:

..'User-Agent': 'python-requests/2.2.1 CPython/2.7.5 Windows/7'..

Why is my 'User-Agent': 'Mozilla/5.0' getting overriden? what am I missing here?

Answer 1

headers are not kept inside session this way.

You need to either explicitly pass them every time you make a request, or set the s.headers once:

with requests.Session() as s:
    s.headers = {'User-Agent': 'Mozilla/5.0'}

You can check that the correct headers were sent via inspecting response.request.headers :

with requests.Session() as s:
    s.headers = {'User-Agent': 'Mozilla/5.0'}

    r = s.post(api_url, data=json.dumps(logindata))
    print(r.request.headers)

Also see how the Session class is implemented - every time you make a request it merges the request.headers with headers you have set on the session object:

headers=merge_setting(request.headers, self.headers, dict_class=CaseInsensitiveDict),

Answer 2

If you want the session to use specific headers for all requests you need to set those headers on the session, explicitly:

with requests.Session() as s:
    s.headers.update(headers)
    s.post(api_url, data=json.dumps(logindata))

    # An authorised request.
    r = s.get(api_url, params=payload)

The s.headers.update(headers) line adds your dictionary to the session headers.

Sessions never copy information from requests to reuse for other requests. Only information from responses (specifically, the cookies) is captured for reuse.

For more details, see the requests Session Objects documentation :

Sessions can also be used to provide default data to the request methods. This is done by providing data to the properties on a Session object.