403 forbidden and CSRF verification failed. Request aborted. python requests

Question

import requests
import json
headers={'User-Agent':'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.87 Safari/537.36'}
post_data={"q":"","filters":{"sizes":["Large","MNE"],"sectors":[18],"countries":[228],"regions":["Northern America"],"years":[2015],"types":[]},"page":1}


with requests.Session() as s:
    for_cookies=s.get('http://database.globalreporting.org/search')
    # print(for_cookies.content)
    p = s.post('http://database.globalreporting.org/search/ajax/',data=json.dumps(post_data), headers=headers)
    print(p.content)

My chrome can visit the website but my code cannot. How to make my code able to visit the website?

Answer 1

I have included the csrf token and tried to call it. But I think the Django website must have used,

if not request.is_ajax():
    return HttpResponse('Only ajax request')

Because I tried the code,

import requests

with requests.Session() as client:
    for_cookies=client.get('http://database.globalreporting.org/search')
    csrf = client.cookies['csrftoken']
    print csrf
    post_data={"csrfmiddlewaretoken": csrf, "q":"","filters":{"sizes":["Large","MNE"],"sectors":[18],"countries":[228],"regions":["Northern America"],"years":[2015],"types":[]},"page":1}
    r = client.post('http://database.globalreporting.org/search/ajax/', data=post_data, headers=dict(Referer='http://database.globalreporting.org/search'))
    print r.text

The response I get is

YrZa9IIoFJZyXqeRXZnZ57s3vaoCUCul
Only ajax request

In general you have to use csrf token in these cases. But we can configure whether to use ajax only.

Hope my answer helps you.

Answer 2

You need to add CSRF token value to your headers:

with requests.Session() as s:
    for_cookies=s.get('http://database.globalreporting.org/search')

    headers =  
    {'X-CSRFToken': for_cookies.headers['Set-Cookie'].split('=')[1].split(';')[0],
    'Referer': 'http://database.globalreporting.org/search/',
    'X-Requested-With':'XMLHttpRequest'}

    p = s.post('http://database.globalreporting.org/search/ajax/',data=json.dumps(post_data), headers=headers)
    print(p.content)

Try this code and let me know in case of any issues