Python requests ajax form authentication issue

Question

I have seemed to be an obvious problem, login to a website programmatically and then get data when authenticated.

I've been reading StackOverflow for a few days and can't find a working solution.

Here is the login form, which is also accessible using a separate URL and after logging in via browser it redirects to the home page:

<strong class="popup-title">i class="fa fa-lock" aria-hidden="true"></i>Login</strong>
<div class="popup-holder">
    <form action="https://test.com/login/" data-form="ajax" method="post">
        <div class="generic-error hidden">
        </div>
        <div>
            <div class="row">
                <label for="login_username" class="field-label required">Username</label>
                <input type="text" name="username" id="login_username" class="textfield"
                    placeholder="Enter your username" />
                <div class="field-error down"></div>
            </div>

            <div class="row">
                <label for="login_pass" class="field-label required">Password</label>
                <input type="password" name="pass" id="login_pass" class="textfield" placeholder="Enter your password" />
                <div class="field-error down"></div>
            </div>

            <div class="row">
                <div class="col-sm-4" style="padding-left: 0;">
                    <input type="checkbox" name="remember_me" id="login_remember_me" class="checkbox" value="1" checked />
                    <label for="login_remember_me">remember me</label>
                </div>
                <div class="col-sm-5 forgot pull-right" style="padding-right: 0px;">
                    <a href="https://test.com/reset-password/" data-fancybox="ajax">Forgot password?</a><br />
                    <a href="https://test.com/resend-confirmation/" data-fancybox="ajax">Missing confirmation email?</a>
                </div>
            </div>

            <div class="row">
                <input type="hidden" name="action" value="login" />
                <input type="hidden" name="email_link" value="https://test.com/email/" />
                <input type="submit" class="btn btn-danger btn-lg btn-block" value="Log in" />
            </div>
            <div class="row">
                <span class="form-separator">Not a member yet? Sign up now for free!</span>
            </div>
            <div class="row">
                <a href="https://test.com/signup/" class="btn btn-info btn-lg btn-block" data-fancybox="ajax">Sign up</a>
            </div>
        </div>
    </form>
</div>

Here is the Python code I've tried:

payload = {
   'username': 'mylogin',
   'pass': 'mypass'
}

with requests.Session() as s:
    r = s.post('https://test.com/login/', data=payload)
    r = s.get('https://test.com/testpage/')

Same logic in PowerShell:

$payload = @{
   username = 'mylogin'
   pass = 'mypass'
}

$r = Invoke-RestMethod 'https://test.com/login/' -Method POST -Body $payload -SessionVariable 'Session'
$r = Invoke-WebRequest -Uri "https://test.com/testpage/" -WebSession $Session

But none of the above is working, I'm still getting results for non-authenticated user.

Answer 1

Here is a working example using one of my Django sites and a demo login account.

requests.Session() is used to manage the cookies. In order to make it work, I had to explicitely manage the header content such as adding the Referer before posting the login.

import requests
import re

base_url = 'https://www.archery-analytics.com/en/'

# use session object to manage cookies and headers
s = requests.Session()
s.headers.update({
    'Host': 'www.archery-analytics.com',
    'Origin': 'https://www.archery-analytics.com',
    })

# get login form and cookies
r1 = s.get(base_url + 'public/home')
print(r1.status_code, r1.url)

# add Referer to header
s.headers.update({
    'Referer': r1.url,
    })

# get csrf token of form (= hidden input element of login form)
reggie = re.compile(rb".*name=\"csrfmiddlewaretoken\" value=\"(?P<csrf>\w+)\".*")
match = reggie.findall(r1.content)

# login data for demo account
payload = {
    'username': 'RyngDyng',
    'password': '123demo123',
    'login': '',
    'csrfmiddlewaretoken': match[0].decode("utf-8")
}

# login post
r2 = s.post(base_url + 'global/login', data=payload)
print(r2.status_code, r2.url)

# check successful login
if r2.status_code == requests.codes.ok:

    # test logged in: access to page for editing user profile
    r3 = s.get(base_url + 'global/edit_profile')
    print(r3.status_code, r3.url)
    
    
    # logout
    r4 = s.get(base_url + 'global/logout')
    print(r4.status_code, r4.url)

Output:

200 https://www.archery-analytics.com/en/public/home
200 https://www.archery-analytics.com/en/public/home
200 https://www.archery-analytics.com/en/global/edit_profile
200 https://www.archery-analytics.com/en/public/home