Python 3 script for logging into a website using the Requests module

Question

I'm trying to write some Python (3.3.2) code to log in to a website using the Requests module. Here is the form section of the login page:

<form method="post" action="https://www.ibvpn.com/billing/dologin.php" name="frmlogin">
<input type="hidden" name="token" value="236647d2da7c8408ceb78178ba03876ea1f2b687" />

<div class="logincontainer">

    <fieldset>

        <div class="clearfix">
            <label for="username">Email Address:</label>
            <div class="input">
                <input class="xlarge" name="username" id="username" type="text" />
            </div>
        </div>

        <div class="clearfix">
            <label for="password">Password:</label>
            <div class="input">
                <input class="xlarge" name="password" id="password" type="password"/>
            </div>
        </div>

        <div align="center">

          <p>
                <input type="checkbox" name="rememberme" /> Remember Me
            </p>

          <p><a href="pwreset.php">Request a Password Reset</a></p>
        </div>

    </fieldset>

</div>

<div class="actions">
<input type="submit" class="btn primary" value="Login" />
</div>

</form>

Here is my code, trying to deal with hidden input:

import requests
from bs4 import BeautifulSoup

url = 'https://www.ibvpn.com/billing/clientarea.php'

body = {'username':'my email address','password':'my password'}

s = requests.Session()

loginPage = s.get(url)

soup = BeautifulSoup(loginPage.text)

hiddenInputs = soup.findAll(name = 'input', type = 'hidden')

for hidden in hiddenInputs:
    name = hidden['name']
    value = hidden['value']
    body[name] = value

r = s.post(url, data = body)

This just returns the login page. If I post my login data to the URL in the 'action' field, I get a 404 error.

I've seen other posts on StackExchange where automatic cookie handling doesn't seem to work, so I've also tried dealing with the cookies manually using:

cookies = dict(loginPage.cookies)

r = s.post(url, data = body, cookies = cookies)

But this also just returns the login page.

I don't know if this is related to the problem, but after I've run either variant of the code above, entering r.cookies returns <<class 'requests.cookies.RequestsCookieJar'>[]>

If anyone has any suggestions, I'd love to hear them.

Answer 1

You are loading the wrong URL. The form has an action attribute:

<form method="post" action="https://www.ibvpn.com/billing/dologin.php" name="frmlogin">

so you must post your login information to:

https://www.ibvpn.com/billing/dologin.php

instead of posting back to the login page. POST to soup.form['action'] instead:

r = s.post(soup.form['action'], data=body)

Your code is handling cookies just fine; I can see that s.cookies holds a cookie after requesting the login form, for example.

If this still doesn't work (a 404 is returned), then the server is using additional techniques to detect scripts vs. real browsers. Usually this is done by parsing the request headers. Look at your browser headers and replicate those. It may just be the User-Agent header that they parse, but Accept-* headers and Referrer can also play a role.