How to login website account with Python & Requests

Question

I'm trying to login onto 2 websites accounts with Python 3 and Requests module.

1st website is : https://www.amundi-ee.com/psf/#login and I need to grab information on this page after : https://www.amundi-ee.com/psf/#avoirs
2nd website is : https://zone.mfgl.com/pensions/burohappold/index.asp (after the login, there is a redirect to https://zone.mfgl.com/pensions/burohappold/login2.asp for another extra random security question within 4/5 choices, like place of birth, or first pet name) and after I need to grab information on this page : https://zone.mfgl.com/pensions/burohappold/statement-page.asp

The idea is to retrieve personnal account information to process it in another script. I know how to use requests (on a basic level), but I'm struggling with those 2 specific website (I believe because of the way the password has to be inputed for the first, by clicking, and because of the redirect on the second).

import requests
import logging

logging.basicConfig(level=logging.DEBUG)

URL1 = 'https://www.amundi-ee.com/psf/#login'
URL2 = 'https://www.amundi-ee.com/psf/#'
URL3 = 'https://zone.mfgl.com/pensions/burohappold/index.asp'
URL4 = 'https://zone.mfgl.com/pensions/burohappold/login2.asp'
URL5 = 'https://zone.mfgl.com/pensions/burohappold/statement-page.asp'

payload1 = {
    'username' : '000000',
    'password' : '000000'
}
payload2 = {
    'username' : '000000',
    'password' : '000000'
}
sub_payload1 = {
    'password' : 'aaaaaa'
}
sub_payload2 = {
    'password' : 'bbbbbb'
}
sub_payload3 = {
    'password' : 'cccccc'
}

headers = {'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.113 Safari/537.36'}


with requests.Session() as c:
    c.post(URL1, headers=headers, data=payload1)
    r = c.get(URL2, headers=headers)
    print(r.text)

with requests.Session() as d:
    d.post(URL3, headers=headers, data=payload2)
    #How to manage the redirection here to URL4?
    #How to test which extra security question is supplied and submit corresponding sub_payload?
    s = d.get(URL5, headers=headers)
    print(s.text)

For the 2nd website, I honestly have no idea how to handle the redirect and question testing...

Do payload 'username'/'password' have to match specific names from those websites forms?

Thanks for any help or guidance (I'm a beginner).

Answer 1

So I managed to make it work for the 2nd website, below the code if someone is interested. There was a hidden form field. I'll open a new topic for the 1st website.

import requests, time
from lxml import html

LOGIN = 'https://zone.mfgl.com/pensions/burohappold/index.asp'
LOGIN2 = 'https://zone.mfgl.com/pensions/burohappold/login2.asp'
PROTECTED_PAGE = 'https://zone.mfgl.com/pensions/burohappold/statement-page.asp'
payload = {
    'username': 'username',
    'password': 'password',
    'action': 'login'
}
s_payload1 = {
    'answer': 'answer',
    'correctanswer': 'answer',
    'action': 'submit'
}
s_payload2 = {
    'answer': 'answer',
    'correctanswer': 'answer',
    'action': 'submit'
}
s_payload3 = {
    'answer': 'answer',
    'correctanswer': 'answer',
    'action': 'submit'
}

with requests.session() as s:
    s.post(LOGIN, data=payload)
    time.sleep(2)
    r = s.get(LOGIN2)
    #2nd login question test to submit appropriate sub_payload
    if 'XXXX' in r.text:
        s.post(LOGIN2, data=s_payload1)
    if 'YYYY' in r.text:
        s.post(LOGIN2, data=s_payload2)
    if 'ZZZZ' in r.text:
        s.post(LOGIN2, data=s_payload3)
    f = s.get(PROTECTED_PAGE)

tree = html.fromstring(f.content)
a = tree.xpath('//td/text()')
index = a.index("Total")
Funds = a[32]
print(Funds)