请求自动登录的Python Web抓取不起作用

Question

I have been attempting to web scrape a website using the python requests module and have needed to login to the site to retrieve the data I want. I have looked around everywhere but cannot find out why it is not working. Here is my code so far:

import requests
import bs4 as bs

login_url = "__withheld__"
target_url = "__withheld__"

login_data = { "username": "my_username", "password": "my_password"}

with requests.Session() as s:
    page = s.get(login_url)
    page_login = s.post(login_url, data = login_data)
    page = s.get(target_url)
    final_page = bs.BeautifulSoup(page.content, 'lxml')
    print(final_page.title)

This is the html of the password box:

<input name="username" type="text" id="username" class="metro-input" placeholder="Username" value="">
<span id="username-error" class=""></span>
<label class="ie789Only"> Password</label>
<input name="password" type="password" id="password" class="metro-input" placeholder="Password">
<input type="submit" name="button1" value="Sign in" id="button1" class="metro-button">

I believe that it may have to do with the website requiring the user to click the button though I could find no solution. I also tried looking for any post forms in the developer console when I login myself but have found no definitive form outlining the password/username. Any help is appreciated.

Update Here is the link to a site run by the same company (privacy) with the same security features if this is any help: https://ashwood-vic.compass.education/login.aspx?sessionstate=disabled

Answer 1

Can you try this below code once

import requests
import bs4 as bs
username = 'username of the site'
password = 'password of the site'

req = requests.get(login_url, auth=(username, password))
final_page = bs.BeautifulSoup(req.content, 'lxml')
print(final_page.title)

- Please refere this http://docs.python-requests.org/en/master/user/authentication/#basic-authentication