使用登录进行 Python 网页抓取

Question

I'm trying to login through a site that is password protected in order to get access to a protected page, i have the email and password names along with the csrf-token.But when i try to access the protected page it doesnt allow me and redirects me back to the login.Any help would be awesome!The site im trying to access is.

https://www.usertesting.com/users/sign_in

import requests
from lxml import html

session_requests = requests.session()

login_url = "https://www.usertesting.com/users/sign_in"
result = session_requests.get(login_url)

tree = html.fromstring(result.text)
authenticity_token = list(set(tree.xpath("//meta[@name='csrf-token']/@content")))[0]

userInfo = {
    "user[email]": "email", 
    "user[password]": "password", 
    "csrf-token": authenticity_token
}

result = session_requests.post(
    login_url, 
    data = userInfo, 
    headers = dict(referer=login_url)
)

url = 'https://www.usertesting.com/my_dashboard'

result = session_requests.get(
    url, 
    headers = dict(referer = url)
)

print result.content

Answer 1

Try taking a look at this https://kazuar.github.io/scraping-tutorial/ for the answer you're looking for. Summarizing, you're going to need to inspect the web page and before you begin your full scraping program you should write another function that will enter the username, password, then enter the site. After that completes, begin the full scripting.