无法从网页上抓取静态信息

Question

I've created a script in python to log in a webpage using credentials and then parse a piece of information SIGN OUT from another link (the script is supposed to get redirected to that link) to make sure I did log in.

Website address

I've tried with:

import requests
from bs4 import BeautifulSoup

url = "https://member.angieslist.com/gateway/platform/v1/session/login"
link = "https://member.angieslist.com/"

payload = {"identifier":"usename","token":"password"}

with requests.Session() as s:
    s.post(url,json=payload,headers={
        "User-Agent":"Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.87 Safari/537.36",
        "Referer":"https://member.angieslist.com/member/login",
        "content-type":"application/json"
        })

    r = s.get(link,headers={"User-Agent":"Mozilla/5.0"},allow_redirects=True)
    soup = BeautifulSoup(r.text,"lxml")
    login_stat = soup.select_one("button[class*='menu-item--account']").text
    print(login_stat)

When i run the above script, I get AttributeError: 'NoneType' object has no attribute 'text' this error which means I went somewhere wrong in my log in process as the information I wish to parse SIGN OUT is a static content.

How can I parse this SIGN OUT information from that webpage?

Answer 1

This website requires JavaScript to work with. Though you generate the login token correctly from the login API, but when you go to the home page, it make multiple additional API calls and then updates the page.

So the issue has nothing to do with login not working. You need to use something like selenium for this

from selenium import  webdriver

driver = webdriver.Chrome()

driver.get("https://member.angieslist.com/member/login")
driver.find_element_by_name("email").send_keys("none@getnada.com")
driver.find_element_by_name("password").send_keys("NUN@123456")
driver.find_element_by_id("login--login-button").click()
import time
time.sleep(3)
soup = BeautifulSoup(driver.page_source,"lxml")
login_stat = soup.select("[id*='menu-item']")

for item in login_stat:
    print(item.text)
print(login_stat)
driver.quit()

I have mixed bs4 and selenium here to get it easy for you but you can use just selenium as well if you want