刮取需要使用Python登录的Javascript呈现页面

Question

My issue is that I can't scrape a website that uses login when it renders the page using Javascript.

I can easily log in using this code:

import requests
from lxml import html

payload ={
    "username":"username",
    "password":"password"
}
session_requests = requests.session()
result = session_requests.get(login_url)
tree = html.fromstring(result.text)
result = session_requests.post(
    login_url,
    data = payload,
    headers = dict(referer=login_url)
)

Then I can get some values using this code:

result = session_requests.get(agent_url, headers = dict(referer = agent_url ))
tree = html.fromstring(result.content)
needed_info = tree.xpath("//div[@class='col-md-6']/div[@class='table-responsive']/table/tbody/tr[22]/td[2]")[0].text

However, not everything is rendered.

I've also tried to use dryscrape, however, it does not work on Windows. Selenium is just too heavy for my needs and I'm having issues installing Spynner (probably because it does not support Python 3.6?)

What would you recommend?

Answer 1

I just went and did it using selenium. Everything else was just too much of a hassle for this little project.