使用 webbot 抓取网页

Question

I am trying to create a simple program to login to a webpage using my credentials and grab the total amount left of flex dollars I have in my account for college. Starting at the log in page, I log in, and am redirected to the page of interest, and I simply want to grab that dollar amount and perform some manipulation on it.

I am currently using webbot for the login portion of this, which works, I have just redacted the credentials:

from webbot import Browser

web = Browser()
web.go_to('insert my url here')
#enter your username and password in the into fields below
web.type('insert email here', into='username')
web.type('insert password here', into='password')
web.click('Login', tag='span')

This works perfectly so far, creating an instance of Chrome and logging into the page I want to grab the dollar amount from. I imagine I might want to proceed using urllib, however, I don't think urllib benefits from my current logged in instance of Chrome. How can I work around this and grab a simple html element from the page?

Answer 1

You first need to get the html source code for the current webpage. You can do that using get_page_source() . You then need to pass the html source code to beautifulsoup

from webbot import Browser
from bs4 import BeautifulSoup
import time

web = Browser()
web.go_to('insert my url here')
#enter your username and password in the into fields below
web.type('insert email here', into='username')
web.type('insert password here', into='password')
web.click('Login', tag='span')
time.sleep(5)

content = web.get_page_source()
soup = BeautifulSoup(content)

#You can now find the element you want
samples = soup.find_all("a", "item-title")