Python web scraper and input

Question

I had started to build a program for my personal work use of web scraping and input for mortgage rates. Essentially what i wanted to do was have my program log into each website, enter the mortgage data necessary, and it would return rates and compare each site so that i wouldnt have to manually do this on each site.

The problem i didnt think of is the login portion. i would have to store tokens and a few other items in order for me to navigate from page to page within each website.

my question is, is this even possible since i dont know the credentials/tokens to send to each page within a site? (i have the login info but unsure if i need more than just the credentials and the tokens)

Answer 1

This is complicated with just the request module.

Note that this approach requires more system resources

You can use PlayWright to control a chromium instance.

Chromium saves the credentials and tokens like nearly every other browser and you just have to program the browser to login and scrape.