如何抓取具有用户名和密码的网站？

Question

I am trying to scrape a website and make my program know all the buttons and links that inside of that website but my problem is that to get to the first page I need to enter a username and a password and then scraping the page that shows after that and every time it's scraping to the page with the password and the username someone knows how to do that? because I don't know-how this is the code that I tried:

import requests
import time
from bs4 import BeautifulSoup
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

PATH = "C:\Program Files (x86)\chromedriver.exe"
driver = webdriver.Chrome(PATH)
driver.get("https://www.ronitnisan.co.il/admin/UnPermissionPage.asp?isiframe=")
try:
    element = WebDriverWait(driver, 10).until(
        EC.presence_of_element_located((By.NAME, "FirstName"))
    )
except:
    driver.quit()
userName = driver.find_element_by_name("FirstName")
userName.clear()
userName.send_keys("username")
password = driver.find_element_by_name("UserIDNumber")
password.clear()
password.send_keys("username")
time.sleep(0.5)
login = driver.find_element_by_name("submit")
login.click()
URL = 'https://www.ronitnisan.co.il/admin/UnPermissionPage.asp?isiframe='
page = requests.get(URL)
soup = BeautifulSoup(page.content, 'html.parser')
print(soup)

Answer 1

You are starting a Chrome 'session' (I don't know if that is the correct word for it) up until and including the try: code block. You use that session to enter the username and password, so far so good.

Then you abandon that session altogether and just use a requests.get() statement to get an URL. That url does not have any login information (either via cookies of sessions via your browser, as the login was done via the driver variable.

The human equivalent of this is login into a website with Firefox and then try to visit the same website with Edge. They won't share the same session and you will have to login again in Edge in that case.

What you might want to try is something like this (after login.click() )

soup = BeautifulSoup(driver.page_source, 'lxml')
print(soup)

Answer 2

Replace

URL = 'https://www.ronitnisan.co.il/admin/UnPermissionPage.asp?isiframe='
page = requests.get(URL)
soup = BeautifulSoup(page.content, 'html.parser')
print(soup)

with

driver.get (URL)

and then use find_element to track down the parts of the page you are interested in.

Otherwise you want to capture the cookies to use with requests.

Answer 3

How about this?

from bs4 import BeautifulSoup
from selenium import webdriver
profile = webdriver.FirefoxProfile()
profile.accept_untrusted_certs = True
import time

wd = webdriver.Firefox(executable_path="C:/Utility/geckodriver.exe", firefox_profile=profile)
url = "https://the_url"
wd.get(url)

# set username
time.sleep(5)
username = wd.find_element_by_id("FirstName")
username.send_keys("your_id")
#wd.find_element_by_id("identifierNext").click()

# set password
#time.sleep(2)
password = wd.find_element_by_id("Password1")
password.send_keys("your_password")
elements = wd.find_elements_by_class_name("submit")
for e in elements:
    e.click()

# wait max 10 seconds until "theID" visible in Logged In page
time.sleep(10)
content = wd.page_source
print(BeautifulSoup(content, 'html.parser'))


file = open('C:\\your_path_here\\test.txt', 'w', encoding='utf-8')
file.write(content)
file.close()