如何使用 Python 中的 selenium 等待该网站加载？

Question

Currently I'm using beautifulSoup in my python web scraping project. However, in one of the pages I need to scrape, I need to interact with a javascript element. So I'm being forced to use selenium (which I'm not that familiar with). This is my code so far:

from selenium.webdriver import Firefox
from selenium.webdriver.firefox.options import Options
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
opts = Options()
opts.add_argument('--headless')
seleniumDriver = Firefox(options=opts, executable_path = 'D:\Programs\Python\Scripts\geckodriver.exe')

seleniumDriver.get("https://www.thecompleteuniversityguide.co.uk/courses/details/computing-bsc/57997898")
driverWait = WebDriverWait(seleniumDriver, 10)
driverWait.until(EC.invisibility_of_element_located((By.ID, "mainBody")))

moduleButton = seleniumDriver.find_element_by_xpath("//div[@class='mdldv']")#.find_element_by_tag_name("span")
print("MODULE BUTTON:", moduleButton)
moduleButton.click()

seleniumDriver.close()

Currently, I'm getting a timeout error, however I'm certain that the mainBody ID element does exist. (I don't know how to use the By class, so I have no idea how it will work). Error Message:

Traceback (most recent call last):
  File "D:/Web Scraping/selenium tests.py", line 12, in <module>
    driverWait.until(EC.invisibility_of_element_located((By.ID, "mainBody")))
  File "D:\Programs\Python\lib\site-packages\selenium\webdriver\support\wait.py", line 80, in until
    raise TimeoutException(message, screen, stacktrace)
selenium.common.exceptions.TimeoutException: Message: 

Answer 1

You are calling:

driverWait.until(EC.invisibility_of_element_located((By.ID, "mainBody")))

Per the doc, this will wait until the element is absent:

class invisibility_of_element_located(object):
    """ An Expectation for checking that an element is either invisible or not
    present on the DOM.

    locator used to find the element
    """

The Timeout exception that was raised means that the element was found but was never removed from the DOM or never became invisible.

What you need it to wait until the element is found (part of the DOM). Use instead, presence_of_element_located

driverWait.until(presence_of_element_located((By.ID, "mainBody")))

The timeout exception will be raised if it is not found within the timeout requested when creating driverWait

(I don't know how to use the By class, so I have no idea how it will work)

The By is used under the hood when calling find_element_by_xpath/id/css_selector.

In your case, when you are using EC, you are giving the locator to use By.ID and its value. You can see it equal to find_element_by_id('yourValue')