How to scrape every specific amount of time with Python Selenium Chrome Driver?

Question

Situation: There is a website which requires me to scrape information from it every x seconds. The site in question has information which requires my input, thus I decided to go with Selenium. The action flow looks like that: User can click in the browser section or interact with the website and the Selenium browser will scrape a specific piece of information every x seconds.

What have I tried?:

• driver.wait (for any kind of element or a specific time); this, unfortunately, doesn't work as I don't have a specific element the browser shall wait for.
• time.sleep(0.5) in a while True loop; this didn't work as the scraping and processing part (which may run simultaneously) took time as well, this time.sleep(0.5) may be off by a few seconds.
• I looked into creating a Google Chrome Plugin which may do actions and send that information to the Python script in charge, though this surpassed the efforts it should, hence I decided against it.

To sum up , how can I scrape information from a Selenium Chrome Driver session every fixed amount of time?

Answer 1

You can simply wait for the difference between when you start and end. You also need to make sure you have a time that is greater than the time it takes your program. Used 5 here so if your program takes 1 second to run than it would wait for 5-1=4 seconds. This does get the difference in floating values so you can switch to int and do some checks for 0-1 second.

import time
while True:
    now = time.time()
    time.sleep(1)
    later = time.time()
    difference = (later - now)
    print(difference)
    driver.implicitly_wait(5-difference)