Python Selenium：如何从页面获取新数据定期刷新？

Question

I have already written a script that opens Firefox with a URL, scrape data and close. The page belongs to a gaming site where page refresh contents via Ajax.

Now one way is to fetch those AJAX requests and get data or refresh page after certain period of time within open browser.

For the later case, how should I do? Should I call the method after certain time period or what?

Answer 1

You can implement so-called smart wait .

1. Indicate the most frequently updating and useful for you web element at the page

2. Get data from it by using JavaScript since DOM model will not be updated without page refresh, eg:

   driver.execute_script('document.getElementById("demo").innerHTML')

3. Wait for certain time, get it again and compare with previous result. If changed - refresh page, fetch data, etc.

Answer 2

You can use the time library to do it. For example:

import time
from selenium import webdriver

driver = webdriver.Firefox()

while <condicion>:
  driver.get("http://www.url.org")
  # extract and save data
  time.sleep(5000) # whaits 5000 seconds

driver.close()

Answer 3

Make sure to call findElement() again after waiting, bc otherwise you might not get a fresh instance. Or use page factory, which will get a fresh copy of the WebElement for you every time the instance is accessed.

Answer 4

Try with refresh the page time by time to get the updated result.

  driver.navigate().refresh();

To refresh page in a period of time refer below link :-

Running a python script for a user-specified amount of time?

Hope it will help you :)